TL;DR
Get monitors, keyboards and dev gear delivered free — and shop member deals
- Fast, free delivery on millions of items
- Access to Prime Big Deal Days deals on October 6–7
- Prime Video, Amazon Music and more included
Polars says version 2.0 makes its streaming engine the default for lazy queries and enables initial out-of-core processing that can spill some workloads to disk. The release also strengthens SQL support and adds a Map dtype; performance comparisons published by the project are based on its own benchmark setup.
Polars has released Polars 2.0, making its streaming engine the default when users collect a LazyFrame and enabling initial out-of-core processing that can spill supported workloads to disk. The project says the changes are intended to improve memory use and performance for many queries, while the major-version change also reflects a difference users may observe: some operations no longer preserve row order by default.
The release post says Polars 2.0 includes first-class SQL support and performance work in its optimizer and engine. The highlighted changes include join reordering, improvements to common-subplan elimination, and dynamic predicates and bloom filters. Polars also introduces a native Map dtype for Arrow MapType data, with operations such as key lookup, checking whether a key exists, and returning keys or values. The project says the release also applies stricter rules around data types and explicitness, aiming to provide faster feedback when queries or expressions do not match expected types.
For lazy queries, calling collect now uses the streaming engine by default, according to the project. Polars warns that the engine does not guarantee observable row order for some operations, including joins, group-bys and unpivots. Users who need ordering can request it with maintain_order=True, where supported. The release also enables spill-to-disk by default, beginning at roughly 80% of available RAM, with a default disk budget of 64 GB. The post says sorting, window functions and many expressions can use this support; joins and group-bys are not yet listed as supported out-of-core operations.
Polars reported benchmark comparisons of its SQL engine with DuckDB and Apache DataFusion on data derived from TPC-H and TPC-DS. It says Polars was fastest on all but one benchmark under the tested conditions, while noting that running with 192 threads added overhead for smaller queries. The tests used two Amazon EC2 machine configurations, repeated each query five times in a hot setting, and compared the best run. Polars published a repository for reproducing the benchmarks and says some DataFusion queries timed out or ran out of memory; those queries were excluded from results for all engines.
Memory and Ordering Changes
The default switch to streaming may affect users without requiring them to opt into a separate execution mode. For analysts working with datasets larger than available memory, spill-to-disk support can let certain operations continue by using disk space rather than relying only on RAM. That does not mean every query is now out-of-core: the release notes limit current support to specified operations and give a default disk budget of 64 GB.
The row-order change is a practical migration concern. A query may produce the same records while presenting them in a different order for operations such as joins or group-bys. Applications that depend on a particular ordering should check their results and use maintain_order=True where appropriate. For teams considering Polars for SQL workloads, the performance results offer a reason to test the release, but they are project-published measurements rather than an independent evaluation.
high performance data analysis laptop
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Why Version 2 Changes Defaults
Polars describes 2.0 as a major version focused on making its existing engine available to more types of workloads, including SQL, rather than as a release planned solely around a large set of new features. The project says the streaming engine’s row-order behavior was one reason the change required a major version bump: its results can differ in ordering from users’ expectations for certain operations, unless ordering is requested explicitly.
The SQL comparison tested Polars against DuckDB 1.5.6, a DuckDB 2.0 development build, and DataFusion 54.0.0. According to the release post, tests ran on machines with 16 and 192 virtual CPUs, with 32 GB and 384 GB of RAM respectively. Queries came from DuckDB’s TPC-H and TPC-DS query generators; data was stored on EBS. Polars says it cleared the file cache between engines and benchmarks, but not between queries, and selected the best of five runs. Those details define the conditions behind its reported ranking.
As an affiliate, we earn on qualifying purchases.
Limits of the Release Claims
The release post does not provide an independent replication of the benchmark findings. Results may depend on hardware, workload, query settings, cache behavior and the treatment of runs that timed out or failed. Polars says its benchmark repository is available for others to reproduce the tests; the supplied material does not establish whether outside researchers have done so.
The exact publication date is not included in the source material. The release also does not specify when out-of-core support for joins and group-bys will arrive, saying only that those operations are on the roadmap. The stated spill threshold of roughly 80% of RAM may need tuning, according to Polars, and users’ actual disk use will depend on their workload and configuration.
As an affiliate, we earn on qualifying purchases.
What Users Should Check
Users upgrading to Polars 2.0 should review queries that rely on row order, especially those involving joins, group-bys or unpivots, and set maintain_order=True when they need order preserved. They can also test memory-intensive workloads against the release’s spill-to-disk behavior, keeping in mind its current operation coverage and default disk budget.
Polars says it hopes to address the overhead it observed when scaling to 192 threads in a future release. Its next stated expansion for out-of-core processing is support for joins and group-bys, but no schedule is given. The project has invited users to reproduce its SQL benchmark results using the published repository; further independent testing would help clarify how broadly the reported performance applies.
As an affiliate, we earn on qualifying purchases.
Key Questions
What is the main change in Polars 2.0?
LazyFrame.collect now defaults to the streaming engine, and initial spill-to-disk support is enabled by default for supported operations.
Does Polars 2.0 preserve row order?
Not by default for some streaming operations, including joins, group-bys and unpivots. The release says users can request ordering with maintain_order=True where supported.
Can every Polars query spill to disk?
No. The release post lists sorting, window functions and many expressions as currently supported. It says out-of-core support for joins and group-bys is planned, but does not give a delivery date.
Did Polars independently verify its benchmark results?
The results in the release post were produced by Polars. The project shared a repository for reproduction, but the supplied source does not report independent verification.
Source: hn
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
