[adapters] size the datafusion sort reservation from the pool - #6870
Open
ryzhyk wants to merge 1 commit into
Open
[adapters] size the datafusion sort reservation from the pool#6870ryzhyk wants to merge 1 commit into
ryzhyk wants to merge 1 commit into
Conversation
We were incorrectly configuring datafusion's `sort_spill_reservation_bytes`. We unconditionally configured 64MiB per worker. It turns out that datafusion insists on reserving the configured number of bytes unconditionally before processing any data, even if it's only sorting a few records. If this reservation (64MiB x the number of workers) exceeds the datafusion memory pool size, it fails. Even if it fits, it can starve other datafusion operators and cause them to fail. The new policy we implement is to allocate 25% of the memory pool to spill-to-disk sorting. This allocation is split across all partitions. If the resulting per-partition allocation is too small, then we fall back to slower single-threaded full sorting. The next issue we fix is that we estimate datafusion partition count as the *local* worker count as opposed to the total number of workers in a multihost config. This still leaves an important gap. The 25% memory pool allocation is per datafusion session. Multiple sessions started by multiple connector can still oversubscribe the pool. Signed-off-by: Leonid Ryzhyk <ryzhyk@gmail.com>
mihaibudiu
approved these changes
Aug 16, 2026
mihaibudiu
left a comment
Contributor
There was a problem hiding this comment.
Frankly, I am sorry for the brain cells I had to use reading this
| /// (`OutputHandle::take_from_all` walks the local mailboxes). `workers` | ||
| /// counts the workers on every host, while the memory pool is per host, so | ||
| /// the local share is what a sort here actually runs. | ||
| fn adhoc_partition_floor(pipeline_config: &PipelineConfig) -> usize { |
Contributor
There was a problem hiding this comment.
and I thought we have many knobs
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
We were incorrectly configuring datafusion's
sort_spill_reservation_bytes. We unconditionally configured 64MiB per worker. It turns out that datafusion insists on reserving the configured number of bytes unconditionally before processing any data, even if it's only sorting a few records. If this reservation (64MiB x the number of workers) exceeds the datafusion memory pool size, it fails. Even if it fits, it can starve other datafusion operators and cause them to fail.The new policy we implement is to allocate 25% of the memory pool to spill-to-disk sorting. This allocation is split across all partitions. If the resulting per-partition allocation is too small, then we fall back to slower single-threaded full sorting.
The next issue we fix is that we estimate datafusion partition count as the local worker count as opposed to the total number of workers in a multihost config.
This still leaves an important gap. The 25% memory pool allocation is per datafusion session. Multiple sessions started by multiple connector can still oversubscribe the pool.
Describe Manual Test Plan
Checklist
Breaking Changes?
Mark if you think the answer is yes for any of these components:
Describe Incompatible Changes