fix[next-dace]: Attempt to achieve deterministic dace auto optimize - #2635
Merged
Conversation
edopao
commented
Jun 8, 2026
There was a problem hiding this comment.
Pull request overview
This PR aims to make DaCe auto_optimize / lowering in the GT4Py “next” backend deterministic by eliminating order-dependence (e.g., set iteration) and replacing unstable name generation with a shared, deterministic ID generator.
Changes:
- Switch several order-sensitive collections from
settoOrderedSetto stabilize traversal/processing order. - Introduce and thread a shared
gtx_utils.IDGeneratorPoolthrough key DaCe transformations to generate deterministic, unique names. - Pin DaCe to a git branch (
dev-sorted_sets) and addordered-setas a runtime dependency to support deterministic ordered data structures.
Reviewed changes
Copilot reviewed 15 out of 16 changed files in this pull request and generated 7 comments.
Show a summary per file
| File | Description |
|---|---|
uv.lock |
Pins dace to a specific git commit/branch and adds ordered-set to the resolved environment. |
pyproject.toml |
Adds runtime dependency on ordered-set and configures dace as a git source. |
tests/next_tests/unit_tests/program_processor_tests/runners_tests/dace_tests/transformation_tests/test_move_tasklet_into_map.py |
Updates tests to pass a shared UID pool into the transformation. |
tests/next_tests/unit_tests/program_processor_tests/runners_tests/dace_tests/transformation_tests/test_fuse_horizontal_conditionblocks.py |
Updates tests to pass a shared UID pool into the transformation. |
src/gt4py/next/program_processors/runners/dace/transformations/utils.py |
Uses OrderedSet for deterministic upstream traversal results. |
src/gt4py/next/program_processors/runners/dace/transformations/splitting_tools.py |
Replaces unordered edge assignment sets with OrderedSet for deterministic splitting. |
src/gt4py/next/program_processors/runners/dace/transformations/split_access_nodes.py |
Uses OrderedSet for deterministic edge reassignment collection. |
src/gt4py/next/program_processors/runners/dace/transformations/simplify.py |
Replaces UUID-based naming with deterministic UID pool; orders dependencies deterministically. |
src/gt4py/next/program_processors/runners/dace/transformations/multi_state_global_self_copy_elimination.py |
Makes pass dependencies ordered (list instead of set). |
src/gt4py/next/program_processors/runners/dace/transformations/move_dataflow_into_if_body.py |
Updates dataflow classification typing to use OrderedSet. |
src/gt4py/next/program_processors/runners/dace/transformations/map_fusion_utils.py |
Makes connector collections and map parameter handling more deterministic. |
src/gt4py/next/program_processors/runners/dace/transformations/inline_fuser.py |
Updates inlining logic to use OrderedSet for deterministic node ordering. |
src/gt4py/next/program_processors/runners/dace/transformations/fuse_horizontal_conditionblocks.py |
Uses the shared UID pool for deterministic renaming during fusion. |
src/gt4py/next/program_processors/runners/dace/transformations/auto_optimize.py |
Creates a UID pool per optimization run and threads it through key transformations. |
src/gt4py/next/program_processors/runners/dace/lowering/gtir_to_sdfg.py |
Stabilizes nested SDFG connector ordering via sorted connector names. |
src/gt4py/next/program_processors/runners/dace/lowering/gtir_dataflow.py |
Stabilizes nested SDFG connector ordering (and adjusts connector representation). |
edopao
changed the title
SortedSet for deterministic dace auto optimize
edopao
changed the title
SortedSet for deterministic dace auto optimizeSortedSet to achieve deterministic dace auto optimize
edopao
changed the title
SortedSet to achieve deterministic dace auto optimizeOrderedSet to achieve deterministic dace auto optimize
tehrengruber
approved these changes
Jun 9, 2026
edopao
changed the title
OrderedSet to achieve deterministic dace auto optimize
philip-paul-mueller
approved these changes
Jun 10, 2026
philip-paul-mueller
left a comment
philip-paul-mueller
left a comment
Contributor
There was a problem hiding this comment.
LGTM
I have some smallish comments.
Contributor
Author
|
cscs-ci run dace-determinism |
Contributor
Author
|
cscs-ci run dace-determinism |
2 tasks
FlorianDeconinck
pushed a commit
that referenced
this pull request
Jul 28, 2026
#2710) ## Description PR #2635 bumped the DaCe version from alpha3 to alpha4. This brought a new simplify pass: `LiftTrivialIf` which aims at inspecting the conditions of if statements and inlines code under trivially true/false conditions, see spcl/dace#2138. The new pass increases the runtime of simplify such that it is unusable for large-scale / real-world SDFGs, e.g. parsing the acoustics loop of fv3 in NDSL with dace orchestration will take >90min with the new pass compared to 10-15min without. In total, the runtime of the acoustics translate test blows up from ~30min to nearly 5h with that pass. Preliminary, crude analysis shows that the bulk of the additional time is spent in the simplify step of sympy when the condition is evluated. This puts the slowdown at the heart of the pass, making an easy fix unlikely. This PR suggests to just skip the new pass until the runtime drops to usable levels. ## Requirements - [ ] All fixes and/or new features come with corresponding tests. N/A. Covered by existing test cases. - [ ] Important design decisions have been documented in the appropriate ADR inside the [docs/development/ADRs/](docs/development/ADRs/README.md) folder. N/A
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR uses ordered data structures and unique node IDs to ensure deterministic lowering to SDFG.
This change was originally implemented by @tehrengruber in #2568.