Repository navigation
Leave computed fields out of serialized pydantic models - #3486
Open
1AbdullahButt wants to merge 2 commits into
Open
1AbdullahButt wants to merge 2 commits into
1AbdullahButt wants to merge 2 commits into
Conversation
The pydantic transformer serialized models with model_dump_json(), which includes @computed_field values. Reading the value back with model_validate_json() then treats them as extra inputs, so a model with extra="forbid" could not be passed between tasks (extra_forbidden), and with the default extra="ignore" the computed values were dropped anyway. Pass exclude_computed_fields=True (pydantic >= 2.12) in both to_literal and the old generic format path; the receiving side computes the fields again when it builds the model. Closes flyteorg/flyte#8342 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: 1AbdullahButt <168034895+1AbdullahButt@users.noreply.github.com>
1AbdullahButt
requested review from
cosmicbboy,
davidmirror-ops,
kumare3,
machichima,
pingsutw,
samhita-alla and
wild-endeavor
as code owners
November 7, 2026 23:29
Collaborator
|
@1AbdullahButt thanks for the clear writeup and tests. A few concerns with fixing this on the write side:
The same discussion is happening on flyteorg/flyte-sdk#1807, so it would be good to land on one approach for both. |
Keep writing model_dump_json() as before, so stored outputs, cache keys and the UI keep the computed values. Before validating, the transformer now drops the computed-field keys of the expected model, walking nested models, lists, tuples, dicts, Optional and Annotated fields and RootModels. That also fixes outputs already stored by older versions, and works on every supported pydantic version. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: 1AbdullahButt <168034895+1AbdullahButt@users.noreply.github.com>
Author
|
@AdilFayyaz thanks, those are all fair. I moved the fix to the read side in 3768cde, so the write path is back to plain
If this approach looks right, the same helper could go into flyte-sdk#1807. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Tracking issue
Closes flyteorg/flyte#8342
Why are the changes needed?
PydanticTransformerserializes a model withmodel_dump_json(), which includes@computed_fieldvalues.to_python_valuereads it back withmodel_validate_json(), and pydantic treats those keys as extra inputs. A model withmodel_config = ConfigDict(extra="forbid")therefore can't be passed from one task to the next (extra_forbiddenbefore the downstream task runs). With the defaultextra="ignore"the computed values were silently dropped on the way in anyway.What changes were proposed in this pull request?
The fix is on the read side, so nothing about what gets written changes:
flytekit/extras/pydantic_transformer/transformer.py: a new_strip_computed_fields(tp, data)drops the computed-field keys (name or alias) of the expected model from the deserialized dict beforemodel_validate_json. It follows the model's field annotations into nested models,list/set/Sequence,tuple(fixed and variadic),dict/Mapping,Optional[...],Annotated[...]andRootModel. A union of several types is left as it is, since it isn't known which one the data is for. Keys that are real fields or field aliases are never dropped.from_binary_idl) and the old protobuf Struct format (to_python_value).Compared with excluding computed fields on write (the first version of this PR):
exclude_computed_fields, so it works the same on pydantic < 2.12, including nested models.How was this patch tested?
New tests in
tests/flytekit/unit/extras/pydantic_transformer/test_pydantic_basemodel_transformer.py:test_basemodel_with_computed_field_and_extra_forbid(msgpack andFLYTE_USE_OLD_DC_FORMAT=true): anextra="forbid"model with a computed field holds the same kind of model directly, in aList, aDictand anOptional. It checks that the stored literal still contains the computed values and that it round-trips.test_basemodel_with_computed_field_and_extra_forbid_between_tasks: a workflow passes anextra="forbid"model and aListof them from one task to another.Run in a
python:3.12-slimcontainer:With the transformer from
master, the new tests fail withextra_forbidden.ruff checkandruff format --checkpass ontransformer.py. I did not run the full unit suite.Setup process
pip install -e . pytest mock pandas pyarrow hypothesis pytest-asyncioCheck all the applicable boxes
Related PRs
flyteorg/flyte-sdk#1807 fixes the same issue in flyte-sdk on the write side; the same read-side approach would carry over there.
This change was written with Claude Code; I reviewed it and ran the tests above.
🤖 Generated with Claude Code