KAFKA-20850: Add system test for suppress() changelog across the HEADERS boundary - #23075
Merged
mjsax merged 1 commit intoAug 21, 2026
Conversation
…ERS boundary The smoke-test application already uses suppress(untilWindowCloses(...)), and streams_application_upgrade_test.py already drives it across versions, but no system test ever set dsl.store.format. With the config unset the headers-aware suppress buffer writes byte-identical V3 records, so the existing matrix passes regardless of the change in KAFKA-20413. Add a HEADERS-configured transition that crosses the 4.3/trunk boundary in both directions. 4.3 has the dsl.store.format config but not the headers-aware buffer, so it writes the whole BufferValue into the record value, while trunk writes only the plain value bytes and ships the value/timestamp/headers prefixes in extra Kafka record headers -- both tagged V3. Since the suppress buffer is in-memory only, every restart replays its full changelog, so the transition exercises a real cross-format restore. - Add an extra_configs pass-through to StreamsSmokeTestBaseService and StreamsSmokeTestJobRunnerService, merged last in prop_file() so a test can override the smoke defaults. - Extract _run_app_transition() from test_app_upgrade() so the target version is a parameter rather than hardcoded to DEV_VERSION, which is what the downgrade direction needs. test_app_upgrade() keeps passing DEV_VERSION, so its matrix behavior is unchanged. - Add test_suppress_headers_app_transition, parameterized on direction. - Assert no instance logged a suppress-changelog restore rejection during the transition. A misread record is otherwise silent, and the existing harness does not compare output topics (KAFKA-10202). Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
github-actions
Bot
added
triage
Contributor
Author
|
|
|
A label of 'needs-attention' was automatically added to this PR in order to raise the |
mjsax
approved these changes
Aug 20, 2026
Member
|
PR merged to |
chia7712
reviewed
Aug 24, 2026
chia7712
left a comment
chia7712
left a comment
Member
There was a problem hiding this comment.
@aliehsaeedii thanks for this patch. Two small comments left. PTAL
|
|
||
| # InMemoryTimeOrderedKeyValueChangeBuffer throws this when it cannot make sense of a | ||
| # suppress-changelog record while restoring. | ||
| INVALID_CHANGELOG_RECORD_MSG = "Restoring apparently invalid changelog record" |
Member
There was a problem hiding this comment.
Should we add a comment to InMemoryTimeOrderedKeyValueChangeBuffer reminding developers not to change this message?
} else {
throw new IllegalArgumentException("Restoring apparently invalid changelog record: " + record);
}| else: | ||
| from_version, to_version = dev_version, old_version | ||
|
|
||
| self._run_app_transition( |
Member
There was a problem hiding this comment.
It seems no test exercises the rolling branch. Should we add tests for rolling or just remove the dead path?
Member
@mjsax +1 to backport to 4.4 branch |
Contributor
Author
chia7712
pushed a commit
that referenced
this pull request
Aug 25, 2026
…ERS boundary (#23075) The smoke-test application already uses `suppress(untilWindowCloses(...))` (`SmokeTestClient#183,195`), and `streams_application_upgrade_test.py` already drives it across versions — but no system test ever set `dsl.store.format`. With the config unset, the headers-aware suppress buffer writes byte-identical V3 records, so the existing matrix passes regardless of KAFKA-20413. This adds a HEADERS-configured transition across the 4.3/trunk boundary in both directions. Reviewers: Matthias J. Sax <matthias@confluent.io>
Member
all tests pass. cherry-picked to 4.4 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The smoke-test application already uses
suppress(untilWindowCloses(...))(SmokeTestClient#183,195), andstreams_application_upgrade_test.pyalready drives it across versions— but no system test ever set
dsl.store.format. With the config unset,the headers-aware suppress buffer writes byte-identical V3 records, so
the existing matrix passes regardless of KAFKA-20413.
This adds a HEADERS-configured transition across the 4.3/trunk boundary
in both directions.
Reviewers: Matthias J. Sax matthias@confluent.io, Chia-Ping Tsai chia7712@gmail.com