Skip to content

feat(telemetry): add the telemetry-3 conformance handler - #882

Merged
hln33 merged 5 commits into
mainfrom
feat/conformance-telemetry-3
Nov 9, 2026
Merged

hln33 merged 5 commits into
mainfrom
feat/conformance-telemetry-3

Conversation

@hln33

@hln33 hln33 commented Nov 9, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds the Java handler for conformance requirement telemetry-3 (per-operation result selection). The requirement is defined in aws/aws-durable-execution-conformance-tests#183 and was raised in aws/aws-durable-execution-conformance-tests#182.

Step fetch-user returns {userId, email, token}. An OperationOverride.withResult("fetch-user", selectUserId) transform returns {userId} alone. The content section sets input(false) and output(false). So the exported record has no input, no output, and operationsByName.fetch-user.result == {"userId": "u-123"}. The execution result still holds all three keys, which the suite's core ExpectedResult check verifies.

Design and alternatives

Problem: Verify that telemetry exports only userId from a step result while the execution still returns userId, email, and token.

Coverage gap: The existing telemetry scenarios do not exercise per-operation result selection.

Alternatives evaluated during review:

  • Add no handler. This leaves the result-selection behavior untested by the deployed conformance suite.
  • Reuse the existing success handler unchanged. It exercises an increment step and includes execution input and output. It cannot produce the required result-selection record.
  • Extend that handler to serve both scenarios. The scenarios require different step results and content settings. Combining them adds scenario selection to the handler and configuration.
  • Return only userId from the step. This changes the execution result. It also avoids testing whether the telemetry transform selects the field correctly.

Chosen approach: Keep a dedicated handler with one step and one result transform. Disable execution input and output in its telemetry configuration. Return the complete step result unchanged. This preserves the existing scenarios without adding scenario-selection logic.

Removals: None. This adds conformance coverage using the existing SDK API.

Java configuration structure: Copying configuration into the new handler would avoid adding overloads. It would duplicate the exporter, emit mode, sampling rate, operation detail, and plugin registration.

The retained overloads share that setup and accept the configuration that differs. The existing entry points retain the settings used by telemetry-1 and telemetry-2. The telemetry settings remain independently testable without initializing the handler's environment-dependent static fields.

So retain the package-private configuration methods. No SDK public API change is needed.

Changes

  • Telemetry3ResultSelect.java: new handler.
  • TelemetryConformanceConfig.java: telemetryConfig(logGroupName, content) and durableConfig(telemetryConfig) overloads; resultSelectTelemetryConfig(logGroupName) and selectUserId(Object) for telemetry-3. The handler's static initializer reads the environment, so the testable pieces live here, following the existing module convention.
  • template.yaml: Telemetry3ResultSelect → TestDescription: [telemetry-3].
  • TelemetryConformanceConfigTest.java: four new tests (transform output, shared settings kept, input/output disabled, override entry targets fetch-user and applies the transform).
  • README.md, pom.xml description.
  • .github/workflows/telemetry-conformance-tests.yml: pinned to the [Docs]: Add missing javadoc #183 merge commit on main (cf13ef2).

Verification

  • JAVA_SDK_VERSION=3.1.1-SNAPSHOT mvn -B package in conformance-tests-telemetry/ (after mvn -Dmaven.test.skip=true install at the root): BUILD SUCCESS, 11 tests, 0 failures.
  • Changed Java files formatted with the root spotless (palantir) configuration. The module is outside the reactor, so the Spotless workflow does not cover it.
  • Telemetry conformance e2e: telemetry-1, telemetry-2, and telemetry-3 passed. 3 passed, 0 failed. Run 42681226152 at d4aca515. Re-run after the re-pin: Run 42714830848 at f8ac60b7, pinned to cf13ef27, 3 passed, 0 failed.

hln33 added 2 commits November 9, 2026 12:12
Requirement telemetry-3 checks that an OperationOverride result transform
exports only the keys it returns. The new handler runs step fetch-user,
which returns {userId, email, token}. The override returns {userId} alone.
The content section also disables input and output, so the record carries
no input and no output key.

TelemetryConformanceConfig gains a telemetryConfig(logGroupName, content)
overload, a durableConfig(telemetryConfig) overload, the telemetry-3
settings builder resultSelectTelemetryConfig, and the selectUserId
transform. The handler static initializer reads the environment, so the
testable pieces live in the config class, as the existing tests require.

The template maps Telemetry3ResultSelect to telemetry-3. Four tests cover
the transform, the shared settings, the disabled input/output, and the
override entry.
The telemetry-3 requirement lives in aws-durable-execution-conformance-tests
PR #183. The shared workflow at the old pin does not know the test, so the
new function would be reported UNCOVERED. Pin to the PR head so CI validates
telemetry-3 now. Re-pin to the #183 merge commit before leaving draft.
@hln33
hln33 marked this pull request as ready for review November 9, 2026 20:22
@hln33
hln33 requested a review from a team November 9, 2026 20:22
@hln33
hln33 deployed to ai-pr-review-runtime November 9, 2026 20:22 — with GitHub Actions Active
@hln33
hln33 deployed to ai-pr-review-runtime November 9, 2026 20:23 — with GitHub Actions Active
@github-actions

This comment has been minimized.

@hln33

hln33 commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

/ai review

@github-actions

This comment has been minimized.

@ParidelPooya ParidelPooya left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good. The handler matches telemetry-3. applyResultOverride parses the checkpoint into a Map before it calls the transform, so selectUserId is right. CI ran telemetry-3 against cb3753e and it passed.

A few small things:

  1. TelemetryConformanceConfig.java:13-19: the class Javadoc still says every handler uses the same plugin settings. telemetry-3 now uses a different content section. Suggest: every handler shares the exporter, emit mode, sampling rate and operation detail, and the content section differs per requirement.
  2. The PR body ("Draft until") and the workflow comment at telemetry-conformance-tests.yml:48 say "before leaving draft", but the PR is not a draft. Suggest rewording it to "before this PR merges".
  3. README.md:23: the new row is 4 characters shorter than the other rows. README.md:36: the layout comment for TelemetryConformanceConfigTest.java does not mention the new telemetry-3 tests.

Review 5832826567 raised three documentation items.

1. The TelemetryConformanceConfig class Javadoc said every handler uses the
   same plugin settings. telemetry-3 uses a different content section. The
   Javadoc now says the handlers share the exporter, emit mode, sampling rate
   and operation detail, and the content section differs per requirement.
2. The workflow re-pin TODO said "before this PR leaves draft". The PR is not
   a draft. It now says "before this PR merges".
3. README: the telemetry-3 coverage row is padded to the table width, and the
   TelemetryConformanceConfigTest layout comment mentions the telemetry-3 tests.
@hln33
hln33 deployed to ai-pr-review-runtime November 9, 2026 21:39 — with GitHub Actions Active
@hln33

hln33 commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Thanks for the review. All three items are addressed in 419e4e7.

  1. TelemetryConformanceConfig.java class Javadoc now says every handler shares the exporter, emit mode, sampling rate and operation detail, and the content section differs per requirement (telemetry-1 and telemetry-2 include input and output; telemetry-3 excludes them and adds one result transform).
  2. The workflow comment at telemetry-conformance-tests.yml:48 and the PR body now say "before this PR merges". The "Draft until" section is renamed "Before this PR merges".
  3. README.md: the telemetry-3 row is padded to the table width, and the TelemetryConformanceConfigTest.java layout comment now mentions the telemetry-3 tests.

The Java change is Javadoc only. Module build: 11 tests, 0 failures.

@github-actions

This comment has been minimized.

Conformance PR #183 merged as cf13ef27. The workflow now pins that commit
and the TODO to re-pin is removed.
@hln33
hln33 deployed to ai-pr-review-runtime November 9, 2026 22:37 — with GitHub Actions Active
@github-actions

This comment has been minimized.

Dependabot #884 bumped the telemetry-suite pin to 41e034a on main. This
branch pins cf13ef27, the #183 merge commit, which is one commit newer.
So the conflict is resolved to cf13ef27.
@hln33
hln33 deployed to ai-pr-review-runtime November 9, 2026 22:47 — with GitHub Actions Active
@github-actions

github-actions Bot commented Nov 9, 2026

Copy link
Copy Markdown
Contributor

Codex AI review

No actionable findings. The prior design-rationale finding is resolved. Residual test risk: this was a static review; the reported test results and updated external workflow pin were not independently verified.

Reviewed commit f8ac60b73a1f78ec51e72af89e3cf00f55d4c787. Workflow run

@hln33

hln33 commented Nov 9, 2026

Copy link
Copy Markdown
Contributor Author

Thanks. All three documentation items were fixed in 419e4e7 (Javadoc, workflow TODO wording, README row width and layout comment).

Since then: e445605 re-pins the workflow to the #183 merge commit cf13ef27 and removes the TODO. Dependabot #884 bumped the same line on main to 41e034a, which made the PR conflict, so f8ac60b merges main and keeps cf13ef27 (41e034a is its parent). The PR body no longer has a "Before this PR merges" section.

Telemetry e2e on the merged head: run 42714830848, telemetry-1, -2, -3 passed.

@ParidelPooya ParidelPooya left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. The class Javadoc, the README and the pin comment are fixed. The suite is pinned to the #183 merge commit cf13ef2, and telemetry-3 passes on it. Approving.

@hln33
hln33 merged commit 9ed457b into main Nov 9, 2026
20 checks passed
@hln33
hln33 deleted the feat/conformance-telemetry-3 branch November 9, 2026 23:10

This branch was successfully deployed

1 active deployment
ai-pr-review-runtime — f8ac60b7 Deployed Nov 9, 2026 by hln33 via ai-pr-review / Codex review / Generate Codex review #2889
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants