Skip to content

feat(telemetry): add telemetry-2 failure handler - #876

Merged
hln33 merged 8 commits into
mainfrom
feat/telemetry-2-failure-handler
Nov 5, 2026
Merged

hln33 merged 8 commits into
mainfrom
feat/telemetry-2-failure-handler

Conversation

@hln33

@hln33 hln33 commented Nov 3, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Adds the Java handler for conformance requirement telemetry-2
(aws/aws-durable-execution-conformance-tests#179).
The JS handler is aws/aws-durable-execution-sdk-js#1168. The Python handler is
aws/aws-durable-execution-sdk-python#907.

The step charge throws RuntimeException("card declined"). The retry strategy is
RetryStrategies.Presets.NO_RETRY, so the step fails on its first attempt. The handler
catches the step error and throws RuntimeException("payment failed") with the step error
as its cause. So the execution fails and the plugin exports one FAILED record.

Design

Problem. The telemetry conformance suite gains requirement telemetry-2: a failed
single-step execution must produce one telemetry record with status FAILED, an execution
error, and a charge STEP operation with failedCount 1 and maxAttempt 1. This module
has no handler that produces that record.

Cause. The module has one handler, Telemetry1Success, and it succeeds. The suite
matches a deployed function to a requirement by function name, so each requirement needs a
function that produces its record.

Options considered.

  1. One input flag on Telemetry1Success selects success or failure. Rejected. The suite
    maps one function to one requirement by name, so one class would still need two template
    functions. The flag adds a branch to the existing handler that would itself need a test.
    The JS (#1168) and Python (#907) packages use one handler per requirement.
  2. Throw from the handler body with no step. Rejected. telemetry-2 requires a failed STEP
    operation record.
  3. Keep the default retry strategy. Rejected. telemetry-2 asserts maxAttempt 1, and the
    default strategy retries the step.
  4. Let the step error propagate unwrapped. Rejected. telemetry-2 pins the execution error
    message payment failed in ExpectedResult.
  5. Copy the log-group guard and the plugin configuration from Telemetry1Success into the
    new handler. Rejected after review. Two copies of the same setup would have to be changed
    together by hand, and the guard sat in a static initializer that read the environment,
    which cannot be unit tested without a subprocess or a custom classloader.
  6. An abstract base handler that implements createConfiguration(). Rejected. It removes
    the duplication, but the environment read would still sit in a static initializer, so the
    guard would stay untested.
  7. Chosen. A second handler, Telemetry2Failure, with one step under NO_RETRY that
    throws card declined; the handler catches the step error and throws payment failed
    with it as the cause. The log-group guard and the plugin configuration move into a
    package-private helper, TelemetryConformanceConfig, with a pure
    requireLogGroupName(String), a pure telemetryConfig(String) that holds the plugin
    settings, and a durableConfig(String) wrap that both handlers call. Reason: it is the
    smallest change that produces the required record, it holds the shared setup once, and
    the guard and the settings become functions with explicit inputs that a unit test can
    exercise.

Test isolation (second review round). DurableConfig.build() fills in a default
DurableExecutionClient when none is given (sdk/.../DurableConfig.java:110–111). That
constructor builds a LambdaClient from environment credentials and sends one
getDurableExecutionState warm-up request (DurableConfig.java:280–296). So a unit test
that calls durableConfig(...) sends a real request whenever AWS credentials are exported.
Options:

  1. Add a durableConfig(String, DurableExecutionClient) overload and pass a no-op client in
    the test, as the SDK's own tests do. Rejected. It is a test-only seam in production code,
    and the test would still build a DurableConfig it does not need.
  2. Add Mockito to the standalone pom and mock the client through the same overload. Rejected
    for the same reason plus a new dependency.
  3. Chosen. Move the plugin settings into telemetryConfig(String), which returns a
    DurableExecutionTelemetryConfig. CloudWatchLogsExporter creates its client on first
    export, so this function constructs no client. The test asserts the settings the runner
    depends on. durableConfig(String) becomes a wrap with no branch of its own; the deployed
    run exercises it.

Removed. Nothing.

Decisions not covered by #179

  • The step throws RuntimeException("card declined"). The handler catches RuntimeException.
    StepOperation.get() rebuilds the step's original exception from the checkpoint and rethrows
    it; StepFailedException is only the fallback when the rebuild fails. So the step surfaces
    as RuntimeException("card declined"), and catching StepFailedException lets it escape
    (the telemetry run on 0bb88f5e3 failed with expected 'payment failed', got 'card declined').
    The handler throws RuntimeException("payment failed", cause).
  • The workflow pin is the [Chore]: publish javadoc #179 merge commit ca2cd02 on main. See "Workflow pin" below.
  • conformance-tests-telemetry is a standalone pom outside the SDK reactor
    (conformance-tests-telemetry/pom.xml:7–15), because SAM builds it in isolation against
    an installed SDK. So the root build never runs its tests. The telemetry conformance
    workflow now runs mvn package without -DskipTests, which runs the module's tests before
    the deploy artifact is built. This mirrors the JS workflow, which runs npm test on its
    telemetry conformance package before building. Python and JS also include the package in
    the repo-wide unit-test run; bringing this module and conformance-tests-otel into the
    reactor is a follow-up, not part of this PR.

Changes

conformance-tests-telemetry/

  • src/main/java/.../Telemetry2Failure.java: new handler.
  • src/main/java/.../TelemetryConformanceConfig.java: new helper holding the log-group
    guard (requireLogGroupName), the plugin settings (telemetryConfig) and the handler
    configuration (durableConfig). Telemetry1Success and Telemetry2Failure call it.
  • src/test/java/.../TelemetryConformanceConfigTest.java: 7 cases. Null, "", " " and
    "\t" each throw IllegalStateException("TELEMETRY_LOG_GROUP_NAME must be set"); a set
    value is returned unchanged; telemetryConfig(...) has one CloudWatchLogsExporter; its
    emit mode is ON_COMPLETE, sampling rate 1.0, operation detail TOP_LEVEL, and input
    and output are included. No test builds a DurableConfig, so no test builds a client.
  • pom.xml: junit-jupiter (test scope) and maven-surefire-plugin, versions repeated from
    the root pom; description lists telemetry-2.
  • template.yaml: Telemetry2Failure function mapped to telemetry-2.
  • README.md: coverage row, layout lines for the helper and the test, local mvn package
    step.

.github/workflows/telemetry-conformance-tests.yml

Error shape

This SDK writes a two-level execution error (payment failed → card declined) and a
one-level operationsByName.charge.error (card declined). The JS SDK writes three and two,
the Python SDK one and one. The team decided this difference is expected. telemetry-2 at
conformance commit 121e832 matches each error with {$object: {}}: present and an object,
keys unchecked. The messages are asserted by the core ExpectedResult and
ExpectedExecutionHistory sections. So this SDK passes telemetry-2 as is.

Workflow pin

telemetry-conformance-tests.yml pins the shared workflow to ca2cd02, the #179 merge commit
on main. Earlier revisions of this PR pinned the #179 head 121e832 so the run exercised
telemetry-2 before #179 merged. The previous mainline pin 0eaf734 predates telemetry-2, and
the runner skipped the handler. telemetry-suite.yml is identical between the commits.

Verification

  • mvn -pl telemetry-exporter -am -DskipTests install from the branch, then
    mvn -B package -Ddurable.sdk.version=3.0.2-SNAPSHOT in conformance-tests-telemetry,
    with fake AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY exported so a client build would
    attempt a request: BUILD SUCCESS; surefire Tests run: 7, Failures: 0, Errors: 0, Skipped: 0 in 0.045 s; the shaded jar contains Telemetry1Success, Telemetry2Failure
    and TelemetryConformanceConfig and no test classes.
  • Guard necessity: with the if block in requireLogGroupName removed, the same command
    fails 4 of the tests. The block was restored before commit.
  • Formatting: the module is outside the reactor, so spotless:check does not cover it. The
    four Java files were formatted with the root spotless configuration (palantir) by
    running it on a temporary copy inside telemetry-exporter.
  • sam validate --lint on template.yaml: valid (template unchanged since).
  • Local run of the handler body through LocalDurableTestRunner with a capturing exporter,
    rendered with record.toByNameWireMap() (the same call CloudWatchLogsExporter makes):
    status FAILED, input: 41, no output,
    error: {RuntimeException, payment failed, cause: {RuntimeException, card declined}},
    operationsByName.charge: STEP, Step, count 1, failedCount 1, maxAttempt 1, FAILED,
    error: {java.lang.RuntimeException, card declined}. Every path matches telemetry-2 except
    the missing cause levels described above. Duration keys are absent only because the local
    runner sets no operation timestamps.
  • Telemetry conformance run (telemetry-2) on the pinned [Chore]: publish javadoc #179 head: PASSED on ccdae02
    and on ecb5e34; rerun on 0ccc30f pending.

@hln33
hln33 marked this pull request as ready for review November 4, 2026 23:30
@hln33
hln33 requested a review from a team November 4, 2026 23:30
@hln33
hln33 deployed to ai-pr-review-runtime November 4, 2026 23:30 — with GitHub Actions Active
hln33 added 2 commits November 4, 2026 16:30
Add the handler for conformance requirement telemetry-2.
The step "charge" throws "card declined" with no retry.
The handler catches the step error and throws "payment failed" with
the step error as its cause. So the execution fails and the exported
record is FAILED with an error on the execution and on the operation.
Add the template function, the README row, and the pom description.
…orkflow to the #179 head

The team decided that error names and cause depth differ per SDK by design.
Conformance PR #179 at 121e832 now matches each telemetry-2 `error` with
`{$object: {}}`: present and an object, keys unchecked. So this SDK's two-
level execution error and one-level step error pass. The README row and the
handler javadoc no longer describe a cause chain.

The workflow pin moves from 0eaf734, which predates telemetry-2 and made the
runner skip the new handler, to the #179 head 121e832 so this PR's run
exercises telemetry-2. telemetry-suite.yml is identical between the two
commits. TEMPORARY: bump to the #179 merge commit on main before merging.
@hln33
hln33 force-pushed the feat/telemetry-2-failure-handler branch from ccdae02 to 3d4eba3 Compare November 4, 2026 23:30
@hln33
hln33 deployed to ai-pr-review-runtime November 4, 2026 23:30 — with GitHub Actions Active
@github-actions

This comment has been minimized.

…uard

Telemetry2Failure copied the TELEMETRY_LOG_GROUP_NAME guard and the full
plugin configuration from Telemetry1Success. Two copies of the same setup
would have to be changed together by hand. So both now live once in
TelemetryConformanceConfig and each handler calls it.

The guard used to run inside a static initializer that read the
environment. A static initializer that reads the environment cannot be
unit tested without a subprocess or a custom classloader. So the guard is
now a pure function, requireLogGroupName(String), and the handler passes
System.getenv to it. Six tests cover it: null and three blank values
throw with the exact message, a set value is returned, and the shared
config registers exactly one plugin factory. Removing the guard fails
four of them.

The module is outside the SDK reactor, so the root build never runs its
tests. The telemetry conformance workflow now runs `mvn package` without
-DskipTests, which runs them before the deploy artifact is built. JUnit
and surefire versions are repeated in the standalone pom from the root
pom for the same reason.
@hln33
hln33 deployed to ai-pr-review-runtime November 5, 2026 00:45 — with GitHub Actions Active
@github-actions

This comment has been minimized.

DurableConfig.build() fills in a default DurableExecutionClient when none
is given. That constructor builds a LambdaClient from environment
credentials and sends one getDurableExecutionState warm-up request. So
the test that called TelemetryConformanceConfig.durableConfig sent a
real request whenever AWS credentials were exported. The helper javadoc
also said the method made no network call, which was false.

The plugin settings now live in a pure function, telemetryConfig(String),
and durableConfig(String) wraps it. The test checks telemetryConfig: one
CloudWatchLogsExporter, emit ON_COMPLETE, sampling rate 1.0, TOP_LEVEL
operation detail, input and output included. The exporter creates its
client on first export, so the test builds no client. The deployed run
exercises durableConfig, which has no branch of its own.

Test declarations use var where the type is obvious, per AGENTS.md.
@hln33
hln33 deployed to ai-pr-review-runtime November 5, 2026 01:03 — with GitHub Actions Active
@github-actions

This comment has been minimized.

AGENTS.md requires static imports for enum constants. The two assertions
used the qualified names EmitMode.ON_COMPLETE and OperationDetail.TOP_LEVEL.
@hln33
hln33 deployed to ai-pr-review-runtime November 5, 2026 01:08 — with GitHub Actions Active
@github-actions

This comment has been minimized.

@ParidelPooya ParidelPooya left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The handler matches the telemetry-2 requirement, the Telemetry1Success refactor does not change its record, and CI passes. One item must change before merge.

Blocking

  1. .github/workflows/telemetry-conformance-tests.yml:50 pins telemetry-suite.yml to 121e832. That commit is only on the branch of aws/aws-durable-execution-conformance-tests#179, which is still open. The conformance repository allows only squash and rebase merges, and it deletes the head branch on merge. So after #179 merges, 121e832 is on no branch. The comment on lines 45-46 says the pinned SHA must be reachable from a branch. Fix: after #179 merges, pin to its merge commit on conformance main and replace the TEMPORARY comment. Marking this PR as draft until then would prevent an early merge.

Non-blocking

  1. Telemetry2Failure.java:47 catches RuntimeException. The step failure is StepFailedException. Other SDK exceptions, such as NonDeterministicExecutionException, also extend RuntimeException. So the handler would rethrow them as payment failed. A catch (StepFailedException stepError) states the intent and avoids this. Suspension is not affected, because SuspendExecutionException extends Error. The JS handler also catches every error, so keep the current code if parity with JS matters more.
  2. conformance-tests-telemetry is not a module in the root pom.xml. So the spotless-check job does not check the new and changed Java files. The PR body lists this as a follow-up. An issue would keep it tracked.

@hln33
hln33 deployed to ai-pr-review-runtime November 5, 2026 17:58 — with GitHub Actions Active
@github-actions

This comment has been minimized.

StepOperation.get() rebuilds the step's original exception from the
checkpoint and rethrows it. StepFailedException is only the fallback
when that rebuild fails. So the step surfaces as
RuntimeException("card declined"), and catch (StepFailedException)
from 0bb88f5 let it escape. The telemetry run failed with
"expected 'payment failed', got 'card declined'". Catch
RuntimeException again and state the reason in a comment.
@hln33

hln33 commented Nov 5, 2026

Copy link
Copy Markdown
Contributor Author

Pushed e8a5b91bc. It reverts 0bb88f5e3 and keeps catch (RuntimeException).

Item 2 rests on the premise that the step failure is StepFailedException. That premise does not hold here.

  1. StepOperation.get() (sdk/src/main/java/software/amazon/lambda/durable/operation/StepOperation.java:229-235) calls deserializeException on the checkpointed error and rethrows the rebuilt original exception with sneakyThrow.
  2. StepFailedException is thrown only when the rebuild fails: class not found, serdes error, or null error data (StepOperationTest.java:157-218).
  3. The step throws RuntimeException("card declined"). java.lang.RuntimeException is on the classpath and the default serdes deserializes it. So the handler receives a RuntimeException, not a StepFailedException.
  4. So catch (StepFailedException) did not match, and card declined escaped. The telemetry run on 0bb88f5e3 failed: ExpectedResult.Error.ErrorMessage: expected 'payment failed', got 'card declined' (run 42079594299).

Telemetry2Failure.java:47-51 now states this reason in a comment. The PR body bullet is updated. Item 3 (reactor) stays a separate PR. The pin bump still waits for aws/aws-durable-execution-conformance-tests#179.

@hln33
hln33 deployed to ai-pr-review-runtime November 5, 2026 18:27 — with GitHub Actions Active
@github-actions

This comment has been minimized.

The previous pin was the head of aws-durable-execution-conformance-tests
PR #179 (121e832), a temporary pin so the run exercised telemetry-2 before
#179 merged. #179 is merged as ca2cd02 on that repository's main branch, so
the pin now points at that merge commit.
@hln33
hln33 deployed to ai-pr-review-runtime November 5, 2026 21:04 — with GitHub Actions Active
@github-actions

github-actions Bot commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

Codex AI review

No actionable findings after reviewing the changes against rules 1–11, SDK invariants, and module guidance. Residual test risk: this read-only review did not execute the unit or cloud conformance suites.

Reviewed commit a86237028d848be3b9125c2d642c48914aba55ea. Workflow run

@ParidelPooya ParidelPooya left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. I checked a862370.

  1. The workflow pins telemetry-suite.yml to ca2cd02, which is the #179 merge commit on conformance main. The telemetry job passes on this pin.
  2. You are right about the catch, and my comment was wrong. StepOperation.java:229-235 rebuilds the original exception from the checkpoint and rethrows it. It throws StepFailedException only when that rebuild fails. So catch (StepFailedException ...) would let card declined escape. catch (RuntimeException ...) is correct, and the new comment explains why.

Approving.

@hln33
hln33 merged commit 27e177c into main Nov 5, 2026
19 checks passed
@hln33
hln33 deleted the feat/telemetry-2-failure-handler branch November 5, 2026 22:28

This branch was successfully deployed

1 active deployment
ai-pr-review-runtime — a8623702 Deployed Nov 5, 2026 by hln33 via ai-pr-review / Codex review / Generate Codex review #2861
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants