Skip to content

feat(sdk): add distributed map operation - #889

Draft
nvasiu wants to merge 1 commit into
mainfrom
feat/distributed-map-operation
Draft

nvasiu wants to merge 1 commit into
mainfrom
feat/distributed-map-operation

Conversation

@nvasiu

@nvasiu nvasiu commented Nov 13, 2026

Copy link
Copy Markdown
Contributor

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

Issue Link, if available

N/A

Description

Adds the distributed map operation (ctx.distributedMap) to the Java SDK:
A map run processes a bounded dataset in parallel. A customer starts a map run from a durable function, naming a source to read items from, a processor function to invoke per batch, and concurrency, retry, and failure settings. The service reads items from the source, groups them into batches, invokes the processor for each batch, retries failures, tracks progress, routes successful results and failed items to destinations, and reports completion.

config/

The input types a customer constructs to describe a map run.

  • DistributedMapConfig: optional settings for distributed map. Its CompletionConfig defines item failure thresholds for marking the overall map run failed.
  • DistributedMapSource: factories naming where map run items come from. inline, s3JsonLines, s3JsonArray, s3Csv, s3Objects, s3FlattenedJsonLines, s3FlattenedJsonArray, s3FlattenedCsv and reader.
  • DistributedMapProcessor: describes the Lambda that processes items. The reportBatchOutcome and reportItemOutcomes factories pick how the processor reports outcomes back. The retry option takes a DistributedMapProcessorRetryConfig holding maxRetryAttempts and maxRetryDuration, with DistributedMapProcessorRetryConfig.UNLIMITED for an unbounded attempt count.
  • DistributedMapDestination: the s3Successes and s3Failures factories for routing item records to S3.
  • Each config validates its own arguments on construction: a destination prefix of at most 900 UTF-8 bytes, a batch size between 1 and 10000, a timeout of at most 90 days, and retry attempts that are non-negative or UNLIMITED.

DurableContext.java, context/DurableContextImpl.java

The customer-facing entry point on the durable context.

  • ctx.distributedMap: the method a customer calls to run a distributed map. Overloads that take a result type return DistributedMapResult, and the others return DistributedMapSummary.

model/

The result types a customer receives back from a map run.

  • DistributedMapSummary: describes the run's overall outcome.
  • DistributedMapResult: returned by the overloads that take a result type. Contains individual map run item outcomes.
  • DistributedMapError: the error on a run or on one item.
  • DistributedMapStatus: the run's terminal status.
  • The item counts and each collected result's item id are required. A terminal operation that omits a count, or a result that omits its item id, throws an error naming the missing field rather than substituting a value the service never sent.

dmap/

Authoring wrappers for the processor Lambda, so a customer writes a plain function rather than the item or batch protocol.

  • DistributedMapHandlers: createDistributedMapItemHandler, createDistributedMapBatchHandler, createDistributedMapReader, and the durable variants createDistributedMapItemHandlerWithDurableExecution and createDistributedMapBatchHandlerWithDurableExecution.
  • The customer's function receives the run ARN, and the item ids where they apply, through ItemContext, BatchContext and ReaderContext, passed as the last argument.
  • The durable variants take a DurableConfig, so a durable processor can register plugins or supply its own client.
  • Responses are built from records (ItemHandlerResponse, ItemResult, ItemFailure, ReaderResponse) rather than hand-built maps.
  • The wrappers validate the invocation input and reject a record that carries no item id, since every outcome is reported against an item id.

operation/DistributedMapOperation.java

The operation that drives the run against the durable execution runtime.

  • Suspends the caller's function while the run executes and resumes it with the finished outcome.
  • Every terminal state resolves, and throwIfError() opts into throwing.
  • All translation between the config types and the service shapes happens here.

execution/

  • CheckpointManager: the checkpoint size estimate counts the distributed map options, so checkpoint batches stay within the limit.
  • DurableExecutor: a processor's invocation input, result and error are read and written with the default JSON serdes, because the service defines their format. Durable operations inside the processor still use the config's serdes.

exception/DistributedMapException.java

The error type a customer catches.

  • DistributedMapException: thrown by throwIfError() when a run or an item fails.

model/OperationSubType.java, operation/BaseDurableOperation.java, operation/ChildContextOperation.java

Operation type registration.

  • Adds DISTRIBUTED_MAP to OperationSubType, and passes a failed run's error to plugins.

Future tasks

  • Add distributed map to LocalDurableTestRunner (in the testing package). When this is done, we can add full end to end tests using the runner.
  • Add distributed map examples and a docs page.

Demo/Screenshots

N/A

Design

See Rules for every change.

  • Design note (rule 3), for a new feature, a new option or a public API change: Feature planned internally.
  • Simpler options considered (rule 2), if the change adds logic: N/A
  • Root cause (rule 5), for a bug fix: N/A
  • Net change in source lines and in public API members:
    • +2522 lines in src/main and +3764 in src/test.
    • New public members: the ctx.distributedMap overloads, 5 config classes in config/ with their nested types and factories, 4 result types in model/, the DistributedMapHandlers wrappers with their 3 functional interfaces, 11 records in dmap/, 1 exception type, and the DISTRIBUTED_MAP value of OperationSubType.

Assumptions

Each assumption about the service, Lambda, the runtime or another library, with its evidence, or marked "unverified" (rule 6):

  • A distributed map operation ends in one of SUCCEEDED, FAILED, STOPPED or TIMED_OUT. Unverified.
  • A terminal operation's DistributedMapDetails carries SuccessCount, FailureCount and ReadCount, carries TotalCount once the run succeeded, and reports a run failure as Error there rather than as an operation-level error. Unverified.
  • Every collected result in Results carries ItemId and a Status of SUCCEEDED or FAILED, with the item's value in Output as a string the customer's serdes decodes. Unverified.
  • A processor invocation payload carries distributedMapRunArn and a records list, each record holding a non-empty string itemId and a body the customer's serdes decodes. Unverified.
  • The service matches a reported outcome back to its item by the itemId the processor echoes, and accepts the batch-failure and item-outcome response shapes the wrappers return. Unverified.
  • A reader invocation payload carries distributedMapRunArn, an integer maxItemsPerPage, and state as the reader last returned it. Unverified.
  • The service accepts the source, processor, completion and destination configuration the checkpoint sends, with MaxRetryAttempts of -1 meaning unlimited attempts and MaxRetryDurationSeconds a whole number of seconds. Unverified.
  • The service's limits are the ones the configs validate against: a destination key prefix of 900 UTF-8 bytes, a batch size of 10000 items, and a map run timeout of 90 days. The ceiling on concurrent processor invocations is the service's alone, since the SDK enforces none. Unverified.

Decisions not covered by the issue or the design note

One line each, with the reason (rule 7):

  • An invalid configuration throws IllegalArgumentException, because the other operations' configuration checks throw it.
  • DistributedMapException stays outside the DurableOperationException hierarchy, because it reports a run's aggregate result rather than one operation's failure.
  • DurableExecutor picks the default JSON serdes when the input type is ProcessorEvent, so a customer serdes cannot change the format the service defines, with no new public method or DurableConfig option.
  • The source factories are flat on DistributedMapSource, named s3JsonLines and so on, so the factory name states the source kind in one call.
  • The destination prefix check is 900 UTF-8 bytes, which is the lower of the service's character and byte limits, so a multi-byte prefix is rejected at the real limit.
  • No upper bound on concurrency is enforced in the SDK, so a service-side limit change needs no SDK release.
  • A processor record with no item id fails the invocation before any item handler runs, rather than failing that record alone, because an outcome cannot be reported for an item that was not named.
  • The context records are passed as the last argument, because Java's other operations pass their context last. The durable wrappers keep DurableContext first.
  • A null inline or reader item is sent as "null", and a non-null value that serializes to null is rejected, because the service requires every item to be a string.
  • Items, Output and the record body are opaque strings, so a customer serdes other than JSON works without an SDK change.

Checklist

  • I have filled out every section of the PR template
  • I have thoroughly tested this change
  • Every new branch has a test that fails without it (rule 4)
  • For a bug fix: the description or commit order shows the regression test failing before the fix and passing after it (rule 5)
  • Every edited test expectation is listed in the description, with the reason (rule 5)

Testing

Unit Tests

Have unit tests been written for these changes?

Yes. Tests are split by layer so each sits alongside its class.

  • config/DistributedMap*Test: config and argument validation.
  • model/DistributedMapSummaryTest and model/DistributedMapResultTest: the result types and throwIfError().
  • operation/DistributedMapOperationTest and operation/DistributedMapOperationTranslationTest: operation behaviour and the translation to and from the service shapes.
  • context/DurableContextDistributedMapTest: the ctx.distributedMap surface.
  • dmap/DistributedMapHandlersTest and dmap/DistributedMapDurableHandlersTest: authoring wrapper tests, checking that they process items, report failures, pass the run ARN and item ids, and reject bad inputs.
  • execution/CheckpointManagerTest: new tests for the size estimate. No existing test expectation was edited.

Integration Tests

Have integration tests been written for these changes?

No. LocalDurableTestRunner does not support distributed map yet (see Future tasks).

Examples

Has a new example been added for the change? (if applicable)

No. Examples will follow with runner support (see Future tasks).

Add distributed map, a durable operation that processes a large set of
items in separate processor Lambda functions run by the service, and
returns the run's outcome to the parent durable function.

- DurableContext.distributedMap starts a run and waits for it to finish.
  It returns a summary of the counts and status, or a result that also
  holds each item's output or error. The run is checkpointed, so it is
  started once and its outcome is reused on replay.
- Items come from an inline list, S3 objects (JSON, JSON Lines or CSV),
  or a custom reader function. Configuration covers the processor, batch
  size, concurrency, failure tolerance and S3 result destinations, and is
  validated before the start checkpoint.
- DistributedMapHandlers wraps a customer's function as a processor or
  reader Lambda handler, with durable and non-durable variants. The
  wrappers handle the service's event and response formats, and pass the
  run ARN and item ids to the customer's function.
@zhongkechen

Copy link
Copy Markdown
Contributor

@nvasiu could you split this PR into smaller ones so that the changes don't rely on new models can be merged early?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants