Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,25 @@

All notable changes to the E2A framework documents and reference code.

## v1.7.1 — Durable outbox stores (design note MP-05)

### Added
- **`e2a.port_contract.check_outbox_store`:** 11 scenarios every `BaseOutboxStore` adapter passes:
- round trip; oldest first; the limit;
- mark once; an unknown id;
- fencing (a stale sweeper cannot mark);
- a race in which exactly one sweeper wins.
- **e2a-adapters 1.7.1, the transactional outbox in each database's own transaction.** A record commits with the business change or not at all:
- **`DynamoDBOutboxStore`** puts `transact_item(record)` in your `TransactWriteItems`. Pending records are one Query in id order (ids sort by time). `mark_published` deletes the record and advances the fence in one transaction.
- **`FirestoreOutboxStore`** takes `add(record, transaction=tx)` (or `batch=`). Pending is ordered by `created_at` with no composite index; `mark_published` is one transaction.
- **`CosmosOutboxStore`** keeps the record in the business document's partition: `batch_operation(record, partition_key)` goes into your transactional batch. Pending is one cross-partition query. The fence and the delete are two conditional writes, because they are in different partitions; a stale double-mark is harmless because consumers deduplicate.
- All three pass the contract on DynamoDB Local and the Firestore and Cosmos DB emulators.
- **The cloud profiles bind `outbox_store`** (key `outbox_name`, default `default`).

### Changed
- **`PubSubBroker` imports its SDK on first use** (MP-05 follow-up): a profile binds and verifies without google-cloud-pubsub installed.
- **`param_dropped` is logged only for a temperature someone set** (keyword, config, profile or environment), not for the registry default (MP-04 follow-up). A model declaring `sampling: unsupported` no longer warns on every call.

## v1.7.0 — Messaging, identity, secrets and the cloud profiles (design note MP-05)

Cloud infrastructure behind E2A's ports, selected by configuration. The contracts are vendor-neutral and
Expand Down
7 changes: 4 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,16 +12,16 @@ It focuses on the boundary where probabilistic reasoning meets deterministic ent

## Install

pip install "e2a-framework @ git+https://github.com/subhamviky/e2a-framework.git@v1.7.0"
pip install "e2a-framework @ git+https://github.com/subhamviky/e2a-framework.git@v1.7.1"

Poetry: `e2a-framework = { git = "https://github.com/subhamviky/e2a-framework.git", tag = "v1.7.0" }`.
Poetry: `e2a-framework = { git = "https://github.com/subhamviky/e2a-framework.git", tag = "v1.7.1" }`.
The framework is the `e2a` package (`from e2a import BaseAgent, BaseWorkflow, Steps, Step`); the release
gate is `from e2a.release import ProjectReleaseGate`; the `e2a_base` module re-exports the same names. Since v1.3 hooks use the v1.2 signature (`state, ctx, …`).

**Production adapters (e2a-adapters, since v1.4.0):** DynamoDB (v1.4.0), Firestore and Cosmos DB (v1.5.0)
implementations of the idempotency, transactional, lease and checkpoint ports; since v1.7.0 also the brokers
(SQS/SNS, Pub/Sub, Service Bus), the secret stores and the cloud profiles `aws`, `gcp`, `azure`, in [`adapters/`](adapters/README.md),
a separate distribution at this tag (`pip install "e2a-adapters[aws,gcp,azure] @ git+https://github.com/subhamviky/e2a-framework.git@v1.7.0#subdirectory=adapters"`;
a separate distribution at this tag (`pip install "e2a-adapters[aws,gcp,azure] @ git+https://github.com/subhamviky/e2a-framework.git@v1.7.1#subdirectory=adapters"`;
pick the extras you need).
Every adapter passes the port contract kit, `e2a.port_contract`.

Expand All @@ -33,6 +33,7 @@ registry: a service's CI builds it from the tag (`mvn -f java/pom.xml install -D

| e2a-framework | a2c-framework | capability-oriented-dsa |
|---|---|---|
| `v1.7.1` (durable outbox stores on DynamoDB, Firestore and Cosmos DB, with an outbox contract; Pub/Sub SDK imported on first use; the default temperature no longer warns) | `v1.4.0` | `v0.2.x`, `v0.3.x` |
| `v1.7.0` (messaging with exactly-once effects, identity at ingress, the literal-secret rule; e2a-adapters `1.7.0`: brokers, secret stores, cloud profiles; e2a-java `1.7.0`) | `v1.4.0` | `v0.2.x`, `v0.3.x` |
| `v1.6.1` (vendor-neutral handling for managed providers: model routes, structured-output checks, scopes, telemetry and cost, adaptive admission, delegated runtimes, reranker port, classifier guardrails, pipeline and MCP checks; the neutrality test) | `v1.3.0` | `v0.2.x`, `v0.3.x` |
| `v1.6.0` (contracts for managed providers: canonical types, typed refusals, bindings and profiles, capabilities, credentials, `check_llm_service`) | `v1.3.0` | `v0.2.x`, `v0.3.x` |
Expand Down
11 changes: 6 additions & 5 deletions adapters/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -14,7 +14,7 @@ An application chooses its cloud by naming a profile; every adapter is built fro
from e2a import bindings_from_profile
bindings = bindings_from_profile("aws", verify=True) # or "gcp", "azure"; or E2A_PROVIDER_PROFILE
config = bindings.config() # idempotency_store, checkpoint_store, transactional_store, lease,
# message_broker, credential_provider, identity_resolver
# message_broker, outbox_store (1.7.1), credential_provider, identity_resolver
```

| Profile | Stores | Messaging | Tool credentials | Identity claims |
Expand Down Expand Up @@ -42,6 +42,7 @@ Clients are created on first use, so a profile binds and verifies without creden
| Google Pub/Sub (since 1.7.0) | `e2a-adapters[gcp]` | `PubSubBroker`: topics and subscriptions, a `<subscription>-dlq` topic and subscription |
| Azure Service Bus (since 1.7.0) | `e2a-adapters[azure]` | `ServiceBusBroker`: queues, topic subscriptions, the native dead-letter subqueue |
| Secret stores (since 1.7.0) | per cloud | `SecretsManagerCredentialProvider`, `SecretManagerCredentialProvider`, `KeyVaultCredentialProvider`: tool auth headers from `secret_headers` |
| Outbox stores (since 1.7.1) | per cloud | `DynamoDBOutboxStore` (`transact_item` for your TransactWriteItems), `FirestoreOutboxStore` (`add(..., transaction=)`), `CosmosOutboxStore` (`batch_operation` for your transactional batch, in the business document's partition) |
| Identity presets (since 1.7.0) | – | `e2a_adapters.identity.PRESETS`: the claim maps of Cognito, Entra ID and Identity Platform |

Firestore and Cosmos DB share one implementation of the ports, `e2a_adapters.cas`. It works over any document store that can create-if-absent and replace-if-unchanged (an update time, an ETag), and each database supplies only a small `CasBackend`. PostgreSQL would be a third backend.
Expand All @@ -51,15 +52,15 @@ Firestore and Cosmos DB share one implementation of the ports, `e2a_adapters.cas
The version follows E2A. Install e2a-framework from its tag first, as every project already does:

```bash
pip install "e2a-framework @ git+https://github.com/subhamviky/e2a-framework.git@v1.7.0" \
"e2a-adapters[aws] @ git+https://github.com/subhamviky/e2a-framework.git@v1.7.0#subdirectory=adapters"
pip install "e2a-framework @ git+https://github.com/subhamviky/e2a-framework.git@v1.7.1" \
"e2a-adapters[aws] @ git+https://github.com/subhamviky/e2a-framework.git@v1.7.1#subdirectory=adapters"
```

Poetry:

```toml
"e2a-framework" = { git = "https://github.com/subhamviky/e2a-framework.git", tag = "v1.7.0" }
"e2a-adapters" = { git = "https://github.com/subhamviky/e2a-framework.git", tag = "v1.7.0", subdirectory = "adapters", extras = ["aws"] }
"e2a-framework" = { git = "https://github.com/subhamviky/e2a-framework.git", tag = "v1.7.1" }
"e2a-adapters" = { git = "https://github.com/subhamviky/e2a-framework.git", tag = "v1.7.1", subdirectory = "adapters", extras = ["aws"] }
```

## DynamoDB
Expand Down
2 changes: 1 addition & 1 deletion adapters/e2a_adapters/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -15,4 +15,4 @@
"""
from . import config_keys # noqa: F401 (registers the cloud keys)

__version__ = '1.7.0'
__version__ = '1.7.1'
4 changes: 3 additions & 1 deletion adapters/e2a_adapters/aws/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,14 @@

DynamoDB* claims, rows, leases, checkpoints
SQSBroker messaging (SQS queues, SNS topics), with dead letters
DynamoDBOutboxStore the transactional outbox (v1.7.1)
SecretsManagerCredentialProvider tool credentials from Secrets Manager
"""
from .dynamodb import (MAX_VALUE_BYTES, DynamoDBCheckpointStore, DynamoDBIdempotencyStore, DynamoDBLease,
DynamoDBTransactionalStore, is_condition_failure)
from .outbox import DynamoDBOutboxStore
from .secrets import SecretsManagerCredentialProvider
from .sqs import SQSBroker

__all__ = ['DynamoDBIdempotencyStore', 'DynamoDBTransactionalStore', 'DynamoDBLease', 'DynamoDBCheckpointStore',
'MAX_VALUE_BYTES', 'is_condition_failure', 'SQSBroker', 'SecretsManagerCredentialProvider']
'MAX_VALUE_BYTES', 'is_condition_failure', 'SQSBroker', 'SecretsManagerCredentialProvider', 'DynamoDBOutboxStore']
103 changes: 103 additions & 0 deletions adapters/e2a_adapters/aws/outbox.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,103 @@
"""The transactional outbox on DynamoDB (v1.7.1, design note MP-05, N1).

A record is written in the SAME transaction as the business change: put transact_item(record) in
the application's TransactWriteItems (or call add() when there is no other write). The
sweeper (e2a.BrokerOutboxSweeper) reads pending() and publishes, and mark_published() deletes
the record in one transaction with the fencing check, so a stale sweeper cannot mark.

Layout, in any table with string keys pk / sk (e2a-adapters' state table by default):

pk = E2A#OUTBOX#<outbox_name> sk = <record id> a pending record
pk = E2A#OUTBOX-FENCE sk = <outbox_name> the newest fencing token seen

Record ids sort by creation time (microseconds, then a random suffix), so pending() is one
Query in key order. All pending records share a partition: one outbox handles what one
partition writes (about 1,000 records a second); use several outbox names to go beyond that.
"""
import json
import time
import uuid
from typing import Any, Callable, Dict, List, Optional

from botocore.exceptions import ClientError
from e2a import resolve_config
from e2a.messaging import BaseOutboxStore

from .. import config_keys # noqa: F401
from .._lazy import Lazy


def _s(value: str) -> Dict[str, str]:
return {'S': str(value)}


class DynamoDBOutboxStore(BaseOutboxStore):
capabilities = frozenset({'outbox_store'})

def __init__(self, client=None, table_name: str = None, name: str = 'default',
clock: Callable[[], float] = time.time) -> None:
if client is None or not table_name:
raise ValueError('DynamoDBOutboxStore needs client= and table_name= (or use from_config)')
self.client, self.table_name, self.name, self.clock = client, table_name, name, clock
self.pk = f'E2A#OUTBOX#{name}'

@classmethod
def from_config(cls, config: Dict[str, Any], **kwargs) -> 'DynamoDBOutboxStore':
import boto3
region, endpoint = resolve_config('aws_region', config), resolve_config('aws_endpoint_url', config)
return cls(Lazy(lambda: boto3.client('dynamodb', region_name=region, endpoint_url=endpoint)),
resolve_config('dynamodb_state_table', config), resolve_config('outbox_name', config), **kwargs)

def _new_id(self) -> str:
return f'{int(self.clock() * 1_000_000):017d}-{uuid.uuid4().hex[:12]}'

def transact_item(self, record: Dict[str, Any]) -> Dict[str, Any]:
"""The Put for the application's own TransactWriteItems: the record is committed if and
only if the business change is. record['id'] is set when absent."""
record.setdefault('id', self._new_id())
payload = {k: record.get(k) for k in ('destination', 'body', 'attributes', 'tenant_id', 'idempotency_key')}
return {'Put': {'TableName': self.table_name,
'Item': {'pk': _s(self.pk), 'sk': _s(record['id']), 'created_at': {'N': repr(self.clock())},
'record': _s(json.dumps(payload, separators=(',', ':'), ensure_ascii=False))},
'ConditionExpression': 'attribute_not_exists(pk)'}}

def add(self, record, config=None, **kwargs):
item = self.transact_item(dict(record))
self.client.put_item(**item['Put'])
return item['Put']['Item']['sk']['S']

def pending(self, limit=100, config=None, **kwargs):
resp = self.client.query(TableName=self.table_name, KeyConditionExpression='pk = :pk',
ExpressionAttributeValues={':pk': _s(self.pk)}, ConsistentRead=True,
Limit=max(1, int(limit)))
out: List[Dict[str, Any]] = []
for item in resp.get('Items', []):
out.append({'id': item['sk']['S'], **json.loads(item['record']['S'])})
return out

def mark_published(self, record_id, config=None, **kwargs):
delete = {'Delete': {'TableName': self.table_name, 'Key': {'pk': _s(self.pk), 'sk': _s(record_id)},
'ConditionExpression': 'attribute_exists(pk)'}}
token: Optional[int] = kwargs.get('fencing_token')
try:
if token is None:
self.client.delete_item(**delete['Delete'])
else:
fence = {'Update': {'TableName': self.table_name,
'Key': {'pk': _s('E2A#OUTBOX-FENCE'), 'sk': _s(self.name)},
'UpdateExpression': 'SET fence = :t',
'ConditionExpression': 'attribute_not_exists(fence) OR fence <= :t',
'ExpressionAttributeValues': {':t': {'N': str(int(token))}}}}
self.client.transact_write_items(TransactItems=[fence, delete])
except ClientError as e:
code = e.response.get('Error', {}).get('Code')
if code == 'ConditionalCheckFailedException':
return False
if code == 'TransactionCanceledException':
# a lost condition (stale fence, already published) or a concurrent transaction
# on the same items is a lost race; throttling or validation is an error
reasons = [r.get('Code') for r in e.response.get('CancellationReasons') or []]
if all(r in (None, 'None', 'ConditionalCheckFailed', 'TransactionConflict') for r in reasons):
return False
raise
return True
4 changes: 3 additions & 1 deletion adapters/e2a_adapters/azure/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -2,12 +2,14 @@

Cosmos* claims, rows, leases, checkpoints
ServiceBusBroker messaging, with native dead letters
CosmosOutboxStore the transactional outbox (v1.7.1)
KeyVaultCredentialProvider tool credentials from Key Vault
"""
from .cosmos import (CosmosBackend, CosmosCheckpointStore, CosmosIdempotencyStore, CosmosLease,
CosmosTransactionalStore)
from .keyvault import KeyVaultCredentialProvider
from .outbox import CosmosOutboxStore
from .servicebus import ServiceBusBroker

__all__ = ['CosmosBackend', 'CosmosIdempotencyStore', 'CosmosTransactionalStore', 'CosmosLease',
'CosmosCheckpointStore', 'ServiceBusBroker', 'KeyVaultCredentialProvider']
'CosmosCheckpointStore', 'ServiceBusBroker', 'KeyVaultCredentialProvider', 'CosmosOutboxStore']
Loading
Loading