Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 22 additions & 4 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -46,22 +46,40 @@ jobs:
python-version: "3.12"
- name: Install the core, then the adapters with every backend and the test extras
run: pip install . && pip install "./adapters[aws,gcp,azure,test]"
- name: Start the emulators (DynamoDB Local, Firestore, Cosmos DB)
# The contract kit's concurrent races need atomic backends: moto's in-process DynamoDB is not one
- name: Start the emulators (DynamoDB Local, Firestore, Cosmos DB, Pub/Sub, Service Bus)
# The contract kit's concurrent races need atomic backends: moto's in-process DynamoDB is not one.
# v1.7.0: the brokers' messaging suite runs on the Pub/Sub and Service Bus emulators (SQS on moto).
# The Service Bus emulator keeps its state in SQL Edge and declares its queues up front.
env:
SQL_PASSWORD: E2a-Emulator-Pass1! # a throwaway password for the CI-only database
run: |
docker run -d --name ddb -p 8000:8000 amazon/dynamodb-local:latest -jar DynamoDBLocal.jar -inMemory -port 8000
docker run -d --name firestore -p 8085:8085 gcr.io/google.com/cloudsdktool/google-cloud-cli:emulators \
gcloud emulators firestore start --host-port=0.0.0.0:8085
docker run -d --name cosmos -p 8081:8081 mcr.microsoft.com/cosmosdb/linux/azure-cosmos-emulator:vnext-preview
for port in 8000 8085 8081; do
docker run -d --name pubsub -p 8086:8086 gcr.io/google.com/cloudsdktool/google-cloud-cli:emulators \
gcloud beta emulators pubsub start --host-port=0.0.0.0:8086 --project=demo-e2a-adapters
docker run -d --name sqledge --network host -e ACCEPT_EULA=Y -e MSSQL_SA_PASSWORD="$SQL_PASSWORD" \
mcr.microsoft.com/azure-sql-edge:latest
sleep 20
docker run -d --name servicebus --network host -e ACCEPT_EULA=Y -e SQL_SERVER=localhost \
-e MSSQL_SA_PASSWORD="$SQL_PASSWORD" -e SQL_WAIT_INTERVAL=5 \
-v "$PWD/adapters/tests/servicebus-emulator.json:/ServiceBus_Emulator/ConfigFiles/Config.json" \
mcr.microsoft.com/azure-messaging/servicebus-emulator:latest
for port in 8000 8085 8081 8086; do
for _ in $(seq 90); do curl -s -o /dev/null "http://127.0.0.1:$port/" && break; sleep 2; done
done
for _ in $(seq 60); do docker logs cosmos 2>&1 | grep -q "Gateway=OK" && break; sleep 2; done
- name: Adapter tests (the port contract kit on every backend)
for _ in $(seq 90); do docker logs servicebus 2>&1 | grep -q "Successfully Up" && break; sleep 2; done
docker logs servicebus 2>&1 | tail -2
- name: Adapter tests (the port contract kit and the messaging suite on every backend)
env:
DYNAMODB_LOCAL_ENDPOINT: http://127.0.0.1:8000
FIRESTORE_EMULATOR_HOST: 127.0.0.1:8085
COSMOS_EMULATOR_ENDPOINT: http://127.0.0.1:8081
PUBSUB_EMULATOR_HOST: 127.0.0.1:8086
# the emulator's documented, fixed connection string (not a secret)
SERVICEBUS_CONNECTION_STRING: "Endpoint=sb://localhost;SharedAccessKeyName=RootManageSharedAccessKey;SharedAccessKey=SAS_KEY_VALUE;UseDevelopmentEmulator=true;"
# -rs lists skips: none is expected here, every emulator is running
run: python -m pytest -q -rs adapters/tests
- name: Governed Entry Point rules over the adapters
Expand Down
55 changes: 55 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,61 @@

All notable changes to the E2A framework documents and reference code.

## v1.7.0 — Messaging, identity, secrets and the cloud profiles (design note MP-05)

Cloud infrastructure behind E2A's ports, selected by configuration. The contracts are vendor-neutral and
live in `e2a`; the cloud adapters live in e2a-adapters 1.7.0 (no AI vendor SDK there, by test).
Additive: no signature changes. e2a-adapters, e2a-java and the pins move to `1.7.0`.

### Added (e2a)
- **`e2a.messaging` (N1): exactly-once effects under at-least-once delivery.**
- `BaseMessageBroker` is the port: publish, receive, ack, nack, dead_letter, receive_dead_letters, ack_dead_letter. `BrokerMessage` is one shape for every broker; tenant and idempotency key travel as attributes.
- **`IdempotentConsumer`**: `consume()` claims every message in a `BaseIdempotencyStore`.
- A duplicate or a lost acknowledgement is acked without a second effect.
- A message another consumer holds is deferred.
- A deterministic error is dead-lettered (`FAILED_FINAL`).
- A transient error is retried.
- An `UNKNOWN` outcome is dead-lettered, never retried blindly.
- Too many deliveries (`consumer_max_deliveries`) dead-letters the message.
- **`reconcile()`** (inherited from `BaseReconciliationService`) works the dead letters. It confirms physical truth through `_confirm` before any replay, and settles a dead letter only when its effect is confirmed or replayed.
- **The outbox:** a `BaseOutboxStore` port with `InMemoryOutboxStore`, and `BrokerOutboxSweeper`. The sweeper publishes only records a consumer can deduplicate, and marks them published under the lease's fencing token.
- `InMemoryBroker` is the reference broker.
- **`e2a.port_contract.check_message_broker`:** 25 scenarios every broker adapter passes, including:
- round trip, ack, nack and redelivery count, dead letters;
- a duplicate delivery, a lost acknowledgement, a poison message;
- an UNKNOWN outcome reconciled both ways;
- the outbox.
- **`e2a.identity` (N3):** `BaseIdentityResolver` / `ClaimMapIdentityResolver.resolve(claims, verified, request_tenant)` gives the tenant, subject and roles from claims the gateway verified, through `identity_claim_map`.
- It raises `IdentityRefused` (a `GovernanceRefusal`) for: unverified claims, an untrusted issuer (`identity_trusted_issuers`), no tenant, or a request naming another tenant.
- `identity_tenant_map` maps an external directory id to a tenant.
- **The literal-secret rule (N2):** a profile that sets a secret key to a literal value instead of `${VAR}` is recorded (`Profile.literal_secrets`, inherited through `extends`). `Bindings.verify()` warns, and `BasePipeline` fails the release.
- **Seams:** `message_broker`, `outbox_store`, `identity_resolver`, `transactional_store`, `lease`. The ones E2A reads are carried in `Bindings.config()`.
- **Capabilities:** `messaging`, `dead_letter`, `outbox_store`, `identity`, `secret_store`.
- **`BaseReconciliationService._settle_dlq_entry(entry, outcome)`:** a hook with a no-op default, called with each entry's decision.
- **Keys:** `consumer_batch_size`, `consumer_wait_seconds`, `consumer_max_deliveries`, `consumer_retry_delay_seconds`, `identity_claim_map`, `identity_trusted_issuers`, `identity_tenant_map`.

### Added (e2a-adapters 1.7.0)
- **Brokers:** `SQSBroker` (SQS queues, SNS topics, `<queue>-dlq`), `PubSubBroker` (topics and subscriptions, `<subscription>-dlq`) and `ServiceBusBroker` (queues and topic subscriptions, the native dead-letter subqueue).
- All three pass `check_message_broker`: SQS on moto, Pub/Sub and Service Bus on their emulators.
- **Secret stores (N2):** `SecretsManagerCredentialProvider`, `SecretManagerCredentialProvider` and `KeyVaultCredentialProvider`. Each gives tool calls their auth headers from `secret_headers` (longest target prefix, optional `json_key`), cached for `secret_cache_seconds` so rotation needs no restart.
- **`from_config(config)` on every store and broker,** from registered keys with environment variables. Clients are created on first use, so a profile binds and verifies offline.
- **The cloud profiles** `aws`, `gcp` and `azure` (shipped in the package): stores, broker, secret store and the identity claim map of their cloud. Provider profiles extend them.
- **`e2a_adapters.identity.PRESETS`:** Cognito, Entra ID and Identity Platform.
- **Capabilities** declared on every store.
- **Extras:** `[gcp]` adds Pub/Sub and Secret Manager; `[azure]` adds Service Bus, Key Vault and azure-identity.

### Changed
- **`e2a.bindings` finds profiles shipped in e2a-adapters** (`e2a_adapters/profiles/`). `available_profiles()` now lists `aws`, `gcp` and `azure` when it is installed.
- **CI:** the adapters job also runs the Pub/Sub emulator, and the Service Bus emulator with SQL Edge.

### Not in this release
- **Durable outbox stores** (DynamoDB, Firestore, Cosmos DB) and **the projects on the brokers:**
- reconciliation's SQS consumer on `IdempotentConsumer`;
- O2C's outbox on Pub/Sub;
- audit-ai on Service Bus.

These are the next step.

## v1.6.1 — Vendor-neutral handling for managed providers (design note MP-04)

What a managed deployment needs from E2A, selected by configuration, capabilities and the
Expand Down
10 changes: 6 additions & 4 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,16 @@ It focuses on the boundary where probabilistic reasoning meets deterministic ent

## Install

pip install "e2a-framework @ git+https://github.com/subhamviky/e2a-framework.git@v1.6.1"
pip install "e2a-framework @ git+https://github.com/subhamviky/e2a-framework.git@v1.7.0"

Poetry: `e2a-framework = { git = "https://github.com/subhamviky/e2a-framework.git", tag = "v1.6.1" }`.
Poetry: `e2a-framework = { git = "https://github.com/subhamviky/e2a-framework.git", tag = "v1.7.0" }`.
The framework is the `e2a` package (`from e2a import BaseAgent, BaseWorkflow, Steps, Step`); the release
gate is `from e2a.release import ProjectReleaseGate`; the `e2a_base` module re-exports the same names. Since v1.3 hooks use the v1.2 signature (`state, ctx, …`).

**Production adapters (e2a-adapters, since v1.4.0):** DynamoDB (v1.4.0), Firestore and Cosmos DB (v1.5.0)
implementations of the idempotency, transactional, lease and checkpoint ports in [`adapters/`](adapters/README.md),
a separate distribution at this tag (`pip install "e2a-adapters[aws,gcp,azure] @ git+https://github.com/subhamviky/e2a-framework.git@v1.6.1#subdirectory=adapters"`;
implementations of the idempotency, transactional, lease and checkpoint ports; since v1.7.0 also the brokers
(SQS/SNS, Pub/Sub, Service Bus), the secret stores and the cloud profiles `aws`, `gcp`, `azure`, in [`adapters/`](adapters/README.md),
a separate distribution at this tag (`pip install "e2a-adapters[aws,gcp,azure] @ git+https://github.com/subhamviky/e2a-framework.git@v1.7.0#subdirectory=adapters"`;
pick the extras you need).
Every adapter passes the port contract kit, `e2a.port_contract`.

Expand All @@ -32,6 +33,7 @@ registry: a service's CI builds it from the tag (`mvn -f java/pom.xml install -D

| e2a-framework | a2c-framework | capability-oriented-dsa |
|---|---|---|
| `v1.7.0` (messaging with exactly-once effects, identity at ingress, the literal-secret rule; e2a-adapters `1.7.0`: brokers, secret stores, cloud profiles; e2a-java `1.7.0`) | `v1.4.0` | `v0.2.x`, `v0.3.x` |
| `v1.6.1` (vendor-neutral handling for managed providers: model routes, structured-output checks, scopes, telemetry and cost, adaptive admission, delegated runtimes, reranker port, classifier guardrails, pipeline and MCP checks; the neutrality test) | `v1.3.0` | `v0.2.x`, `v0.3.x` |
| `v1.6.0` (contracts for managed providers: canonical types, typed refusals, bindings and profiles, capabilities, credentials, `check_llm_service`) | `v1.3.0` | `v0.2.x`, `v0.3.x` |
| `v1.5.1` (model-call retries, provider metadata kept, rerank query; e2a-adapters and e2a-java `1.5.1`) | `v1.3.0` | `v0.2.x`, `v0.3.x` |
Expand Down
47 changes: 41 additions & 6 deletions adapters/README.md
Original file line number Diff line number Diff line change
@@ -1,13 +1,48 @@
# e2a-adapters

Production adapters for the E2A foundation ports (design note D-02). Each one passes E2A's
port contract kit (`e2a.port_contract`), the same suite as the in-memory reference adapters.
E2A's ports on the clouds' infrastructure (design notes D-02, D-03, MP-05). Each adapter passes
E2A's port contract kit (`e2a.port_contract`), the same suite as the in-memory reference adapters;
each broker passes its messaging suite. No AI vendor SDK is imported here: AI services live in
the provider layer (`e2a-managed-providers`).

## Cloud profiles (since 1.7.0)

An application chooses its cloud by naming a profile; every adapter is built from configuration
(`from_config`), so nothing is constructed by hand:

```python
from e2a import bindings_from_profile
bindings = bindings_from_profile("aws", verify=True) # or "gcp", "azure"; or E2A_PROVIDER_PROFILE
config = bindings.config() # idempotency_store, checkpoint_store, transactional_store, lease,
# message_broker, credential_provider, identity_resolver
```

| Profile | Stores | Messaging | Tool credentials | Identity claims |
|---|---|---|---|---|
| `aws` | DynamoDB | SQS / SNS (`SQSBroker`) | Secrets Manager | Cognito |
| `gcp` | Firestore | Pub/Sub (`PubSubBroker`) | Secret Manager | Identity Platform |
| `azure` | Cosmos DB | Service Bus (`ServiceBusBroker`) | Key Vault | Entra ID |

Every setting is a registered key with an environment variable (`AWS_REGION`, `DYNAMODB_IDEMPOTENCY_TABLE`,
`GOOGLE_CLOUD_PROJECT`, `COSMOS_ENDPOINT`, `SERVICEBUS_NAMESPACE`, `KEYVAULT_URL`, ...; `python -c
"from e2a import config_reference_markdown; import e2a_adapters; print(config_reference_markdown())"`).
Credentials come from the platform identity; the connection string and account key keys exist for
the emulators. A provider profile in `e2a-managed-providers` extends the profile of its cloud.

Clients are created on first use, so a profile binds and verifies without credentials or network.

## Adapters

| Backend | Install | Adapters |
|---|---|---|
| AWS DynamoDB | `e2a-adapters[aws]` | `DynamoDBIdempotencyStore`, `DynamoDBTransactionalStore`, `DynamoDBLease`, `DynamoDBCheckpointStore` |
| GCP Firestore (since 1.5.0) | `e2a-adapters[gcp]` | `FirestoreIdempotencyStore`, `FirestoreTransactionalStore`, `FirestoreLease`, `FirestoreCheckpointStore` |
| Azure Cosmos DB, NoSQL API (since 1.5.0) | `e2a-adapters[azure]` | `CosmosIdempotencyStore`, `CosmosTransactionalStore`, `CosmosLease`, `CosmosCheckpointStore` |
| Amazon SQS / SNS (since 1.7.0) | `e2a-adapters[aws]` | `SQSBroker`: queues, SNS topics, a `<queue>-dlq` dead-letter queue |
| Google Pub/Sub (since 1.7.0) | `e2a-adapters[gcp]` | `PubSubBroker`: topics and subscriptions, a `<subscription>-dlq` topic and subscription |
| Azure Service Bus (since 1.7.0) | `e2a-adapters[azure]` | `ServiceBusBroker`: queues, topic subscriptions, the native dead-letter subqueue |
| Secret stores (since 1.7.0) | per cloud | `SecretsManagerCredentialProvider`, `SecretManagerCredentialProvider`, `KeyVaultCredentialProvider`: tool auth headers from `secret_headers` |
| Identity presets (since 1.7.0) | – | `e2a_adapters.identity.PRESETS`: the claim maps of Cognito, Entra ID and Identity Platform |

Firestore and Cosmos DB share one implementation of the ports, `e2a_adapters.cas`. It works over any document store that can create-if-absent and replace-if-unchanged (an update time, an ETag), and each database supplies only a small `CasBackend`. PostgreSQL would be a third backend.

Expand All @@ -16,15 +51,15 @@ Firestore and Cosmos DB share one implementation of the ports, `e2a_adapters.cas
The version follows E2A. Install e2a-framework from its tag first, as every project already does:

```bash
pip install "e2a-framework @ git+https://github.com/subhamviky/e2a-framework.git@v1.6.0" \
"e2a-adapters[aws] @ git+https://github.com/subhamviky/e2a-framework.git@v1.6.0#subdirectory=adapters"
pip install "e2a-framework @ git+https://github.com/subhamviky/e2a-framework.git@v1.7.0" \
"e2a-adapters[aws] @ git+https://github.com/subhamviky/e2a-framework.git@v1.7.0#subdirectory=adapters"
```

Poetry:

```toml
"e2a-framework" = { git = "https://github.com/subhamviky/e2a-framework.git", tag = "v1.6.0" }
"e2a-adapters" = { git = "https://github.com/subhamviky/e2a-framework.git", tag = "v1.6.0", subdirectory = "adapters", extras = ["aws"] }
"e2a-framework" = { git = "https://github.com/subhamviky/e2a-framework.git", tag = "v1.7.0" }
"e2a-adapters" = { git = "https://github.com/subhamviky/e2a-framework.git", tag = "v1.7.0", subdirectory = "adapters", extras = ["aws"] }
```

## DynamoDB
Expand Down
24 changes: 14 additions & 10 deletions adapters/e2a_adapters/__init__.py
Original file line number Diff line number Diff line change
@@ -1,14 +1,18 @@
"""e2a-adapters: production adapters for E2A's foundation ports (design note D-02).
"""e2a-adapters: E2A's ports on the clouds' infrastructure (design notes D-02, D-03, MP-05).

Each backend is an optional extra and a subpackage, so a project installs only what it uses:
Each cloud is an optional extra and a subpackage, so a project installs only what it uses:

e2a_adapters.aws DynamoDB (pip install "e2a-adapters[aws]")
e2a_adapters.gcp Firestore (pip install "e2a-adapters[gcp]")
e2a_adapters.azure Cosmos DB (pip install "e2a-adapters[azure]")
e2a_adapters.cas the ports over any document store with optimistic concurrency, which
Firestore and Cosmos DB use (and a new backend can too)
e2a_adapters.aws DynamoDB, SQS/SNS, Secrets Manager (pip install "e2a-adapters[aws]")
e2a_adapters.gcp Firestore, Pub/Sub, Secret Manager (pip install "e2a-adapters[gcp]")
e2a_adapters.azure Cosmos DB, Service Bus, Key Vault (pip install "e2a-adapters[azure]")
e2a_adapters.cas the ports over any document store with optimistic concurrency
e2a_adapters.identity claim maps of Cognito, Entra ID and Identity Platform

Every adapter passes E2A's port contract kit (e2a.port_contract), the same suite the in-memory
reference adapters pass.
v1.7.0: every adapter has from_config(config), and the cloud profiles `aws`, `gcp` and `azure`
bind them (bindings_from_profile("aws")); a provider profile extends one. Every adapter passes
E2A's port contract kit (e2a.port_contract), the brokers its messaging suite, on emulators.
No AI vendor SDK is imported here (tests/test_boundary.py).
"""
__version__ = '1.6.0'
from . import config_keys # noqa: F401 (registers the cloud keys)

__version__ = '1.7.0'
22 changes: 22 additions & 0 deletions adapters/e2a_adapters/_lazy.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,22 @@
"""A client built on first use, so from_config() never opens a connection. A profile can then be
bound and verified (bindings.verify()) without cloud credentials or network, and the client is
created where the adapter first talks to its service."""
import threading
from typing import Any, Callable


class Lazy:
def __init__(self, factory: Callable[[], Any]) -> None:
object.__setattr__(self, '_factory', factory)
object.__setattr__(self, '_target', None)
object.__setattr__(self, '_lock', threading.Lock())

def _get(self) -> Any:
if self._target is None:
with self._lock:
if self._target is None:
object.__setattr__(self, '_target', self._factory())
return self._target

def __getattr__(self, name: str) -> Any:
return getattr(self._get(), name)
11 changes: 9 additions & 2 deletions adapters/e2a_adapters/aws/__init__.py
Original file line number Diff line number Diff line change
@@ -1,6 +1,13 @@
"""AWS adapters for E2A's foundation ports (design note D-02). Needs boto3: install e2a-adapters[aws]."""
"""AWS adapters for E2A's ports (design notes D-02, MP-05). Needs boto3: install e2a-adapters[aws].

DynamoDB* claims, rows, leases, checkpoints
SQSBroker messaging (SQS queues, SNS topics), with dead letters
SecretsManagerCredentialProvider tool credentials from Secrets Manager
"""
from .dynamodb import (MAX_VALUE_BYTES, DynamoDBCheckpointStore, DynamoDBIdempotencyStore, DynamoDBLease,
DynamoDBTransactionalStore, is_condition_failure)
from .secrets import SecretsManagerCredentialProvider
from .sqs import SQSBroker

__all__ = ['DynamoDBIdempotencyStore', 'DynamoDBTransactionalStore', 'DynamoDBLease', 'DynamoDBCheckpointStore',
'MAX_VALUE_BYTES', 'is_condition_failure']
'MAX_VALUE_BYTES', 'is_condition_failure', 'SQSBroker', 'SecretsManagerCredentialProvider']
Loading
Loading