feat: create credential providers before synthesizing a deploy - #2123
feat: create credential providers before synthesizing a deploy#2123notgitika wants to merge 21 commits into
Conversation
There was a problem hiding this comment.
AgentCore Harness Review
Verdict: Looks good
Nice PR. The design of running credential provisioning before cdk synth and threading the ARNs through deployed-state.json is well-motivated and the comments do a great job of capturing why. The seam boundary (IdentityProviderClient) is drawn at the SDK client rather than at fs/process boundaries, so the tests avoid excessive mocking while still exercising the real spec + .env.local parsing paths. Sharing credentialEnvVarName/CLIENT_SECRET_SUFFIX between add and deploy via envLocal.ts (with a re-export from shared.ts) removes a latent format-drift bug.
A couple of small things that aren't blockers but worth confirming intentional:
-
Stale credential entries when the spec goes to zero credentials. In
src/core/project/backends/cdk.ts(~L115)updateTargetStateis only called whenObject.keys(provisioned).length > 0. If a user deletes their last credential fromagentcore.jsonand re-deploys,provisionedis{}, the state write is skipped, and the previousresources.credentialsmap is left on disk. The doc-comment onupdateTargetStatepromises "A resource map provided in the patch replaces the previous map for that kind wholesale, so a credential dropped from the spec stops being advertised" — that guarantee is only actually delivered when at least one credential remains. Since the synthesized CDK app looks up credentials by name, this is likely inert in practice, but if you want the drop-to-zero case to behave the same as drop-one-of-many, you'd either always callupdateTargetState({ resources: { credentials: provisioned } })or explicitly write{}whendeclaredis non-empty on the spec side but you provisioned nothing. -
parseEnvcast inEnvLocalFile.read(src/core/project/envLocal.tsL90):parseEnv's declared return type isRecord<string, string | undefined>(last-write-wins across duplicate keys), but you cast toRecord<string, string>. All callers happen to useif (!value)so undefined is handled safely today; just be aware the type is a small lie and a future caller doingenv[k].trim()would compile but crash.
Neither of these needs to block the merge.
4b787a8 to
0db4266
Compare
The synthesized CDK app reads credential provider ARNs out of deployed-state.json and fails to synth a project that declares credentials until they exist. Provision them between the account preflight and the build, then record their ARNs via updateTargetState so the assembly is synthesized against a state file that already describes them. Providers are created when absent and reused when present, never updated, so a redeploy neither mints a new secret version nor overwrites one rotated outside the CLI. Payment credentials are rejected up front (agentcore.json can't express the vendor config they need). Secrets come from the same place 'project add credentials' writes them, so the env-var name is now derived from one function in envLocal.ts that both sides share.
0db4266 to
3c09c23
Compare
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## refactor #2123 +/- ##
============================================
- Coverage 97.22% 97.12% -0.11%
============================================
Files 507 508 +1
Lines 33809 34308 +499
============================================
+ Hits 32872 33321 +449
- Misses 937 987 +50 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
|
Claude Security Review: no high-confidence findings. (run) |
…v type - Add SDK-mocked coverage for createIdentityProviderClient (the real Identity factory the provisioner tests bypass): ~55% -> ~95% on credentials.ts. - Always record the provisioned credential set, so removing the last credential from the spec clears the stale entry instead of leaving it advertised. - EnvLocalFile.read returns Record<string, string | undefined> (parseEnv's real type) rather than casting it away. - Tighten a few verbose comments.
|
Claude Security Review: no high-confidence findings. (run) |
The existing collision check compares the .env.local variables the credentials in the spec write today, so it cannot catch a clash with a field the CLI no longer writes but still reads — an OAuth client id, which pre-0.29 projects keep in AGENTCORE_CREDENTIAL_<NAME>_CLIENT_ID — or with one a later credential type adds. An api-key credential named 'svc-client-id' therefore added cleanly and its key was then read as the client id of an OAuth credential named 'svc'. Names whose derived variable ends in a field suffix are now refused at add time, the way main's validateCredentialNameEncryptable refuses them. The check stays in the add flow rather than the schema so that a project already holding such a name keeps loading and can be repaired.
…ider Deploy reused an existing provider untouched, so editing a secret in .env.local and redeploying had no effect on AWS: the provider kept the value it was created with, and nothing said so. main's deploy updates instead — `// Always update to ensure provider has current credentials` — and losing that on the rewrite would be a silent regression for anyone rotating a key. A credential whose secret the CLI can see is now written on every deploy: created when the provider is absent, updated when it is present. A credential with no secret to offer — nothing in .env.local and no external reference — leaves an existing provider exactly as it is, so a project that provisioned once and no longer keeps the secret on disk still deploys; only an absent provider fails, as before.
Provisioning read agentcore/.env.local and nothing else, so a deploy from CI had to write its secrets to disk first. main reads every credential variable from process.env and merges it over the file, which is what makes a non-interactive deploy possible without persisting secrets. Variables carrying the credential prefix now override the file. The filter keeps the deploy from reading anything else out of the environment, and the provisioner takes the environment as an argument so tests do not mutate the process's own.
`project add credentials payment` and `add payment-connector` write a payment credential to the spec, but deploy refused to provision one — it threw and told the user to remove it — so a project could be assembled that added cleanly and then could not deploy at all. main provisions them, and the CDK construct that wires payment connectors already reads their ARNs out of the credentials map in deployed-state.json. - Core's Identity client gains the payment provider operations. The pinned SDK carries them, so unlike main there is no hand-signed HTTP request. - A payment credential is created when absent and updated when present, from the vendor's variables: CoinbaseCDP's api key id, api key secret and wallet secret, or StripePrivy's app id, app secret, authorization private key and authorization id. Every variable that is unset is named in one error rather than one per attempt, and an existing provider is left alone when they are all absent. - Teardown deletes the payment providers the project declares, as main's cleanupPaymentCredentialProviders does, after the stack rather than before, since a resource in it may still be using one. Only payment providers: an api-key or OAuth provider is named account-globally and may be shared with another project. The payment collision test now asserts what actually guards that case: a name ending in a payment field suffix is refused on its own, so a credential can no longer be created that would collide with a payment credential's variables.
Resolving every credential before writing any removes the likeliest cause of a half-provisioned deploy — a missing secret — but not the rest: a create that fails on throttling or permissions after an earlier one succeeded left a provider in AWS that deployed-state.json never recorded. Retrying adopted it by name, but abandoning the deploy or dropping the credential orphaned it. A failure during the write loop now deletes the providers that same run created, newest first, and rethrows the original error. A provider that already existed is not deleted: this deploy only updated its secret, and undoing that would need the value it held before, which the CLI never had. A deletion that fails is reported — naming the provider and saying the next deploy will adopt it — rather than replacing the error that stopped the deploy.
|
Claude Security Review: no high-confidence findings. (run) |
Conflicts were additive on both sides: - envLocal: this branch added `read`, refactor added `removeKeys`; both kept. - CdkBackend: this branch added the credential provisioner and payment remover, refactor replaced the stack probe with `describeStack`; both kept, and teardown now asks `describeStack` whether the stack is still there.
|
Claude Security Review: no high-confidence findings. (run) |
e2e run — account
|
| # | Scenario | Result |
|---|---|---|
| 1 | First deploy creates the provider before synth | Preparing credential provider 'e2ekey2' precedes Synthesizing CloudFormation templates; provider ARN + secret ARN written to deployed-state.json, then stackArn merged into the same target entry |
| 2 | Rotate the secret in .env.local, redeploy |
Provider updated: new AWSCURRENT secret version at 22:59:05 against a create at 22:57:22, and GetSecretValue returned the rotated value |
| 3 | Variable in the process environment | .env.local held one value, the environment another → the environment's value is what landed in AWS |
| 4 | No secret in the file or the environment | Deploy succeeded, no new secret version, stored value unchanged — the existing provider is left alone rather than failing the deploy |
| 5 | Second credential fails after the first is created | Removing credential provider 'rbkeya' this deploy created; rbkeya gone from AWS, pre-existing e2ekey2 untouched, deployed-state.json unchanged, exit code 1 with the original service error |
| 6 | Payment credential (CoinbaseCDP) | paymentcredentialprovider/e2epay created and its ARN recorded. With two of three variables unset, one error named both: missing the values its payment provider needs: ..._API_KEY_SECRET, ..._WALLET_SECRET |
| 7 | Payment update, then teardown | Rotated apiKeyId reached AWS on redeploy. Teardown removed the stack, deleted e2epay, kept e2ekey2 (account-global, may be shared), and emptied targets |
Two notes for anyone re-running this:
- The service validates vendor key formats. Placeholder payment secrets are rejected (
Invalid apiKeySecret format: Expected base64-encoded Ed25519 private key, thenInvalid walletSecret format: Expected base64-encoded EC P-256 private key). I used structurally valid throwaway keys, so create/update/delete and the request shape are verified — but not against real Coinbase credentials. Those service messages surface as-is rather than CLI-framed. lastUpdatedTimeon an API-key provider does not move on update, soget-api-key-credential-provideralone can't confirm a rotation; Secrets Manager version history can.
Unrelated papercut noticed on the way: project remove harness <name> fails with "too many arguments" — it wants --name <name>.
Hweinstock
left a comment
There was a problem hiding this comment.
I wasn't able to full follow everything done here, so I left some random comments about where I was confused.
I was kind of envisioning this as a single function executed on deploy to reconcile the local env with the deployed environment, but it feels like there's a lot more going on. Is there a way to descope this into something simpler as a start?
| @@ -6,6 +6,7 @@ import type { AddResourceInput } from "../../types"; | |||
| import { | |||
There was a problem hiding this comment.
OOS here, but I think we should rename this file to avoid it becoming a dumping ground.
There was a problem hiding this comment.
Agreed. I'll leave it in this PR to keep the diff reviewable and rename in a follow-up, probably addFlow.ts or credentialInput.ts, since what's actually in here is the shared add-flow + flag parsing
|
|
||
| // The collision check above only compares the fields credentials write today. A name | ||
| // ending in a field suffix would also shadow a field the CLI no longer writes but | ||
| // still reads (an OAuth client id in a pre-0.29 project) or one a later credential |
There was a problem hiding this comment.
can we link to more context here? I'm not sure what a pre-0.29 project means
There was a problem hiding this comment.
ah I meant pre v0.29. fair, I'll reword to say what actually changed: _CLIENT_ID used to be written into .env.local, an OAuth client id now lives in agentcore.json, but deploy still reads the old variable so projects created before that move keep working. That's the field the collision check can't see, which is why the name is refused outright
| export type Credential = z.infer<typeof CredentialSchema>; | ||
|
|
||
| /** The prefix every variable carrying credential material shares. */ | ||
| export const CREDENTIAL_ENV_PREFIX = "AGENTCORE_CREDENTIAL_"; |
There was a problem hiding this comment.
i'm not super familiar with these schemas, but I'm suprised to find helper functions in projectSchemas. I would have expected it to be strictly zod schemas or their related types/constants.
There was a problem hiding this comment.
ah, yes I found that the directory already does this, runtime.ts, harness.ts, etc. all export helpers alongside their schemas. these three derive their output from this schema's own field names (credentialEnvironmentVariableNames switches on authorizerType), so splitting them out would mean a second file that imports the schema and has to be kept in sync with it.
happy to move them if you'd rather the directory be strictly schemas, but it'd be a broader cleanup than this PR intends
| @@ -22,11 +29,12 @@ export interface CoreOptions { | |||
| export interface ClientConfig { | |||
There was a problem hiding this comment.
Should these two types be merged? The difference is less clear to me now.
There was a problem hiding this comment.
thhey're deliberately different shapes. ClientConfig is spread straight into the SDK constructor, so its fields have to be the SDK's names: endpoint.
CoreOptions is the handler-facing shape and uses endpointUrl, which is what the flag and context call it. toClientConfig is the translation. I'll make that explicit in the comment, since it's not visible from the type alone.
| // AwsCredentials is an explicit credential source for a call: either resolved | ||
| // credentials or a provider that resolves them. Callers that rely on the SDK's own | ||
| // default credential chain leave it unset. | ||
| export type AwsCredentials = NonNullable<CloudFormationClientConfig["credentials"]>; |
There was a problem hiding this comment.
q: is the credentials field in CloudFormationClientConfig valid for all aws sdk clients?
There was a problem hiding this comment.
yes! every v3 client takes the same AwsCredentialIdentity | AwsCredentialIdentityProvider. but you're right that deriving it from the CFN client looks arbitrary.
I can look into importing the type from @smithy/types directly
|
|
||
| // credentialsId assigns each credential source a stable id for the lifetime of the | ||
| // object, so the same source reuses its client and a different one gets its own. | ||
| function credentialsId(credentials: NonNullable<ClientConfig["credentials"]>): number { |
There was a problem hiding this comment.
I'm not really following why we need this id?
There was a problem hiding this comment.
The bug it fixes: credentials is either a provider function or an object of resolved credentials, and JSON.stringify silently drops functions, so two deploys against different accounts in the same region got the same cache key and shared one client, i.e. the second one ran with the first one's credentials.
It needs to be keyed by identity, not by value. but i can see how this is more complicated.
let me try switching to a 2 level cache: WeakMap<credentials, Map<string, Client>>, plus a plain Map for the no-credentials case. Same semantics, no global id, no string suffix
|
|
||
| export class IdentityClient implements CoreIdentityClient { | ||
| constructor(private readonly clients: AwsClients) {} | ||
| // Only the control plane is used, so the dependency is narrowed to it: CoreClient |
There was a problem hiding this comment.
i feel like the code expresses this.
There was a problem hiding this comment.
I feel like the Pick is visible, but why it's a Pick rather than the whole AwsClients isn't. I'll shorten it
| CDK: new CdkBackend({ | ||
| logger: config.logger, | ||
| createCloudFormationClient: config.createCloudFormationClient, | ||
| identity: config.identity ?? createIdentityClient(), |
There was a problem hiding this comment.
whats the advantage to making this optional? It looks like we always pass it and I wonder if we it would simplify some of the core changes.
There was a problem hiding this comment.
ah I see what you mean. making it required and updating the call sites.
| yield { | ||
| message: | ||
| `Could not remove credential provider '${name}': ${(error as Error).message}. ` + | ||
| `Delete it with 'aws bedrock-agentcore-control delete-payment-credential-provider'.`, |
There was a problem hiding this comment.
isn't there a way to remove these via the resource based commands in our cli? Or did we only do readonly?
There was a problem hiding this comment.
Not for payment providers, there is CRUDL for api and oauth only
| @@ -0,0 +1,711 @@ | |||
| import { afterEach, describe, expect, test } from "bun:test"; | |||
There was a problem hiding this comment.
this testing setup feels extremely complex and coupled to the underlying implementation. Is there a way we can decouple it, and simplify?
There was a problem hiding this comment.
I'll make the fake a small stateful in-memory Identity that mutates a Record<name, provider>, and assert the resulting store plus the returned DeployedCredentials instead of the call log. I'd keep call-order assertions in the few places where the order is the contract, get-before-create, and rollback deleting only what the run created. What do you think?
…rialized config `cacheKey` stringified the whole ClientConfig, which was wrong in two ways once a config could carry credentials. `credentials` is either a provider function or an object of resolved credentials, and JSON.stringify drops a function silently. Two targets in the same region therefore produced the same key and shared one client, so the second ran with the first one's credentials. The previous fix appended a WeakMap-assigned id to the serialized key; the cache is two-level instead — a WeakMap keyed by the credential object, holding a map of the serializable fields — which drops the id counter and the string suffix. Serializing the object also made the key depend on property order. Every config comes out of `toClientConfig` today so the order is fixed in practice, but `configKey` now names the fields it cares about rather than relying on that. `ClientCache` also absorbs the get-or-create block the four client accessors each repeated. AwsCredentials comes from `@smithy/types` rather than being derived from CloudFormation's client config: every v3 client accepts the same shape, so deriving it from one client read as though CloudFormation were special.
`identity` was optional and fell back to `createIdentityClient()`, which built a real BedrockAgentCoreControlClient. CoreClient always passes one, so the fallback existed only for the tests that construct a manager without it — and it made those tests silently reach for AWS instead of failing, which is the opposite of what a default should do here. It is required now, and the call sites pass TestIdentityClient. With the fallback gone `createIdentityClient` has no callers and is deleted, which also removes IdentityClient's last path to constructing its own SDK client: it can only work through the control client a CoreClient hands it.
…t a call log The fake Identity recorded every call, and the tests asserted the sequence — that a create was preceded by a get, that a rollback emitted exactly one delete. That pins the implementation rather than the behaviour: reordering calls without changing what ends up in the account broke them. The fake is a small in-memory account instead: providers can be looked up, created, updated and deleted, and each provider records the request body it was last written with and whether this run created or updated it. Tests assert what the account holds afterwards. "Created when absent, updated when present" is now checked by the recorded operation rather than by a get/create pair, and "left alone" by the absence of one. Deletions stay a list, because a provider created and then rolled back is indistinguishable from one never created by looking at the contents alone, and that distinction is the whole point of the rollback path. Three cases got stronger in the process: the teardown test seeds both an api-key and a payment provider and asserts the api-key one survives, where it previously only counted calls against an empty account; and both "creates nothing" cases now assert the account is empty rather than counting lookups.
The comment cited a "pre-0.29 project", a version that means nothing in a repo on 1.0.0. Name the change instead: an OAuth client id now lives in agentcore.json, but deploy still reads it from .env.local for projects created before it moved, so it is a field the collision check cannot see. Point at CREDENTIAL_FIELD_SUFFIXES for the rest.
|
Claude Security Review: no high-confidence findings. (run) |
| yield { message: `Removing stack ${artifact.stackName}` }; | ||
| await this.cdk({ kind: "destroy", stackArtifactId: artifact.id }, options); | ||
| // After the stack, since a resource in it may still be using the provider. | ||
| yield* this.removePaymentCredentials(project, { |
There was a problem hiding this comment.
I think teardown needs to use the payment providers recorded for the deployed target, not the current project.spec.credentials. project remove all writes credentials: [] before this deploy, so the remover gets no names; removeTargetState then drops the only tracking while the providers remain in AWS. Could we persist the payment-provider names/type in deployed state and delete from that snapshot before removing the target entry, with a remove all -> deploy regression test?
|
|
||
| const fields = paymentFields(credential, env); | ||
| if ("missing" in fields) { | ||
| if (existing) return { reuse: paymentProvision(name, existing) }; |
There was a problem hiding this comment.
Should we verify existing.credentialProviderVendor before reusing this ARN? I used a CoinbaseCDP credential with an existing same-name StripePrivy provider and no local values; deploy accepted it and recorded the Stripe ARN as the Coinbase credential. The OAuth reuse branch above has the same issue, so maybe a shared mismatch check and tests for both provider types would help.
| provisioned[credential.name] = provision.reuse; | ||
| continue; | ||
| } | ||
| provisioned[credential.name] = await provision.write(); |
There was a problem hiding this comment.
I think one rollback window remains here. write() can successfully create the provider and then throw while requireArn() validates a malformed response, but the credential is not added to created until the next line. I ran that response and got get -> create with no delete; could the create path record its side effect immediately after the service call returns and before response validation?
| * CLI cannot see — nothing in `.env.local` and no external reference — is left | ||
| * exactly as it is rather than failing the deploy. | ||
| */ | ||
| export function createCredentialProvisioner( |
There was a problem hiding this comment.
I may be missing an intentional compatibility change, but released v0.28.0 always configured the token-vault CMK before API-key provider creation (GetTokenVault, CreateKey, and SetTokenVaultCMK via enableKmsEncryption: true) and persisted identityKmsKeyArn. This path does none of that, so new refactor deployments no longer proactively get the CLI-managed CMK behavior. Should we preserve the released behavior here, or explicitly call out that encryption-default change?
There was a problem hiding this comment.
deferred to a follow up
…view Three separate defects on the provisioning path, all reported by @aidandaly24. **A teardown deleted the payment providers the spec still declared, not the ones the target actually provisioned.** A teardown is reached by declaring nothing to deploy, and `project remove all` gets there by emptying the spec — including the credentials naming the providers to delete. Worse, the deploy rewrites the credentials map to empty on its way past, so by the time teardown ran the only record was already gone and `removeTargetState` then dropped the target entry, leaving the providers orphaned in AWS with nothing tracking them. The credentials recorded for the target are now read before provisioning overwrites them and handed to the remover, which deletes the payment providers named there as well as any the spec still declares. Each entry records its `authorizerType` so the remover knows which providers it owns; entries written before that was persisted are classified by the provider type in their ARN. **An existing provider was reused without checking its vendor.** Provider names are shared across an account within a kind, so a CoinbaseCDP credential whose local values were absent reused a same-named StripePrivy provider and recorded the Stripe ARN as the Coinbase credential's. OAuth had the same hole. A vendor mismatch is now refused for both, at the lookup rather than only on the reuse branch, so pushing one vendor's configuration at another's provider is caught too. **A create whose response failed validation was not rolled back.** The create was recorded after `write()` returned, but `write()` validates the response before returning: a service call that created the provider and then reported no ARN threw past the line that records it, so rollback never deleted it. The create is recorded before the call it describes; rollback treats a provider that is not there as already gone, which is what a create that never reached the service leaves behind. The fake Identity in the tests now throws on deleting a provider it does not hold, as the service does — without that it recorded rollback deletions of providers that were never created, hiding the distinction the rollback tests exist to check. Not addressed here: released v0.28.0 also configured a token-vault CMK (GetTokenVault, CreateKey, SetTokenVaultCMK) and persisted `identityKmsKeyArn`, which this path does not. Tracked as a follow-up — the token vault is account-scoped rather than per project, so who owns the key and its lifecycle needs deciding rather than porting as-is.
|
Claude Security Review: no high-confidence findings. (run) |
Creates or updates a project's credential providers before synthesis, so the synthesized CDK app can read their ARNs out of
deployed-state.json. Without this, deploying a project that declares any credential fails insidecdk synth.What it does
CdkBackend.deployprovisions each declared credential provider and records the ARNs viaupdateTargetState. Local prerequisites are checked first, so a setup error never mutates AWS.maindoes). A credential with no secret the CLI can see leaves an existing provider untouched instead of failing..env.local(AGENTCORE_CREDENTIAL_<NAME>) or a Secrets ManagersecretRef; process-environment variables override the file, so CI needs no secrets on disk.add credentials paymentcould build a project that couldn't deploy.core.identityclient instead of a second SDK client; the target's credentials travel viaCoreOptions.Notes
.env.localsilently had no effect.CoreClient's client cache is now keyed by credential identity — credentials are a function, whichJSON.stringifydrops, so different credentials in one region shared a cached client._CLIENT_ID,_APP_SECRET, …) are refused ataddtime; their variable would shadow another credential's field.mainguarded this; the rewrite had lost it.@aws/agentcore-cdk: its API-key Gateway path doesn't grantGetSecretValueon an external secret ARN, so an API-keysecretRefdeploys and then fails at retrieval. The CLI already records the ARN.Tested
_CLIENT_IDfallback, payment vendors, teardown deletion, rollback.add credentials api-key→deploycreated the provider before synth, wrote it todeployed-state.json, then mergedstackArninto the same entry.