feat(project): add `project add evaluator code-based` by jariy17 · Pull Request #2144 · aws/agentcore-cli

jariy17 · 2026-08-28T22:14:37Z

Command structure

agentcore project add evaluator          add a custom evaluator to the current project
├── llm-as-a-judge                       existing — LLM prompted to score a session
└── code-based                           NEW — a Lambda that scores a session
agentcore project remove evaluator --name <name>   NEW — enabled via the generic remove

agentcore project add evaluator code-based --help:

Usage: agentcore project add evaluator code-based [options]

add a code-based evaluator — a Lambda that scores a session. Pass a 3P metric,
an existing Lambda, or neither to scaffold an empty evaluator you fill in

Options:
  --name <name>                        the name of the evaluator
  --level <level>                      what to score: SESSION, TRACE, or TOOL_CALL
  --metric <metric>                    3P metric to scaffold as <library.Metric>,
                                       e.g. deepeval.FaithfulnessMetric or autoevals.Factuality
  --model <model>                      judge model for the 3P metric,
                                       e.g. bedrock/anthropic.claude-3-5-sonnet-20240620-v1:0
  --lambda-arn <lambda-arn>            ARN of an existing Lambda that scores a session
  --timeout-seconds <timeout-seconds>  Lambda timeout in seconds (1-300)
  --description <description>          a description of what this evaluator measures
  --kms-key-arn <kms-key-arn>          customer-managed KMS key ARN to encrypt the evaluator
  --tags <tags>                        tags to apply (JSON object of key/value strings)
  -h, --help                           display help for command

Commits

c4430c03 feat — the command + 3 scaffold templates + remove evaluator
baec1630 fix — guard app/<name> collisions (up-front, no partial writes)
e7bc3675 fix — validate --metric class + require a Bedrock --model
9bd79980 fix — echo the inferred mode + caveats at add time
32a10ef9 refactor — share toPythonPackageName via fsUtils; DEFAULT_TIMEOUT const
853dcf86 refactor — move template knowledge into templates/evaluator.ts (runtime layering)

Testing

bun run build OK · bun test src/handlers/project src/core/project → 597 pass / 0 fail.
Cloud bug bash (5 parallel agents; 2 deployed to a non-prod account, us-west-2, then tore down): all 5 flows + a 13-case error matrix pass at the CLI/scaffold/synth layer. The managed evaluator synthesizes correctly into AWS::BedrockAgentCore::Evaluator + Lambda + role + permissions, and generated Python ast.parses for deepeval + autoevals (bedrock + openai branches). Full report shared separately.

agentcore-devx-automation

AgentCore Harness Review

Verdict: Looks good

Small, focused fix that surfaces two real footguns of the code-based evaluator scaffold:

Empty stub silently returns Pass for every session (verified against src/assets/evaluators/python-lambda/lambda_function.py, which returns label="Pass").
Managed code-based evaluators aren't yet provisioned by project deploy.

Logic in index.ts (lines 158–166) matches the commit message: the "returns Pass" note is gated on !hasLambda && !hasMetric, and the "not yet provisioned" note is gated on !hasLambda, so --lambda-arn (external) correctly prints neither.

Tests in index.test.ts use real temp directories via mkdtemp and drive the handler through the router — no excessive mocking — and cover both the stub and external paths. Telemetry isn't warranted here since this only adds informational stderr output, not a new feature.

Nothing blocking.

codecov-commenter · 2026-08-28T22:18:03Z

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 97.24%. Comparing base (d304147) to head (5d5e079).
⚠️ Report is 3 commits behind head on refactor.

Additional details and impacted files

@@             Coverage Diff              @@
##           refactor    #2144      +/-   ##
============================================
+ Coverage     97.22%   97.24%   +0.01%     
============================================
  Files           507      509       +2     
  Lines         33809    34009     +200     
============================================
+ Hits          32872    33073     +201     
+ Misses          937      936       -1

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

agentcore-devx-automation · 2026-08-28T22:18:32Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T14:53:59Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T17:28:09Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T17:31:40Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-08-31T17:35:47Z

Claude Security Review: no high-confidence findings. (run)

Hweinstock

Looks good, two comments worth thinking about before merging related to the dependency inversion and version pinning.

Hweinstock · 2026-09-01T14:58:43Z

+    # autoevals grades via LiteLLMClient -> Bedrock (Converse); litellm replaces the openai judge
+    "litellm>=1.60,<1.85",
+{{else}}
+    "openai>=1.0.0",


could we use compatible releases or pin this above to avoid the templates breaking on version bumps?

https://packaging.python.org/en/latest/specifications/version-specifiers/#compatible-release

Sure thing.

Hweinstock · 2026-09-01T15:08:07Z

+  EVALUATOR_LIBRARIES,
+  type EvaluatorLibrary,
+  type ManagedEvaluatorScaffoldInput,
+} from "../../../../../core/project/templates/evaluator";


isn't this breaking the dependency inversion? The downside being that we're coupling the UI to the underlying core implementation, which I think the dependency inversion aims to avoid.

Yeah i missed this completely. Ill fix this.

…(DI) Addresses PR feedback: the handler was importing EVALUATOR_LIBRARIES + ManagedEvaluatorScaffoldInput from core/project/templates, coupling the UI layer to core. Mirror runtime's ScaffoldRuntimeInput placement: define EVALUATOR_LIBRARIES (names) + EvaluatorLibrary + ManagedEvaluatorScaffoldInput in handlers/project/types.ts; templates/evaluator.ts imports those and keeps the asset/timeout map (EVALUATOR_ASSETS). Arrow now points core -> handler-types.

agentcore-devx-automation · 2026-09-01T16:06:23Z

Claude Security Review: no high-confidence findings. (run)

Addresses PR feedback on version pinning. The ported templates left bedrock-agentcore (incl. [deepeval]/[autoevals] extras), openai, and aiobotocore with >= but no upper bound; cap each at the next major so a release bump can't silently break a scaffolded evaluator. deepeval (<3), autoevals (<1), and litellm (<1.85) were already capped.

agentcore-devx-automation · 2026-09-01T16:11:03Z

Claude Security Review: no high-confidence findings. (run)

Addresses PR feedback (parity with the runtime templates, which ship a README). Each README explains the files, how to customize the handler, and that 'project deploy' packages the dir into the evaluator Lambda; rendered through the same handlebars context (Name, EvaluatorClass).

agentcore-devx-automation · 2026-09-01T16:40:40Z

Claude Security Review: no high-confidence findings. (run)

Hweinstock

LGTM!

notgitika · 2026-09-01T19:22:33Z

Blocking issue: the generated DeepEval scaffold is not currently installable/runnable.

With --model, uv lock fails because the generated bedrock-agentcore, deepeval, aiobotocore, and botocore constraints are incompatible.
Without --model, dependencies resolve, but importing lambda_function fails because the selected AgentCore SDK does not contain third_party.deepeval.DeepEvalAdapter.

Could we align the DeepEval/AgentCore dependency versions and add a scaffold smoke test that runs uv lock and imports the generated handler?

…ld installs The generated deepeval scaffold was not installable/runnable: - 'uv lock' failed. Our explicit deepeval>=2,<3 pin fought the version the [deepeval] extra owns (4.x), and aiobotocore<3.0.0 excluded the only line (3.x) that accepts the botocore bedrock-agentcore requires. - Without a model the deps resolved but importing the handler failed: the resolved bedrock-agentcore predated third_party.deepeval.DeepEvalAdapter. Raise the floor to 1.20.0 (first release with the deepeval/autoevals adapters), drop the explicit deepeval/autoevals pins so the extras own them, and require aiobotocore 3.x for the Bedrock judge. Verified: uv lock + uv sync + import of the rendered handler now succeed for deepeval(bedrock), autoevals(bedrock), autoevals(default), and the empty stub.

agentcore-devx-automation · 2026-09-01T20:47:19Z

Claude Security Review: no high-confidence findings. (run)

agentcore-devx-automation · 2026-09-01T20:54:26Z

Claude Security Review: no high-confidence findings. (run)

github-actions Bot added the size/xl PR size: XL label Aug 28, 2026

agentcore-devx-automation Bot added agentcore-harness-reviewing AgentCore Harness review in progress claude-security-reviewing Claude Code /security-review in progress labels Aug 28, 2026

agentcore-devx-automation Bot reviewed Aug 28, 2026

View reviewed changes

agentcore-devx-automation Bot removed the agentcore-harness-reviewing AgentCore Harness review in progress label Aug 28, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Aug 28, 2026

jariy17 force-pushed the feat/project-add-evaluator-code-based branch from 303c8db to 9bd7998 Compare August 31, 2026 14:51

github-actions Bot added size/l PR size: L and removed size/xl PR size: XL labels Aug 31, 2026

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Aug 31, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Aug 31, 2026

jariy17 commented Aug 31, 2026

View reviewed changes

Comment thread src/handlers/project/add/evaluator/code-based/index.ts Outdated

github-actions Bot added size/l PR size: L and removed size/l PR size: L labels Aug 31, 2026

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Aug 31, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Aug 31, 2026

github-actions Bot added size/l PR size: L and removed size/l PR size: L labels Aug 31, 2026

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Aug 31, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Aug 31, 2026

jariy17 force-pushed the feat/project-add-evaluator-code-based branch from e78e9bc to 32a10ef Compare August 31, 2026 17:33

github-actions Bot added size/l PR size: L and removed size/l PR size: L labels Aug 31, 2026

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Aug 31, 2026

Hweinstock reviewed Sep 1, 2026

View reviewed changes

github-actions Bot added size/l PR size: L and removed size/l PR size: L labels Sep 1, 2026

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Sep 1, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Sep 1, 2026

github-actions Bot added size/l PR size: L and removed size/l PR size: L labels Sep 1, 2026

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Sep 1, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Sep 1, 2026

github-actions Bot added size/l PR size: L and removed size/l PR size: L labels Sep 1, 2026

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Sep 1, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Sep 1, 2026

Hweinstock previously approved these changes Sep 1, 2026

View reviewed changes

jariy17 dismissed Hweinstock’s stale review via 5d5e079 September 1, 2026 20:43

github-actions Bot added size/xl PR size: XL and removed size/l PR size: L labels Sep 1, 2026

agentcore-devx-automation Bot added the claude-security-reviewing Claude Code /security-review in progress label Sep 1, 2026

agentcore-devx-automation Bot removed the claude-security-reviewing Claude Code /security-review in progress label Sep 1, 2026

jariy17 force-pushed the feat/project-add-evaluator-code-based branch from 5d5e079 to 220a2f5 Compare September 1, 2026 20:51

Uh oh!

Conversation

jariy17 commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Command structure

Commits

Testing

Uh oh!

agentcore-devx-automation Bot left a comment

Choose a reason for hiding this comment

AgentCore Harness Review

Uh oh!

codecov-commenter commented Aug 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

agentcore-devx-automation Bot commented Aug 28, 2026

Uh oh!

agentcore-devx-automation Bot commented Aug 31, 2026

Uh oh!

Uh oh!

agentcore-devx-automation Bot commented Aug 31, 2026

Uh oh!

agentcore-devx-automation Bot commented Aug 31, 2026

Uh oh!

agentcore-devx-automation Bot commented Aug 31, 2026

Uh oh!

Hweinstock left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Hweinstock Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

jariy17 Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Hweinstock Sep 1, 2026

Choose a reason for hiding this comment

Uh oh!

jariy17 Sep 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

agentcore-devx-automation Bot commented Sep 1, 2026

Uh oh!

agentcore-devx-automation Bot commented Sep 1, 2026

Uh oh!

agentcore-devx-automation Bot commented Sep 1, 2026

Uh oh!

Hweinstock left a comment

Choose a reason for hiding this comment

Uh oh!

notgitika commented Sep 1, 2026

Uh oh!

agentcore-devx-automation Bot commented Sep 1, 2026

Uh oh!

agentcore-devx-automation Bot commented Sep 1, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

jariy17 commented Aug 28, 2026 •

edited

Loading

codecov-commenter commented Aug 28, 2026 •

edited

Loading

jariy17 Sep 1, 2026 •

edited

Loading