feat(evaluations): integrate offline judges and deterministic scorers

AGENTS.md

@@ -703,3 +703,10 @@ response = await graph(
     },
 ).invoke(user_input, context)
 ```
+
+## Maintaining this file
+
+Keep this file for knowledge useful to almost every future agent session in this project.
+Do not repeat what the codebase already shows; point to the authoritative file or command instead.
+Prefer rewriting or pruning existing entries over appending new ones.
+When updating this file, preserve this bar for all agents and keep entries concise.

CLAUDE.md

@@ -0,0 +1,2 @@
+<!-- Points Claude at AGENTS.md via import; edit AGENTS.md, not this file. -->
+@AGENTS.md

packages/ai/README.md

@@ -52,6 +52,30 @@ if result["enabled"]:
 
 Never raises. Returns `{"enabled": bool, "config": dict | None, "meta": dict | None}`.
 
+## Evaluations from code
+
+`init_evaluations` and the evaluations result types are also re-exported:
+
+```python
+from launchdarkly_ai_python import Accuracy, Scorer, init_evaluations
+
+evals = init_evaluations()
+result = await evals.run(
+    project_key="my-project",
+    key="unique-evaluation-key",
+    dataset="golden-dataset",
+    handler=my_handler,
+    generation={"provider": "OpenAI", "model": "gpt-4o"},
+    judges=[
+        Accuracy(),
+        Scorer(name="exact-match", fn=lambda row, output: output == row.expected_output),
+    ],
+)
+print(result.evaluation_results)
+```
+
+`LD_API_TOKEN` is required, and LaunchDarkly judges also require `LD_SDK_KEY`; deterministic `Scorer` values do not. Use `LD_API_BASE_URI` for staging or local management API traffic; it is separate from the SDK delivery setting `LD_BASE_URI`. See the [core evaluations guide](../client/README.md#run-an-evaluation-from-code).
+
 ---
 
 All exports, types, and behaviors are identical to `launchdarkly-ai-server`. See the [core client README](../client/README.md) for the full API reference.

packages/client/README.md

@@ -39,6 +39,53 @@ No code changes are required — `init_client()` detects the packages at runtime
 | `LD_SERVICE_NAME` | No | OTel `service.name` resource attribute (default: `python-sdk`) |
 | `LD_ENVIRONMENT` | No | `deployment.environment` resource attribute attached to telemetry |
 | `OTEL_EXPORTER_OTLP_ENDPOINT` | No | OTLP endpoint override (default: LaunchDarkly Observability backend) |
+| `LD_API_TOKEN` | For evaluations | API access token used by the evaluations management API |
+| `LD_API_BASE_URI` | No | Evaluations management API host override; intentionally separate from `LD_BASE_URI` |
+
+### Run an evaluation from code
+
+The evaluations harness reads an LD-hosted dataset, creates a new evaluation and client-source run, invokes your handler once per row, optionally runs typed LaunchDarkly judges and deterministic scorers in the same worker, uploads the generations, and returns LaunchDarkly's stored verdict. Evaluation keys must be unique because every call creates a new evaluation with `POST`.
+
+```python
+import asyncio
+import sys
+
+from launchdarkly_ai_openai_messages import create_openai_messages_handler
+from launchdarkly_ai_server import Accuracy, Judge, Scorer, init_evaluations
+
+
+async def main() -> int:
+    evals = init_evaluations()  # LD_API_TOKEN + LD_SDK_KEY for LD judges
+    result = await evals.run(
+        project_key="my-project",
+        key="support-qa-2026-08-20",
+        dataset="support-golden",
+        handler=create_openai_messages_handler(),
+        generation={
+            "provider": "OpenAI",
+            "model": "gpt-4o",
+            "instructions": "You are a support agent.",
+        },
+        judges=[
+            Accuracy(),
+            Judge(key="security-judge", threshold=0.7),
+            Scorer(
+                name="exact-match",
+                fn=lambda row, output: output == row.expected_output,
+            ),
+        ],
+    )
+    print(result.url, result.summary)
+    print(result.evaluation_results)
+    return 0 if result.passed else 1
+
+
+sys.exit(asyncio.run(main()))
+```
+
+`project_key` is supplied per run rather than during initialization. `generation.instructions` is shorthand for one system message; use `generation.messages` instead for a full message list, but do not supply both. The harness never retries a handler invocation because doing so could repeat tool side effects. Its retries apply only to LaunchDarkly management API requests.
+
+`judges` accepts only typed `JudgeReference` values (`Accuracy`, `AnswerRelevancy`, `Likeness`, `Bias`, `Toxicity`, `Misinformation`, or `Judge`) and `Scorer` values. LaunchDarkly judges require `LD_SDK_KEY` and a generation handler built with `create_handler()` so the resolved judge model can be routed safely. Scorers may be synchronous or asynchronous and receive `(row, generation_output)`; `row` includes the rendered row index, input, expected output, variables, and metadata. Results are returned in `EvalRunResult.evaluation_results`. This judging foundation does not yet submit those local scores to LaunchDarkly's evaluation-results endpoint, so `result.passed` remains the stored generation-run verdict.
 
 The client uses **lazy initialization**: importing the package does not connect to LaunchDarkly. The singleton is created automatically on the first API call that needs it (`config().invoke()`, `graph().invoke()`, `resolve_graph()`, etc.), as long as `LD_SDK_KEY` is set in the environment.
 

packages/client/agents.md

@@ -30,6 +30,7 @@ No other `launchdarkly-ai-*` package may define or duplicate these. They import
 | `src/launchdarkly_ai_server/utils.py` | `parse_template`, `parse_json_with_possible_fences`, `create_handler`, `parse_usage`, `make_track_data`, `to_ld_context` |
 | `src/launchdarkly_ai_server/registry.py` | `Registry`, `global_registry`, `compose`, `resolve_handlers`, `resolve_tools` |
 | `src/launchdarkly_ai_server/judges.py` | `run_judges`, `build_judge_tasks`, `run_judge` |
+| `src/launchdarkly_ai_server/evaluations/` | `init_evaluations`, the private management API operations, offline judge/scorer resolution, and `EvaluationsModule.run()` orchestration |
 | `src/launchdarkly_ai_server/__init__.py` | Public barrel — the only surface handler packages import from |
 
 ---
@@ -66,7 +67,7 @@ from launchdarkly_ai_server import Registry, global_registry, compose, resolve_h
 from launchdarkly_ai_server import execute_and_track, execute_and_stream, wrap_tool_handlers
 
 # Entry points
-from launchdarkly_ai_server import config, graph, resolve_graph
+from launchdarkly_ai_server import config, graph, resolve_graph, init_evaluations
 ```
 
 When adding a new export, add it to `__init__.py`'s imports and `__all__`. Handler packages must never import from sub-paths (e.g. `launchdarkly_ai_server.client`).
@@ -123,6 +124,12 @@ Handlers may return any of these — the client normalizes them before emitting
 
 ---
 
+## SDK-run evaluations
+
+`init_evaluations()` creates an evaluations harness using `LD_API_TOKEN` and the management API host `LD_API_BASE_URI`. Do not reuse `LD_BASE_URI`: that variable configures SDK flag delivery and may point at a relay proxy. `LD_SDK_KEY` is optional for generation-only or deterministic-scorer runs, and required when `judges` contains a LaunchDarkly judge reference.
+
+`await EvaluationsModule.run(...)` takes `project_key` per call. Dataset lookup/row pagination, evaluation creation, and run creation are private helpers; only `run()` is public. Each call creates a new evaluation with `POST`, so its key must be unique. The harness directly invokes the supplied handler once per row, never retries it, then runs typed `JudgeReference` / `Scorer` values with the generated output and full rendered row context. Generation results are batch-ingested; local evaluation outcomes are returned in `EvalRunResult.evaluation_results`, while `passed` remains the server's stored generation-run verdict until evaluation-results ingest lands.
+
 ## OTel Setup
 
 The core client owns all OTel initialization. `init_client()` configures a `TracerProvider` with a `BatchSpanProcessor` and an OTLP HTTP exporter when the optional OTel packages are installed.

packages/client/pyproject.toml

@@ -2,7 +2,7 @@
 name = "launchdarkly-ai-server"
 version = "0.1.3"
 requires-python = ">=3.12"
-dependencies = ["opentelemetry-api>=1.25"]
+dependencies = ["opentelemetry-api>=1.25", "pydantic>=2"]
 description = "LaunchDarkly AI SDK core client for Python"
 readme = "README.md"
 license = "Apache-2.0"

packages/client/src/launchdarkly_ai_server/__init__.py

@@ -16,6 +16,34 @@
     text_message,
     to_semconv_finish_reason,
 )
+from .evaluations import (
+    Accuracy,
+    AnswerRelevancy,
+    Bias,
+    EvalRunResult,
+    EvaluationMethod,
+    EvaluationsError,
+    EvaluationsModule,
+    GenerationConfig,
+    Judge,
+    JudgeEvaluationError,
+    JudgeEvaluationResult,
+    JudgeIdentity,
+    JudgeReference,
+    JudgeUsage,
+    LaunchDarklyJudgeEvaluation,
+    Likeness,
+    Misinformation,
+    RunSummary,
+    Scorer,
+    ScorerError,
+    ScorerResult,
+    ScorerRow,
+    ScoreValue,
+    Toxicity,
+    init_evaluations,
+    resolve_launchdarkly_judges,
+)
 from .graph import GraphInstance, graph, resolve_graph
 from .judges import build_judge_tasks, run_judge, run_judges
 from .lifecycle import (
@@ -150,6 +178,33 @@
     "text_message",
     "to_semconv_finish_reason",
     "VariationMeta",
+    # evaluations
+    "Accuracy",
+    "AnswerRelevancy",
+    "Bias",
+    "EvalRunResult",
+    "EvaluationMethod",
+    "EvaluationsError",
+    "EvaluationsModule",
+    "GenerationConfig",
+    "Judge",
+    "JudgeEvaluationError",
+    "JudgeEvaluationResult",
+    "JudgeIdentity",
+    "JudgeReference",
+    "JudgeUsage",
+    "LaunchDarklyJudgeEvaluation",
+    "Likeness",
+    "Misinformation",
+    "RunSummary",
+    "ScoreValue",
+    "Scorer",
+    "ScorerError",
+    "ScorerResult",
+    "ScorerRow",
+    "Toxicity",
+    "init_evaluations",
+    "resolve_launchdarkly_judges",
     # utils
     "create_handler",
     "make_track_data",

packages/client/src/launchdarkly_ai_server/evaluations/__init__.py

@@ -0,0 +1,67 @@
+"""Run LaunchDarkly evaluations from your own environment."""
+
+from .api import (
+    DEFAULT_BASE_URI,
+    EvaluationsError,
+    HttpResponse,
+    LDApiClient,
+    LDApiError,
+    Transport,
+    urllib_transport,
+)
+from .judges import (
+    Accuracy,
+    AnswerRelevancy,
+    Bias,
+    EvaluationMethod,
+    Judge,
+    JudgeEvaluationError,
+    JudgeEvaluationResult,
+    JudgeIdentity,
+    JudgeReference,
+    JudgeUsage,
+    LaunchDarklyJudgeEvaluation,
+    Likeness,
+    Misinformation,
+    Toxicity,
+    resolve_launchdarkly_judges,
+)
+from .module import EvaluationsModule, init_evaluations
+from .scorers import Scorer, ScorerError, ScorerResult, ScorerRow, ScoreValue
+from .types import EvalRunResult, GenerationConfig, RunSummary, Usage
+
+__all__ = [
+    "DEFAULT_BASE_URI",
+    "Accuracy",
+    "AnswerRelevancy",
+    "Bias",
+    "EvalRunResult",
+    "EvaluationMethod",
+    "EvaluationsError",
+    "EvaluationsModule",
+    "GenerationConfig",
+    "HttpResponse",
+    "Judge",
+    "JudgeEvaluationError",
+    "JudgeEvaluationResult",
+    "JudgeIdentity",
+    "JudgeReference",
+    "JudgeUsage",
+    "LDApiClient",
+    "LDApiError",
+    "LaunchDarklyJudgeEvaluation",
+    "Likeness",
+    "Misinformation",
+    "RunSummary",
+    "ScoreValue",
+    "Scorer",
+    "ScorerError",
+    "ScorerResult",
+    "ScorerRow",
+    "Toxicity",
+    "Transport",
+    "Usage",
+    "init_evaluations",
+    "resolve_launchdarkly_judges",
+    "urllib_transport",
+]