Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions packages/client/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -220,6 +220,45 @@ asyncio.run(main())

---

### Agent Skills

Skills are versioned `SKILL.md` documents managed in LaunchDarkly and attached to AI Config
variations by reference. This release adds the first layer: discovering which skills a
resolved config references. Retrieving their content and materializing them onto disk follow.

```python
import asyncio

from launchdarkly_ai_server import init_client, inspect_config, skill_refs

async def main():
await init_client()

info = await inspect_config("doc-agent", {"kind": "user", "key": "user-123"})
refs = skill_refs(info["config"]) # [SkillReference(key='pdf-extraction', version=2)]

for ref in refs:
print(ref.key, ref.version)

asyncio.run(main())
```

**`skills` is now a validated field.** Config parsing fails closed on a `skills` value that
is not a list of `{key, version}` objects (key matching `^[a-z0-9][a-z0-9-]*$`, version an
integer ≥ 1): the whole variation is rejected, `inspect_config` returns `config: None`, and
`extract_variation` raises. A variation that previously carried its own custom `skills`
field of a different shape must rename it before upgrading.

| Export | Description |
|---|---|
| `skill_refs(config)` | Project a config's `skills` array into `list[SkillReference]`. Pure — no client, store, or network needed. Returns `[]` when absent. |

> `Skill.content` is `bytes` — the verified verbatim bytes LaunchDarkly delivered, exactly
> what was hashed. The SDK never parses or interprets them; if you want the frontmatter,
> decode and parse the content on your side.

---

### Utility Helpers

```python
Expand Down Expand Up @@ -250,3 +289,5 @@ All types are exported from this package. Handler packages import them from here
| `GraphNode` / `GraphEdge` | A dataclass node (`.key`, `.config`, `.meta`, `.edges`, `.is_terminal`) and a dataclass directed edge (`.key`, `.source_key`, `.target_key`, `.handoff`) |
| `ProviderGraphResponse` | A dataclass returned by `graph(...).invoke()`: `.response`, `.usage`, `.judge_results` |
| `GraphTopology` | The parsed graph flag shape (`root` + `edges`) |
| `Skill` | A frozen skill document: `.key`, `.version`, `.content` (verified verbatim `bytes`), `.content_hash`, `.name?`, `.description?` |
| `SkillReference` | A frozen version-pinned pointer to a skill: `.key`, `.version` |
79 changes: 78 additions & 1 deletion packages/client/agents.md
Original file line number Diff line number Diff line change
Expand Up @@ -27,7 +27,8 @@ No other `launchdarkly-ai-*` package may define or duplicate these. They import
| `src/launchdarkly_ai_server/tracking.py` | `execute_and_track`, `execute_and_stream`, `wrap_tool_handlers`, `parse_usage` |
| `src/launchdarkly_ai_server/graph.py` | `graph()`, `resolve_graph()`, `GraphInstance` |
| `src/launchdarkly_ai_server/types.py` | All shared Python types — `AiConfigRep`, `ProviderHandler`, `LDContext`, `NativeTool`, etc. |
| `src/launchdarkly_ai_server/types_validation.py` | `parse_ai_config` — validates flag variation shape |
| `src/launchdarkly_ai_server/types_validation.py` | `parse_ai_config` — validates flag variation shape; `is_valid_skill_key` / `is_valid_skill_version` / `skill_key_rejection_reason` (the canonical key-grammar explanation every layer quotes) |
| `src/launchdarkly_ai_server/skills.py` | Agent Skills — `skill_refs`, the projection of a config's `skills` array into typed references |
| `src/launchdarkly_ai_server/utils.py` | `parse_template`, `parse_json_with_possible_fences`, `create_handler`, `parse_usage`, `make_track_data`, `to_ld_context` |
| `src/launchdarkly_ai_server/registry.py` | `Registry`, `global_registry`, `compose`, `resolve_handlers`, `resolve_tools` |
| `src/launchdarkly_ai_server/judges.py` | `run_judges`, `build_judge_tasks`, `run_judge` |
Expand All @@ -53,6 +54,7 @@ from launchdarkly_ai_server import (
TrackData, UsageDict, HandlerResult, HandlerStreamEvent,
StreamEvent, StreamChunkEvent, StreamDoneEvent, ExecuteStreamEvent, ExecuteStreamDoneEvent,
VariationMeta, InitClientOptions, JudgeResult, ParseResult, ParseSuccess, ParseFailure,
Skill, SkillReference,
)

# Utilities
Expand All @@ -69,6 +71,9 @@ from launchdarkly_ai_server import execute_and_track, execute_and_stream, wrap_t

# Entry points
from launchdarkly_ai_server import config, graph, resolve_graph

# Agent Skills
from launchdarkly_ai_server import skill_refs
```

When adding a new export, add it to `__init__.py`'s imports and `__all__`. Handler packages must never import from sub-paths (e.g. `launchdarkly_ai_server.client`).
Expand Down Expand Up @@ -155,6 +160,32 @@ This is an OTel context value, not W3C baggage, so the id does not leak onto out

---

## Agent Skills

Versioned `SKILL.md` documents attached to AI Config variations by reference, retrieved
through an injectable store, and materialized onto disk for agent runtimes to discover.
Three layers, in increasing order of blast radius. Only the first is implemented here:

1. **Reference discovery** — `skill_refs(config)` projects the config's `skills` array into
typed `SkillReference` values. Pure: no network, no client, no store, no telemetry.
Validation of the array itself lives in `parse_ai_config` and is **fail closed** — one
malformed reference fails the whole config parse.
2. **Content accessors** — reading skill content through a store seam.
3. **Materialization** — writing skills onto disk under a manifest.

### Security posture — do not relax any of this

- **Skill content is an opaque byte buffer.** `Skill.content` is `bytes` — the verified
verbatim bytes LaunchDarkly delivered, exactly what was hashed. This SDK never parses,
decodes, or interprets them anywhere: not in the integrity path, not in an accessor, not
during materialization. Consumers who want frontmatter parse it themselves.
- **A key is untrusted input everywhere it appears.** `skill_key_rejection_reason` is the
single canonical explanation, so the config parser and the reference projection reject a
key for the same stated reason — and so does every layer added later. A silently
shortened projection is not acceptable: every dropped entry is logged.

---

## OTel Setup

The core client owns all OTel initialization. `init_client()` configures a `TracerProvider` with `ConversationIdSpanProcessor` and a `BatchSpanProcessor` plus an OTLP HTTP exporter when the optional OTel packages are installed.
Expand Down Expand Up @@ -230,6 +261,43 @@ When `enabled` is `False`, `config` is always `None`. When `enabled` is `True` b

---

## Dependencies

Tier 0, so the runtime surface is deliberately tiny: **one** hard dependency, and everything else either an optional extra, resolved dynamically at runtime, or dev-only. Nothing here may grow without a reason recorded in this table.

### Runtime (`[project] dependencies`)

| Package | Why |
|---|---|
| `opentelemetry-api>=1.25` | The tracer/span API used on every instrumented path (`tracking.py`, `graph.py`, `content.py`, `conversation.py`, `utils.py`). API-only — the *SDK* half is an optional extra, so a consumer that never configures OTel gets no-op spans rather than an `ImportError`. `conversation.py` imports `opentelemetry.sdk.trace.SpanProcessor` under `TYPE_CHECKING` only, for exactly this reason. |

There is deliberately **no** `python-dotenv` here: `lifecycle.py` reads `os.environ` directly, so loading a `.env` file is the application's job rather than the SDK's. `python-dotenv` is in the workspace dev group for the examples only.

### Optional extra (`[project.optional-dependencies] otel`)

| Package | Why |
|---|---|
| `opentelemetry-sdk>=1.25` | Tracer provider, resources, and the batch span processor, imported inside `_setup_telemetry()` in `lifecycle.py`. Optional so telemetry is opt-in; absent ⇒ a `logger.warning` and no spans, never a raise. |
| `opentelemetry-exporter-otlp-proto-http>=1.25` | OTLP/HTTP span export and its compression enum. Same optionality, same loader. |

Install with `pip install "launchdarkly-ai-server[otel]"`; see [OTel Setup](#otel-setup) for the endpoint variables.

### Resolved dynamically, declared nowhere

| Package | Why |
|---|---|
| `launchdarkly-server-sdk` | The LaunchDarkly server SDK, reached by `importlib.import_module("ldclient")` (falling back to `launchdarkly_server_sdk`) inside `init_client()`'s options path. Undeclared on purpose: the BYOC path (`init_client(client=...)`) targets environments that supply their own client, and a hard dependency would force an unused SDK into every such install. So it is imported late and raises actionably when missing — absent ⇒ a `RuntimeError` naming the `pip install`, and only on the path that needs it. |

### Dev-only (workspace root `[dependency-groups] dev`) — the ones with a contract attached

| Package | Why |
|---|---|
| `launchdarkly-server-sdk>=9.0`, and the `otel` extra mirrored (`opentelemetry-sdk`, `opentelemetry-exporter-otlp-proto-http`) | Each dynamically-resolved or optional package is repeated in the dev group so the test suite can import it. Something that is *only* optional would not be installed in this workspace and the tests covering its present-and-working path could not run. |
| `pytest>=8`, `pytest-asyncio>=0.24` | Test runner and the async support the whole suite relies on. `asyncio_mode = "auto"` is set at the workspace root, which is why no test in this package carries an `@pytest.mark.asyncio`. |
| `mypy>=1.10` (`strict`), `ruff>=0.15` | Type checker and linter/formatter. |

---

## Common Pitfalls

### 1. Calling `get_client()` before `init_client()` resolves
Expand All @@ -240,6 +308,14 @@ When `enabled` is `False`, `config` is always `None`. When `enabled` is `True` b

`execute_and_track` expects the handler to return a plain `dict` with at least `output` and `usage` keys. Do not return a custom class — `parse_usage` and the telemetry pipeline both access dict keys.

### 3. Interpreting skill content anywhere

`Skill.content` is opaque `bytes` by construction. Do not add a parser, a decoder, or a
convenience accessor that reads meaning into it — no YAML/frontmatter parsing, no
"decode as UTF-8 for display", nothing. The SDK's whole contract is that content is the
verified verbatim byte buffer and nothing more; a consumer who wants structure parses it
on their side of the boundary.

---

## Adding a New Export
Expand All @@ -255,3 +331,4 @@ When `enabled` is `False`, `config` is always `None`. When `enabled` is `True` b
- Handler packages must import `LDContext` from `launchdarkly-ai-server` — not directly from any LD SDK.
- Do not weaken the `parse_ai_config` validation — handler packages rely on `config` being valid when they receive it.
- `parse_usage` must continue to accept `input_tokens/output_tokens`, `inputTokens/outputTokens`, and `input/output` as all existing handlers return one of these variants.
- `Skill.content` is opaque `bytes`. Do not add anything that parses or interprets it — no YAML library in this package's dependencies at any tier, and no accessor that decodes content.
9 changes: 9 additions & 0 deletions packages/client/src/launchdarkly_ai_server/__init__.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,9 @@
resolve_handlers,
resolve_tools,
)
from .skills import (
skill_refs,
)
from .tracking import execute_and_stream, execute_and_track, wrap_tool_handlers
from .types import (
NATIVE_TOOL_KEY,
Expand Down Expand Up @@ -66,6 +69,8 @@
ProviderGraphResponse,
ProviderHandler,
ProviderResponse,
Skill,
SkillReference,
StreamChunkEvent,
StreamDoneEvent,
StreamEvent,
Expand Down Expand Up @@ -126,6 +131,8 @@
"ProviderGraphResponse",
"ProviderHandler",
"ProviderResponse",
"Skill",
"SkillReference",
"StreamChunkEvent",
"StreamDoneEvent",
"StreamEvent",
Expand Down Expand Up @@ -199,4 +206,6 @@
"graph",
"resolve_graph",
"GraphInstance",
# skills
"skill_refs",
]
72 changes: 72 additions & 0 deletions packages/client/src/launchdarkly_ai_server/skills.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,72 @@
"""
Agent Skills — reference discovery.

Projects the skill references a resolved AI Config carries into typed values. A
pure projection: no network, no client, no store, no telemetry. Retrieving the
content those references point at is a separate layer.
"""

from __future__ import annotations

import logging

from .types import AiConfigRep, SkillReference
from .types_validation import (
is_valid_skill_key,
is_valid_skill_version,
skill_key_rejection_reason,
)

logger = logging.getLogger(__name__)


# ---------------------------------------------------------------------------
# Reference discovery
# ---------------------------------------------------------------------------
def skill_refs(config: AiConfigRep | None) -> list[SkillReference]:
"""
Projects a resolved AI Config's ``skills`` array into typed references.

A pure projection — no network, no client, no store, no telemetry. Returns
``[]`` when the config carries no skills.

A config that came through ``parse_ai_config`` never contains an invalid
entry — parsing fails closed on one. A hand-built dict can, and a silently
shortened projection would leave a caller materializing a skill set it
believes is complete, so every dropped entry is logged.
"""
if not isinstance(config, dict):
return []

raw = config.get("skills")
if not isinstance(raw, list):
return []

refs: list[SkillReference] = []
for index, entry in enumerate(raw):
if not isinstance(entry, dict):
logger.warning(
"skills[%d] is not a {key, version} object; it was dropped "
"from the projection",
index,
)
continue
key = entry.get("key")
version = entry.get("version")
# Branch on the TypeGuard predicate (not the reason string) so the type
# checker narrows ``key`` to ``str`` for the reference below.
if not is_valid_skill_key(key):
logger.warning(
"skills[%d].key %s; it was dropped from the projection",
index,
skill_key_rejection_reason(key),
)
elif not is_valid_skill_version(version):
logger.warning(
"skills[%d].version must be an integer >= 1; it was dropped "
"from the projection",
index,
)
else:
refs.append(SkillReference(key=key, version=version))
return refs
42 changes: 41 additions & 1 deletion packages/client/src/launchdarkly_ai_server/types.py
Original file line number Diff line number Diff line change
Expand Up @@ -92,7 +92,8 @@ class Message:
AiConfigRep = dict[str, Any]
"""
Raw AI config dict as returned by ``parse_ai_config``. Fields include
``model``, ``provider``, and at least one of ``instructions`` / ``messages``.
``model``, ``provider``, at least one of ``instructions`` / ``messages``, and an
optional ``skills`` array of ``{key, version}`` references (see ``skill_refs``).
"""

VariationMeta = dict[str, Any]
Expand Down Expand Up @@ -412,6 +413,45 @@ class ProviderGraphResponse:
"""Results from a graph-level judge, if configured."""


# ---------------------------------------------------------------------------
# Agent Skills
# ---------------------------------------------------------------------------


@dataclass(frozen=True)
class SkillReference:
"""A version-pinned pointer to a skill, as attached to an AI Config variation."""

key: str
"""Immutable skill key — ``^[a-z0-9][a-z0-9-]*$``, at most 256 characters."""
version: int
"""Immutable skill version — an integer >= 1."""


@dataclass(frozen=True)
class Skill:
"""
A single verbatim ``SKILL.md`` document.

Only ever constructed after integrity verification passes, so ``content``
holds the exact byte sequence LaunchDarkly delivered and ``content_hash``
is its sha256. Instances are immutable.
"""

key: str
version: int
content: bytes
"""The verified verbatim bytes, exactly as LaunchDarkly delivered and
hashed them. Opaque to this SDK: no encoding is claimed and nothing here
ever parses or interprets them."""
content_hash: str
"""sha256, lowercase hex, over the verbatim bytes of ``content``."""
name: str | None = None
"""Display name from LaunchDarkly metadata; never parsed from the content."""
description: str | None = None
"""Description from LaunchDarkly metadata; never parsed from the content."""


# ---------------------------------------------------------------------------
# Model / graph options
# ---------------------------------------------------------------------------
Expand Down
Loading
Loading