Replace the leads demo with examples that run - #29
Open
arjunvlama wants to merge 46 commits into
Open
Conversation
`charter pending` and `charter describe` suggested an approve, reject or answer command without `--instance`. Every applied agent has an instance, so copying the suggestion exits 1 with "has 1 instance — say which". Both now name the instance they resolved. A test runs every command a gate prints and asserts it exits 0, so a newly required flag fails here rather than in someone's terminal. `demo/leads/approve.py` is deleted. It opened with "Charter has no approve command", which stopped being true, and a Python script for approvals reads as missing product in a demo. The README now uses `charter pending` and `charter approve`, and points at `charter ui` for the same thing in a browser.
`examples/` named real Zendesk and Stripe servers and was documented as not running. `demo/leads` ran, but needed four terminals, a simulated professional network and you role-playing the people being contacted, and an agent writing cold outreach at scale is the wrong thing for a governance product to demonstrate. Both are now one toy support desk. `examples/desk.py` is a small MCP server with four tickets and the charges behind them; `refund-triage` reads a ticket and gates the refund, `ticket-summarizer` reads and reports. One domain across both, so what differs is the capability rather than the scenario. They need a model key and nothing else. `refund-triage` gains the skill the deleted demo was carrying, so skills are still shown somewhere. `worker.yaml` keeps its notification block, which resolves lazily, and comments out the OTel sink, which does not and stopped the worker booting. Tests that loaded the demo or hardcoded the old server names move to the desk. `test_an_ordinary_tool_failure_does_not_end_the_task` was passing on a tool name no config declared, which made it vacuous; it now names a declared tool that is not fail-fast.
arjunvlama
changed the title
The worker spawns the tool server as `python desk.py`. With the venv inactive that resolves elsewhere, mcp is missing, and the agent is quarantined at boot.
Spawned as `python desk.py`, the server ran under whatever `python` the worker's PATH resolved to. With the venv inactive that is a different interpreter, `mcp` is missing, and both agents quarantine at boot. It now serves streamable-http on 127.0.0.1:8931 and the agents name it by URL, which is how an MCP server is usually reached anyway. You start it yourself, as its own step in the README. The worker boots clean with no venv active. Loopback http is already allowed by the url rule, so nothing in the schema changed. The three McpServer tests that assumed the example spawns a process now assume it names a URL.
'desk' is help-desk jargon, and it leaked into every tool name as desk__create_refund. The server is support_server.py, the server is named support, and the tools read support__get_ticket and support__create_refund. tests/e2e/test_live.py keeps 'desk.get_ticket' in its docstring: that is the actual name from the incident it describes.
The rename swept tests/e2e, which loads playground/, not examples/. Its server is still named desk, so those assertions have to keep desk__ names.
`charter agents` filters instances whose lifecycle_state is deleted; the picker behind --instance did not, so the two disagreed about what exists and the picker suggested a deleted one as the default. Its docstring already said 'every live instance'.
`charter run` loaded the agent's YAML to validate flags, so triggering a task needed the repo. Nothing else does: Prefect validates against a schema the server holds, Temporal sends an opaque payload. Inputs now go as given. The worker already fills declared defaults and refuses a task missing a required input, from the versioned config it is serving, so the CLI was keeping a second copy of a spec it could not guarantee was current. Flag values are text, so numbers and booleans are read off the value, the same guess YAML makes. --path is gone from run, and _coerce and _show_inputs with it.
Rejecting produced a failed task: the objective said nothing about being refused, so the agent reopened the question, ended its turn on prose, and never called submit_result. A task with no answer in the shape response_format declares is a failed task. It now revises from the reason it was given. Rejecting $48.00 with 'only half is ours, refund 24.00' brings back a $24.00 proposal and a second gate, and approving that finishes the task with both charges accounted for. Attempts are capped in runtime.yaml rather than the objective, at three calls to support__create_refund. The prompt asks it to revise rather than repeat; the ceiling is what holds when it doesn't.
Comments in the example files explained reasoning and history. They now state the field. Dropped an orphaned note in v1.yaml describing approval_timeout_seconds, which lives in runtime.yaml, and the em dashes.
`describe` printed each rule as '6 of 4 -> pause window=10'. The count is the current version's total and the threshold is scoped to a window of recent tasks, so it read as progress toward something it was not measuring. Rules are now a table of metric, threshold, window, action and tool. Version totals move to their own block with the version as a field. The arrow is gone from apply, describe, runs and audit, which now print metric=, threshold= and action= like the rest of the output.
`window` is a lookback over recent runs and a minimum sample size: a rule does not evaluate until that many runs exist. At window 10 none of these rules could fire until someone had run the agent ten times, so nothing in the example was demonstrable. Rejections and failures now look back 3 runs, spend 5. The comment says what window means, because the behaviour is not obvious from the field name.
One run can propose at most 3 refunds, so a threshold of 4 cannot be reached inside a single run. The rule now demonstrates what a lifecycle policy is for: a pattern across runs, not a limit within one.
The rules existed in the config and nothing in the README made one fire, so a reader saw policy and took it on faith. ticket-summarizer is the rollback: v2 is a pricier model on a schedule, v1 is neither, and crossing the spend threshold puts v1 back. refund-triage is the pause: four rejections across three runs and it stops itself, which one run cannot reach because the tool cap is three. The audit trail printed WorkflowPolicyAction.SET_VERSION, an SDK enum repr. It now reads set_version.
`charter agents` printed 'N waiting on a human' and 'N stopped — no new tasks will start' beneath a table whose STATUS and ACTIVITY columns already said both. What the table does not carry is the command for each, so that is what is left. `describe` printed the same sentence under a line showing the same state.
The field descriptions said window was how many recent tasks the metric sums over, and left out that a rule is not evaluated until that many tasks exist. An agent with window 10 and four tasks behind it has no rule evaluated, whatever its metrics say. Reading it as a plain lookback is how I spent an afternoon concluding a working feature was broken. DESIGN.md also now says why set_version has no window: it compares totals for the running version.
The minimum-sample-size note goes away with the behaviour it described. Cooldown keeps a description, which it never had. DESIGN.md keeps the one part still true: set_version takes no window, because it compares totals for the version now running.
"if a worker dies / another picks it up" described one cause of a resumed run, and `resumable` covers any infrastructure failure: an expired lease, a cancelled operation, a worker that went away. Each row is now a field and a value, using the config's own names, so a value in `describe` can be found in the YAML that set it. triggerable and invoke mode were folded into prose in other rows and are their own now.
The gate printed a sentence built from two of its five fields and dropped the rest. metadata carries the tool and its arguments as data, and opened_at says whether a decision has been waiting two minutes or two hours. Both are now on screen, with the approval id as a field rather than only inside a command. status gains the request's kind, sequence, status and timeout, and the runtime policy that was in force for that run, which is what it was actually held to rather than whatever apply has changed since. The metrics block gains working time and per-tool failures. Working time is BoundFlow's total_latency_seconds: summed across runs, excluding time parked at a gate, which is why it is not called latency next to a wall-clock took. create_refund takes a reason, so the agent's rationale reaches the approver instead of staying in its head. playground's tool always had one.
`runs` folded status and outcome into one column, so a request that completed with an unsuccessful run looked like one that failed, and dropped request_type entirely. Both are columns now. `agents` sorted an agent in cooldown into 'stopped' and told you to resume it, which is not a thing you need to do: it starts again on its own. Cooling down and deleting are their own groups, carrying the times BoundFlow already gave us.
The reason said what it was for and never what it must contain, so the agent filled it with a summary the approver cannot check. The tool description now asks for the ticket, the charges compared, and why this amount, with an example of each kind. The objective says the same, and says why: the approver cannot see the tickets the agent read.
The gate printed 'agent wants to call TOOL with args...' and then listed tool and args underneath. The sentence stays in , which a notification and the console carry on their own, and the CLI now prints only what the harness added beyond it. Only when metadata carries the tool: a justification with nothing below it repeating it is still printed whole.
The arguments were joined into one line, which put the agent's reasoning at the end of a string nobody reads to the end of. The reason it passes is the point of the screen, so it gets a row like every other argument.
A gate could only ever show the call and its arguments: the harness hands Charter a tool call and nothing else, and the description it attaches says "Tool execution requires approval". So an approver saw what was about to happen and no account of why. Charter now adds a required `why` to the schema of every tool marked `approval: always`, and strips it before dispatch, since the server never declared it. It leads the justification, which is the field a notification and the console carry on their own, and it gets its own row on the gate. Only gated tools: nobody reviews the rest, so asking the model to justify them spends tokens on text nobody reads. The example's create_refund had grown a `reason` parameter to work around this. It is gone, and the example no longer needs a tool it controls to demonstrate the feature.
`justify` on a gated tool, default true. A call whose arguments already say everything does not need a sentence about them, and the tokens are not free. The field the model fills is `justification`, which is what BoundFlow calls the thing it becomes, rather than a Charter word for the same idea.
The two policy paragraphs were written rather than instructional: "the ceiling is what holds when it doesn't", "says something about the agent rather than the task". They now say what the file does and what to run to see it, with the describe output the reader will get.
A rejection that crosses the threshold mid-run leaves the agent active until that run finishes, which reads as the rule not working.
The README named T-1041 and nothing else, so there was no way to know the other three existed or which of them propose a refund. Reaching the pause needs a second run, which the reader had no way to set up.
The ticket list claimed T-1043 and T-1044 finish without a refund. T-1043 does, which I ran; T-1044 was a guess. The model reads the ticket and the refund policy and decides, so the list now says what each ticket is and leaves the outcome to the run.
Nobody reading this cares what the fictional customers wanted. They need an input to pass.
v2 differed from v1 by model and a schedule. Neither is what versioning is for in practice: production teams change prompts 5 to 15 times a week, and prompt regressions are the leading cause of agent quality drift. Every tool in the space is built around prompt versions and rollback. v2 now differs only in its objective: it opens each ticket and the charge behind it rather than working from subject lines. Better summaries, more model calls, which is why a cost rule rolls it back. Dropping the schedule also removes an ambiguity: with v2 changing two things at once, a rollback for cost silently turned scheduling off as well. The two tests that read model and schedule off the example now build what they assert.
The reader is trying a new version of a prompt and wants it rolled back if it costs too much. What the prompt was changed to is the example's business.
`max_calls` counts runs, and a rejected gated call never runs, so nothing bounded how often an agent could put the same decision in front of someone. A task could propose a refund all afternoon. `max_proposals` sits on the same tool_call_limits entry and bounds proposals. Past it the agent is told and carries on, the way a spent call cap works, and the call does not run because nobody approved it. Not routed through `on_reject`: nobody refused. Failing a task because we declined to ask would punish the agent for a limit the operator set. The example now allows one refund to go through and four proposals, which leaves room to correct it from a rejection and then stop asking.
It is its own field on the approval, and it was also left in the metadata arguments, so anything rendering both printed the agent's sentence once as the justification and again inside args. The CLI hid it at render time; the console did not. Verified against the console, which now carries what the CLI does: the approval id, the justification, the tool and its arguments, when it opened and when it times out. The examples README says it is there.
… pending `charter pending` had been lost in an edit, so "that prints the call it wants to make" referred to nothing and there was no way to see a gate or get an approval id. The console is introduced right after the first run, which is Prefect's placement for the same reason: the task appears, then the gate does, on a refresh. It is also more than approvals, so the line says what else it holds.
Introduced right after the agent exists, then pointed at when the task appears, when a gate opens, and when the run picks back up. One instruction to open it rather than three, and each stage says what it shows.
Keeping it out of metadata.args stopped the console printing it twice, and it also removed what the CLI was rendering its row from, so the gate lost the one line that says why. The console was unaffected: it reads the field. Verified on BoundFlow 0.7.0, both surfaces, with a test that the row survives arguments that do not carry it.
CI installs the SDK from PyPI and runs it against `:latest`, so 0.7.0 with 0.7.0 is the only combination that gets exercised. The floor said 0.6.0, which allowed a pairing nobody checks.
`custom` is Charter's half of the runtime policy: BoundFlow carries it and reads none of it, so it arrived as one opaque dict and printed as one. Eight limits an operator looks for by name were behind it. Two rendering bugs it was hiding: capability limits name a capability rather than a tool, so every one read `None=30`, and a tool limit's proposal ceiling was dropped.
BoundFlow's ToolCallLimit caps calls and has no field for asks, so max_proposals was dropped at compile and only ever read from the worker's own runtime.yaml. It travels in `custom` now, beside the other limits Charter enforces itself, and the loop reads it from the governor's policy like it already reads max_wait. Two things this fixes. Lowering the ceiling took a worker restart instead of a `charter apply`. And a worker serving a pulled artifact has no runtime.yaml at all, so its gates ran with no proposal ceiling.
A worker serving a checkout read its caps from the runtime.yaml on its disk; only a worker serving a pulled artifact read them back from the control plane. So `charter apply` could not lower a cap without a restart, and two workers on different checkouts of the same agent enforced different numbers. Both paths now read the applied policy, and the local file stands in only until the first apply. Fixes a crash this exposed: tool_failure_limits is emitted per declared tool, so an agent that calls none applies a policy carrying an empty list, and rebuilding runtime.yaml from it fed 0 to a field requiring a positive number. That is the agent `charter init` writes.
`load_agent(policy=False)` leaves runtime.yaml and lifecycle.yaml unopened, and `charter worker` uses it. The worker never acted on lifecycle.yaml, and no longer needs runtime.yaml now that its caps come from the control plane — so a malformed policy file in the directory stopped a worker booting over something it would never read. Every other command reads them as before. The proposal cap follows: one read, from applied policy, with no fall back to a file the worker no longer opens.
The SDK returns a typed object when a policy is written and protobuf JSON — a plain dict — when one is read back. `_of` only ever looked at the attribute, so every reader of `custom` saw an empty dict on the read path: an allowlist read as no allowlist, a cap as no cap, and `allowed_spawns` as nothing at all. Already broken for a worker serving a pulled artifact, which has always taken its policy from the wire. Reading applied policy on the checkout path too is what made it visible. Also prints counts as integers. Protobuf JSON has one number type, so every ceiling came back as 40.0.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Two changes that turned out to be one: the demo's approval flow was wrong, and then the demo itself was the wrong thing to ship.
The printed command did not run
charter pendingandcharter describerender a gate with the commands that answer it, and left out--instance. Every applied agent has one, so copying the suggestion exits 1 with "has 1 instance — say which". Same for thecharter answerhint. Both call sites now name the instance they already resolved, and a test runs every command a gate prints and asserts it exits 0.examples/ did not run, and the demo ran but was off-message
examples/named real Zendesk and Stripe servers and the README said it "does not run as-is". None of the reference repos ship that: Prefect's examples are runnable scripts, NeMo's are runnable configs, LangGraph's are executable notebooks.demo/leadsdid run, but needed four terminals, a simulated professional network, and you role-playing the people being contacted. It is also an agent writing cold outreach at scale, which is the use case people are worried about rather than the one to lead with.Both are replaced by one toy support desk, following NeMo's pattern of reusing a single trivial domain across examples:
examples/desk.py, a small MCP server with four tickets and their chargesrefund-triagereads a ticket, decides, and gates the refundticket-summarizerreads and reports, at two versions so a rollback has somewhere to goThey run with a model key and nothing else.
demo/is deleted, andapprove.pywith it.Verified
Ran end to end against a local control plane and a real model:
charter run refund-triage --ticket_id T-1041reached a gate in 24 seconds asking to refund $48.00 on the duplicate charge, approved it, task completed in 40s with the structured resolution. Also killed the worker mid-gate, approved with nothing running, restarted, and watched the task resume.315 unit tests pass. Several needed repointing, and one was found to be vacuous:
test_an_ordinary_tool_failure_does_not_end_the_taskpassed because it named a tool no config declared, so nothing could have been fail-fast.