[joelteply]

Slice 1 of continuum card 494c99f1 (the learning blocker). A cache is never need. On the M5, a 4 MB learner waited forever because the ~13 GB host prompt cache counted as serving's need.

Change: POST /prompt-cache/limit {limit_mib >= 1}.

• It runs on the serving loop, which owns the cache.
• It sets the limit and evicts the oldest states until the cache fits. The cache's own update() already did this; only --cache-ram could set the limit, and only at launch.
• It replies with {refusal, limit_mib, held_bytes, freed_bytes, states}, so the core frees exactly what was measured.
• 0 is refused, because to the cache it means no limit.
• Router mode proxies it.

Fixture: tests/prompt-cache-limit.py. It fills the cache, checks 0 is refused, lowers the limit to 1 MiB and asserts bytes were freed and the cache fits, then raises the limit again.

Next slices (core): serving's cache becomes its own reclaimable lease, whose eviction calls this verb. A learner's refused lease then reclaims cache bytes before waiting.

🤖 Generated with Claude Code

[joelteply]

Review of fork #110 at 357096e (Cormac): CHANGES REQUESTED, one fix; the rest is right.

Verified by reading (not run here: the patch does not apply on top of #109, so running its fixture needs its own build tree):

• server_prompt_cache::update() (server-task.cpp:1930) evicts the oldest states while size() > limit_size and rescales its token limit to the new byte limit, so lowering the limit and calling update() frees exactly what the reply's freed_bytes reports. Raising it allocates nothing. Correct.
• The task runs on the serving loop between batches and only drops host copies of states, never a live slot. Correct.
• 0 refused at the route because the cache reads limit_size == 0 as unlimited (if (limit_size > 0) in update()). Correct and well tested.
• Router mode proxies the route. Correct.

The fix: prompt_cache->limit_size = 1024ull * 1024ull * (size_t) task.prompt_cache_limit_mib; overflows for a large limit_mib (anything at or above 2^44 MiB on a 64-bit size_t): the product wraps, and a wrap to exactly 0 is "unlimited" while any other wrap is an arbitrary small limit that empties the cache. The route only checks >= 1. Refuse limit_mib above SIZE_MAX >> 20 (or above what the host can hold) as invalid, and add that case to the fixture next to the 0 case.

Nit (non-blocking, card it or fix here): the reply reuses server_task_result_learn_grants for a non-learning operation; a generic receipt result type would read truer.

Owner question for the core slice: a lowered limit outlives a core that dies before raising it again; who restores it (a lease the core holds, released on its death)?

[joelteply]

Re-review (Cormac): APPROVE. Verified by reading the new head's diff: max_mib = SIZE_MAX >> 20 (2^44 - 1 on 64-bit), so limit_mib * 2^20 < 2^64 and never wraps; 0 and anything above max_mib are refused 400; a JSON integer past INT64_MAX reads back negative and fails >= 1; the fixture asserts 2^62 refused. The non-blocking nit (generic receipt type) and the core-side question (who restores a lowered limit after the core dies) stand for the core slice.