Antigravity CLI provider: bounded tail latency

[andreBurnt]

Antigravity CLI provider: bounded tail latency

Follow-up to #10. Routing tiers through antigravity-cli runs 4.8s to 30s+ against sub-second direct-API equivalents. I benched the provider's exact argv against agy 1.1.23, fixed what is fixable client-side, and documented the rest where the next reader needs it. 2 commits, 3 files, +183/-31, no new dependencies.

Tier defaults stop riding auto

defaultTierModels now maps every tier to an explicit effort-suffixed id: fast to gemini-3.7-flash-low, balanced to -medium, smart and code to -high. The sharing is deliberate (the pro models reject --effort and pay pro latency, wrong for a tier default). auto stays in the model list for anyone who wants it. Tier-routed calls resolve through the same map the ledger reports, so a tier never silently rides auto. The map's values are compile-time-tied to the catalog (AgyModelId), so a typo'd id is a tsc error, not a runtime surprise.

The measured basis, so you can check my reasoning. On a trivial prompt the floor is model-invariant: auto ran 2.82/3.04/3.19/3.05s over 4 runs, flash-low ran 2.74/2.97/3.08/3.40s over 4 runs plus one 4.71s outlier I cannot explain. The floor is process boot plus a 24.2k-token, 57-tool catalog prefill that never cache-hit across requests in any of my 15 single-turn runs (caching does work between turns inside one multi-turn call). On a hard prompt the arms diverge: auto spent 725 and 773 thinking tokens (6.98s, 5.67s) while flash-low spent 0 and 504 (4.20s, 5.30s), 2 runs per arm. Small sample, consistent direction: explicit effort bounds the thinking budget, auto picks the high end on exactly the prompts that were already slow. I did not reproduce the 30s+ band in these runs. Its known ingredients (retry sleeps, multi-turn tool loops) are addressed or documented below.

Empty completions resample instantly

An empty completion with SUCCESS status is the model abandoning the turn after a tool denial. It is stochastic, not load-shedding: identical requests varied from 0 to 568 thinking tokens in my probes. So waiting 1s/2s/4s between identical resamples was up to 7 dead seconds on the worst path. RetryOptions gains an optional delayMs hook, and this provider returns 0 for EmptyCompletionError while transient CLI errors keep exponential backoff. Attempt cap unchanged at 3.

On the shared-infra blast radius: every existing withRetries caller leaves delayMs undefined and behaves exactly as before. The hook itself is clamped to [0, maxDelayMs] and wrapped like onRetry. A future policy that throws or returns Infinity falls back to the plain backoff instead of crashing or sleeping forever. One design note: stream() keeps its own retry ladder because withRetries cannot wrap a generator. Both ladders now share the one policy function, which is as close as I could bring them without restructuring streaming.

gemini-3.5 leaves the catalog because agy sunset it

The registry check in the new test file diffs AGY_MODELS against live agy models output. It failed on its first run: the 3.5 flash line no longer exists upstream. Removed here, with modelMigrations carrying saved 3.5 routings to the 3.7 line at the same effort. Same mechanism as the codex gpt-5/gpt-5-mini to auto migration at codex-cli.ts:192.

One test file ships in-tree, on purpose

I know the convention from #11 is lab scripts in the PR body, and the one-shot behavioral proofs are below as exactly that. But antigravity-cli.test.ts (5 checks: tier map validity, resolution order, retry policy, migrations, live registry diff) is a standing guard, not a proof of this PR. It caught the 3.5 sunset the day it was written, and it will catch the day Google sunsets 3.7. As a PR-description script it runs once and rots. It follows the health-monitor.test.ts shape (node:test, zero deps) and the live check skips cleanly when agy is not installed. If you want it out of the tree, one git rm and it rides this description instead. My case is: this one earns its place.

The header now says what I measured

57 tools and ~24.2k input tokens per request, never cache-hit across requests. No isolation mechanism in agy 1.1.23: no --safe-mode or --config-dir, --sandbox does not shrink the catalog, config paths are hardcoded to ~/.gemini/config/*. A HOME override forks agy's OAuth state, so I rejected it. And the sharpest one: headless sessions EXECUTE allow-ruled tools rather than denying them. My probes watched run_command list $HOME and write_to_file create files (corralled in agy's per-session brain/ scratch). The header's "treat as a provider with tool access" warning was true before. Now it is specific.

What this does not do

1. No warm PTY transport. Boot is roughly 0.5-1s of a prefill-dominated ~2.8s floor. Scraping an undocumented TUI to save a third of a second is a bad trade. The real floor-breaker is the cross-request cache miss, which only the agy side can fix.
2. No skill/config isolation. There is no mechanism. Verified against the binary, documented in the header instead.
3. Nothing for sub-second tiers. That is the direct gemini provider's job, and it already registers when a key credential exists.

Evidence

npx tsc --noEmit clean. 13/13 across the suite: 3 in-tree health-monitor, 5 in-tree registry/policy guards, 5 lab checks below (pnpm tsx --test lab/*.test.ts, scripts kept in untracked lab/ per convention).

Lab script 1 - RetryOptions.delayMs override behavior (3 checks)

/**
 * RetryOptions.delayMs override — lets a provider substitute its own
 * per-error delay policy for the exponential backoff.
 * Run: pnpm tsx --test src/llm/provider-retry-delay.test.ts
 */
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { withRetries, EmptyCompletionError } from '../src/llm/provider.js';

test('delayMs override replaces exponential backoff', async () => {
  const delays: number[] = [];
  let calls = 0;
  const started = Date.now();
  const result = await withRetries(async () => {
    calls++;
    if (calls < 3) throw new EmptyCompletionError('empty', 'stop');
    return 'ok';
  }, {
    isRetryable: () => true,
    delayMs: (err, _attempt, dflt) => (err instanceof EmptyCompletionError ? 0 : dflt),
    onRetry: (_e, _a, d) => { delays.push(d); },
  });
  assert.equal(result, 'ok');
  assert.deepEqual(delays, [0, 0]);
  assert.ok(Date.now() - started < 500, `took ${Date.now() - started}ms — backoff not bypassed`);
});

test('delayMs return value is clamped to [0, maxDelayMs]; throwing or non-finite falls back to backoff', async () => {
  const seen: number[] = [];
  const run = (hook: (err: unknown, attempt: number, dflt: number) => number) => {
    let calls = 0;
    return withRetries(async () => {
      calls++;
      if (calls < 2) throw new Error('transient');
      return 'ok';
    }, {
      isRetryable: () => true,
      initialDelayMs: 10,
      maxDelayMs: 50,
      delayMs: hook,
      onRetry: (_e, _a, d) => { seen.push(d); },
    });
  };
  await run(() => 99_999);              // over the cap
  await run(() => -5);                  // negative
  await run(() => { throw new Error('bad policy'); }); // throwing hook
  await run(() => Number.NaN);          // non-finite
  assert.deepEqual(seen, [50, 0, 10, 10]);
});

test('without delayMs the exponential defaults stand', async () => {
  const delays: number[] = [];
  let calls = 0;
  await withRetries(async () => {
    calls++;
    if (calls < 2) throw new Error('transient');
    return 'ok';
  }, {
    isRetryable: () => true,
    initialDelayMs: 10,
    onRetry: (_e, _a, d) => { delays.push(d); },
  });
  assert.deepEqual(delays, [10]);
});

Lab script 2 - instant-resample behavioral proofs with a stub agy (2 checks)

/**
 * One-shot behavioral proofs for the empty-completion instant-resample
 * change (see PR body). Uses a stub `agy` that always returns an empty
 * SUCCESS result, so the retry ladder's timing is observable.
 *   pnpm tsx --test lab/agy-provider-checks.test.ts
 */
import { test } from 'node:test';
import assert from 'node:assert/strict';
import { mkdtempSync, writeFileSync, chmodSync } from 'node:fs';
import { tmpdir } from 'node:os';
import { join } from 'node:path';
import { AntigravityCliProvider } from '../src/llm/antigravity-cli.js';
import { EmptyCompletionError } from '../src/llm/provider.js';

/** A fake agy that ends a turn cleanly with an empty response every time. */
function fakeEmptyAgy(): string {
  const dir = mkdtempSync(join(tmpdir(), 'fake-agy-'));
  const bin = join(dir, 'agy');
  writeFileSync(bin, '#!/bin/sh\n'
    + 'echo \'{"event":"result","result":{"status":"SUCCESS","response":"","usage":{"input_tokens":1,"output_tokens":0}}}\'\n');
  chmodSync(bin, 0o755);
  return bin;
}

test('empty-completion retries resample instantly on the stream path', async () => {
  const p = new AntigravityCliProvider({ bin: fakeEmptyAgy() });
  const started = Date.now();
  const chunks = [];
  for await (const c of p.stream([{ role: 'user', content: 'hi' }])) chunks.push(c);
  const elapsed = Date.now() - started;
  const last = chunks[chunks.length - 1];
  assert.equal(last.done, true);
  assert.equal(last.stopReason, 'stop');
  // 3 attempts with the old 1s+2s ladder floor at >=3000ms; instant resample
  // is spawn-bound (~tens of ms per attempt).
  assert.ok(elapsed < 1500, `3 empty attempts took ${elapsed}ms — backoff not bypassed`);
});

test('empty-completion retries resample instantly on the complete path', async () => {
  const p = new AntigravityCliProvider({ bin: fakeEmptyAgy() });
  const started = Date.now();
  await assert.rejects(
    () => p.complete([{ role: 'user', content: 'hi' }]),
    (e: unknown) => e instanceof EmptyCompletionError,
  );
  assert.ok(Date.now() - started < 1500, 'complete() backoff not bypassed');
});