Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 3 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,6 +17,9 @@ Every entry says what changed as a caller sees it, why, and what to watch for

### Added

- Drive research acceptance selects exact version-specific export and memory-search contracts,
and records aggregate read-only Google import/recovery results without publishing source data.

- The installed Drive research check now verifies person links, stale-note revision refusal,
scoped source-backed memories and local source retirement through the released CLI. It
checks persisted memory staleness and owner-note preservation with synthetic data only.
Expand Down
66 changes: 59 additions & 7 deletions performance/documents/offline-catalog/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,61 @@ The release check also asks whether the new JSON envelope and config migration p
workflows for installed-package consumers.

The [controlled Google checklist](live-checklist.md#use-the-prepared-checker) now includes a
reusable isolated acceptance runner. Google checks remain unrun until the owner connects a
dedicated test account; its synthetic and installed-package validation is recorded below.
reusable isolated acceptance runner. The owner later explicitly authorized read-only checks with
an existing profile; aggregate results are recorded below, alongside synthetic package validation.

## Read-only Google acceptance

The owner asked to validate an existing selected folder using its already connected profile.
On 2026-10-15, Linux/Node 24.19.0 and installed Drive 0.7.0 (Core 0.278.0/Base 0.28.2)
read Google while keeping all content, database, checkpoints and model-cache paths in a new
private workspace. Original authentication was reused; the owner's database was not opened.
Google files, sharing and permissions were never modified.

| Check | Observed result |
| --- | --- |
| Complete selected catalog | 443 entries: 65 folders and 378 files |
| Full import | 197 text-indexed files; 181 metadata-only; zero pending jobs |
| Metadata-only reasons | 179 unsupported types; one scanned PDF without a text layer; one unreadable XLSX |
| Cached hierarchy and metadata | Complete folder ancestry/tree and retained provider metadata |
| Initial offline batch | 20 files; 56 assertions passed with network/browser/keyring denied |
| Reader coverage | Exact word search and byte-exact Markdown export passed for one file each in eight categories |
| Live Google Docs comparison | Five single-tab Docs; 937 fragments and two tables matched; provider versions stayed stable |
| Interruption and resume | Own CLI exited 130 after SIGINT; resumed and drained pending jobs |
| Local document context | Temporary tag, note and proposed source-backed memory survived interruption and repeated sync |
| Repeat import | Zero new text saves; zero pending jobs; source references preserved |

The eight categories were Google Docs, Google Sheets, PDF, DOCX, XLSX, CSV, Markdown and TXT.
Exported files had mode 0600. The TXT test initially normalized CRLF while comparing Python
strings; comparing UTF-8 bytes established exact preservation. This was a harness issue.

Private report and scripts: `cli-private/reports/drive/2026-10-15-live-read/`. Queries, source IDs,
per-file outcomes and local workspace paths stay there. These scripts describe historical staged
checks on that authorized workspace; they are not an invitation to use arbitrary owner accounts.
The reusable synthetic runner below remains independent of this live run.

This establishes reading and offline retrieval for this selected tree. It does not certify fresh
OAuth consent, Google verification, all fields on every account, remote writes, AI extraction,
embeddings or OCR. No footnotes or multiple/nested tabs occurred in the five-Doc live sample.

## Fresh-install compatibility

Drive 0.8.0 changes export to `--output` and memory search to a list envelope. The version-aware
checker accepts exact reviewed pins and selects the corresponding contract, retaining 0.7.0's
`--out` and rich memory-search result. Ten synthetic guard tests pass; 0.7.0's 8 initial, 8 repeat
and 7 knowledge lifecycle groups still pass after these checker changes.

A fresh npm 0.8.0 install passed the same lifecycle groups but failed broader discovery checks:
Core 0.281.0 asks for Base 0.28.x, while Drive pins Base 0.31.0. npm installs two instances and
generated API commands lose their operation identity/effects. Do not interpret the lifecycle
pass as full standards acceptance. The producer correction is
[Core PR 1257](https://github.com/WireCatLabs/cli-core/pull/1257).

The aligned candidate (Core 0.283.0/Base 0.31.1) passes 41 CLI invocations and three MCP
validation/audit calls. Its installed Node and Bun smoke checks require generated read/write
operation identities and correct remote effects. Drive also preserves its existing one-time
camelCase config migration after the producer stopped converting old settings. Candidate
results must be distinguished from the eventual published patch's installation results.

## Live-checker preparation

Expand Down Expand Up @@ -288,8 +341,7 @@ and the [controlled live checklist](live-checklist.md) for the separately author

## Limits

This is a functional contract check, not a latency benchmark or Google API integration test.
It does not establish extraction fidelity, live metadata completeness, Google verification or
live user sign-in. Metadata-only backfill never refreshes old Google responses, and
catalog retirement differs from regular sync's indexed-content removal. No owner account data
is used or published.
The synthetic runner is a functional contract check, not a latency benchmark or Google API
integration test. It does not establish extraction fidelity, live metadata completeness, Google
verification or live user sign-in; the bounded real-data observations above are reported separately. Metadata-only backfill never refreshes old Google responses, and
catalog retirement differs from regular sync's indexed-content removal. Synthetic fixtures contain no owner data. The separately authorized live run publishes totals only.
7 changes: 4 additions & 3 deletions performance/documents/offline-catalog/knowledge-installed.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ import assert from "node:assert/strict"
import { existsSync } from "node:fs"
import { join } from "node:path"

export const verifyKnowledge = async ({ invoke, openStore, env, workspace, reference }) => {
export const verifyKnowledge = async ({ invoke, openStore, env, workspace, reference, release }) => {
const checks = []
const item = (words, options) => {
const answer = invoke(words, options)
Expand Down Expand Up @@ -50,8 +50,9 @@ export const verifyKnowledge = async ({ invoke, openStore, env, workspace, refer
checks.push("memory-evidence-and-review")
assert.equal(list(["memories", "list", "doc", "--scope", "personal"]).length, 0)
assert.ok(list(["memories", "list", "doc", "--scope", "work"]).some((entry) => entry.ref === memory.ref))
const found = item(["memories", "search", "doc", "lifecyclemarker", "--scope", "work"])
assert.ok(found.items.some((entry) => entry.ref === memory.ref))
const search = ["memories", "search", "doc", "lifecyclemarker", "--scope", "work"]
const found = release.memory_search === "items" ? list(search) : item(search).items
assert.ok(found.some((entry) => entry.ref === memory.ref))
checks.push("memory-scope-and-source-search")

item(["folders", "remove", "root"])
Expand Down
4 changes: 3 additions & 1 deletion performance/documents/offline-catalog/live-check.mjs
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import assert from "node:assert/strict"
import { mkdtempSync, readFileSync } from "node:fs"
import { join } from "node:path"
import { releaseOf } from "./live-input.mjs"

const tag = "wirecat-acceptance"
const note = "Synthetic acceptance note; preserve across rename and resync."
Expand All @@ -9,6 +10,7 @@ const annotated = (context) =>
context.tags.some((entry) => entry.tag === tag) && context.notes.items.some((entry) => entry.text === note)

export const checkFixture = ({ manifest, invoke, workspace, refresh = false, annotate = false, baseline }) => {
const release = releaseOf(manifest.drive_version)
const checks = []
const check = (name, fn) => {
fn()
Expand Down Expand Up @@ -74,7 +76,7 @@ export const checkFixture = ({ manifest, invoke, workspace, refresh = false, ann
check("old-marker-removed", () => assert.ok(search.hits.every((hit) => hit.ref !== document.reference)))
}
const out = join(exports, `${index}.md`)
item(["documents", "export", entry.id, "--out", out])
item(["documents", "export", entry.id, release.export_flag, out])
check("markdown-export", () => {
const markdown = readFileSync(out, "utf8")
for (const text of entry.markdown_markers ?? []) assert.ok(markdown.includes(text))
Expand Down
12 changes: 11 additions & 1 deletion performance/documents/offline-catalog/live-input.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,16 @@ import { mkdirSync, readFileSync, realpathSync, writeFileSync } from "node:fs"
import { join, resolve } from "node:path"

const marker = "drive-acceptance-workspace.json"
const releases = {
"0.7.0": { export_flag: "--out", memory_search: "item", core: "0.278.0", framework: "0.28.2" },
"0.8.1": { export_flag: "--output", memory_search: "items", core: "0.283.0", framework: "0.31.1" },
"0.8.0": { export_flag: "--output", memory_search: "items", core: "0.281.0", framework: "0.31.0" },
}
export const releaseOf = (version) => {
assert.equal(typeof version, "string")
assert.ok(Object.hasOwn(releases, version), "unreviewed Drive release")
return releases[version]
}
const object = (value, keys) => {
assert.ok(value && typeof value === "object" && !Array.isArray(value), "invalid manifest object")
assert.ok(
Expand All @@ -20,7 +30,7 @@ const strings = (value) => {
export const validateManifest = (value) => {
object(value, ["schema_version", "drive_version", "folder_id", "folders", "documents", "absent_ids"])
assert.equal(value.schema_version, 1)
assert.equal(value.drive_version, "0.7.0", "review a new package before changing the accepted version")
releaseOf(value.drive_version)
id(value.folder_id)
assert.ok(Array.isArray(value.folders) && value.folders.length > 0)
assert.ok(Array.isArray(value.documents) && value.documents.length > 0)
Expand Down
12 changes: 7 additions & 5 deletions performance/documents/offline-catalog/live-installed.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -6,13 +6,15 @@ import { tmpdir } from "node:os"
import { join, resolve } from "node:path"
import { fileURLToPath, pathToFileURL } from "node:url"
import { verifyKnowledge } from "./knowledge-installed.mjs"
import { initialize, workspaceEnvironment } from "./live-input.mjs"
import { initialize, releaseOf, workspaceEnvironment } from "./live-input.mjs"

const [input, ...rest] = process.argv.slice(2)
assert.ok(input && rest.length === 0, "usage: node live-installed.mjs <installed-drive-0.7.0-package>")
assert.ok(input && rest.length === 0, "usage: node live-installed.mjs <installed-reviewed-drive-package>")
const pkg = resolve(input)
const metadata = JSON.parse(readFileSync(join(pkg, "package.json"), "utf8"))
assert.equal(metadata.version, "0.7.0")
const release = releaseOf(metadata.version)
assert.equal(metadata.dependencies["@wirecat/cli-core"], release.core)
assert.equal(metadata.dependencies["@wirecat/cli-base"], release.framework)
const require = createRequire(join(pkg, "package.json"))
const { openStore } = await import(pathToFileURL(require.resolve("@wirecat/cli-core/store")))
const root = mkdtempSync(join(tmpdir(), "drive-live-installed-"))
Expand Down Expand Up @@ -104,7 +106,7 @@ writeFileSync(
manifest,
JSON.stringify({
schema_version: 1,
drive_version: "0.7.0",
drive_version: metadata.version,
folder_id: "root",
folders: [{ id: "root", parents: [] }],
documents: [
Expand Down Expand Up @@ -161,7 +163,7 @@ const invoke = (words, { offline = true, failure = false } = {}) => {
if (!failure) assert.equal(answer.meta.complete, true)
return answer
}
const knowledge = await verifyKnowledge({ invoke, openStore, env, workspace, reference: note.ref })
const knowledge = await verifyKnowledge({ invoke, openStore, env, workspace, reference: note.ref, release })
process.stdout.write(
`${JSON.stringify({ passed: true, data: "synthetic", live_google: false, drive: metadata.version, core: metadata.dependencies["@wirecat/cli-core"], first_checks: annotated.checks.length, repeat_checks: repeated.checks.length, knowledge_checks: knowledge })}\n`,
)
5 changes: 4 additions & 1 deletion performance/documents/offline-catalog/live.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ import { closeSync, openSync, readFileSync, writeFileSync } from "node:fs"
import { join, resolve } from "node:path"
import { parseArgs } from "node:util"
import { checkFixture } from "./live-check.mjs"
import { initialize, validateManifest, workspaceEnvironment } from "./live-input.mjs"
import { initialize, releaseOf, validateManifest, workspaceEnvironment } from "./live-input.mjs"

let report
let stage = "preflight"
Expand Down Expand Up @@ -64,6 +64,9 @@ try {
const pkg = JSON.parse(readFileSync(join(values.drive, "package.json"), "utf8"))
assert.equal(pkg.name, "@wirecat/drive-cli")
assert.equal(pkg.version, manifest.drive_version)
const release = releaseOf(pkg.version)
assert.equal(pkg.dependencies?.["@wirecat/cli-core"], release.core)
assert.equal(pkg.dependencies?.["@wirecat/cli-base"], release.framework)
const baseline = values.baseline ? JSON.parse(readFileSync(values.baseline, "utf8")) : undefined
if (baseline) {
assert.equal(baseline.passed, true)
Expand Down
31 changes: 23 additions & 8 deletions performance/documents/offline-catalog/live.test.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,11 @@ import { checkFixture } from "./live-check.mjs"
import { initialize, validateManifest, workspaceEnvironment } from "./live-input.mjs"

const runner = fileURLToPath(new URL("./live.mjs", import.meta.url))
const testPackage = {
name: "@wirecat/drive-cli",
version: "0.7.0",
dependencies: { "@wirecat/cli-core": "0.278.0", "@wirecat/cli-base": "0.28.2" },
}
const fixture = () => ({
schema_version: 1,
drive_version: "0.7.0",
Expand Down Expand Up @@ -57,7 +62,9 @@ const setup = (change = {}) => {
if (group === "documents" && action === "show")
return item({ metadata: { id: "doc" }, text: "acceptance-new", tabs: ["one", "two"] })
if (group === "documents" && action === "export") {
writeFileSync(words[words.indexOf("--out") + 1], "# Acceptance")
const offset = words.indexOf(change.version && change.version !== "0.7.0" ? "--output" : "--out")
assert.ok(offset >= 0)
writeFileSync(words[offset + 1], "# Acceptance")
return item({})
}
if (group === "documents" && action === "context")
Expand All @@ -69,7 +76,12 @@ const setup = (change = {}) => {
})
throw new Error("unexpected fixture command")
}
return { workspace, calls, invoke, manifest: validateManifest(fixture()) }
return {
workspace,
calls,
invoke,
manifest: validateManifest({ ...fixture(), drive_version: change.version ?? "0.7.0" }),
}
}

test("manifest rejects unknown fields, wrong release, duplicate ids and undeclared parents", () => {
Expand All @@ -78,7 +90,7 @@ test("manifest rejects unknown fields, wrong release, duplicate ids and undeclar
f.extra = true
},
(f) => {
f.drive_version = "0.8.0"
f.drive_version = "0.9.0"
},
(f) => {
f.documents[0].id = "root"
Expand Down Expand Up @@ -124,6 +136,12 @@ test("cached verification checks hierarchy, own-source search, tabs, exports and
assert.ok(args.calls.every((call) => call.offline))
assert.equal(result.references.length, 3)
})
test("reviewed 0.8 uses --output while 0.7 keeps --out", () => {
const args = setup({ version: "0.8.0" })
checkFixture(args)
assert.ok(args.calls.some((call) => call.words.includes("--output")))
assert.ok(args.calls.every((call) => !call.words.includes("--out")))
})
test("wrong root refuses before any Google access or local annotation", () => {
const args = setup({ extraRoot: true })
assert.throws(() => checkFixture({ ...args, refresh: true, annotate: true }))
Expand Down Expand Up @@ -158,10 +176,7 @@ test("entry point refuses unknown flags, missing profile and existing report bef
assert.ok(!existsSync(join(workspace, "missing-profile")))
initialize(join(workspace, "state"), "synthetic")
mkdirSync(join(workspace, "package"))
writeFileSync(
join(workspace, "package/package.json"),
JSON.stringify({ name: "@wirecat/drive-cli", version: "0.7.0" }),
)
writeFileSync(join(workspace, "package/package.json"), JSON.stringify(testPackage))
writeFileSync(join(workspace, "manifest.json"), JSON.stringify(fixture()))
const output = join(workspace, "report.json")
writeFileSync(output, "preserve")
Expand All @@ -187,7 +202,7 @@ test("failed CLI output never leaks through stdout, diagnostics or a private fai
initialize(join(root, "workspace"), "synthetic")
const pkg = join(root, "package")
mkdirSync(join(pkg, "dist/bin"), { recursive: true })
writeFileSync(join(pkg, "package.json"), JSON.stringify({ name: "@wirecat/drive-cli", version: "0.7.0" }))
writeFileSync(join(pkg, "package.json"), JSON.stringify(testPackage))
writeFileSync(
join(pkg, "dist/bin/drive.js"),
'process.stdout.write(JSON.stringify({error:{code:"synthetic_failure",message:"SYNTHETIC_TOKEN_CANARY"}})); process.stderr.write("SYNTHETIC_TOKEN_CANARY"); process.exitCode=1;',
Expand Down
Loading