KIWI CLI

Command line setup helper for KIWI.

Install

Because the repository is private, authenticate GitHub CLI before installing:

gh auth login

Linux

gh release download --repo OFFIS-RIT/kiwi_cli --pattern kiwi-linux-x86_64.tar.gz --output - \
  | sudo tar -xz -C /usr/local/bin kiwi
kiwi --help

This installs kiwi to /usr/local/bin.

macOS

gh release download --repo OFFIS-RIT/kiwi_cli --pattern kiwi-macos.tar.gz --output - \
  | sudo tar -xz -C /usr/local/bin kiwi
kiwi --help

This installs kiwi to /usr/local/bin.

Windows

Run this in PowerShell:

$InstallDir = "$env:LOCALAPPDATA\Kiwi\bin"
New-Item -ItemType Directory -Force -Path $InstallDir | Out-Null

$Zip = "$env:TEMP\kiwi-windows-x86_64.zip"
gh release download --repo OFFIS-RIT/kiwi_cli --pattern kiwi-windows-x86_64.zip --output $Zip

Expand-Archive -Force -Path $Zip -DestinationPath $InstallDir

$UserPath = [Environment]::GetEnvironmentVariable("Path", "User")
if (($UserPath -split ";") -notcontains $InstallDir) {
  [Environment]::SetEnvironmentVariable("Path", "$UserPath;$InstallDir", "User")
}

$env:Path = "$InstallDir;$env:Path"
kiwi --help

This installs kiwi.exe to %LOCALAPPDATA%\Kiwi\bin and adds that directory to your user Path.

Usage

kiwi setup

Run an evaluation against an existing KIWI graph:

kiwi eval --questions questions.csv

When started inside a KIWI install directory, kiwi eval discovers .env from the current directory or a parent directory. It derives the API endpoint from KIWI_BASE_URL, KIWI_API_URL, or the KIWI API_URL/APP_DOMAIN settings, and it uses KIWI_API_KEY or MASTER_USER_API_BYPASS when present. Missing values are prompted interactively.

The command lists visible graphs from GET /graphs, lets you select one, then asks every question from the CSV three times via POST /chat/:graph_id. The CSV may have a question/frage header or just one question per row. Results are written next to the input as <name>.kiwi-eval.csv unless --output is passed. KIWI citation fences are stripped from the answer text and appended as compact references such as (file.pdf S.13-15) in the same answer cell.

Useful non-interactive options:

kiwi eval \
  --endpoint https://kiwi.example.com/api \
  --api-key kiwi_... \
  --graph graph_id \
  --questions questions.csv \
  --iterations 3 \
  --output answers.csv

Shell completions

kiwi autocomplete prints a completion script for your shell. The shell is auto-detected from $SHELL, or you can name it explicitly (bash, zsh, fish, powershell, elvish):

kiwi autocomplete            # detect $SHELL, print script + install hint
kiwi autocomplete zsh        # print the zsh script

Pass --install to write the script into the shell's standard per-user completions location (~/.local/share/bash-completion/completions/kiwi for bash, ~/.zfunc/_kiwi for zsh, ~/.config/fish/completions/kiwi.fish for fish), then restart your shell:

kiwi autocomplete fish --install

For PowerShell, add this to your $PROFILE:

kiwi autocomplete powershell | Out-String | Invoke-Expression

Benchmarks

Built-in benchmark suites can bootstrap their own cases and source documents as part of run. If --graph is omitted, the command creates a graph, uploads the prepared sources, waits for indexing, writes manifest.json, and then starts the benchmark against that graph:

kiwi benchmark run \
  --suite crag \
  --output .kiwi-bench/crag \
  --endpoint https://kiwi.example.com/api \
  --api-key kiwi_...

Supported suites are printed by kiwi benchmark list-suites. Passing --source uses local files instead of downloading suite sources. With --graph --no-upload, run only prepares cases for the existing graph and does not download suite source bundles. custom still requires --cases. Pass --manifest to rerun an already prepared benchmark without setup.

Benchmark questions run with up to 50 concurrent workers by default. Pass --concurrency N to tune the limit for the KIWI and judge endpoints, or use --concurrency 1 for sequential execution.

Graph indexing is awaited without a time limit by default. Pass --wait-timeout-seconds N to bound the wait, or 0 to skip waiting entirely.

Suite downloads and the generated source archives are cached per user in ~/.cache/kiwi-cli/suite-data/<suite> ($XDG_CACHE_HOME and $KIWI_CLI_CACHE_DIR are honored), so reruns and fresh output directories do not re-download or rebuild the dataset. Upload progress is tracked in <output>/upload-state.json: when a run aborts mid-upload, rerunning it reuses the already created graph and skips the batches that were uploaded before. If the recorded graph was deleted in the meantime, the rerun creates a fresh graph and uploads everything again. Without a state file (deleted, or the graph was created by an older CLI), an existing graph with the intended name is reused instead of creating a duplicate; already ingested documents are skipped server-side. Delete the state file to force a fresh start.

For automatic RAG-QA Arena setup, select the technology domain:

kiwi benchmark run \
  --suite rag-qa-arena \
  --domain technology \
  --output .kiwi-bench/rag-qa-arena-technology \
  --endpoint https://kiwi.example.com/api \
  --api-key kiwi_...

This downloads the public AWS Labs technology annotations, fetches the LoTTE archive, and builds KIWI source bundles from the referenced RobustQA gold_doc_ids. Other RAG-QA Arena domains still require local RobustQA source files via --source or an existing corpus-backed graph via --graph --no-upload.

Suite provenance

Benchmark setup prints the dataset source and research citations used by the selected suite. kiwi benchmark list-suites prints the same information for all suites.

Suite	Dataset source	Research paper
frames	google/frames-benchmark and MediaWiki source pages	Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generation
rgb	chen700564/RGB	Benchmarking Large Language Models in Retrieval-Augmented Generation
alce	princeton-nlp/ALCE-data	Enabling Large Language Models to Generate Text with Citations
crag	facebookresearch/CRAG Task 1 and 2 dev v5	CRAG - Comprehensive RAG Benchmark
rag-qa-arena	awslabs/rag-qa-arena annotations with LoTTE and RobustQA corpora	RAG-QA Arena, ColBERTv2, and RobustQA
contextual-bench	Salesforce/ContextualBench PopQA long-tail split	SFR-RAG and PopQA

The custom suite uses user-provided cases and sources and has no associated paper.

Per-suite evaluation

kiwi benchmark run scores answers with the evaluation flow implemented for each suite and writes the canonical metric into summary.json/summary.csv (usable with --fail-under):

Suite	Canonical metric(s)	Method
rag-qa-arena	judge_win_rate, judge_win_tie_rate	Pairwise LLM judge vs. the human reference (position-bias controlled)
frames	frames_accuracy	Binary LLM accuracy judge (prediction conveys the ground truth)
crag	crag_truthfulness (+ crag_accuracy/crag_hallucination/crag_missing)	Two-step auto-eval: rule-based missing detection, exact-match shortcut, LLM accurate/incorrect; truthfulness = accuracy − hallucination
rgb	rgb_accuracy	Gold answer string present in the prediction
contextual-bench	contextual_exact_match, contextual_token_f1, contextual_accuracy	Standard QA exact-match / token-F1 / containment
alce	alce_correctness_recall	Deferred — only deterministic correctness recall; citation quality (NLI) and fluency (MAUVE) are not yet evaluated

kiwi benchmark run writes setup_stats.csv beside the manifest when it prepares sources or a graph, and writes bench_stats.csv beside attempts.jsonl. These CSVs capture wall-clock timings, API request counts/retries, upload batches, judge requests, and KIWI response metadata such as input/output/total tokens when the API returns them.

The FRAMES/CRAG judges reuse the configured AI_TEXT_* judge model; that model is recorded in the run log for reproducibility (neither benchmark fixes an official judge model).

CRAG corpus note: the official CRAG Task 1 provides each question its own 5 retrieved pages. KIWI builds a single shared graph from every question's pages, so retrieval runs against the full pooled corpus. The truthfulness metric is computed exactly as published, but because the retrieval setup differs, absolute CRAG numbers are not directly comparable to the per-question leaderboard.