diff --git a/.agents/skills/sys1rust-develop/SKILL.md b/.agents/skills/sys1rust-develop/SKILL.md new file mode 100644 index 0000000..3253d8a --- /dev/null +++ b/.agents/skills/sys1rust-develop/SKILL.md @@ -0,0 +1,81 @@ +--- +name: sys1rust-develop +description: Build sys1rust from source and change its Rust code (laya-core, laya-mlx with its Metal kernels, the sys1rust server, sys1-bench), run its unit, GPU and live test suites, check that answers stay the same, measure speed, and follow the repository's rules for commits, docs and releases. Use when modifying, testing, benchmarking, reviewing or releasing sys1rust itself, not when installing or calling it. +--- + +# Develop sys1rust + +Read the code you change and the write-up behind it first. `results/SPEED.md` explains every engine setting, `results/SERVER.md` the server's design, and `bench/PLAN.md` the benchmark rules. + +## Build + +You need an Apple silicon Mac, Rust 1.89 or newer, CMake, the Xcode command line tools and Python 3.10 or newer. Python only fetches MLX, and `sys1rust` never runs it. From the repository root: + +```sh +# A prebuilt MLX 0.32.2 (the Python wheel ships libmlx and its CMake files). +python3 -m venv .mlx && .mlx/bin/pip install mlx==0.32.2 +export MLX_SYS_PREBUILT_DIR="$(.mlx/bin/python -c 'import mlx.core, os; print(os.path.dirname(mlx.core.__file__))')" + +# The binaries go to runtime/target/release/. +cargo build --release --manifest-path runtime/Cargo.toml +``` + +Every later `cargo` command needs `MLX_SYS_PREBUILT_DIR` set the same way. The build gives `sys1rust`, `sys1-bench` and `sys1-probe`. The binary loads MLX from the venv by its absolute path, so keep `.mlx/` where it is. Release bundles build with the Rust version in `packaging/rust-toolchain-version`. For benchmark work, `source bench/env.sh` instead, as the README's "Build and run inside the benchmark setup" section says. It keeps the caches and the Cargo output under `bench/`. + +## Where the code lives + +- `runtime/crates/laya-core` is request parsing, question checks, tokenization, sequence layout, answer decoding and the action head. It has no GPU code. It reproduces the Python `laya` package step for step. +- `runtime/crates/laya-mlx` is the forward pass on MLX through mlx-rs. Its engine settings, the `--tuning` list, are the `Knobs` in `src/lib.rs`. The Metal kernels are in `src/kernels/`, `src/split_rope.rs` (`fuserope`, `band`) and `src/nax_gemm.rs` (`nax`). +- `runtime/crates/sys1rust` is the command. `cli.rs` holds the subcommands, `config.rs` the flags and model resolution, `http.rs` the router, `validate.rs` upstream's request checks, `worker.rs` the inference thread, `agent.rs` the warm-up, `models.rs` the pinned manifest and `download.rs` the downloader. +- `runtime/crates/sys1-bench` is the adapter for `bench/harness` and `sys1-probe`, which times 2 settings against each other in one process and with `--check` compares their answers. +- `runtime/vendor/mlx-sys` is mlx-sys 0.6.0 with a build that links a prebuilt MLX. +- `bench/models.lock.json` pins the model revisions, `bench/workloads/` holds the requests and `bench/reference/` the upstream fp32 answers. + +## Test + +Run the suites in this order, and stop at the first failure. + +1. CPU tests, with no GPU and no model: + ```sh + cargo test --release --manifest-path runtime/Cargo.toml -p laya-core -p sys1rust + ``` +2. The whole workspace. This adds laya-mlx's kernel tests, which run on the GPU but need no model: + ```sh + cargo test --release --manifest-path runtime/Cargo.toml --workspace + ``` +3. The ignored suites, which load models on the GPU. Run each alone: + ```sh + ( + set -e + for t in settings loading reference; do + SYS1_TEST_ALL_CHECKPOINTS=1 cargo test --release --manifest-path runtime/Cargo.toml \ + -p laya-mlx --test $t -- --ignored --test-threads 1 + done + cargo test --release --manifest-path runtime/Cargo.toml -p sys1rust --test live -- --ignored + cargo test --release --manifest-path runtime/Cargo.toml -p sys1rust --test live_pull -- --ignored + ) + ``` + - `settings` takes about 14 minutes with all 3 models. Its exact cases, including `fuserope` alone, `band`, `nax` and the loading settings, require identical answer JSON and raw logits and pooled outputs under `f32::to_bits`. The `dense_upto`, `headprune`, `unpad`, their combinations and boolean-mask cases require the same chosen answers and allow probability, answer-confidence and action-probability differences up to 0.001. `reference` checks the answers against `bench/reference`. `live` starts the real binary and requires each reply to equal an in-process prediction byte for byte. `live_pull` downloads typed-decisions (846 MB) into an empty cache, so it needs the network. + - The laya-mlx suites load the snapshot that a model's `refs/main` names, or else its only snapshot, and `sys1rust pull` writes no `refs/main`. Point `HF_HUB_CACHE` at a cache that holds only the pinned snapshots, filled with `sys1rust pull ` for each of the 3 models. Step 3 of `packaging/RELEASE.md` sets one up. + - Without `SYS1_TEST_ALL_CHECKPOINTS=1`, a model missing from the cache is skipped with a note. + - The `settings` tests need MLX's NAX gemms, so they pass only on an M5-class GPU (generation 17 or later) with macOS 26.2 or later and the macOS 26 MLX build. + - `laya-core`'s `fixtures_parity` and laya-mlx's `parity` and `bench` tests need fixtures that aren't in the repository. Skip them. + +## Rules + +- **One GPU job at a time.** A running `sys1rust serve`, the laya-mlx tests, the live tests, `sys1-probe` and the benchmarks all use the GPU. Run each with nothing else on the GPU, and stop any server you started before the next job. 2 jobs at once slow each other and make every timing worthless. +- **Format only your own lines.** The workspace isn't rustfmt-clean, so `cargo fmt` rewrites files you didn't touch. Never run it on the whole workspace. Check a new file with `rustfmt --edition 2021 --check `, which also checks the modules a `lib.rs` or `main.rs` declares. In an existing file, keep your lines in rustfmt's style and leave the rest alone. +- **Answers must not change.** A new speed change must give bit-identical results to the current default. Require identical answer JSON and raw outputs under `f32::to_bits`, using an exact case in `tests/settings.rs`. Add such a test for a new setting, and make it fail if a kernel fell back to MLX's ops without saying so. Some existing work-reduction and boolean-mask cases use the 0.001 tolerance described above. Their passing tests do not prove bit identity. A change that is meant to change numbers must say how it was checked against `bench/reference`. The correctness workload agrees with upstream on 1,498 of 1,500 answers today, and 99% is the floor. `sys1-probe --ab SPEC_A SPEC_B --check` compares 2 settings' answers. +- **Kernels fall back.** A custom kernel that can't build or run on a Mac must be found at model load, with a line on stderr, and the MLX path it replaces stays behind a setting. It never fails in the middle of a request. +- **The server matches `laya serve`.** Status codes, `detail` strings, header names, response key order and request limits match upstream. Document an intentional difference in the code and in the README's "Differences from `laya serve`". +- **Speed claims need measurements.** Name the machine, the macOS version, the MLX build, the model, the request shapes, the number of runs and processes, and report medians. Compare against a baseline measured in the same session. `sys1-probe --ab` alternates the 2 settings in one process, because separate runs on one Mac vary by about 5%. Write the result up in `results/`, as `results/SPEED.md` does. 1 run proves nothing. +- **No Python at run time.** Code under `runtime/crates` must not call Python. Build time may use the MLX wheel. +- **Bound every cache keyed by request shape**, or document why its keys are few. A long-running server sees many lengths. +- **No machine paths.** Scripts and docs must not hard-code a home directory or another machine-specific absolute path. Resolve paths from the script or the repository root. +- **Releases follow `packaging/RELEASE.md`.** Pushing a `v*` tag publishes a release, so tag only with the maintainer's go-ahead. + +## Writing + +- Each Rust file starts with a `//!` header that says what the file does. Comments say why, and match the surrounding code's density and naming. +- In docs, use sentence case headings, active voice and digits for numbers. Write MB as 10^6 bytes. Use no em dashes, and no colon that joins 2 clauses. Every number must trace to the code or to a write-up in `results/`. +- A commit subject is an imperative sentence in sentence case with no type prefix and no final period, such as `Stop on a leftover draft release instead of deleting it`. The body says in prose what changed, why, and how it was checked. Make one logical change per commit and stage files by path. diff --git a/.agents/skills/sys1rust-setup/SKILL.md b/.agents/skills/sys1rust-setup/SKILL.md new file mode 100644 index 0000000..c47c57e --- /dev/null +++ b/.agents/skills/sys1rust-setup/SKILL.md @@ -0,0 +1,220 @@ +--- +name: sys1rust-setup +description: Install, update, start, check, stop or uninstall sys1rust, the local server for Laya's System 1 models, on the user's Apple silicon Mac from its prebuilt GitHub release. Use when the user asks to set up, install, run, update, repair or remove sys1rust, or gives you https://github.com/krishhgg/sys1rust and asks you to set it up. +--- + +# Set up sys1rust + +This skill installs the prebuilt `sys1rust` with `install.sh`, downloads a model, starts the server and checks that it answers. Follow the steps in order, and run each command from the root of the sys1rust clone. Without a clone, run each `./install.sh ARGS` below as `curl -fsSL https://raw.githubusercontent.com/krishhgg/sys1rust/main/install.sh | sh -s -- ARGS`. + +Put any environment assignments for the installer after `|`, immediately before `sh`, or before `arch` under Rosetta. This passes the settings to the interpreter that runs the installer. Step 5 gives the exact service commands for a chosen port. + +Write only to the clone, `~/.local/share/sys1rust`, `~/.local/bin`, `~/Library/LaunchAgents`, `~/Library/Logs` and the Hugging Face cache. Ask the user before you write anywhere else, edit a shell startup file or delete a model. Never use `sudo`. If your sandbox blocks the network or the GPU, ask the user to let these commands run outside it. + +## 1. Check the Mac + +```sh +sysctl -n hw.optional.arm64 # 1 on Apple silicon, an error on an Intel Mac +sysctl -n sysctl.proc_translated # 0 in a native shell, 1 under Rosetta +sw_vers -productVersion # the macOS version +df -H ~ | awk 'NR==2 {print $4}' # free space on the home volume +``` + +- If `hw.optional.arm64` prints 0 or an error, the Mac has an Intel chip. Stop and tell the user that sys1rust runs only on Apple silicon. +- If `sysctl.proc_translated` prints 1, your shell runs under Rosetta, and the installer stops in it. Run the installer in a native shell instead. Write `arch -arm64 /bin/sh ./install.sh` wherever this skill says `./install.sh`, and pipe into `arch -arm64 /bin/sh -s -- ARGS` in place of `sh -s -- ARGS` without a clone. Variables set in front, such as `LAYA_PORT=8001 arch -arm64 /bin/sh ./install.sh`, still reach the installer. +- If macOS is older than 14.0, stop and tell the user that sys1rust needs macOS 14.0 or later. +- On macOS 26.2 or later the installer picks the `macos26` bundle. On 14.0 to 26.1 it picks the `macos14` bundle, which is about 3x slower on the M5. If the Mac runs 26.0 or 26.1, tell the user that updating macOS to 26.2 and rerunning the installer gets the faster bundle. +- The install takes about 215 MB, and each model 846 MB (multilingual 678 MB). With less than 1.1 GB free, tell the user and stop. + +## 2. Install + +```sh +./install.sh +``` + +The installer downloads the latest release for this Mac, checks it against the release's `SHA256SUMS`, unpacks it into `~/.local/share/sys1rust/-./`, points `~/.local/share/sys1rust/current` at it and links `~/.local/bin/sys1rust`. It records its paths in `~/.local/share/sys1rust/.sys1rust-install` and refuses a nonempty prefix without that registry. It locks the prefix, bin directory and shared LaunchAgent path so installs with different prefixes cannot overwrite shared paths. It downloads no model. `--version vX.Y.Z` installs a given release instead, and `./install.sh --help` lists every option. Check the result: + +```sh +~/.local/bin/sys1rust --version +``` + +It prints `sys1rust (MLX 0.32.2, macos26 build)`, or `macos14 build` for the other bundle. From here on, run `~/.local/bin/sys1rust` by its full path, as the commands below do, so that no other `sys1rust` on `PATH` runs in its place. + +## 3. Check PATH for the user + +The user will type `sys1rust` in their own terminal, so check what that name finds: + +```sh +command -v sys1rust +``` + +- If it prints the full path of `~/.local/bin/sys1rust`, such as `/Users//.local/bin/sys1rust`, go to step 4. +- If it prints another path, an older sys1rust, from another installer or a source build, comes first on `PATH`. The installer said so too, with ` comes before on PATH`. Tell the user that `sys1rust` in a terminal runs that older binary. Ask whether to remove it, or to put `~/.local/bin` ahead of it in `~/.zshrc`. Change nothing until they answer. +- If it prints nothing, `~/.local/bin` is not on your shell's `PATH`, and the installer printed the line that adds it. Check whether `~/.zshrc` has such a line already with `grep -n '\.local/bin' ~/.zshrc`. If it doesn't, ask the user whether to add it. If they agree, append the exact line the installer printed to `~/.zshrc`. If `echo $SHELL` is not `/bin/zsh`, ask the user which file to use. + +## 4. Download the model + +```sh +~/.local/bin/sys1rust pull typed-decisions +``` + +This downloads typed-decisions, the default model, at the revision the release pins (846 MB) into the Hugging Face cache. The cache is `~/.cache/huggingface/hub` unless `HF_HUB_CACHE`, `HF_HOME` or `XDG_CACHE_HOME` is set. `pull` prints its progress on stderr and the snapshot directory on stdout. If it stops partway, run it again, and it resumes where it stopped. Download another model only if the user asks for it, with `~/.local/bin/sys1rust pull multilingual` (678 MB) or `~/.local/bin/sys1rust pull english` (846 MB). Then check: + +```sh +~/.local/bin/sys1rust models +``` + +The typed-decisions row says `downloaded, 846 MB`, and the last line names the cache. + +If `HF_HUB_CACHE`, `HF_HOME` or `XDG_CACHE_HOME` is set, `./install.sh --service` in step 5 writes it into the service, so the service reads the cache that `pull` filled. Run both commands in the same shell. + +## 5. Start the server + +The server and the service listen on port 8000 unless you pick another. Check that the port is free: + +```sh +PORT=${PORT:-8000} +lsof -nP -iTCP:$PORT -sTCP:LISTEN +``` + +- No output means the port is free. +- A `sys1rust` row means a sys1rust server already runs. If the user wants the service, stop that server first, because the installer won't start the service while another server answers on the port. Otherwise go to step 6. +- Any other program means you must ask the user whether to stop it. If they keep it, pick another port, such as 8001, and check it the same way. + +Set `PORT` to the chosen port and keep it for every later command. Each block defaults it to 8000 only when it is unset. If your tool starts a fresh shell for each command, include the chosen value in each block, such as `PORT=8001`. + +Ask the user how they want to run it, unless their request already says: + +- **At every login.** A LaunchAgent runs `sys1rust serve` now and at each login, and restarts it if it crashes. The model stays in memory while the user is logged in. Recommend this when apps or agents will call sys1rust often. +- **Only when they start it.** You start it now in the background, and later the user runs `sys1rust serve` when they need it. + +If you can't ask, choose "only when they start it". + +For the service from a clone, run: + +```sh +PORT=${PORT:-8000} +LAYA_PORT="$PORT" ./install.sh --service +``` + +Without a clone, run: + +```sh +PORT=${PORT:-8000} +curl -fsSL https://raw.githubusercontent.com/krishhgg/sys1rust/main/install.sh | + LAYA_PORT="$PORT" sh -s -- --service +``` + +Under Rosetta, run the installer with a native interpreter and keep the assignment before `arch`: + +```sh +PORT=${PORT:-8000} +curl -fsSL https://raw.githubusercontent.com/krishhgg/sys1rust/main/install.sh | + LAYA_PORT="$PORT" arch -arm64 /bin/sh -s -- --service +``` + +The installer writes `~/Library/LaunchAgents/io.github.krishhgg.sys1rust.plist` with `LAYA_PORT` in it, starts the service and waits up to 60 s until `GET /health` answers on that port. The server logs to `~/Library/Logs/sys1rust.log`. If the installer exits with an error, read the last lines of that log. + +To start it yourself, run: + +```sh +PORT=${PORT:-8000} +mkdir -p ~/Library/Logs +env -u LAYA_API_KEY -u SYS1_REVISION nohup ~/.local/bin/sys1rust serve \ + --host 127.0.0.1 --model typed-decisions --port "$PORT" >> ~/Library/Logs/sys1rust-serve.log 2>&1 & +pid=$! +echo "$pid" +ready=0 +for i in $(seq 60); do + kill -0 "$pid" 2>/dev/null || break + if curl -fs 127.0.0.1:$PORT/health; then ready=1; break; fi + sleep 1 +done +if [ "$ready" = 0 ]; then + kill "$pid" 2>/dev/null || true + wait "$pid" 2>/dev/null || true + tail -n 30 ~/Library/Logs/sys1rust-serve.log + false +fi +``` + +The command selects typed-decisions on `127.0.0.1` without authentication, matching step 6. `env -u` removes an inherited API key and revision only for the child process. The shell keeps its settings. If the user requested another model or authentication, use their choices and adapt step 6 to that model and its required authorization. + +It prints the process id, then the `/health` reply once the server answers. Keep the process id, because the user needs it to stop the server. A start with the model already downloaded takes a few seconds at most. The first start of a new install takes about 1.7 s longer while macOS compiles the GPU kernels. When the server is ready, its log ends with `sys1rust: listening on http://127.0.0.1: (auth off)`. If no `/health` reply follows the process id, read the log with `tail -n 30 ~/Library/Logs/sys1rust-serve.log` and see "When something fails" below. + +## 6. Check that it answers + +```sh +PORT=${PORT:-8000} +curl -s 127.0.0.1:$PORT/health +echo +curl -s 127.0.0.1:$PORT/v1/systemone -H 'content-type: application/json' -d '{ + "state": "Customer: I was charged twice for my subscription this month and I need it fixed today.", + "questions": { + "team": {"type": "choice", "instructions": "Which team should handle this?", + "criteria": {"billing": "payments, charges and refunds", "tech": "bugs and outages", "sales": "new purchases"}}, + "urgency": {"type": "score", "instructions": "How urgent is this?", + "criteria": ["none", "low", "high", "now"]}, + "money": {"type": "noul", "instructions": "Is money involved?"} + } +}' +``` + +The first reply has `"status":"ok"` and `"loaded":["typed-decisions"]`. The second sends the README's example request, and its `answers` holds `team`, `urgency` and `money`. With typed-decisions, `team.choice` is `billing`, `urgency.score` is about 2.5 and `money.noul` is about 0.73. A reply with `detail` instead of `answers` is an error. Read the server log. + +## 7. Tell the user + +Report in a few lines: + +- the version and bundle that `~/.local/bin/sys1rust --version` prints; +- where it lives, `~/.local/bin/sys1rust` linked into `~/.local/share/sys1rust/current`, and the model cache that `~/.local/bin/sys1rust models` names on its last line; +- what runs, either the LaunchAgent `io.github.krishhgg.sys1rust` with its log `~/Library/Logs/sys1rust.log`, or the process id you started with its log `~/Library/Logs/sys1rust-serve.log`; +- the address `http://127.0.0.1:` with the port from step 5, and the README's "Try it" section for calling it, which uses port 8000; +- how to stop and start it, from the table below; +- what step 3 found, so the user knows whether `sys1rust` in a new terminal runs this install, or still needs the `PATH` line or the older binary removed. + +| it runs as | stop it with | start it again with | +| --- | --- | --- | +| the LaunchAgent | `launchctl bootout gui/$(id -u)/io.github.krishhgg.sys1rust` | `launchctl bootstrap gui/$(id -u) ~/Library/LaunchAgents/io.github.krishhgg.sys1rust.plist`, or the next login | +| a process you started | `kill ` | Repeat the start block in step 5 with the chosen port. | +| `sys1rust serve` in a terminal | Ctrl-C | the same command | + +`launchctl bootout` stops the service until the next login. `./install.sh --uninstall` removes it. + +## Update + +Refresh the clone first. Run the clone block in `AGENTS.md` again if you made the scratch clone with it, or `git pull --ff-only` in a clone of the user's own. Then run: + +```sh +SERVICE_PORT=$(plutil -extract EnvironmentVariables.LAYA_PORT raw ~/Library/LaunchAgents/io.github.krishhgg.sys1rust.plist 2>/dev/null || echo 8000) +LAYA_PORT="$SERVICE_PORT" ./install.sh +``` + +The first line reads the service's port into `SERVICE_PORT`, with 8000 as the fallback, and leaves the chosen `PORT` unchanged. The installer itself also reads an existing service's port and cache from that plist before checking a restarted service. It installs the latest release, moves `current` to it, keeps the version before it as `previous` and deletes older ones. If the service runs, the installer restarts it on the new version and waits for `/health` on that port. + +For the LaunchAgent, set `PORT=$SERVICE_PORT` before repeating step 6. For a server you started yourself, keep its chosen `PORT`, stop it with `kill ` and repeat the start block in step 5 with that port, then step 6. If the new release pins a new model revision, run `~/.local/bin/sys1rust pull` to download it ahead of time, or the next `serve` downloads it. + +Rerun an interrupted update to finish any pending service restart. The installer accepts `/health` only when the LaunchAgent's PID owns the listening port, so a foreground server cannot hide a failed service restart. + +## Uninstall + +1. Stop any server you started. `pgrep -fl 'sys1rust serve'` lists them, and `kill ` stops one. +2. Run `./install.sh --uninstall`. It removes only the version directories, links, service and log that its registry owns. It refuses a prefix without its registry and leaves unrelated files, including replaced links, alone. If the LaunchAgent runs a sys1rust other than `~/.local/bin/sys1rust`, it leaves the LaunchAgent and its log and says so. It keeps the downloaded models, and prints the cache path, their size and the command that deletes them. Run that command only if the user asks for the models to go too. +3. If you added the `PATH` line, ask the user whether to remove it from `~/.zshrc`. Delete `~/Library/Logs/sys1rust-serve.log` if you created it. + +## When something fails + +| what you see | what to do | +| --- | --- | +| `sys1rust: error: bind 127.0.0.1:: Address already in use (os error 48)`, or the installer says another server answers on 127.0.0.1: | Another program holds the port. Follow the port check in step 5. | +| The installer finds no release | No release is published yet. Check `https://github.com/krishhgg/sys1rust/releases` for a prerelease. A tag alone does not prove that its release bundles are ready. If a `vX.Y.Z-rcN` tag exists and the user accepts a prerelease, run `./install.sh --version vX.Y.Z-rcN`. Otherwise build from source as the README's "Build from source" section says, or wait for a release. | +| The installer says the shell runs under Rosetta | Run it in a native shell, as step 1 says. | +| The installer reports an old macOS | sys1rust needs macOS 14.0 or later. Tell the user and stop. | +| A download stopped partway | Run the same command again. `sys1rust pull` and `sys1rust serve` resume a partial model download, and the installer is safe to rerun. | +| The installer reports another installer is running | Wait for that operation to finish, then rerun. For a stale lock, check that no installer runs and use the recovery command in the error. Never remove a live lock. | +| The installer reports no installer registry | Check that you used the intended prefix. Do not delete foreign files or make a registry by hand. Use an empty prefix for a new install. | +| The Mac is offline | Add `--offline` to the start command in step 5 to load only what the cache holds and never download. `HF_HUB_OFFLINE=1` also turns this on. A missing model is then an error that names the `sys1rust pull ` command to run. Run it while online. The installer needs the network, unless `--from DIR` points it at release files downloaded earlier (see `./install.sh --help`). | +| `laya-mlx: nax is off for this load, MLX's gemms run instead: ...` on stderr | This is expected with the `macos14` bundle and on GPUs older than the M5's, and it isn't an error. sys1rust runs MLX's own matmuls instead and serves normally. Other `laya-mlx: ... is off for this load` lines are fallbacks of the same kind. | +| `curl` can't connect | The server isn't up, has exited or listens on another port. Read the end of its log, `~/Library/Logs/sys1rust.log` for the service or `~/Library/Logs/sys1rust-serve.log` for one you started. | +| `/health` answers 503 with `"worker":"not running"` | The inference thread stopped. Restart the server and read its log. | +| `/v1/systemone` answers 401 | The server has an API key (`--api-key` or `LAYA_API_KEY`). Send `Authorization: Bearer `. | diff --git a/.agents/skills/sys1rust-use/SKILL.md b/.agents/skills/sys1rust-use/SKILL.md new file mode 100644 index 0000000..bba1574 --- /dev/null +++ b/.agents/skills/sys1rust-use/SKILL.md @@ -0,0 +1,214 @@ +--- +name: sys1rust-use +description: Call a running sys1rust server from code, a script or another agent through its /v1/systemone API, read its choice, score and noul answers and their confidences, handle its errors, and choose the Laya model, port, API key or offline mode it runs with. Use when writing a client for sys1rust or laya serve, sending System 1 questions about a text, or changing how sys1rust serves. +--- + +# Use sys1rust + +sys1rust answers questions about a text, the state, with one of Laya's System 1 models on the Mac's GPU. It speaks the `/v1/systemone` API of upstream `laya serve`, so a client written for one works with the other. Check that a server runs before you call it: + +```sh +curl -s 127.0.0.1:8000/health +``` + +A running server answers 200 with `"status":"ok"` and names its model in `loaded`. If nothing answers, follow the setup skill, [`.agents/skills/sys1rust-setup/SKILL.md`](../sys1rust-setup/SKILL.md). + +## Endpoints + +- `GET /health` answers 200 with `{"status":"ok","loaded":["typed-decisions"],"revisions":{...},"device":"gpu","engine":"mlx(gpu,f16)","tuning":"...","worker":"running"}`. It answers 503 with `"status":"error"` when the inference thread has stopped. +- `POST /v1/systemone` takes one state and its questions and returns one answer per question. + +## Request + +```json +{ + "state": "Customer: I was charged twice for my subscription this month and I need it fixed today.", + "questions": { + "team": {"type": "choice", "instructions": "Which team should handle this?", + "criteria": {"billing": "payments, charges and refunds", "tech": "bugs and outages", "sales": "new purchases"}}, + "urgency": {"type": "score", "instructions": "How urgent is this?", + "criteria": ["none", "low", "high", "now"]}, + "money": {"type": "noul", "instructions": "Is money involved?"} + } +} +``` + +- `state` is required. It is the text to judge, as a string or any other JSON value except `null`, up to 50,000 characters. A missing or `null` state gets 400 `'state' is required`. A value that isn't a string counts as the length of Python's `str()` of it. +- `questions` is required. It maps each question id to a question, up to 64 of them. Every question needs `type` and `instructions`: + - `choice` takes `criteria` as an object of label to description, or as a list of labels, up to 100 options. + - `score` takes `criteria` as a list of level descriptions, level 0 first, up to 32 levels. + - `noul` is a yes or no question. Its `criteria` is optional, an object with a `"true"` description, a `"false"` one or both. + - All the questions together take at most 512 options. +- `model` is optional. Leave it out to use the served checkpoint. The names `typed-decisions`, `multilingual` and `english`, and the repo ids for typed-decisions and multilingual, get 400 when they name a different checkpoint from the one served. The server ignores other values, `null` included. + + In this request field, `convaiinnovations/laya` means "use the served checkpoint", so it is accepted even on a typed-decisions server. Use `english` to name the English checkpoint explicitly. On the command line, `--model convaiinnovations/laya` selects English. +- `max_len` and `head_max_len` get 422 when they hold any value but `null`, because sys1rust doesn't support them. Absent or `null`, they are ignored. +- The body must be standard JSON in UTF-8, at most 2,097,152 bytes. + +Put every question about one state into one request. They run through the model together, one row per question, which costs less than separate requests. + +## Response + +The request above returns: + +```json +{ + "model": "laya-rl-agent", + "answers": { + "team": {"type": "choice", "choice": "billing", "probabilities": {"billing": 0.7941, "tech": 0.1012, "sales": 0.1047}, "confidence": 0.4073, "answer_confidence": 0.7941, "action": {"act_probability": 1.0}}, + "urgency": {"type": "score", "score": 2.5407, "legend": {"0": "none", "1": "low", "2": "high", "3": "now"}, "probabilities": {"0": 0.0101, "1": 0.0452, "2": 0.3384, "3": 0.6062}, "confidence": 0.3821, "answer_confidence": 0.6062, "action": {"act_probability": 1.0}}, + "money": {"type": "noul", "noul": 0.7302, "confidence": 0.7302, "answer_confidence": 0.7302, "action": {"act_probability": 1.0}} + }, + "usage": {"input_tokens": 150, "output_tokens": 0}, + "routing": {"model": "typed-decisions", "repo": "convaiinnovations/laya-typed-decisions", "reason": "only checkpoint served"} +} +``` + +`answers` has one entry per question id, in request order. Each entry has `type`, the answer, `confidence`, `answer_confidence` and `action`. + +| type | answer fields | the decision | +| --- | --- | --- | +| `choice` | `choice`, the most likely label, and `probabilities` for every label | `choice` | +| `score` | `score`, the expected level, which can fall between levels; `probabilities` per level, keyed `"0"`, `"1"` and so on; `legend`, each level's description | `score` for a number, or the level with the highest probability for a level | +| `noul` | `noul`, the probability that the answer is yes | yes when `noul` is 0.5 or more | + +- `answer_confidence` is the probability of the most likely option. For `choice` that is the chosen label, for `score` the most likely level, which can differ from the expected `score`, and for `noul` the larger of yes and no. Use it to decide whether to trust an answer. +- For `choice` and `score`, `confidence` is 1 minus the normalized entropy of the probabilities. It is 0 for an even split and 1 when everything is on one option. For `noul`, it equals `answer_confidence`, which is 0.5 for an even split. Don't compare the 2 against the same threshold. +- `action.act_probability` is the model's probability, from a second output, for acting on the answer rather than escalating. +- The server rounds every probability and score to 4 decimals. `usage.input_tokens` counts the tokens of all the rows, and `routing` names the model that answered. + +The `X-Inference-Time-Ms` and `Server-Timing: inference;dur=` headers give the time spent tokenizing, running the model and decoding. + +## Errors + +Every error body is `{"detail": ""}`, and `detail` says what was wrong. + +| status | cause | what to do | +| --- | --- | --- | +| 400 | invalid JSON, a missing or `null` `state`, no `questions` object, or a recognized `model` name or repo id for another checkpoint, as described above | Fix the request. | +| 401 | the server has an API key, and the request sent no `Authorization: Bearer ` or a wrong one | Send the key. | +| 408 | the body took over 10 s to arrive | Send it again. | +| 413 | the body, the state, the questions or the options went over a limit | Split the request or shorten the state. | +| 422 | a malformed question, such as a `choice` without `criteria`, or a `max_len` or `head_max_len` that isn't `null` | Fix what `detail` names. | +| 503 with `Retry-After: 1` | the server already holds as many requests as `--max-concurrent` allows, 16 by default | Wait 1 s and retry. | +| 503 without `Retry-After` | the inference thread stopped | Restart the server. | +| 500 | inference failed | Read the server's log. | + +The GPU runs one request at a time. The server holds up to 16, one running and the rest waiting their turn (`--max-concurrent`), and answers any request past that with 503 at once. More parallel clients don't get more answers per second. + +## Examples + +With curl, add `-i` to see the timing headers: + +```sh +curl -s 127.0.0.1:8000/v1/systemone -H 'content-type: application/json' \ + -d '{"state": "My card was charged twice.", "questions": {"money": {"type": "noul", "instructions": "Is money involved?"}}}' +``` + +With Python's standard library: + +```python +import json +import time +import urllib.error +import urllib.request + +URL = "http://127.0.0.1:8000/v1/systemone" + + +def ask(state, questions, api_key=None, tries=5): + """Send one state and its questions, and return the answers object.""" + body = json.dumps({"state": state, "questions": questions}).encode() + headers = {"content-type": "application/json"} + if api_key: + headers["authorization"] = f"Bearer {api_key}" + for attempt in range(tries): + req = urllib.request.Request(URL, data=body, headers=headers) + try: + with urllib.request.urlopen(req, timeout=60) as resp: + return json.load(resp)["answers"] + except urllib.error.HTTPError as e: + retry = e.headers.get("Retry-After") + if e.code == 503 and retry and attempt + 1 < tries: + time.sleep(int(retry)) + continue + raise RuntimeError(f"{e.code}: {json.load(e)['detail']}") from e + + +answers = ask( + "Customer: I was charged twice for my subscription this month and I need it fixed today.", + { + "team": {"type": "choice", "instructions": "Which team should handle this?", + "criteria": {"billing": "payments, charges and refunds", + "tech": "bugs and outages", "sales": "new purchases"}}, + "money": {"type": "noul", "instructions": "Is money involved?"}, + }, +) +print(answers["team"]["choice"], answers["team"]["answer_confidence"]) +print("money involved:", answers["money"]["noul"] >= 0.5) +``` + +A program can also start its own server on a free port and read the address from the JSON line that `serve` prints on stdout once the model is loaded: + +```python +import json +import os +import subprocess + +env = os.environ.copy() +for name in ("LAYA_API_KEY", "SYS1_REVISION"): + env.pop(name, None) +proc = subprocess.Popen( + ["sys1rust", "serve", "--host", "127.0.0.1", "--model", "typed-decisions", "--port", "0"], + env=env, stdout=subprocess.PIPE, text=True, +) +try: + line = proc.stdout.readline() + if not line: + raise RuntimeError(f"sys1rust serve exited with code {proc.wait()}, see its stderr") + ready = json.loads(line) # {"event": "listening", "addr": "127.0.0.1:", ...} + URL = f"http://{ready['addr']}/v1/systemone" + # ... call ask() as above ... +finally: + proc.terminate() # SIGTERM lets requests in flight finish before it exits + try: + proc.wait(timeout=30) + except subprocess.TimeoutExpired: + proc.kill() + proc.wait() +``` + +This example selects typed-decisions on `127.0.0.1` without authentication and clears an inherited revision for the child process. The parent keeps its settings. The `finally` block stops the server even when the start or a request fails, so no server is left holding the GPU. If `serve` fails, it prints the error on stderr and exits without the ready line, so `readline()` returns an empty string. A server already running on the Mac shares the GPU with this one, so prefer one server per Mac. + +## Models + +One server runs one model, `typed-decisions` unless `--model` or `SYS1_MODEL` selects another. + +| `--model` | Hugging Face repo | encoder | download | +| --- | --- | --- | --- | +| `typed-decisions` (default) | `convaiinnovations/laya-typed-decisions` | ModernBERT-large | 846 MB | +| `multilingual` | `convaiinnovations/laya-multilingual` | mmBERT-base | 678 MB | +| `english` | `convaiinnovations/laya` | ModernBERT-large | 846 MB | + +To switch, stop the server and start it with the model you want: + +```sh +sys1rust serve --model multilingual +``` + +That start downloads the model if the cache lacks it, and `sys1rust pull multilingual` downloads it ahead of time. `sys1rust models` lists what the cache holds. `--model` also takes a repo id from the table or a local checkpoint directory. The LaunchAgent that `install.sh --service` installs runs the default model. To serve another one, stop the service with `launchctl bootout gui/$(id -u)/io.github.krishhgg.sys1rust` and run `sys1rust serve --model ` yourself. + +## Options + +Each flag falls back to an environment variable. `sys1rust serve --help` lists all of them. + +| flag | environment variable | default | effect | +| --- | --- | --- | --- | +| `--model` | `SYS1_MODEL` | `typed-decisions` | the model, from the table above | +| `--host` | `LAYA_HOST` | `127.0.0.1` | the bind address | +| `--port` | `LAYA_PORT` | `8000` | the port. `0` picks a free one and prints it in the `addr` of the stdout line | +| `--api-key` | `LAYA_API_KEY` | none | when set, `/v1/systemone` needs `Authorization: Bearer ` | +| `--max-concurrent` | `LAYA_MAX_CONCURRENT` | `16` | requests held at once, past which the server answers 503 | +| `--offline` | `HF_HUB_OFFLINE` | off | never download. A model that the cache lacks is then an error. `HF_HUB_OFFLINE` turns it on when set to `1`, `ON`, `YES` or `TRUE`, in any letter case | + +Keep the default host. sys1rust speaks plain HTTP with no TLS, so an API key alone doesn't make it safe to reach from other machines. The README's "Serving other machines" section says how to put a reverse proxy in front. diff --git a/.claude/skills/sys1rust-develop b/.claude/skills/sys1rust-develop new file mode 120000 index 0000000..2b4704d --- /dev/null +++ b/.claude/skills/sys1rust-develop @@ -0,0 +1 @@ +../../.agents/skills/sys1rust-develop \ No newline at end of file diff --git a/.claude/skills/sys1rust-setup b/.claude/skills/sys1rust-setup new file mode 120000 index 0000000..6018773 --- /dev/null +++ b/.claude/skills/sys1rust-setup @@ -0,0 +1 @@ +../../.agents/skills/sys1rust-setup \ No newline at end of file diff --git a/.claude/skills/sys1rust-use b/.claude/skills/sys1rust-use new file mode 120000 index 0000000..7122c18 --- /dev/null +++ b/.claude/skills/sys1rust-use @@ -0,0 +1 @@ +../../.agents/skills/sys1rust-use \ No newline at end of file diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..7298820 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,59 @@ +# AGENTS.md + +sys1rust runs Laya's System 1 decision models on an Apple silicon Mac's GPU, in Rust on MLX. Its server, `sys1rust serve`, speaks the `/v1/systemone` API of `laya serve` on `http://127.0.0.1:8000`. This file tells coding agents which instructions to follow. Read the skill for your task before you run anything. + +## Pick the skill for the task + +| the user wants to | follow | +| --- | --- | +| install, update, start, stop or uninstall sys1rust on their Mac | [`.agents/skills/sys1rust-setup/SKILL.md`](.agents/skills/sys1rust-setup/SKILL.md) | +| call a running sys1rust from a program, a script or an agent | [`.agents/skills/sys1rust-use/SKILL.md`](.agents/skills/sys1rust-use/SKILL.md) | +| change the code, run the tests, measure speed or make a release | [`.agents/skills/sys1rust-develop/SKILL.md`](.agents/skills/sys1rust-develop/SKILL.md) | + +A request like "set up sys1rust on my Mac" goes to the setup skill. If you are reading this on GitHub and have no clone, make a scratch one first, outside any project of the user's: + +```sh +dir=$(mktemp -d "${TMPDIR:-/tmp}/sys1rust-agent.XXXXXX") +if xcode-select -p >/dev/null 2>&1 && git --version >/dev/null 2>&1; then + git clone --depth 1 https://github.com/krishhgg/sys1rust "$dir" +else + curl -fsSL -o "$dir/source.tar.gz" https://codeload.github.com/krishhgg/sys1rust/tar.gz/refs/heads/main && + tar -xzf "$dir/source.tar.gz" --strip-components 1 -C "$dir" && rm "$dir/source.tar.gz" +fi && cd "$dir" +``` + +On a new Mac, `git` can be a stub that asks the user to install Apple's command line tools. The block runs `git` only after `xcode-select -p` finds those tools, and otherwise downloads the same files as a tarball with `curl`. Don't install the command line tools for this. The block enters the folder only when the download worked. If it fails, show the user its error and stop. The installed sys1rust doesn't need the folder afterwards. + +Agents that support skills also find these 3 files as the skills `sys1rust-setup`, `sys1rust-use` and `sys1rust-develop`. Codex, [Cursor](https://cursor.com/docs/skills) and [GitHub Copilot](https://docs.github.com/en/copilot/concepts/agents/about-agent-skills) read `.agents/skills/`, and Claude Code reads the links to them in `.claude/skills/`. Without skill support, open the file and follow it. Edit the skills only in `.agents/skills/`. + +## Rules for every agent + +- Never use `sudo`. Nothing here needs it. +- When you set sys1rust up for a user, write only to the clone, `~/.local/share/sys1rust`, `~/.local/bin`, `~/Library/LaunchAgents`, `~/Library/Logs` and the Hugging Face cache. Ask the user before you write anywhere else or edit a shell startup file such as `~/.zshrc`. +- Don't delete downloaded models without asking. The Hugging Face cache can hold other tools' models. +- Run one GPU job at a time on a Mac. A running `sys1rust serve`, a test suite that loads a model and a benchmark all use the GPU, and 2 at once slow each other. +- Keep the server on `127.0.0.1`. Bind a wider address only when the user asks, and then follow "Serving other machines" in the [README](README.md). +- Take commands, flags and numbers from `sys1rust --help`, `./install.sh --help`, the README and the code. Don't guess an option name. +- Don't commit, push or tag unless the user asks. + +## Facts to get right + +- sys1rust needs Apple silicon, a native shell rather than Rosetta, and macOS 14.0 or later. On macOS 26.2 or later the installer picks the `macos26` bundle, which is about 3x faster on the M5 than the `macos14` bundle. +- `./install.sh` installs the latest release into `~/.local/share/sys1rust` and links `~/.local/bin/sys1rust`. `--service` adds the LaunchAgent `io.github.krishhgg.sys1rust`, which logs to `~/Library/Logs/sys1rust.log`. `--uninstall` removes both and keeps the models. +- `sys1rust serve` downloads its model on the first start, typed-decisions (846 MB) by default, into the Hugging Face cache. `sys1rust pull [model]` downloads one ahead of time, and `sys1rust models` lists the 3 models and what the cache holds. +- `GET /health` says whether the server is up. `POST /v1/systemone` takes a state and its questions and answers each one with a choice, a score or a yes probability. + +## Repository map + +- `install.sh` installs a prebuilt release from GitHub Releases. +- `runtime/` is the Rust workspace, `runtime/Cargo.toml`. + - `crates/laya-core` parses requests, tokenizes, lays out sequences and decodes answers. It has no GPU code. + - `crates/laya-mlx` runs the forward pass on MLX through mlx-rs, with sys1rust's own Metal kernels. + - `crates/sys1rust` is the `sys1rust` command (`serve`, `pull`, `models`) and its HTTP server. + - `crates/sys1-bench` is the benchmark adapter and `sys1-probe`, which compares engine settings. + - `vendor/mlx-sys` is mlx-sys 0.6.0 with a build that links a prebuilt MLX. +- `bench/` holds the benchmark harness, the workloads, the upstream reference answers and the pinned model revisions in `bench/models.lock.json`. [`bench/PLAN.md`](bench/PLAN.md) has its ground rules. +- `packaging/` builds the release bundles, smoke-tests them and holds the release checklist, [`packaging/RELEASE.md`](packaging/RELEASE.md). +- `results/` holds the measured write-ups, from the bake-off ([`results/REPORT.md`](results/REPORT.md)) to the speed rounds ([`results/SPEED.md`](results/SPEED.md)). +- `research/` holds sourced reports on the models, runtimes and hardware. +- `docs/assets/` holds the README's diagrams. diff --git a/CLAUDE.md b/CLAUDE.md new file mode 100644 index 0000000..7b01b01 --- /dev/null +++ b/CLAUDE.md @@ -0,0 +1,7 @@ +@AGENTS.md + +## Claude Code + +- `.claude/skills/` holds links to the 3 skills in `.agents/skills/`, so `/sys1rust-setup`, `/sys1rust-use` and `/sys1rust-develop` load the same files that AGENTS.md lists. Edit the files in `.agents/skills/`. +- Don't use the Bash tool's `run_in_background` for a server that the user keeps after the session. Claude Code stops background tasks, and the processes they started, when it exits. Start it with `nohup` in an ordinary Bash call, as the setup skill does, or install the service. +- A server you start only to test something can run in the background. Stop it before you finish, and check with `pgrep -fl 'sys1rust serve'` that none is left. diff --git a/README.md b/README.md index 46c7ca5..620284b 100644 --- a/README.md +++ b/README.md @@ -21,7 +21,15 @@ ## Install -To install on an Apple silicon Mac with macOS 14 or later, run: +Paste this line into Claude Code, Codex, Cursor or another coding agent: + +```text +Set up sys1rust on my Mac from https://github.com/krishhgg/sys1rust +``` + +The agent clones this repository and follows [`AGENTS.md`](AGENTS.md), which has it check your Mac, run `install.sh`, download the model, start the server and send it a test request. + +To install it yourself on an Apple silicon Mac with macOS 14 or later, run: ```sh curl -fsSL https://raw.githubusercontent.com/krishhgg/sys1rust/main/install.sh | sh @@ -135,7 +143,7 @@ export MLX_SYS_PREBUILT_DIR="$(.mlx/bin/python -c 'import mlx.core, os; print(os cargo build --release --manifest-path runtime/Cargo.toml ``` -Then run `runtime/target/release/sys1rust serve`. Where this README runs `sys1rust`, use that path, or add `runtime/target/release` to your `PATH`. The binary loads MLX from that venv by its absolute path, so keep `.mlx/` where it is. +Then run `runtime/target/release/sys1rust serve`. Where this README runs `sys1rust`, use that path, or add `runtime/target/release` to your `PATH`. The binary loads MLX from that venv by its absolute path, so keep `.mlx/` where it is. To change the code and run the tests, follow [`.agents/skills/sys1rust-develop/SKILL.md`](.agents/skills/sys1rust-develop/SKILL.md). ## More @@ -216,6 +224,7 @@ $CARGO_TARGET_DIR/release/sys1rust serve --model typed-decisions --port 8000 Repository layout - `install.sh`: the installer for prebuilt releases. +- `AGENTS.md`, `CLAUDE.md` and `.agents/skills/`: instructions for coding agents that set up, call or change sys1rust. - `runtime/`: the Rust workspace. - `laya-core`: request parsing, tokenization, sequence layout and answer decoding, with no GPU code. - `laya-mlx`: the forward pass on MLX through mlx-rs.