Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions .dockerignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
**
!Dockerfile
!maple_metrics_auth.py
!scripts/
!scripts/install_metrics_auth.py
45 changes: 45 additions & 0 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,45 @@
name: Validate

on:
pull_request:
push:
branches: [main]

permissions:
contents: read

concurrency:
group: validate-${{ github.ref }}
cancel-in-progress: true

jobs:
validate:
runs-on: ubuntu-latest
timeout-minutes: 60
steps:
- name: Check out the repository
uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
with:
persist-credentials: false

- name: Test the metrics guard and the release inputs
run: PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests -v

- name: Check the committed configuration
run: |
python3 render_config.py --check
git diff --exit-code -- tinfoil-config.yml

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0

# The build itself is the test that the guard installs into the real
# image: the installer fails if vLLM is not the build it expects.
- name: Build the image without publishing
uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0
env:
SOURCE_DATE_EPOCH: 0
with:
context: .
platforms: linux/amd64
push: false
10 changes: 5 additions & 5 deletions .github/workflows/tinfoil-release-publish.yml
Original file line number Diff line number Diff line change
Expand Up @@ -30,18 +30,18 @@ jobs:
with:
persist-credentials: false

- name: Validate the tagged staging configuration
- name: Validate the tagged configuration
shell: bash
run: |
git ls-files --error-unmatch tinfoil-config.yml >/dev/null
python3 render_config.py --check
python3 render_config.py --check --require-pinned-image
git diff --exit-code -- tinfoil-config.yml

- name: Measure, attest, and publish the staging release
- name: Measure, attest, and publish the release
uses: tinfoilsh/measure-image-action@59f7ab5dbe472b3097c888ebe64e6a4131fc0975 # v0.14.0
with:
config-file: ${{ github.workspace }}/tinfoil-config.yml
github-token: ${{ secrets.GITHUB_TOKEN }}
# Publish the isolated staging release as latest so Tinfoil can
# issue a freshness witness for its deployment digest.
# Publish the release as latest so Tinfoil can issue a freshness
# witness for its deployment digest.
mark-as-latest: 'true'
119 changes: 98 additions & 21 deletions .github/workflows/tinfoil-release.yml
Original file line number Diff line number Diff line change
@@ -1,31 +1,38 @@
# Two-phase Tinfoil release. Builds the image, pins its digest in the
# configuration on a release-only commit, tags that commit, then dispatches
# measurement on the tag.
name: Tinfoil Release
run-name: Prepare Tinfoil release ${{ inputs.version }}

on:
workflow_dispatch:
inputs:
version:
description: Stable staging release version (for example, v0.1.0)
description: Stable release version (for example, v0.2.0)
required: true
type: string

concurrency:
group: tinfoil-release-prepare-${{ inputs.version }}

jobs:
prepare-release:
build:
runs-on: ubuntu-latest
timeout-minutes: 60
permissions:
contents: write
actions: write
contents: read
packages: write
outputs:
digest: ${{ steps.build.outputs.digest }}

steps:
- name: Check out the requested commit
uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
with:
persist-credentials: false

- name: Validate the committed staging configuration
- name: Validate the source and the requested version
id: release
shell: bash
env:
VERSION: ${{ inputs.version }}
Expand All @@ -36,31 +43,101 @@ jobs:
exit 1
fi
if [[ ! "$VERSION" =~ ^v(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)\.(0|[1-9][0-9]*)$ ]]; then
echo "::error::Choose a stable version such as v0.1.0."
echo "::error::Choose a stable version such as v0.2.0."
exit 1
fi
if git ls-remote --exit-code --tags origin "refs/tags/$VERSION" >/dev/null; then
echo "::error::Tag $VERSION already exists. Choose a new version."
exit 1
else
status=$?
if [[ "$status" != 2 ]]; then exit "$status"; fi
fi
PYTHONDONTWRITEBYTECODE=1 python3 -m unittest discover -s tests -v
git ls-files --error-unmatch tinfoil-config.yml >/dev/null
python3 render_config.py --check
git diff --exit-code -- tinfoil-config.yml
echo "image-tag=${VERSION#v}" >> "$GITHUB_OUTPUT"

- name: Set up Docker Buildx
uses: docker/setup-buildx-action@4d04d5d9486b7bd6fa91e7baf45bbb4f8b9deedd # v4.0.0

- name: Log in to GitHub Container Registry
uses: docker/login-action@4907a6ddec9925e35a0a9e82d7399ccc52663121 # v4.1.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}

- name: Extract metadata for Docker
id: meta
uses: docker/metadata-action@030e881283bb7a6894de51c315a6bfe6a94e05cf # v6.0.0
with:
images: ghcr.io/${{ github.repository }}
flavor: latest=false
tags: |
type=raw,value=${{ steps.release.outputs.image-tag }}
labels: |
org.opencontainers.image.version=${{ inputs.version }}

- name: Tag the validated commit and dispatch measurement
- name: Build and push the image
id: build
uses: docker/build-push-action@bcafcacb16a39f128d818304e6c9c0c18556b85f # v7.1.0
env:
SOURCE_DATE_EPOCH: 0
with:
context: .
platforms: linux/amd64
push: true
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}

prepare-release:
needs: build
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: write
actions: write

steps:
- name: Check out the source commit
uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
with:
ref: ${{ github.sha }}
persist-credentials: false

- name: Check out the pinned configuration updater
uses: actions/checkout@8e8c483db84b4bee98b60c0593521ed34d9990e8 # v6.0.1
with:
repository: tinfoilsh/update-container-action
ref: 5e0a3789316941664cee055706799114e0180565 # v0.6.0
path: .release-tools
persist-credentials: false

- name: Pin the built image and create the release tag
shell: bash
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GH_REPO: ${{ github.repository }}
VERSION: ${{ inputs.version }}
IMAGE_REFS: |
ghcr.io/mapleprivacylabs/inference-penclaw@${{ needs.build.outputs.digest }}
run: |
tag_ref="refs/tags/$VERSION"
existing_commit=$(git ls-remote --refs origin "$tag_ref" | cut -f 1)
if [[ -z "$existing_commit" ]]; then
gh api --method POST "repos/$GH_REPO/git/refs" \
-f ref="$tag_ref" -f sha="$GITHUB_SHA" --silent
elif [[ "$existing_commit" != "$GITHUB_SHA" ]]; then
echo "::error::The release tag already points to another commit. Choose a new version."
exit 1
elif gh release view "$VERSION" >/dev/null 2>&1; then
echo "Release $VERSION is already published."
exit 0
fi
python3 .release-tools/update_config.py
python3 render_config.py --check --require-pinned-image
git config user.name "github-actions[bot]"
git config user.email "41898282+github-actions[bot]@users.noreply.github.com"
source_date=$(git show -s --format=%cI "$GITHUB_SHA")
export GIT_AUTHOR_DATE="$source_date" GIT_COMMITTER_DATE="$source_date"
git add tinfoil-config.yml
git commit -m "chore: prepare $VERSION release"
git tag "$VERSION"
gh auth setup-git
git push origin "refs/tags/$VERSION"

gh workflow run tinfoil-release-publish.yml --ref "$VERSION"
- name: Dispatch measurement on the new tag
env:
GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
GH_REPO: ${{ github.repository }}
VERSION: ${{ inputs.version }}
run: gh workflow run tinfoil-release-publish.yml --ref "$VERSION"
37 changes: 37 additions & 0 deletions CHANGES.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@
# Differences from stock

v0.1.4 equals the stock Maple Kimi v0.0.3 configuration apart from the primary
checkpoint fields. Every later difference is listed here.

`render_config.py` lists every difference from stock in `EDITS` and refuses to
render a template that differs in any other way.

## Current configuration (v0.2.0 onwards)

| Difference from stock | Why |
|---|---|
| `cpus` 120 instead of 32 | The host is already paid for. Measured throughput is unchanged up to 32 concurrent requests. |
| Three DSpark draft tokens instead of five | About 15% more output at 32 concurrent requests in the same clean test (see v0.1.7 below). |
| Image built by this repository from Tinfoil's image | The [Dockerfile](Dockerfile) adds one file, [`maple_metrics_auth.py`](maple_metrics_auth.py), and changes nothing else. The build fails if the base image's vLLM is not the build the guard was written against. |
| `--middleware maple_metrics_auth.MetricsAuthMiddleware` | Loads the guard. It requires the `METRICS_API_KEY` bearer on every path under `/metrics` and passes all other requests through untouched. |
| Served model name `kimi-k3-1-pro-uncensored` instead of `kimi-k3-1-pro` | The node serves Audn's abliterated checkpoint, so it answers under a name that says so. Maple offers it as a separate model beside the stock Kimi K3.1 Pro. |
| `/metrics` in the shim paths | vLLM's Prometheus metrics (queue, cache, latency, token and speculative-decoding counters) become readable with the metrics key. Without the guard this path would be open: vLLM's own bearer check does not cover it, and the shim's only gate for it is Tinfoil's online admin-key validator. |

The same key already protects the shim's own `/.well-known/tinfoil-metrics` and
`/.well-known/metrics`. Metrics are counters and histograms; they contain no
request content.

## Release history

| Release | Change from the previous release | Why |
|---|---|---|
| v0.1.5 | `cpus` 32 to 112; `/metrics` added to the shim paths. **Never deployed:** the platform accepts only 2, 4, 8, 16, 32, 56, 64, 120 or 128 vCPUs and rejected 112 at update planning | The identical host class runs two 56-vCPU guests, so 32 vCPUs for eight GPUs looked low. vLLM's Prometheus metrics give running and waiting requests, token counters, latency histograms and speculative-decoding acceptance, which have never been readable on this node. |
| v0.1.6 | `cpus` 120 in place of 112; otherwise v0.1.5 | Nearest accepted size above the 112 vCPUs that two DeepSeek guests use on the identical host class. |
| v0.1.7 | DSpark draft tokens 5 to 3 | On v0.1.6 the draft tokens were accepted 65%, 40%, 23%, 13% and 7% of the time by position: the fourth and fifth add about 0.2 tokens per step while every step carries a third more rows. Result on v0.1.6 for comparison: 128-134 tokens/s for one request, 549-603 at 8, 1,358-1,378 at 32. |
| v0.1.8 | `/v1/messages` and `/v1/messages/count_tokens` added to the shim paths | The published Terminal-Bench 4.1 result for Kimi K3.1 Pro was run with Claude Code, which speaks the Anthropic Messages format. vLLM already serves that format at this build; only the shim blocked the path. No engine setting changes. |
| v0.2.0 | Image built here with the metrics guard, loaded by `--middleware`; served model name `kimi-k3-1-pro-uncensored`; `/v1/messages` and `/v1/messages/count_tokens` removed from the shim paths again | First release meant to serve customers. `/metrics` needs the metrics key from this release on. The Anthropic Messages paths were only needed for the benchmark run. |

**Result of v0.1.7, and the configuration kept:** three draft tokens were faster at every level in the same clean test: 140-146 tokens/s for one request, 600-642 at 8, 1,568-1,612 at 32 (about 15% more than v0.1.6), and 1,493 at 32 with real prompts against 1,281. Tokens per step fell only from about 2.5 to about 2.25.

In v0.1.6 to v0.1.8 `/metrics` was reachable without a key. Those releases were
staging experiments and carried no customer traffic.
9 changes: 9 additions & 0 deletions Dockerfile
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
# syntax=docker/dockerfile:1.6
# Tinfoil's qualified Kimi K3.1 Pro runtime, unchanged: same vLLM build, kernels,
# confidential-computing settings and entrypoint. The only addition is one file,
# a guard that requires the Maple metrics key on vLLM's /metrics.
FROM ghcr.io/tinfoilsh/confidential-kimi-k31-pro@sha256:0a81868bb275945f23e0e71d53524dd9a416bfcde3acb14903af0c2139495bc0

COPY maple_metrics_auth.py scripts/install_metrics_auth.py /tmp/
RUN python3 -B /tmp/install_metrics_auth.py && \
rm /tmp/install_metrics_auth.py /tmp/maple_metrics_auth.py
21 changes: 0 additions & 21 deletions EXPERIMENTS.md

This file was deleted.

34 changes: 25 additions & 9 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,15 +1,18 @@
# Penclaw Kimi K3.1 Pro staging
# Penclaw Kimi K3.1 Pro

Releases after v0.1.4 are performance experiments and differ from the stock
configuration in the ways listed in [EXPERIMENTS.md](EXPERIMENTS.md).
Releases after v0.1.4 differ from the stock configuration in the ways listed
in [CHANGES.md](CHANGES.md). From v0.2.0 the serving image is built by this
repository: Tinfoil's image plus one file that requires the metrics key on
vLLM's `/metrics`.

The v0.1.4 configuration serves Audn's Penclaw checkpoint using the
[Maple Kimi v0.0.3 configuration](https://github.com/MaplePrivacyLabs/inference-kimi-k3-1-pro/blob/06a62d7829bdb30237846544874830a86305e9f8/tinfoil-config.yml).
It changes the primary checkpoint repository and revision, Modelwrap tuple,
pack schema, and vLLM model path. It uses the same eight B300 GPUs, serving
image, Inferact DSpark draft, and native vLLM bearer authentication. The
`kimi-k3-1-pro` API model name is retained for compatibility. The signed
Penclaw release identifies the actual checkpoint served under that name.
image, Inferact DSpark draft, and native vLLM bearer authentication. Through
v0.1.8 it answered to the stock API model name `kimi-k3-1-pro`. From v0.2.0
the API model name is `kimi-k3-1-pro-uncensored`. The signed release
identifies the actual checkpoint served under either name.

The [snapshot manifest](snapshot-manifest.json) records the exact Audn checkpoint
revision and Hugging Face's checksums. The measured configuration pins each
Expand Down Expand Up @@ -41,6 +44,19 @@ that commitment. The checkpoint is distributed under the Kimi K3.1-Pro license.
Pass the full `root_offset_uuid` tuple from the completed schema-2 wrap to
`python3 render_config.py --mpk TUPLE`. Run `python3 render_config.py --check`
before publishing. The renderer verifies that the template matches the pinned
stock configuration outside the four primary pack fields. The release workflow
checks the rendered config on the default branch and at the release tag before
measuring it.
stock configuration outside the four primary pack fields and the differences
listed in `EDITS`. The source names the image without a digest. The release
workflow builds the image, writes its digest into the configuration on a
release-only commit, tags that commit and measures it. The renderer checks the
configuration on the default branch and again, with the pinned digest, at the
release tag.

## The serving image

The [Dockerfile](Dockerfile) starts from the exact image the stock
configuration pins and adds [`maple_metrics_auth.py`](maple_metrics_auth.py)
through [`scripts/install_metrics_auth.py`](scripts/install_metrics_auth.py).
The installer checks the vLLM version and the three places in vLLM's source the
guard depends on, and fails the build if any has changed. Tests in `tests/`
cover the guard and check that the Dockerfile, the configuration and the guard
agree.
Loading