diff --git a/reference/operations-api/operations.md b/reference/operations-api/operations.md index 5a54dca7..d755a047 100644 --- a/reference/operations-api/operations.md +++ b/reference/operations-api/operations.md @@ -577,27 +577,38 @@ Operations for deploying and managing Harper components (applications, plugins). Detailed documentation: [Components Overview](../components/overview.md) -| Operation | Description | Role Required | -| ---------------------- | ----------------------------------------------------------------------- | ------------- | -| `add_component` | Creates a new component project from a template | super_user | -| `deploy_component` | Deploys a component via payload (tar) or package reference (NPM/GitHub) | super_user | -| `package_component` | Packages a component project into a base64-encoded tar | super_user | -| `drop_component` | Deletes a component or a file within a component | super_user | -| `get_components` | Lists all component files and config | super_user | -| `get_component_file` | Returns the contents of a file within a component | super_user | -| `set_component_file` | Creates or updates a file within a component | super_user | -| `add_ssh_key` | Adds an SSH key for deploying from private repositories | super_user | -| `update_ssh_key` | Updates an existing SSH key | super_user | -| `delete_ssh_key` | Deletes an SSH key | super_user | -| `list_ssh_keys` | Lists all configured SSH key names | super_user | -| `set_ssh_known_hosts` | Overwrites the SSH known_hosts file | super_user | -| `get_ssh_known_hosts` | Returns the contents of the SSH known_hosts file | super_user | -| `install_node_modules` | _(Deprecated)_ Run npm install on component projects | super_user | +| Operation | Description | Role Required | +| --------------------------- | ----------------------------------------------------------------------- | ------------- | +| `add_component` | Creates a new component project from a template | super_user | +| `deploy_component` | Deploys a component via payload (tar) or package reference (NPM/GitHub) | super_user | +| `package_component` | Packages a component project into a base64-encoded tar | super_user | +| `drop_component` | Deletes a component or a file within a component | super_user | +| `get_components` | Lists all component files and config | super_user | +| `get_component_file` | Returns the contents of a file within a component | super_user | +| `set_component_file` | Creates or updates a file within a component | super_user | +| `list_deployments` | Lists deployment records with optional filters | super_user | +| `get_deployment` | Fetches a single deployment record by ID; supports SSE streaming | super_user | +| `get_deployment_payload` | Returns the tarball stored for a deployment | super_user | +| `delete_deployment_payload` | Removes the stored tarball to free space | super_user | +| `add_ssh_key` | Adds an SSH key for deploying from private repositories | super_user | +| `update_ssh_key` | Updates an existing SSH key | super_user | +| `delete_ssh_key` | Deletes an SSH key | super_user | +| `list_ssh_keys` | Lists all configured SSH key names | super_user | +| `set_ssh_known_hosts` | Overwrites the SSH known_hosts file | super_user | +| `get_ssh_known_hosts` | Returns the contents of the SSH known_hosts file | super_user | +| `install_node_modules` | _(Deprecated)_ Run npm install on component projects | super_user | ### `deploy_component` Deploys a component. The `package` option accepts any valid NPM reference including GitHub repos (`HarperDB/app#semver:v1.0.0`), tarballs, or NPM packages. The `payload` option accepts a base64-encoded tar string from `package_component`. Supports `"replicated": true` and `"restart": true` or `"restart": "rolling"`. +Additional parameters: + +- `urlPath` — override the HTTP URL path the component is mounted at (e.g. `"/api/v2"`) +- `install_allow_scripts` — set to `true` to allow npm pre/post install scripts (disabled by default) + +The response includes a `deployment_id` that can be used to query the deployment record: + ```json { "operation": "deploy_component", @@ -608,6 +619,95 @@ Deploys a component. The `package` option accepts any valid NPM reference includ } ``` +Response: + +```json +{ + "deployment_id": "a3f8c2d1...", + "message": "Component deployed successfully" +} +``` + +### Deployment Operations + +Harper records every `deploy_component` call in the `system.hdb_deployment` table, capturing the full lifecycle of a deployment including phase transitions (prepare → load → replicate → restart → success/failed), per-node outcomes, and a bounded event log of install output. + +### `list_deployments` + +Returns a list of deployment records, newest first. All filter parameters are optional. + +| Parameter | Type | Description | +| --------- | ------ | ------------------------------------------------ | +| `project` | string | Filter to a specific component project | +| `status` | string | Filter by status: `pending`, `success`, `failed` | +| `since` | number | Start of time range (Unix timestamp ms) | +| `until` | number | End of time range (Unix timestamp ms) | +| `limit` | number | Maximum number of results (default: 100) | +| `offset` | number | Pagination offset | + +```json +{ + "operation": "list_deployments", + "project": "my-app", + "status": "success", + "limit": 20 +} +``` + +Response includes a `deployments` array and a `total` count. The `payload_blob` field is stripped from list responses for size; use `get_deployment_payload` to retrieve the tarball. + +### `get_deployment` + +Returns a single deployment record by `deployment_id`. When called on an in-progress deployment via a request that accepts `text/event-stream`, the response streams live phase events and install output as Server-Sent Events, replaying the buffered event log then tailing until the deployment reaches a terminal status. + +```json +{ + "operation": "get_deployment", + "deployment_id": "a3f8c2d1..." +} +``` + +The deployment record includes: + +| Field | Description | +| -------------------- | ----------------------------------------------------------------------- | +| `deployment_id` | Unique identifier (content hash) | +| `project` | Component project name | +| `package_identifier` | Package reference or `payload` for tar uploads | +| `status` | `pending`, `success`, `failed`, or `rolled_back` | +| `phase` | Current lifecycle phase: `prepare`, `load`, `replicate`, `restart` | +| `event_log` | Bounded log of install output and phase transitions (up to 200 entries) | +| `peer_results` | Per-node outcome map for replicated deployments | +| `payload_hash` | SHA-256 hash of the deployment tarball | +| `payload_size` | Byte size of the deployment tarball | +| `started_at` | Timestamp when deployment began | +| `completed_at` | Timestamp when deployment finished | +| `user` | User who initiated the deployment | +| `rollback_of` | `deployment_id` of the deployment this rolls back, if applicable | +| `error` | Error message for failed deployments | + +### `get_deployment_payload` + +Returns the raw tarball for a deployment. Useful for inspecting or re-deploying a specific version. + +```json +{ + "operation": "get_deployment_payload", + "deployment_id": "a3f8c2d1..." +} +``` + +### `delete_deployment_payload` + +Removes the tarball blob from a deployment record. The deployment record itself is retained; only the binary payload is deleted. Use this to reclaim storage after confirming a deployment is stable. + +```json +{ + "operation": "delete_deployment_payload", + "deployment_id": "a3f8c2d1..." +} +``` + ### `add_ssh_key` Adds an SSH key (must be ed25519) for authenticating deployments from private repositories. diff --git a/release-notes/v5-lincoln/5.1.md b/release-notes/v5-lincoln/5.1.md new file mode 100644 index 00000000..cfc7aa9d --- /dev/null +++ b/release-notes/v5-lincoln/5.1.md @@ -0,0 +1,234 @@ +--- +title: '5.1' +--- + +# 5.1 Release Notes + +### Patch Releases + +All patch release notes for 5.1.x are available on the [releases page](https://github.com/HarperFast/harper/releases?q=v5.1&expanded=true). + +## AI Models Integration + +Harper 5.1 introduces a built-in models layer that provides a unified interface for AI model backends. This enables embedding generation and text generation from directly within Harper applications, without managing external API connections per-application. + +The models layer is exposed via `scope.models` in application code: + +```js +// Generate embeddings +const vector = await scope.models.embed('text to embed', { model: 'my-embedding-model' }); + +// Generate text +const response = await scope.models.generate([{ role: 'user', content: 'Hello' }], { model: 'my-chat-model' }); + +// Streaming generation +for await (const chunk of scope.models.generateStream(messages, { model: 'my-chat-model' })) { + // ... +} +``` + +Supported backends are Anthropic, AWS Bedrock, OpenAI, and Ollama, configured under the `models` key in `harper-config.yaml`: + +```yaml +models: + anthropic: + apiKey: your-api-key + openai: + apiKey: your-api-key + baseUrl: https://api.openai.com/v1 # optional override + ollama: + baseUrl: http://localhost:11434 + bedrock: + region: us-east-1 +``` + +### `@embed` Schema Directive + +The `@embed` directive automates vector embedding at the schema level, eliminating the need for application code to compute and store embeddings on every write. Add it to any `Float` array field with a `source` pointing to the text field to embed: + +```graphql +type Document @table { + id: ID @primaryKey + content: String + embedding: [Float] @embed(source: "content", model: "my-embedding-model") @indexed(type: "HNSW") +} +``` + +On every write, Harper automatically calls the specified model to compute the embedding from `source` and stores it on the record. The `@indexed(type: "HNSW")` index is attached automatically when not explicitly specified. + +### Agent Loop + +`scope.models.generate` now supports `toolMode: 'auto'`, which runs an agentic loop — the model can invoke tools and Harper will automatically dispatch them until the model produces a final non-tool response. This makes it straightforward to build tool-using agents directly in Harper application code. + +## MCP Server + +Harper 5.1 includes a built-in [Model Context Protocol](https://modelcontextprotocol.io/) server, allowing LLM clients such as Claude Desktop, Cursor, and Zed to connect directly to a Harper instance and interact with its data and operations. + +The MCP server exposes two profiles: + +- **Operations profile** — wraps Harper's operations catalog as tools, with a curated default allow-list of read-only operations +- **Application profile** — auto-generates tools from a Harper application's Resource verb methods + +The `harper mcp` CLI command provides a stdio bridge for use with MCP clients: + +```bash +# Generate a config block for Claude Desktop +harper mcp print-config --client claude-desktop + +# Run diagnostics against a running instance +harper mcp doctor +``` + +MCP is enabled by adding an `mcp` block to `harper-config.yaml`. See the MCP reference documentation for full configuration options, authentication, and tool customization. + +## Application Routing & Middleware + +Harper 5.1 adds a named middleware ordering and URL routing system for application components, giving fine-grained control over how request handlers are composed. + +### `urlPath` routing + +A component's HTTP handler can now declare a `urlPath` to scope its routes to a URL prefix. Harper strips the prefix before dispatching to the handler, so the handler's logic sees clean relative paths regardless of where it's mounted. + +```js +// This handler only receives requests under /api/v2/ +export const handleHttpRequest = { + urlPath: '/api/v2', + async handleRequest(request) { + // request.url here is the path *after* /api/v2/ is stripped + }, +}; +``` + +### `before`/`after` ordering + +Components declare their execution order relative to other named middleware using `before` or `after`. The primary use case is ordering relative to Harper's built-in `authentication` middleware: + +```js +export const handleHttpRequest = { + name: 'my-rate-limiter', + before: 'authentication', // run this handler before authentication + async handleRequest(request) { ... }, +}; + +export const handleAuthenticatedRequests = { + name: 'my-data-handler', + after: 'authentication', // only runs after authentication has completed + async handleRequest(request) { ... }, +}; +``` + +The `name` field makes a handler addressable so that others can reference it in their `before`/`after` declarations. Cyclic ordering is detected at startup and logged as a warning. + +These options apply uniformly to HTTP handlers (`handleHttpRequest`), WebSocket handlers (`onWebSocket`), and protocol upgrade handlers (`onUpgrade`). + +### Node.js middleware adapter + +`request.getNodeRequestResponse()` returns a `{ nodeRequest, nodeResponse, response }` triple that bridges Harper's W3C-style Request/Response to Node.js's `IncomingMessage`/`ServerResponse` API. This makes it possible to integrate third-party Node.js middleware (Express, Koa, Passport, etc.) directly inside a Harper application handler without wrapping the entire server. + +```js +import someMiddleware from 'some-package'; + +export async function handleRequest(request) { + const { nodeRequest, nodeResponse, response } = request.getNodeRequestResponse(); + someMiddleware(nodeRequest, nodeResponse, () => {}); + return response; +} +``` + +`nodeRequest` mirrors the current request state (including any mutations from earlier middleware), while `nodeResponse` captures headers and body written by the consumer and resolves `response` once headers are available. + +## Deployment Tracking + +`deploy_component` now records a full audit trail for every deployment in the `system.hdb_deployment` system table. Each deployment gets a `deployment_id` and tracks phases (prepare → load → replicate → restart → success), per-node outcomes, and a bounded event log capturing install output. + +The response from `deploy_component` now includes a `deployment_id`: + +```json +{ + "deployment_id": "a3f8c2...", + "message": "Component deployed successfully" +} +``` + +New operations provide access to deployment history: + +- `list_deployments` — query deployment history with filters +- `get_deployment` — fetch a single deployment record; supports live SSE streaming for in-progress deploys +- `get_deployment_payload` — retrieve the stored tarball for a deployment +- `delete_deployment_payload` — free storage by removing the payload blob after deployment + +See [Deployment Operations](#deployment-operations) in the Operations API reference for details. + +## HNSW int8 Quantization + +HNSW vector indexes now support `int8` quantization, reducing index storage by approximately 3× and improving search throughput approximately 5× with around 1% recall loss at recall@10: + +```graphql +type Document @table { + embedding: [Float] @indexed(type: "HNSW", quantization: "int8") +} +``` + +Search uses asymmetric scoring: queries use full-precision float vectors while the index graph uses int8, and results are reranked against full-precision vectors before returning. The full-precision vector is always stored on the record itself. + +Per-query `ef` can be overridden at query time for applications that need to tune the recall/latency tradeoff dynamically. + +`int8` quantization is on by default for new HNSW indexes in 5.1. Existing indexes can be reindexed to take advantage of it. + +## Replication Improvements + +Several replication reliability improvements are included in 5.1: + +- **Resumable bulk clone** — interrupted full-table copies resume from where they left off rather than restarting from the beginning +- **Client-side receive watchdog** — dead WebSocket connections are now detected and recovered without waiting for a server-side timeout +- **Wedge detection and recovery** — stalled replication streams are detected and re-subscribed automatically +- **`isLeader` flag on `add_node`** — explicitly request a full-table copy when joining a cluster, independent of the normal subscription logic +- **`replication.pingInterval` / `replication.pingTimeout`** — configurable keepalive intervals for replication connections (values in milliseconds) +- **Reconnect backoff cap** — reconnect retry backoff is now capped at 30 seconds; previously it could grow unbounded and leave a peer effectively unable to reconnect +- **Boot replay bounded** — the startup transaction-log replay is now bounded so it cannot stall indefinitely on corrupted or out-of-order entries; corrupt frames are skipped, and aged transaction logs are purged before replay begins + +## HTTP Caching + +Harper 5.1 improves HTTP caching behavior in the `CacheOfHttp` resource. Responses are now cached based on RFC 9111 cacheable status codes rather than a hardcoded list. The `allowStaleWhileRevalidate` option enables serving stale cached content while refreshing in the background, reducing perceived latency for read-heavy workloads. The `sourcedFrom` configuration field identifies the upstream source for a cache table and drives the eviction and revalidation logic. + +## `LOCAL_ONLY` Writes + +A new `LOCAL_ONLY` metadata flag allows writes that are persisted locally but never replicated to cluster peers: + +```js +await table.put({ id: 'node-local-key', value: 42 }, { metadata: { LOCAL_ONLY: true } }); +``` + +This is useful for node-local state — per-node counters, cache metadata, or analytics that don't belong in the replicated dataset. + +## Configuration + +### `HARPER_CONFIG` environment variable + +`HARPER_CONFIG` is now the recommended way to specify the configuration file location: + +```bash +HARPER_CONFIG=/etc/harper/harper-config.yaml harper +``` + +Previously only `HARPERDB_SETTINGS` and CLI flags were available for this purpose. + +### RocksDB memory configuration + +`storage.rocks.blockCacheSize` explicitly sets the RocksDB block cache size. When not set, Harper auto-sizes it based on available memory. The WriteBufferManager is now on by default at 1/3 of the block cache, which improves memory pressure handling under write-heavy workloads. + +### `migrateOnStart` + +Setting `storage.migrateOnStart: true` automatically migrates LMDB databases to RocksDB on the next startup. This provides a path to migrate existing LMDB-backed instances without manual tooling, at the cost of a longer first startup time. + +## v4 to v5 Upgrade Improvements + +Several reliability fixes landed for in-place v4→v5 upgrades: + +- The node `hostname` field no longer defaults to `localhost` during an in-place upgrade. Previously, all nodes in a cluster would end up with `hostname: localhost` after a v4→v5 upgrade, causing silent split-brain where nodes couldn't locate each other correctly. +- The `__dbis__` structure dictionary is now correctly persisted to RocksDB during the LMDB→RocksDB migration. Without this, cold restarts after migration would fail to decode records. +- Blob file references, expiry metadata (`expiresAt`), and residency flags are preserved during migration rather than being dropped. + +These fixes make the in-place upgrade path substantially more reliable for clustered deployments. See the [migration guide](./v5-migration.md) for the recommended upgrade procedure. + +Please see the [migration guide](./v5-migration.md) for suggestions on how to migrate from v4 to v5.