From 1f82b9c5762fb1053409549fff3b6f6536eec2e5 Mon Sep 17 00:00:00 2001 From: Julian Payne Date: Fri, 14 Aug 2026 16:06:25 +0200 Subject: [PATCH 1/2] fix(docs): add missing server start steps to Quick Taste and Quick Start The Quick Taste on home.mdx jumped straight from pip install to evalhub eval run with no mention of starting either a model server or the EvalHub service, causing "connection refused" for new users. - Expand Quick Taste into a 4-step sequence: install SDK with [server,cli], start Ollama, start EvalHub, run evaluation - Add a "Connect to EvalHub" step to Quick Start before Submit - Link to Local Mode guide for full configuration details Closes #69 Assisted-by: Cursor Signed-off-by: Julian Payne Co-authored-by: Cursor --- .../docs/getting-started/quickstart.mdx | 17 ++++++- src/content/docs/home.mdx | 50 ++++++++++++++----- 2 files changed, 53 insertions(+), 14 deletions(-) diff --git a/src/content/docs/getting-started/quickstart.mdx b/src/content/docs/getting-started/quickstart.mdx index 3fd3320..3a4b2e1 100644 --- a/src/content/docs/getting-started/quickstart.mdx +++ b/src/content/docs/getting-started/quickstart.mdx @@ -78,7 +78,20 @@ Run your first evaluation with EvalHub. -3. **Submit an Evaluation** +3. **Connect to EvalHub** + + The commands below assume an EvalHub service is reachable. On a cluster, use + the route or service URL. For local development, start the server first: + + ```bash + pip install "eval-hub-sdk[server,cli]" + evalhub server start + ``` + + See [Local Mode](/guides/local-mode/) for full local setup including + configuration, MLflow, and provider registration. + +4. **Submit an Evaluation** @@ -156,7 +169,7 @@ Run your first evaluation with EvalHub. -4. **Wait for Results** +5. **Wait for Results** diff --git a/src/content/docs/home.mdx b/src/content/docs/home.mdx index 2045b5e..d91cfb6 100644 --- a/src/content/docs/home.mdx +++ b/src/content/docs/home.mdx @@ -2,7 +2,7 @@ title: "EvalHub" --- -import { Card, CardGrid, LinkCard } from '@astrojs/starlight/components'; +import { Card, CardGrid, LinkCard, Steps } from '@astrojs/starlight/components'; Open source evaluation orchestration platform for Large Language Models. @@ -62,17 +62,43 @@ Community providers with a `provider.yaml` are listed in the [Provider Catalog]( ## Quick Taste -```bash -pip install "eval-hub-sdk[cli]" - -evalhub eval run \ - --name my-first-eval \ - --model-url http://localhost:11434/v1 \ - --model-name qwen2.5:1.5b \ - --provider lm_evaluation_harness \ - --benchmark mmlu \ - --wait -``` + + +1. **Install the SDK** (includes the server and CLI): + + ```bash + pip install "eval-hub-sdk[server,cli]" + ``` + +2. **Start a model server** (e.g. [Ollama](https://ollama.com)): + + ```bash + ollama run qwen2.5:1.5b + ``` + +3. **Start EvalHub locally**: + + ```bash + evalhub server start + ``` + +4. **Run an evaluation**: + + ```bash + evalhub eval run \ + --name my-first-eval \ + --model-url http://localhost:11434/v1 \ + --model-name qwen2.5:1.5b \ + --provider lm_evaluation_harness \ + --benchmark mmlu \ + --wait + ``` + + + +See the [Quick Start](/getting-started/quickstart/) for a full walkthrough or +[Local Mode](/guides/local-mode/) for advanced local configuration (MLflow, +custom server config, provider registration). ## Get Started From 1c1be81e2f964de94afc43893a2facb7549f388b Mon Sep 17 00:00:00 2001 From: Julian Payne Date: Fri, 14 Aug 2026 16:16:10 +0200 Subject: [PATCH 2/2] docs: clarify CLI setup and terminal usage in quick-start guides Add explicit base_url configuration steps so readers don't hit connection-refused errors. Home page now notes Ollama must stay running in its own terminal. Quickstart splits Step 3 into Cluster/Local tabs with server_config_file and base_url commands for each path. Assisted-by: Cursor Signed-off-by: Julian Payne Co-authored-by: Cursor --- .../docs/getting-started/quickstart.mdx | 21 ++++++++++++++++--- src/content/docs/home.mdx | 11 ++++++++++ 2 files changed, 29 insertions(+), 3 deletions(-) diff --git a/src/content/docs/getting-started/quickstart.mdx b/src/content/docs/getting-started/quickstart.mdx index 3a4b2e1..2ac21db 100644 --- a/src/content/docs/getting-started/quickstart.mdx +++ b/src/content/docs/getting-started/quickstart.mdx @@ -80,17 +80,32 @@ Run your first evaluation with EvalHub. 3. **Connect to EvalHub** - The commands below assume an EvalHub service is reachable. On a cluster, use - the route or service URL. For local development, start the server first: + + + + Point the CLI at your EvalHub route or service URL: ```bash - pip install "eval-hub-sdk[server,cli]" + evalhub config set base_url https://evalhub.apps.my-cluster.example.com + ``` + + + + + Start the server with a config file, then point the CLI at it: + + ```bash + evalhub config set server_config_file server-config.yaml evalhub server start + evalhub config set base_url http://localhost:8080 ``` See [Local Mode](/guides/local-mode/) for full local setup including configuration, MLflow, and provider registration. + + + 4. **Submit an Evaluation** diff --git a/src/content/docs/home.mdx b/src/content/docs/home.mdx index d91cfb6..13c0911 100644 --- a/src/content/docs/home.mdx +++ b/src/content/docs/home.mdx @@ -76,12 +76,23 @@ Community providers with a `provider.yaml` are listed in the [Provider Catalog]( ollama run qwen2.5:1.5b ``` + Keep Ollama running in this terminal and execute Steps 3–4 in a separate terminal. + 3. **Start EvalHub locally**: ```bash evalhub server start ``` + Point the CLI at the local server: + + ```bash + evalhub config set base_url http://localhost:8080 + ``` + + For provider registration and other advanced options (e.g. `server_config_file`), + see [Local Mode](/guides/local-mode/). + 4. **Run an evaluation**: ```bash