Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 30 additions & 2 deletions src/content/docs/getting-started/quickstart.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -78,7 +78,35 @@ Run your first evaluation with EvalHub.
</TabItem>
</Tabs>

3. **Submit an Evaluation**
3. **Connect to EvalHub**

<Tabs>
<TabItem label="Cluster">

Point the CLI at your EvalHub route or service URL:

```bash
evalhub config set base_url https://evalhub.apps.my-cluster.example.com
```

</TabItem>
<TabItem label="Local">

Start the server with a config file, then point the CLI at it:

```bash
evalhub config set server_config_file server-config.yaml
evalhub server start
evalhub config set base_url http://localhost:8080
```

See [Local Mode](/guides/local-mode/) for full local setup including
configuration, MLflow, and provider registration.

</TabItem>
</Tabs>

4. **Submit an Evaluation**

<Tabs syncKey="interface">
<TabItem label="CLI">
Expand Down Expand Up @@ -156,7 +184,7 @@ Run your first evaluation with EvalHub.
</TabItem>
</Tabs>

4. **Wait for Results**
5. **Wait for Results**

<Tabs syncKey="interface">
<TabItem label="CLI">
Expand Down
61 changes: 49 additions & 12 deletions src/content/docs/home.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
title: "EvalHub"
---

import { Card, CardGrid, LinkCard } from '@astrojs/starlight/components';
import { Card, CardGrid, LinkCard, Steps } from '@astrojs/starlight/components';

Open source evaluation orchestration platform for Large Language Models.

Expand Down Expand Up @@ -62,17 +62,54 @@ Community providers with a `provider.yaml` are listed in the [Provider Catalog](

## Quick Taste

```bash
pip install "eval-hub-sdk[cli]"

evalhub eval run \
--name my-first-eval \
--model-url http://localhost:11434/v1 \
--model-name qwen2.5:1.5b \
--provider lm_evaluation_harness \
--benchmark mmlu \
--wait
```
<Steps>

1. **Install the SDK** (includes the server and CLI):

```bash
pip install "eval-hub-sdk[server,cli]"
```

2. **Start a model server** (e.g. [Ollama](https://ollama.com)):

```bash
ollama run qwen2.5:1.5b
```

Keep Ollama running in this terminal and execute Steps 3–4 in a separate terminal.

3. **Start EvalHub locally**:

```bash
evalhub server start
```

Point the CLI at the local server:

```bash
evalhub config set base_url http://localhost:8080
```

For provider registration and other advanced options (e.g. `server_config_file`),
see [Local Mode](/guides/local-mode/).

4. **Run an evaluation**:

```bash
evalhub eval run \
--name my-first-eval \
--model-url http://localhost:11434/v1 \
--model-name qwen2.5:1.5b \
--provider lm_evaluation_harness \
--benchmark mmlu \
--wait
```

</Steps>

See the [Quick Start](/getting-started/quickstart/) for a full walkthrough or
[Local Mode](/guides/local-mode/) for advanced local configuration (MLflow,
custom server config, provider registration).

## Get Started

Expand Down
Loading