Skip to content

Commit b0c6f8e

Browse files
UnlikedOneUnlikedOne
authored
Merge pull request #7 from UnlikedOne/docs/redesign
Redesign docs home and correct CLI reference
2 parents 7fa8704 + ef5b36b commit b0c6f8e

5 files changed

Lines changed: 489 additions & 115 deletions

File tree

‎docs/CLI.md‎

Lines changed: 126 additions & 54 deletions
Original file line numberDiff line numberDiff line change
@@ -1,90 +1,162 @@
1-
# CLI
1+
# CLI reference
22

3-
This page describes the diffuse command line interface. It covers joining a
4-
mesh, serving a slice of a model, and running an interactive chat.
3+
Every command Diffuse ships today, with the flags that exist and the behavior you
4+
should expect. Anything not listed here does not exist yet.
55

6-
The examples assume the daemon binary is on your path. Build it from the
7-
repository root with cargo if it is not.
6+
## Install
87

9-
## Overview
8+
One command, Linux x86_64:
109

11-
Diffuse is run as a daemon. The daemon joins the gossip network, optionally
12-
serves a slice of layers, and exposes a local chat. A node needs to know at
13-
least one sentinel to reach the rest of the mesh.
10+
```bash
11+
curl -fsSL https://raw.githubusercontent.com/UnlikedOne/diffuse/main/install.sh | bash
12+
```
13+
14+
This fetches the `diffuse` binary, sets up the Python worker in `~/.diffuse/worker`,
15+
and puts everything in place. No account, no API key, no server of your own.
16+
17+
The script is [install.sh](https://github.com/UnlikedOne/diffuse/blob/main/install.sh)
18+
if you would rather read it before piping it to a shell.
1419

15-
Ports used by a node are 9440 for gossip and 10440 for compute. The compute port
16-
is always the gossip port plus 1000. Make sure both are reachable if you run a
17-
node with a public IP.
20+
## How a node is put together
1821

19-
## Global behavior
22+
Every command starts a daemon. The daemon joins the gossip network, supervises a
23+
local Python worker on port 50051 bound to localhost, and speaks to other nodes
24+
over gRPC. You never start the worker yourself.
2025

21-
Every invocation starts the daemon, which supervises the local Python worker on
22-
port 50051 bound to localhost. You do not start the worker yourself. The daemon
23-
launches it when a slice needs to be served and talks to it over gRPC.
26+
Two ports matter. Gossip runs on 9440. Compute runs on 10440, which is always the
27+
gossip port plus 1000. If you run a node with a public IP, both need to be
28+
reachable.
2429

25-
## Joining a mesh
30+
## diffuse chat
2631

27-
To join the network through the live sentinel:
32+
Interactive chat against a model served on the network.
2833

34+
```bash
35+
diffuse chat
2936
```
30-
diffuse join --sentinel 204.168.151.107
37+
38+
You pick a model from the ones currently live, and the client builds an encrypted
39+
route to the nodes serving it. Your machine tokenizes the prompt and runs the
40+
first layers locally, so the prompt itself is never transmitted. Tokens stream
41+
back one at a time as they are generated.
42+
43+
To join through a specific sentinel rather than the built-in ones:
44+
45+
```bash
46+
diffuse chat --bootstrap http://204.168.151.107:9440
3147
```
3248

33-
This connects to the sentinel, exchanges signed gossip, and builds a local view
34-
of which slices are live. Joining alone does not serve any layers. It is enough
35-
to act as a client that runs prompts against models served by others.
49+
## diffuse query
3650

37-
## Serving a slice
51+
A single question, no interactive session:
3852

39-
To contribute a slice of layers to a model:
53+
```bash
54+
diffuse query --prompt "Explain black holes in two sentences." --model <model>/<model>
55+
```
56+
57+
With an explicit model and sentinel:
4058

59+
```bash
60+
diffuse query \
61+
--prompt "Explain black holes in two sentences." \
62+
--model Qwen/Qwen2.5-0.5B-Instruct \
63+
--bootstrap http://204.168.151.107:9440
4164
```
42-
diffuse serve --model Qwen2.5-7B-Instruct --layers 8:16 --sentinel 204.168.151.107
65+
66+
Useful for scripting, and for reading the latency instrumentation on a single
67+
request without the chat UI in the way.
68+
69+
## diffuse models
70+
71+
List the models currently served by the network:
72+
73+
```bash
74+
diffuse models
4375
```
4476

45-
The layers argument is a half open range, so 8:16 serves layers 8 through 15.
46-
The daemon loads only those layers into the worker, announces the slice over
47-
gossip, and starts accepting encrypted hidden states for it.
77+
Shows each live model, the layer coverage, and how many replicas hold each slice.
78+
A model is only usable when its slices are fully covered.
4879

49-
For the model to be usable end to end, every slice must be covered by at least
50-
one live node. If a slice has no server, clients see a dead replica.
80+
## diffuse host
5181

52-
Serving from a machine with a public IP works directly. Serving from a machine
53-
behind NAT relies on the relay, which is not yet reliable across independent
54-
networks. See [Architecture](ARCHITECTURE.md) for the current status.
82+
Contribute your machine. The daemon measures how much of the model your hardware
83+
can hold, takes the slice the network needs most, loads it, and starts serving.
5584

56-
## Running a chat
85+
```bash
86+
diffuse host --model Qwen/Qwen2.5-0.5B-Instruct
87+
```
5788

58-
To open an interactive chat against a model that is fully served on the mesh:
89+
Against a specific sentinel:
5990

91+
```bash
92+
diffuse host --model Qwen/Qwen2.5-0.5B-Instruct --bootstrap http://204.168.151.107:9440
6093
```
61-
diffuse chat --model Qwen2.5-7B-Instruct --sentinel 204.168.151.107
94+
95+
The model argument is a Hugging Face model id. The slice assignment is automatic:
96+
you do not choose a layer range, the daemon picks one based on your available
97+
memory and where the network is thin.
98+
99+
To keep contributing after you close the terminal:
100+
101+
```bash
102+
nohup diffuse host --model Qwen/Qwen2.5-0.5B-Instruct > ~/.diffuse/host.log 2>&1 &
62103
```
63104

64-
The client runs the first layers of the model locally, so your prompt is never
65-
sent in the clear. Tokens stream back one at a time as they are decoded.
105+
Follow it with `tail -f ~/.diffuse/host.log`, stop it with `pkill diffuse`. For a
106+
node that survives a reboot, see `deploy/README.md` in the repository for the
107+
systemd unit.
108+
109+
## What the encryption actually does
110+
111+
This is the part worth reading carefully, because it is the reason the project
112+
exists.
113+
114+
Session keys are established with **X25519**, an elliptic-curve Diffie-Hellman
115+
exchange. Each client generates an ephemeral keypair per session, so two requests
116+
from the same person are not trivially linkable by a shared key. The public key of
117+
every node is published in gossip and **signed with Ed25519**, so a client can
118+
verify it is negotiating with the node it intends to, and forged or tampered peer
119+
records are rejected.
120+
121+
Traffic is encrypted with **ChaCha20-Poly1305**, an authenticated cipher: a node
122+
cannot silently alter a hidden state in flight without the tampering being
123+
detected on decryption. There is no certificate authority anywhere in the design;
124+
keys are bound to node identities directly.
125+
126+
The encryption is genuinely end to end between your machine and each serving node.
127+
A node in the middle of the pipeline decrypts only the hidden state addressed to
128+
it, runs its slice, and re-encrypts the result for the next hop. It never holds a
129+
key that would let it read another hop's traffic.
66130

67-
The chat prints latency instrumentation that separates compute time from network
68-
time, so you can see where time is going on your connection and hardware.
131+
Two honest caveats. This is a hand-rolled protocol built on sound primitives, and
132+
a hand-rolled protocol has not had the review a standard transport has. And
133+
encryption protects the wire, not the computation: the node running your slice
134+
sees a plaintext tensor of activations in its own memory. What that does and does
135+
not reveal is covered in [Privacy](PRIVACY.md).
69136

70-
## Running your own sentinel
137+
## Reaching nodes behind NAT
71138

72-
A sentinel is a node with a public IP that other nodes use as an entry point. To
73-
run one, start the daemon on a public host and let others point their sentinel
74-
flag at its address. On the live deployment this runs as a systemd service named
75-
diffuse on the Hetzner host.
139+
A node with a public IP is dialed directly and works today. A node behind NAT has
140+
no address a client can dial, so Diffuse routes it through a sentinel relay
141+
instead.
76142

77-
Make sure ports 9440 and 10440 are open to the internet on the sentinel host, or
78-
nodes will not be able to join or compute through it.
143+
That relay path works in a single-machine test but is **not yet reliable between
144+
independent networks**. If a node on one network hosts a model and a client on
145+
another network tries to reach it, the client currently fails with `all replicas
146+
dead` or a failure to pre-connect to `0.0.0.0:10440`. Serving from behind NAT
147+
across networks should be treated as non-functional until this is fixed.
148+
Consuming from behind NAT works fine. The design and current status are in
149+
[NAT relay design](nat-relay-design.md).
79150

80151
## Known rough edges
81152

82-
The startup banner still prints version 0.1.0 even on a v0.2.0 build. The version
83-
string is hardcoded and does not yet reflect the release tag.
153+
The startup banner still prints `version 0.1.0` on a v0.2.0 build. The version
154+
string is hardcoded and does not yet track the release tag.
84155

85-
Pressing Ctrl+C during generation currently tears down the whole chat rather
86-
than interrupting only the current generation, and can leave the Python worker
87-
process orphaned. Both are known and tracked.
156+
Pressing `Ctrl+C` during generation tears down the whole chat session instead of
157+
interrupting only the current generation, and can leave an orphaned Python worker
158+
process behind.
88159

89-
Very small models such as a 0.5B model run greedy decoding with no sampling, so
90-
they can loop on repetitive output. Sampling is planned.
160+
Very small models such as a 0.5B run greedy decoding with no sampling, so they can
161+
loop on repetitive output. Sampling is planned. This is a property of the model
162+
and the decoding strategy, not of the network carrying it.

‎docs/index.md‎

Lines changed: 20 additions & 36 deletions
Original file line numberDiff line numberDiff line change
@@ -1,36 +1,20 @@
1-
# Diffuse
2-
3-
Distributed peer-to-peer LLM inference where your prompt never leaves your
4-
device.
5-
6-
Diffuse runs a large language model across several machines that do not trust
7-
each other. The model is split vertically by layers, and each machine holds only
8-
a slice. Your own machine always runs the first layers, so the raw tokens of
9-
your prompt never leave it. Every other node sees only an encrypted hidden
10-
state.
11-
12-
This is the property the whole project is built around. It is the difference
13-
between Diffuse and other distributed inference systems, where peers can see the
14-
prompt.
15-
16-
## Status
17-
18-
The current release is v0.2.0. It ships real token by token streaming in the
19-
chat, latency instrumentation that separates compute from network, and
20-
persistent connections that cut per token latency from 498 ms to 335 ms.
21-
22-
Serving a model from a machine with a public IP works. Serving from behind NAT
23-
across independent networks does not yet work reliably and is in progress. This
24-
is stated plainly rather than hidden. See [Architecture](ARCHITECTURE.md).
25-
26-
## Start here
27-
28-
Read [Privacy](PRIVACY.md) for why the prompt stays local and what that does and
29-
does not protect. Read [Architecture](ARCHITECTURE.md) for how the system fits
30-
together. Read [CLI](CLI.md) to join a mesh, serve a slice, or run a chat. Read
31-
[Benchmarks](BENCHMARKS.md) for what has actually been measured.
32-
33-
## Principle
34-
35-
No marketing, no lies. Performance numbers are real, limitations are documented,
36-
and work in progress is labeled as such.
1+
---
2+
hide:
3+
- navigation
4+
- toc
5+
title: Diffuse
6+
---
7+
8+
<div class="dx-home">
9+
<p class="dx-home__eyebrow">📖&nbsp;&nbsp;Diffuse Documentation</p>
10+
<h1 class="dx-home__title">Your prompt never <em>leaves your device</em></h1>
11+
<p class="dx-home__lede">
12+
Architecture, privacy model, CLI reference, and measured performance for
13+
distributed peer-to-peer inference.
14+
</p>
15+
<div class="dx-home__actions">
16+
<a class="dx-btn dx-btn--primary" href="CLI/">Getting started</a>
17+
<a class="dx-btn dx-btn--ghost" href="https://github.com/UnlikedOne/diffuse">GitHub</a>
18+
</div>
19+
<div class="dx-home__rule"></div>
20+
</div>

0 commit comments

Comments
 (0)