Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
24 changes: 20 additions & 4 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -69,15 +69,31 @@ jobs:
# because it shares the toolchain, and a console that does not compile is a console nobody
# notices is broken until they try to open it.
#
# On this runner that means the data layer and the four command-line tools -- the tray and
# its two dependencies are declared for macOS only, since the schedules it reads are
# launchd's. Everything the tests cover is in the part that builds here.
- name: Build the console (data layer + CLI tools; the tray is macOS-only)
# On this runner that means the data layer and the four command-line tools with the default
# feature set: the tray's GUI dependencies are declared for macOS unconditionally, and for
# Linux only behind the `tray` feature (off by default), specifically so this step needs no
# GUI library at all. Everything the tests cover here is in the part that builds without it.
- name: Build the console (data layer + CLI tools; no GUI libraries needed)
working-directory: console
run: cargo build --release --locked
- name: Console unit tests
working-directory: console
run: cargo test --locked
# The Linux tray, behind its feature: exercises the `tray-icon` + GTK path this runner does
# not otherwise touch. Its own dependencies need real headers, hence the apt step -- and
# that step is confined to here so the two steps above stay proof that nothing else in this
# crate needs them.
- name: Build the Linux tray (GTK + appindicator)
working-directory: console
run: |
sudo apt-get update
# libxdo-dev is easy to miss: tray-icon's menu-accelerator handling links -lxdo, and the
# link fails without it even though nothing above mentions xdo by name.
sudo apt-get install -y libgtk-3-dev libayatana-appindicator3-dev libxdo-dev
cargo build --release --locked --features tray
- name: Linux tray unit tests
working-directory: console
run: cargo test --locked --features tray

schema:
runs-on: ubuntu-latest
Expand Down
3 changes: 3 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
@@ -1,5 +1,8 @@
# Rust build artifacts
mcp-server/target/
# CARGO_TARGET_DIR for the console crate, when set to the repo root's build/ (keeps target output
# out of an immutable-OS host's toolbox checkout and in one place regardless of which crate built).
/build/

# Python
__pycache__/
Expand Down
49 changes: 33 additions & 16 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -270,10 +270,14 @@ holds, whether the scheduled passes are running, and what the tunables are set t
writing a query.

```bash
cd console && cargo build --release # two dependencies, both only for the tray
cd console && cargo build --release # the CLI tools; zero dependencies
./target/release/hypermnesia-setup # the walkthrough: from nothing to a menu-bar icon
```

The tray itself needs a GUI toolkit, so it is not part of that plain build: unconditional on
macOS, and on Linux behind `cargo build --release --features tray` (GTK3 +
libayatana-appindicator, see [docs/INSTALL.md](docs/INSTALL.md)).

It reaches the database exactly one way: a command that receives SQL on stdin. Direct psql,
`docker exec`, `kubectl exec`, ssh to a machine that has kubectl — all of them are one string with
different contents, which is why there is one setting and not five. The wizard tries the command
Expand All @@ -282,7 +286,7 @@ cannot be told from an empty store.

| Command | What |
|---------|------|
| `hypermnesia` | the menu-bar tray (macOS) |
| `hypermnesia` | the menu-bar tray (macOS) / system tray (Linux, `--features tray`) |
| `hypermnesia-stats` | the same numbers on stdout |
| `hypermnesia-jobs` | scheduled passes: what is configured, when each last worked, run one now, change a schedule |
| `hypermnesia-settings` | the tunables, each with the value in force and where that value came from |
Expand All @@ -295,33 +299,46 @@ cannot be told from an empty store.
</p>

A menu, not a window. Everything the console has to show is a dozen lines and a dozen buttons; a
window would mean a GUI framework for the same result. Two dependencies, both macOS-only, and the
data layer under them has none at all — a console that takes a minute to build is a console nobody
rebuilds.
window would mean a GUI framework for the same result. The tray's GUI dependencies are declared
per platform and behind a feature on Linux, and the data layer under them has none at all — a
console that takes a minute to build is a console nobody rebuilds.

What is in the menu:

- **the first line is the age of what you are reading.** Not a status light: "updated 4m ago, in
1.2s", and after a failure "! not updated: <reason>, showing state from 12m ago". It is computed
when the menu is drawn, from the wall clock, so a Mac that slept for two hours says two hours.
The store's own numbers refresh every minute; the jobs — a local, cheap read on either platform
— refresh every few seconds, faster still for a little while after a button is pressed, so
pressing one does not mean waiting out the idle cadence to see whether it worked.
- **the volumes** — memories active of total, knowledge pages, documents and chunks, how many
chunks have no embedding, how many embedding models are in the store, the review queue, the
stale count, the database size.
- **Run now** — every scheduled pass, with its schedule, when it last wrote to its log, and its
last exit code. Pressing one runs it through launchd and reports what launchd then did, not that
the request was accepted.
- **Schedule** — the common intervals and times, per job. It edits the plist, validates it, and
reloads the job, because launchd keeps its own copy from the moment it loaded it: writing the
file without the reload would show a new schedule while the old one is in force.
last exit code. Pressing one runs it — through launchd on macOS, through `systemctl --user start
--no-block` on Linux — and reports what the service manager then did, not that the request was
accepted: a program that is not there fails immediately, and that is read back rather than taken
on faith.
- **Schedule** — the common intervals and times, per job. It writes the unit, validates it, and
reloads the job, because both launchd and systemd keep their own copy from the moment they
loaded it: writing the file without the reload would show a new schedule while the old one is
still in force. A backup goes down next to the file first, and on any slip along the way it is
restored — what actually landed is read back and compared to what was asked before the tray
calls it done.
- **Timers** (Linux only) — arm or disarm a job's schedule without removing it. systemd tracks
"loaded but not armed" as a state of its own, which the read side of this tray could already
show as a fault; this is what fixes it. launchd has no such state — a job it has loaded runs on
its schedule, full stop — so there is nothing here to switch on macOS.
- **Refresh now** and **Quit**.

The menu-bar title carries one mark, `HM !`, and it is derived from the lines below rather than
computed beside them: anything the menu would show with a `!` puts the mark in the title. That is
the whole design in one detail — the icon is where a problem is noticed, so the icon must not be
able to disagree with the menu.
The tray carries one mark for "something here needs a look", derived from the lines below rather
than computed beside them: anything the menu would show with a `!` raises it. That is the whole
design in one detail — the mark is where a problem is noticed, so it must not be able to disagree
with the menu. On macOS the mark is in the menu-bar title, `HM !`; a system tray icon has no text
of its own, so on Linux it is the icon itself, calm green or warned red.

`hypermnesia --install` puts it in launchd to start at login, restarted if it crashes and not if
you quit it. `--uninstall` takes it back out.
`hypermnesia --install` puts it in autostart to start at login, restarted if it crashes and not if
you quit it — launchd on macOS, a systemd user unit on Linux. `--uninstall` takes it back out.

**Try it without a database.** The console's only connection setting is a command that prints the
query's answer, so a file works:
Expand Down
2 changes: 2 additions & 0 deletions console/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

13 changes: 13 additions & 0 deletions console/Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,19 @@ license = "MIT"
tray-icon = "0.25"
winit = "0.30"

# The Linux tray, behind a feature and off by default -- for the same reason the macOS deps are
# per-target: the data layer and the four command-line tools must keep building on a machine with
# no GUI libraries at all, which is what CI's bare runner is. `tray-icon` drives the same crate as
# macOS, through `libayatana-appindicator`; on Linux it needs a GTK main loop rather than winit's,
# so `gtk` is a dependency here and `winit` is not.
[target.'cfg(target_os = "linux")'.dependencies]
tray-icon = { version = "0.25", optional = true }
gtk = { version = "0.18", optional = true }
glib = { version = "0.18", optional = true }

[features]
tray = ["dep:tray-icon", "dep:gtk", "dep:glib"]

[[bin]]
name = "hypermnesia-stats"
path = "src/bin/stats.rs"
Expand Down
108 changes: 79 additions & 29 deletions console/src/bin/jobs.rs
Original file line number Diff line number Diff line change
@@ -1,8 +1,12 @@
//! The pipeline's scheduled jobs: show them and run them.
//! The pipeline's scheduled jobs: show them, run them, and change when they run.
//!
//! hypermnesia-jobs what is configured and when it last worked
//! hypermnesia-jobs run <short> run now, without touching the schedule
//! hypermnesia-jobs log <short> the last lines of the log
//! hypermnesia-jobs every <short> 4h an interval schedule
//! hypermnesia-jobs at <short> 05:30 a calendar schedule
//! hypermnesia-jobs enable <short> arm the schedule (systemd only)
//! hypermnesia-jobs disable <short> disarm it without removing it (systemd only)
//!
//! The short name is the tail of the label: extract, consolidate, reflect, freshness, rerank.

Expand All @@ -11,15 +15,15 @@ use hypermnesia_console::jobs::{self, Job};
fn main() {
let args: Vec<String> = std::env::args().skip(1).collect();
// Through jobs::job_prefix rather than the variable: only that path also consults the
// config file, and the tray -- started by launchd with a minimal environment -- has
// nothing else to read.
// config file, and the tray -- started by the service manager with a minimal environment --
// has nothing else to read.
let prefix = jobs::job_prefix();
let all = match jobs::list(&prefix) {
Ok(j) => j,
Err(e) => fail(&e),
};
if all.is_empty() {
fail(&format!("no job with the prefix {prefix} found in ~/Library/LaunchAgents"));
fail(&format!("no job with the prefix {prefix} found in {}", jobs::UNITS_LOCATION));
}

match args.first().map(String::as_str) {
Expand Down Expand Up @@ -69,7 +73,21 @@ fn main() {
let Some((hour, minute)) = parse_hhmm(time) else { fail("a time is written as HH:MM") };
apply(find(&all, name), &jobs::Schedule::at(hour, minute, day));
}
Some("-h") | Some("--help") => print!("{HELP}"),
Some("enable") => {
let Some(name) = args.get(1) else { fail("give the job's short name") };
match jobs::set_enabled(find(&all, name), true) {
Ok(msg) => println!("{msg}"),
Err(e) => fail(&e),
}
}
Some("disable") => {
let Some(name) = args.get(1) else { fail("give the job's short name") };
match jobs::set_enabled(find(&all, name), false) {
Ok(msg) => println!("{msg}"),
Err(e) => fail(&e),
}
}
Some("-h") | Some("--help") => print!("{}", help_text()),
Some(other) => fail(&format!("unknown command: {other}")),
}
}
Expand Down Expand Up @@ -116,9 +134,12 @@ fn apply(j: &Job, sched: &jobs::Schedule) {
Ok(msg) => {
println!("{msg}");
println!("was: {was}");
// launchd keeps its own copy of the schedule from the moment it loaded the job, so the
// job is reloaded. That resets the run counter -- which has to be said out loud, or
// the next look at the list will be alarming: a zero where everything is fine.
// The service manager keeps its own copy of the schedule from the moment it loaded
// the job, so the job is reloaded. That resets whatever run counter it keeps -- which
// has to be said out loud on the platform that has one, or the next look at the list
// will be alarming: a zero where everything is fine. systemd keeps no such counter
// (`Job::runs` is always `None` there), so there is nothing to warn about on Linux.
#[cfg(target_os = "macos")]
println!("the job was reloaded into launchd; the run counter started over");
}
Err(e) => fail(&e),
Expand Down Expand Up @@ -159,10 +180,11 @@ fn parse_weekday(s: &str) -> Option<u32> {

fn find<'a>(all: &'a [Job], name: &str) -> &'a Job {
match all.iter().find(|j| j.short() == name || j.label == name) {
Some(j) if j.broken() => fail(&format!(
"{name}: this plist cannot be read ({}). launchd may still be running the copy it \
loaded before the file broke -- what it will do at the next login is the question. \
Fix the file first.", j.fault.clone().unwrap_or_default())),
Some(j) if j.unwritable() => fail(&format!(
"{name}: this {} cannot be read ({}). {} may still be running the copy it loaded \
before the file broke -- what it will do at the next login is the question. Fix \
the file first.", jobs::UNIT_NOUN, j.fault.clone().unwrap_or_default(),
jobs::BACKEND_NAME)),
Some(j) => j,
None => fail(&format!("no such job: {name}. There is: {}",
all.iter().map(Job::short).collect::<Vec<_>>().join(", "))),
Expand All @@ -178,10 +200,11 @@ fn list(all: &[Job]) {
println!("{:<13} {:<22} {:<14} {}", "JOB", "SCHEDULE", "LAST OUTPUT", "STATE");
for j in all {
if let Some(why) = &j.fault {
// An unreadable plist is one launchd also refused at login: the job is not running.
// It used to be dropped from the list entirely, which is the one case the console had
// nothing at all to say about.
println!("{:<13} {:<22} {:<14} ! unreadable plist: {why}", j.short(), "—", "—");
// An unreadable unit is one the service manager also refused: the job is not
// running. It used to be dropped from the list entirely, which is the one case the
// console had nothing at all to say about.
println!("{:<13} {:<22} {:<14} ! unreadable {}: {why}", j.short(), "—", "—",
jobs::UNIT_NOUN);
continue;
}
let when = match j.since_last_output() {
Expand All @@ -198,7 +221,7 @@ fn list(all: &[Job]) {
None => "never ran".to_string(),
}
} else if j.last_exit.is_none() {
"not loaded into launchd".to_string()
format!("not loaded into {}", jobs::BACKEND_NAME)
} else if j.runs == Some(0) {
// Loaded, and launchd has not started it since this login. Its exit column says 0,
// which it prints both for "finished successfully" and for "never finished at all" --
Expand All @@ -213,7 +236,7 @@ fn list(all: &[Job]) {
// has nothing to tell them apart by.
Some(0) => format!("last run ok{}", runs_note(j)),
Some(c) => format!("last run: code {c}{}", runs_note(j)),
None => "not loaded into launchd".to_string(),
None => format!("not loaded into {}", jobs::BACKEND_NAME),
}
};
println!("{:<13} {:<22} {:<14} {}", j.short(), j.schedule.human(), when, state);
Expand All @@ -227,7 +250,7 @@ fn list(all: &[Job]) {
// job installed later than its own last slot honestly shows a zero and is not a
// complaint: a false alarm here would teach one to scroll past this line.
let what = if j.never_ran() {
"the period has already passed and launchd still never ran it"
"the period has already passed and it still never ran"
} else {
"it ran at some point, but the log has not moved for more than a period"
};
Expand All @@ -237,6 +260,9 @@ fn list(all: &[Job]) {
println!();
println!("run <name> — run now; log <name> [N] — the tail of the log; \
every/at <name> … — change the schedule (--help)");
if jobs::SUPPORTS_ENABLE {
println!("enable/disable <name> — arm or disarm the schedule without removing it");
}
}

fn runs_note(j: &Job) -> String {
Expand All @@ -254,22 +280,46 @@ fn ago(d: std::time::Duration) -> String {
else { format!("{} d ago", s / 86400) }
}

const HELP: &str = "\
hypermnesia-jobs — the memory pipeline's scheduled jobs (launchd).
// One text, not two: both backends now write, validate, reload and read back a schedule, and
// differ only in the nouns `jobs::UNIT_NOUN`/`BACKEND_NAME`/`UNITS_LOCATION` already carry.
// `enable`/`disable` is the one command that is not portable -- launchd has no separate "loaded
// but not armed" state, so it is documented as systemd-only rather than interpolated away.
#[cfg(target_os = "macos")]
const RUN_NOW: &str = "run immediately (launchctl kickstart -k)";
#[cfg(target_os = "linux")]
const RUN_NOW: &str = "run immediately (systemctl --user start --no-block)";
#[cfg(not(any(target_os = "macos", target_os = "linux")))]
const RUN_NOW: &str = "run immediately";

fn help_text() -> String {
// launchd has no separate "loaded but not armed" state -- a job it has loaded is armed, full
// stop -- so `enable`/`disable` is documented only where `jobs::SUPPORTS_ENABLE` says it does
// something, rather than interpolating a noun into a paragraph that would be false on macOS.
let enable = if jobs::SUPPORTS_ENABLE {
format!(" hypermnesia-jobs enable <name> arm the schedule\n\
\x20 hypermnesia-jobs disable <name> disarm it without removing it\n")
} else {
String::new()
};
format!("\
hypermnesia-jobs — the memory pipeline's scheduled jobs ({backend}).

hypermnesia-jobs what is configured, when it worked, the last exit code
hypermnesia-jobs run <name> run immediately (launchctl kickstart -k)
hypermnesia-jobs log <name> [N] the last N lines of the log (40 by default)
hypermnesia-jobs run <name> {run_now}
hypermnesia-jobs log <name> [N] the last N lines of the log (40 by default), where one \
is configured
hypermnesia-jobs every <name> 4h an interval schedule (30s, 15m, 4h, 2d)
hypermnesia-jobs at <name> 05:30 daily at that time
hypermnesia-jobs at <name> mon 06:10 weekly on that day

Editing a schedule writes the plist, validates it and reloads the job into launchd: without the
reload launchd keeps working from its own copy, and the new schedule would be displayed where the
old one is in force. Before the edit a .bak is put down next to the file, and on any slip the
{enable}
Editing a schedule writes the {noun}, validates it and reloads the job into {backend}: without
the reload {backend} keeps working from its own copy, and the new schedule would be shown where
the old one is in force. Before the edit a .bak is put down next to the file, and on any slip the
file is restored.

The name is the tail of the label: extract, consolidate, reflect, freshness, rerank.
The label prefix comes from HM_JOB_PREFIX, then the console config file, then the
default com.hypermnesia.
";
default {prefix}.
", backend = jobs::BACKEND_NAME, run_now = RUN_NOW, noun = jobs::UNIT_NOUN,
prefix = jobs::DEFAULT_JOB_PREFIX)
}
Loading
Loading