Skip to content

API.md

Latest commit

 

History

History
618 lines (479 loc) · 26.6 KB

API.md

File metadata and controls

618 lines (479 loc) · 26.6 KB
Raw

crosspoint-sync API

The contract for firmware (CrossPoint / CrossInk) and any other client. Two API surfaces share one account system and the same auth headers:

  1. KOSync-compatible API at the root — byte-compatible with sync.koreader.rocks. Stock KOReader and current CrossPoint firmware work by changing only the server URL.
  2. Extended API under /api/v1 — lossless CrossPoint positions, bookmarks, clippings, reading stats, and document metadata.

All requests and responses are JSON (Content-Type: application/json). The Accept: application/vnd.koreader.v1+json header is accepted but never required.

Authentication

Every endpoint except POST /users/create and GET /healthz requires:

x-auth-user: <username>
x-auth-key:  <MD5 hex of the account password>

The key is the lowercase 32-hex MD5 of the plain password — exactly what KOReader and CrossPoint already send. The server never sees the plain password and stores only a salted PBKDF2 of the key.

Error responses

Errors reuse the stock kosync codes so existing client error handling keeps working:

HTTP code Meaning
401 2001 Missing/invalid auth headers
402 2002 Username already registered
403 2003 Invalid request body / registration disabled
403 2004 Missing or malformed document
429 2001 Rate limited (registration / repeated auth failures)

Body shape is always {"code": 2001, "message": "Unauthorized"}.

Document identity

document is an opaque key chosen by the client, up to 64 chars of [A-Za-z0-9._-]. CrossPoint sends the KOReader 32-hex MD5 (partial-binary or filename method, per device setting). The server never tries to unify the two hash methods; a book synced under both appears as two documents.

KOSync-compatible API

POST /users/create

Open registration (disable with REGISTRATION_DISABLED=true).

// request
{"username": "justin", "password": "0f359740bd1cda994f8b55330c86d845"}
// 201 response
{"username": "justin"}

Usernames: 1–64 chars of [A-Za-z0-9._@+-]. The password field is the MD5 auth key.

GET /users/auth

Validates credentials. 200 {"authorized": "OK"} or 401.

PUT /syncs/progress

// request
{
  "document": "a1b2c3d4e5f60718293a4b5c6d7e8f90",
  "progress": "/body/DocFragment[8]/body/div[2]/p[4]/text()[1].96",
  "percentage": 0.4867,
  "device": "CrossPoint",
  "device_id": "crossink-device"
}
// 200 response
{"document": "a1b2c3d4e5f60718293a4b5c6d7e8f90", "timestamp": 1752345678}

  • One row is stored per (user, document, device_id) — unlike stock kosync, devices never overwrite each other. If device_id is absent, device is used as the device key.

  • percentage is a float in [0, 1]. timestamp is unix seconds (integer).

  • Superset capture: the body may include the extended position object (see PUT /api/v1/progress) and/or a metadata object (below). If present and valid they are stored; unknown fields are ignored. This lets firmware ship rich sync against this endpoint with a one-line change.

  • Metadata capture (KOReader PR #15306 shape, sent by CrossPoint when the "send metadata" setting is on):

    "metadata": {
  "filename": "Foundryside - Robert Jackson Bennett.epub",
  "title": "Foundryside",
  "authors": "Robert Jackson Bennett",
  "bookfusion_id": "36835"
}

    All fields optional strings (≤512 chars). filename/title/authors are stored per (user, document); fields the client omits never overwrite previously stored values. Retrieve via GET /api/v1/documents or joined into GET /api/v1/progress.

    Service ids. Any extra <service>_id field (from a CrossPoint plugin's book sidecar, e.g. bookfusion_id) pre-seeds an exact match for the connector of that name: progress is pushed straight to that record, skipping title/author search. Unknown-connector ids are ignored; a user's manual match is never overridden.

    BookFusion positions. For a BookFusion download identified by its sidecar, the server retrieves the EPUB and resolves the existing KOSync XPath against its chapter XHTML. It sends the resulting CFI, chapter index, and text-based spine-normalized position to BookFusion. This requires no additional firmware fields. Precision follows the supplied XPath: a text-node offset identifies that text position; a paragraph/chapter-only XPath identifies its start. An unresolved XPath fails without posting an estimated position.

    When a device requests GET /syncs/progress/:document or GET /api/v1/progress/:document, the server first fetches the matched BookFusion book's latest position, resolves its point CFI in the EPUB, and then reads the stored progress for the response. This requires a linked, enabled account and a previously received sidecar match; a GET cannot discover metadata that the device has not sent. BookFusion is not polled in the background. Concurrent requests for the same user's book share a lookup; subsequent requests check the provider again.

    Refreshes have a 10-second total deadline. A timeout returns HTTP 504; provider or conversion failures return HTTP 502. Stored progress is preserved and the next GET retries the refresh. Books without an enabled sidecar match use the normal KOSync response. No firmware update is needed. Conversion uses the provider's update timestamp, skips older positions, and preserves text offsets (converting UTF-16 CFI offsets to CrossPoint codepoints). Delayed outbound events also check the remote timestamp before posting, to avoid replacing newer BookFusion progress. The provider does not offer an atomic conditional update, so a simultaneous edit between that check and the POST is still possible.

    BookFusion's storage ignores Range requests, so an EPUB is downloaded whole (limited to 200 MiB and 30 seconds), and only its text entries (container, package, and XHTML) are decompressed; each is limited to 1 MiB. The server then stores a redacted position map per linked book in epub_maps: chapter text is replaced with same-length filler and only id attributes are kept, so no readable book text is stored, yet positions resolve identically. Later syncs use the map without downloading; a position that fails against it triggers one fresh download and rebuild (the book may have changed). Maps are removed when the connector is unlinked or the account deleted, and pruned when their book is no longer matched or unused for 30 days. Books matched by title or manual selection retain percentage-only outbound updates and are excluded from inbound position conversion because those matches do not establish that the reader has the same EPUB edition.

GET /syncs/progress/{document}

Returns the newest progress across all of the user's devices (most recent timestamp wins):

{
  "document": "a1b2c3d4e5f60718293a4b5c6d7e8f90",
  "progress": "/body/DocFragment[8]/body/div[2]/p[4]/text()[1].96",
  "percentage": 0.4867,
  "device": "CrossPoint",
  "device_id": "crossink-device",
  "timestamp": 1752345678
}

Quirk (stock-compatible): if no progress exists, the response is 200 with {} — not 404. KOReader clients check for field presence, not status codes.

Extended API (/api/v1)

Same auth headers. Designed for the ESP32: responses are small (< 8 KB at default limits), keys are short, batches are capped.

Delta-sync protocol (bookmarks & clippings)

  • GET ...?since=<unix>&limit=<n> returns items with updated_at > since, oldest first, including tombstones ("deleted": 1). limit defaults to 50, max 100.
  • The response carries until (persist it as the next since cursor) and more (repeat with since = until until more is false).
  • PUT batches set updated_at on the server clock — the server is the last-write-wins authority because device RTCs drift. A delete is {"id": "...", "deleted": 1} (no other fields needed). Deleted items keep tombstones forever so late-syncing devices converge.
  • Merge semantics: set-union of ids, last-write-wins per id.

Rich progress

PUT /api/v1/progress

The kosync body plus a position object mapping 1:1 to the firmware's CompactPosition:

{
  "document": "a1b2c3d4e5f60718293a4b5c6d7e8f90",
  "progress": "/body/DocFragment[8]/body/div[2]/p[4]/text()[1].96",
  "percentage": 0.4867,
  "device": "CrossPoint",
  "device_id": "A1B2C3D4",
  "position": {
    "pctQ": 486700,
    "spine": 7,
    "page": 143,
    "pages": 412,
    "para": 96,
    "li": 0,
    "anchor": "ch08-sec2",
    "xpath": "/body/DocFragment[8]/body/div[2]/p[4]/text()[1].96"
  }
}
field type notes
pctQ uint percentage × 1,000,000 (0–1,000,000). Layout-independent; authoritative.
spine uint16 spine (chapter) index
page / pages uint16 page within spine / spine page count — layout hints, depend on font settings
para uint16, optional synthetic 1-based paragraph index; omit when unavailable (hasParagraphIndex=false)
li uint16, optional running <li> count; omit when unavailable
anchor string ≤ 48 bytes, optional nearest <a id> anchor
xpath string ≤ 120 bytes, optional KOReader-style xpath

Response: {"document": "...", "timestamp": 1752345678}.

An invalid position is ignored (the kosync fields still sync); a missing position on a later PUT keeps the previously stored one for that device.

GET /api/v1/progress

Lists every synced document — newest progress across devices, joined with any stored metadata. This is how clients/UIs discover documents without knowing hashes. ?limit= defaults to 100 (max 500), ordered newest-first.

{
  "items": [
    {"document": "25f8abb4f4f5594f02f361726814fea1",
     "title": "Foundryside", "author": "Robert Jackson Bennett",
     "filename": "Foundryside - Robert Jackson Bennett.epub",
     "percentage": 0.2853, "progress": "/body/DocFragment[16]/body/div[1]/p[143]",
     "device_id": "crosspoint-reader", "device": "CrossPoint", "timestamp": 1783913361}
  ]
}

title/author/filename are null until some client sends metadata for that document. Items also carry status, status_at (when a manual status was set, else null), cover_url, page_count, and book details from Hardcover's catalog when the server sets HARDCOVER_API_KEY: hardcover_slug, moods, genres, content_warnings (lists), rating, series, series_position, release_year (empty lists / null until looked up; lookups run in the background, a few books per list request, one per distinct book across users).

GET /api/v1/progress/{document}

All device rows, newest first — the client decides what to apply:

{
  "document": "a1b2c3d4e5f60718293a4b5c6d7e8f90",
  "devices": [
    {"device_id": "A1B2C3D4", "device": "CrossPoint", "percentage": 0.4867,
     "progress": "...", "position": { "pctQ": 486700, "spine": 7, "page": 143, "pages": 412 },
     "timestamp": 1752345678},
    {"device_id": "koreader-boox", "device": "boox", "percentage": 0.41,
     "progress": "...", "position": null, "timestamp": 1752300000}
  ]
}

position is null for rows written by plain kosync clients.

DELETE /api/v1/progress/{document}

Removes a synced book completely. Deletes the kosync progress for every device plus everything else stored server-side for that book: position samples, bookmarks, clippings, per-book reading stats, connector matches and any queued connector events. Document metadata goes too, so the book disappears from GET /api/v1/progress and GET /api/v1/documents, and GET /syncs/progress/{document} goes back to returning {}.

{"document": "a1b2c3d4e5f60718293a4b5c6d7e8f90", "deleted": true, "rows": 14}

rows is the number of database rows removed. Returns 404 {"code": 2003, "message": "Unknown document"} when the user has no data for that document.

Bookmarks and clippings are hard-deleted rather than tombstoned — there is no book left to delta-sync against. A device that still holds the file simply re-uploads its state on the next sync, so this is a server-side reset, not a device-side delete.

Bookmarks

Item ids are client-derived: id = first 16 hex chars of SHA-256(xpath). Deterministic, so re-adding the same bookmark is idempotent and delete-by-id needs no new device-side state. All CrossPoint devices must compute ids identically.

GET /api/v1/bookmarks/{document}?since=&limit=

{
  "document": "a1b2c3d4e5f60718293a4b5c6d7e8f90",
  "until": 1752345678,
  "more": false,
  "items": [
    {"id": "9f86d081884c7d65",
     "xpath": "/body/DocFragment[3]/body/p[12]/text().0",
     "percentage": 0.35,
     "summary": "It was the best of times, it was the worst of",
     "si": 3, "pc": 120, "pp": 42,
     "deleted": 0, "updated_at": 1752340000}
  ]
}

si/pc/pp are CrossPoint's spine index / chapter page count / page-in-chapter hints (nullable — layout dependent).

PUT /api/v1/bookmarks/{document}

Batch upsert, max 50 items:

{"items": [
  {"id": "9f86d081884c7d65", "xpath": "...", "percentage": 0.35,
   "summary": "...", "si": 3, "pc": 120, "pp": 42},
  {"id": "deadbeef00112233", "deleted": 1}
]}

Response: {"until": 1752345679, "accepted": 2}.

Clippings

Mirrors CrossInk's Clipping struct (src/ClippingStore.h) 1:1. Item ids are client-derived: id = first 16 hex chars of SHA-256(created_at_decimal + text) — both fields are immutable after creation.

Item shape:

{
  "id": "c0ffee0011223344",
  "spine": 7,
  "start_page": 12, "end_page": 13, "pages": 40,
  "start_word": 5, "end_word": 22, "words": 30,
  "para": 96,
  "chapter": "Chapter 8",
  "text": "So we beat on, boats against the current...",
  "note": null, "color": null,
  "created_at": 1752300000,
  "deleted": 0, "updated_at": 1752340000
}
  • para optional — omit when the firmware has paragraphIndex == UINT16_MAX.
  • Page/word fields are layout hints, kept verbatim for CrossInk↔CrossInk restore; para + text are the portable anchors.
  • text ≤ 4096 UTF-8 bytes, chapter ≤ 64 chars, note ≤ 4096 bytes.
  • note and color are preserved when omitted from updates.
  • No per-book cap server-side; the current firmware stores up to 256 active clippings per book.

Endpoints: GET /api/v1/clippings/{document}?since=&limit= and PUT /api/v1/clippings/{document} (batch max 50).

Clipping clients should use GET ...?cursor=0&limit=1&format=reader. Responses advertise sync_version: 2, include each item's revision, and return a monotonic cursor for the next page. Continue until more is false. This avoids losing items that share a second-resolution timestamp. format=reader omits note and color to bound firmware response memory. Legacy since requests remain supported.

start_offset and end_offset are optional chapter-visible Unicode-codepoint positions, [start, end), independent of pagination. Supply both or neither. layout_signature preserves compatibility with older page-local word ranges. A deleted ID cannot be resurrected by a delayed upload; create a new ID to save the same quote again. Enable Sync Clippings in the firmware's CrossPoint Sync settings to sync clippings alongside the current book's manual progress sync.

Reading stats

Model (matches the firmware's nearby P2P stats sync): each device uploads its own snapshot; snapshots replace, never accumulate; the server aggregates across devices on read. Never merge another device's stats into local ones.

PUT /api/v1/stats/global

The device's GlobalReadingStats as JSON:

{
  "device_id": "A1B2C3D4", "device": "CrossInk",
  "v": 5,
  "sessions": 312, "seconds": 184300, "pages": 9120, "completed": 14,
  "tod": [1200, 84000, 60100, 39000],
  "dow": [8000, 9000, 7000, 11000, 12000, 60000, 77300],
  "anchor_day": 9650,
  "history_b64": "<base64 of the 92-byte reading-history bitmap>",
  "streak": 21
}
  • tod = seconds in [morning, afternoon, evening, night]; dow = Mon..Sun seconds.
  • anchor_day = days since 2000-01-01 of the bitmap's most recent day; bit N of the bitmap = anchor_day - N (LSB-first within each byte, 730 days / 92 bytes).
  • streak = device-reported all-time longest streak (may predate the bitmap window).

Response: {"until": 1752345678}.

PUT /api/v1/stats/books

Batch (max 20) of per-book stats_v5 snapshots for one device:

{"device_id": "A1B2C3D4", "items": [
  {"document": "a1b2c3d4e5f60718293a4b5c6d7e8f90", "v": 5,
   "sessions": 9, "seconds": 8400, "pages": 310, "completed": false,
   "avg_fwd": 12, "pace_n": 250, "eta": 5400,
   "start_manual": false, "finish_manual": false,
   "start_date": 1751000000, "finished_date": 0,
   "tod": [0, 3000, 4000, 1400], "dow": [0, 0, 1200, 0, 2000, 3000, 2200]}
]}

GET /api/v1/stats/summary

Server-side aggregate across all of the user's devices:

  • scalars and tod/dow are summed;
  • history bitmaps are OR-ed after re-anchoring to the newest anchor_day (each device's bits shift by its anchor delta; days older than the 730-day window drop);
  • streak = max(longest run in the combined bitmap, any device-reported streak);
  • current_streak = run ending at the anchor day (the anchor day itself may be unset — today isn't over).
{
  "sessions": 402, "seconds": 190300, "pages": 9500, "completed": 15,
  "tod": [...], "dow": [...],
  "anchor_day": 9650, "history_b64": "...", "streak": 21, "current_streak": 6,
  "devices": [{"device_id": "A1B2C3D4", "device": "CrossInk", "updated_at": 1752345678}]
}

GET /api/v1/stats/global

Raw per-device snapshots ({"devices": [{"device_id", "device", "updated_at", "stats": {...}}]}) for clients that render per-device numbers like the P2P screen does.

GET /api/v1/stats/books/{document}

Per-device rows plus a combined object (scalars/buckets summed, completed OR-ed, avg_fwd weighted by pace_n, start_date = earliest non-zero, finished_date = latest).

Merged documents (POST /api/v1/documents/merge, {"document", "into", "stats"?}): stats rows stay under the hash each device uploaded them with and are combined on read. With "stats": true (the default) the alias's rows count toward the canonical book, and each row carries its document; with "stats": false they are left out. Unmerging hands the alias its stats back.

Documents metadata (optional)

PUT /api/v1/documents — {"items": [{"document": "...", "title": "...", "author": "...", "filename": "...", "filesize": 812345}]} (max 50); omitted fields never overwrite stored values. GET /api/v1/documents lists them. Most clients don't need this endpoint — progress-PUT metadata capture populates the same table.

PUT /api/v1/documents/{document}/status

Manual reading status: {"status": "reading" | "paused" | "finished" | "dnf"}, or {"status": null} to go back to deriving it from progress. finished also fans out a finished event to linked write-connectors. GET /api/v1/progress items carry the effective status (manual, else finished at ≥ 98%, else reading) and cover_url.

PUT /api/v1/documents/{document}/dates

Manual reading dates: {"start_date": "2026-08-11", "finished_date": "2026-08-16"} as calendar dates (YYYY-MM-DD, no later than tomorrow). An omitted field is left alone; null clears it back to the CrossInk / sync-history date. A start after the finish is a 400. Manual dates win over device and derived dates in /stats/activity (which flags them with start_manual / finish_manual). Setting a finish date also marks the book finished and fans out a finished event dated that day. Response: {"document", "start_date", "finished_date", "status"} with dates as unix seconds at UTC midnight.

GET /api/v1/documents/{document}/cover

{"url": "https://..." | null, "pages": 433 | null}: cover URL and print-edition page count, resolved from the document's title/author and cached (a miss is retried after 7 days). Covers come from iTunes Search (ebooks), then Open Library; page counts from Open Library, then Google Books when the server has GOOGLE_BOOKS_API_KEY, then Amazon's "Print length" via SearchAPI when it has SEARCHAPI_KEY. GET /api/v1/progress items also carry page_count.

PUT /api/v1/documents/{document}/info

Manual fixes when the lookup got it wrong: {"cover_url": "https://...", "page_count": 433} (either field optional). Manual values stick, since lookups only fill blanks; null clears a field so it's looked up again. Returns the stored {document, cover_url, page_count}.

GET /api/v1/documents/{document}/cover/candidates?q=

Up to 12 covers to choose from ({"items": [{url, title, author, source, pages}]}), best matches first, from Apple Books and Open Library. q searches a different title.

GET /api/v1/clippings

Every live clipping across all books, newest first (max 5000): {"items": [{document, id, spine, para, start_offset, chapter, text, note, created_at}]}. Read-only convenience for apps; devices keep using the per-document delta sync.

GET /api/v1/stats/activity?tz=

Pages and books derived from the progress history, for readers that never send stats (stock CrossPoint, KOReader). Every progress change is logged server-side. tz is the client's Date#getTimezoneOffset() so days bucket locally.

{"pages_total": 1923,
 "books": [{"document": "...", "started_at": 1790000000, "last_at": 1790600000, "percentage": 0.997,
            "finished_at": 1790600000, "page_count": 400, "pages_read": 399,
            "days_to_finish": 7}],
 "days": [{"day": "2026-09-28", "pages": 56, "syncs": 4,
           "books": [{"document": "...", "pages": 56, "syncs": 4, "from": 0.41, "to": 0.55}]}]}

Pages are print pages (furthest percent x page count), not screen pages. A day's pages counts only forward progress after a book's first logged sync, as a running max across devices; syncs counts every logged sync that day, so any day with a sync is a reading day. books breaks the day down per book for a reading timeline; from/to are the furthest position before and after that day. History from before the log existed is backfilled from progress samples (migration 0013). started_at is CrossInk's start_date (pinned to local noon on that day) when a device reported one, else the first sync. finished_at is CrossInk's finished_date (pinned to local noon on that day) when a device reported one, else the first sync at

= 98%, overridden by a manual status. A CrossInk finish day with no sync still gets a books entry (syncs: 0) so the timeline can show it. days_to_finish counts calendar days from start to finish, both included (a one-day read is 1); it is null when unfinished, finished only by a manual status, or already finished on the first sync with no CrossInk start date. This never estimates reading time.

Connectors (master sync hub)

Pair external services (Hardcover, Readwise, …) to the account; reading activity fans out to them server-side. See docs/design/sync-hub.md. All under /api/v1, same auth headers. Requires the server to have TOKEN_ENC_KEY set (credentials are encrypted at rest) — otherwise these endpoints report encryption: "disabled" and linking returns 403.

GET /api/v1/connectors

Lists available connectors and this account's link status.

{
  "encryption": "enabled",
  "connectors": [
    {"id": "hardcover", "name": "Hardcover", "tier": 1, "beta": false,
     "carries": ["progress", "finished"], "capabilities": {"read": false, "write": true},
     "credential_kind": "token", "linked": true, "status": "ok", "account": "julia",
     "queue": {"pending": 0, "dead": 0}},
    {"id": "readwise", "name": "Readwise", "tier": 1, "beta": false,
     "carries": ["highlight"], "capabilities": {"read": true, "write": true},
     "credential_kind": "token", "linked": false, "status": null, "account": null}
  ]
}

PUT /api/v1/connectors/{id}

Link/re-link by validating and storing a credential. Body: {"credential": { ... }} — shape is connector-specific ({"token": "..."} for Hardcover and Readwise). The server validates against the service before storing; returns 400 if rejected. {"id": "hardcover", "linked": true, "account": "julia"} on success.

DELETE /api/v1/connectors/{id}

Unlink; wipes the stored credential, all matches, and queued work.

GET /api/v1/connectors/{id}/matches

Lists resolved book matches (for a review UI): {"connector": "hardcover", "matches": [{"document", "external_id", "confidence", "source": "auto|manual|none", "query_used", "updated_at"}]}. The review endpoint (GET /connectors/{id}/review) additionally carries push_note: a per-book condition from the last successful push (e.g. Hardcover has no page count for the book, so progress cannot sync), or null. Cleared automatically when the condition resolves or the match changes.

PUT /api/v1/connectors/{id}/matches/{document}

Manually set a match (sticky — never auto-recomputed). Body {"external_id": "42"}, or {"external_id": null} to mark "never sync this document".

POST /api/v1/connectors/{id}/rematch/{document}

Force (re)matching now; returns the resolved match or null. Preserves manual overrides.

POST /api/v1/connectors/{id}/library/refresh

Force-refresh the connector's server-side library list. {"count": 2}. 400 for connectors without a refreshable library.

POST /api/v1/connectors/{id}/lookup

Verify an externally-supplied book id against the user's account at the service. Body {"external_id": "…"}. {"found": true, "book": {"externalId", "title", "author", "edition"}} or {"found": false}. 400 for connectors without lookup.

Matching is server-side from the document's title/author (the EPUB metadata the firmware sends — so connectors need "Send Metadata" on). Fan-out is automatic: a progress PUT enqueues a progress/finished event to write-connectors that carry it; a clippings PUT enqueues highlight events to highlight-connectors (Readwise). A background worker delivers them with retry/backoff. Fan-in (read connectors: Audiobookshelf, BookOrbit, Readwise Reader, BookFusion) is pulled on a background interval for library-wide providers, and on-demand — when a device asks for progress on a matched book — for per-book providers.

GET /healthz

Unauthenticated. {"status": "ok", "version": "0.1.0"}.

Limits

What Limit
Batch items per PUT (bookmarks, clippings, documents) 50
Batch items per PUT (per-book stats) 20
List page size default 50, max 100
progress string 4096 bytes
position anchor / xpath 48 / 120 bytes
clipping text / note / chapter 2048 / 4096 bytes / 64 chars
bookmark xpath / summary 512 / 256 chars
username / password key 64 / 128 chars