From b330650f602b1b7318dd6165643c84aa4f9c941e Mon Sep 17 00:00:00 2001 From: Prannaya Gupta Date: Tue, 3 Nov 2026 19:27:22 -0500 Subject: [PATCH 01/23] docs: plan evidence-backed paper citation indexing --- docs/citation-indexing-investigation.md | 70 +++++++ .../2026-11-02-paper-citation-indexing.md | 191 ++++++++++++++++++ 2 files changed, 261 insertions(+) create mode 100644 docs/citation-indexing-investigation.md create mode 100644 docs/superpowers/plans/2026-11-02-paper-citation-indexing.md diff --git a/docs/citation-indexing-investigation.md b/docs/citation-indexing-investigation.md new file mode 100644 index 0000000..4846aeb --- /dev/null +++ b/docs/citation-indexing-investigation.md @@ -0,0 +1,70 @@ +# Citation indexing investigation + +Investigated November 2, 2026. Scope: repository inspection and provider documentation. No live database audit, provider coverage benchmark, or citation backfill was performed. This is a proposed design, not an implemented feature. + +Recommendation: build a persistent directed citation index using OpenAlex first, Crossref reference metadata where available, and PDF bibliography extraction for gaps. Match references against local paper identities and retain evidence for each link. PostgreSQL is sufficient for this feature. + +## Existing foundations and gaps + +- `backend/services/related_paper_service.py` already resolves papers through OpenAlex and reads `referenced_works`. Its response includes `already_exists` and `existing_paper_id`, so parts of the local matching flow exist. +- This is a discovery feature, not a complete index: `find_related_papers` combines semantic neighbors with citation links, caps fetched candidates at three times the result limit, ranks existing library papers last, and returns at most 30 candidates. Connections between existing papers can disappear from the response. +- `_resolve_seed_work` returns the first title search result when there is no exact title match. That is insufficient evidence for creating a durable citation edge. Even equal titles need disambiguation when identifiers are absent. +- The paper model stores DOI and arXiv ID but no persistent OpenAlex identity. The checked migrations define no paper-to-paper citation table. +- `backend/services/import_service.py::_fetch_doi` already requests Crossref records, but its returned metadata omits reference lists. Capturing these during import could avoid another request. +- `backend/services/dedup_service.py` offers DOI, arXiv, and normalized-title matching. It loads the library for every call; reusing it for each reference would multiply database reads. Its title normalization removes non-ASCII characters, and identifier handling needs stronger canonicalization for this use. +- `backend/services/pdf_text_service.py` truncates cached text at 120,000 characters. Bibliographies near the end can be lost. The upload metadata extractor reads the first three pages and at most 8,000 characters, so it is not a reference extractor. +- Import-time PDF text extraction in `_auto_generate_notes` is conditional on automatic notes being enabled. Reference indexing needs an independent hook. + +## Options + +| Source | What it supplies | Recommended role | Limitation | +| --- | --- | --- | --- | +| OpenAlex | Outgoing reference IDs in `referenced_works` | First implementation; existing integration | Coverage and matching are incomplete; a missing edge is not proof of no citation | +| Crossref | Deposited reference records, including identifiers or bibliographic fields when supplied | Reuse DOI import responses and supplement coverage | Depends on what is deposited; not every reference has a DOI | +| Semantic Scholar | Reference endpoint and identifier-based paper lookup, including arXiv | Evaluate as a fallback for unresolved records | Additional integration, rate limiting, and coverage evaluation | +| GROBID | Structured bibliography extraction from PDFs, with optional raw citation strings | Recover references from stored PDFs and supply inspectable evidence | Requires a service and reliable PDF extraction; scanned PDFs need a separate OCR path | +| Existing PDF-to-Markdown extraction | Text that may contain a bibliography | Low-cost feasibility experiment | Needs segmentation, complete bibliography text, and validation of parsed entries | + +OpenAlex constructs its graph by matching extracted references to indexed works. It explicitly documents missing references and imperfect metadata matches; preserve provider provenance and distinguish provider-reported links from PDF-verified evidence. See [OpenAlex citations](https://help.openalex.org/data/works/citations/). + +Crossref documents reference-bearing records and its reference representation in the [REST filters](https://www.crossref.org/documentation/retrieve-metadata/rest-api/rest-api-filters/) and [response format](https://github.com/CrossRef/rest-api-doc/blob/master/api_format.md). An absent reference field should be treated as unavailable data. + +Semantic Scholar's official [API examples](https://api.semanticscholar.org/api-docs/snippets) show reference retrieval and arXiv identifier lookup. The documentation endpoint returned HTTP 429 during this investigation; pagination and field details should be verified during integration. + +GROBID's [`processReferences`](https://grobid.readthedocs.io/en/latest/Grobid-service/#apiprocessreferences) extracts bibliography entries into TEI XML or BibTeX. `includeRawCitations=1` preserves reference text. Full-text processing can additionally support citation context. Prefer a local service for stored PDFs; keep metadata consolidation configurable. + +## Proposed data flow + +1. Page through all papers in the chosen scope. Build normalized DOI/arXiv lookup maps and persist validated provider identities. The existing unpaginated `list_papers` must not silently define the complete backfill population. +2. Resolve source papers by exact identifiers first. Use title, authors, and publication year together for unresolved candidates; retain ambiguous matches for review. Normalize DOI prefixes and casing, support old and new arXiv formats, and retain version information even when matching by base arXiv ID. +3. Fetch the full provider reference list for every resolved paper. For OpenAlex, intersect outgoing IDs with the local OpenAlex-ID map. Fetching metadata for every external cited work is unnecessary just to find local connections. +4. Supplement missing coverage with Crossref, then evaluate Semantic Scholar and PDF extraction based on measured gaps. For PDF evidence, preserve the original reference string and document version or file hash. Do not use truncated LLM context as the authoritative bibliography. +5. Resolve reference targets by exact identifiers. Title/author/year matches remain candidates until validated; do not invent a numeric confidence score without calibration. Preprint and published versions may be related without having identical bibliographies, so retain aliases and evidence of version equivalence. +6. Persist directed edges: `A -> B` means A cites B. Query the same table in reverse for "Cited by in this library." Reciprocal citation means both edges exist; it should not be inferred from a single edge. +7. Reconcile unresolved references when new papers arrive. Schedule refresh when identifiers or PDFs change, and support a resumable backfill plus per-paper reindexing. + +Semantic similarity and note wiki-links are separate relationships. Neither establishes that one paper cites another. + +## Suggested storage + +| Table | Purpose and key fields | +| --- | --- | +| `paper_external_ids` | Local paper, provider, external ID, resolution method/status, resolved timestamp; index `(provider, external_id)` and preserve multiple local copies | +| `paper_reference_runs` | Source paper, provider/parser version, source fingerprint, status, retrieved count, timestamps, retry/error information | +| `paper_references` | Run, stable provider reference key or reference fingerprint, target identifiers, raw bibliography text/metadata, resolution status; retain references without a local target | +| `paper_citations` | Unique `(citing_paper_id, cited_paper_id)`; foreign keys to papers and reverse index on `cited_paper_id` | +| `paper_citation_evidence` | Citation edge linked to one or more reference records, matching method, review status; allows providers to support the same edge independently | + +Use foreign keys, reject edges from a paper to itself, and exclude links between known duplicate representations of the same work. Preserve ambiguous targets instead of selecting the first duplicate. Scope results explicitly: default to the current library, and make database-wide analysis an explicit scope that respects the caller's access. A provider ID may map to several library copies, so it must not be globally unique across local papers. + +Publish refreshed evidence atomically after successful indexing. Failed or partial runs must not erase prior good evidence. Track `pending`, `running`, `succeeded`, `partial`, `not_found`, and `failed` separately; successful provider retrieval does not imply a complete bibliography. Report local resolved citations, unresolved references, and indexing status separately. + +## Implementation sequence and validation + +Start with provider identity persistence, reference/evidence tables, a separate citation service, and a resumable OpenAlex backfill. Add outgoing/incoming citation API responses and simple paper-detail lists before building a graph visualization. Capture Crossref references during DOI import next. Add PDF processing where the coverage pilot demonstrates value. + +Use a persistent job record or queue for bulk work, bounded concurrency, provider-aware retries, and idempotent writes. Existing FastAPI background tasks can launch small jobs but are not durable across restarts. OpenAlex supports [batch ID retrieval](https://help.openalex.org/how-to/api-recipes/); use it where helpful and configure `OPENALEX_API_KEY` for bulk access according to the current [authentication documentation](https://help.openalex.org/api/authentication/). Avoid hard-coding historical quotas. + +Before estimating full-library coverage, run a stratified sample containing DOI papers, arXiv-only papers, and PDF-only imports. Report identity resolution rate, reference availability, local edges found, ambiguous matches, and provider failures separately. Manually inspect sampled edges against bibliographies and check known positive pairs to estimate missed citations. This investigation has not measured those rates or identified actual citing pairs in the live database. + +Implementation tests should cover citation direction, DOI/arXiv normalization, ambiguous titles, version aliases, duplicate library copies, missing references versus failed retrieval, pagination beyond database/provider page limits, retry/resume behavior, and reindexing without duplicate or lost evidence. The most important product wording is "No indexed citations found" rather than "These papers do not cite each other." diff --git a/docs/superpowers/plans/2026-11-02-paper-citation-indexing.md b/docs/superpowers/plans/2026-11-02-paper-citation-indexing.md new file mode 100644 index 0000000..9fdcf33 --- /dev/null +++ b/docs/superpowers/plans/2026-11-02-paper-citation-indexing.md @@ -0,0 +1,191 @@ +# Paper citation indexing execution plan + +Date: November 2, 2026 +Branch: `feat/paper-citation-indexing` +Status: OpenAlex/Crossref implementation locally validated; production rollout and PDF fallback pending +Basis: [citation indexing investigation](../../citation-indexing-investigation.md) + +## Execution record - November 3, 2026 + +Implemented the OpenAlex/Crossref index, migration 033, durable worker/backfill tooling, APIs, paper-detail lists, and regression coverage. See [operator guide](../../guides/citation-indexing.md) and [approved read-only pilot](../../citation-indexing-pilot-report.md). + +The pilot sampled 30 of 391 papers, made 56 provider requests, and found one local pair: MAGiC-SLAM cites SplaTAM. The source PDF confirms entry [15] on page 9. OpenAlex resolved 18 source identities, but ten returned empty reference arrays. Nine of ten sampled arXiv papers had no provider references. Crossref supplied 460 reference records from nine DOI sources but no additional local pairs in this sample. PDF extraction remains necessary follow-up work; Task 9 is not complete. + +Implementation adjustments: + +- Database triggers enqueue and invalidate work for every creation/edit/deletion path. Queue creation is part of the paper transaction; provider work runs separately. A database enqueue failure fails the create transaction explicitly rather than committing an unqueued paper. Captured Crossref payload failure is recoverable by refetching. +- Local DOI/arXiv identities are registered before provider indexing. Each published identity triggers SQL reconciliation, so correctness does not depend on a separate global resolution pass. Provider calls are per source; reference matching uses indexed SQL joins without one remote lookup per reference. The read-only pilot batches referenced-work metadata requests. +- Durable jobs are stored in PostgreSQL; the backfill's frozen selection/enqueue checkpoint is an ignored local JSON manifest. Keep the manifest for batch-specific resume. Individual job recovery does not depend on that file. +- Fixtures are embedded in Python, SQL, component, and browser tests rather than a separate fixtures directory. A pinned disposable PGlite engine runs the SQL suite in CI. +- Title hits retain available author/year metadata as unaccepted candidates. No manual acceptance UI or automatic fuzzy matching is implemented. Source versions remain on the paper; raw provider evidence is retained. +- Access follows the repository's existing shared self-hosted model. Library matching is enforced server-side, but this change does not introduce authentication. + +No migration or backfill has been applied to the live database. Production permissions, multi-connection contention, worker deployment, and full-library persistence remain rollout validation. Semantic Scholar and GROBID were not benchmarked; the external-data approval covered only OpenAlex and Crossref. + +Validation: 340 backend tests and 401 frontend tests passed; the full frontend suite used `TZ=UTC` for the existing Dashboard fixture's date assumptions. Migration replay and SQL transaction checks passed. The production build passed with the existing bundle-size warning. The citation browser fixture passed. Combined citation/library browser checks passed 13 of 14 tests: the unchanged row-context-menu test timed out opening/clicking its Copy & export submenu and also timed out when rerun alone. No context-menu implementation was changed. A 391-paper dry-run inventory completed without enqueueing jobs. + +## Outcome and scope + +Let a user open a paper and see which papers in the current library it cites and which library papers cite it. Each connection must have retrievable evidence, a reliable direction, and an indexing status that distinguishes missing data from a failed lookup. + +Milestone 1 delivers persistent OpenAlex identities/references, a resumable indexing worker, local citation matching, API responses, paper-detail lists, and a measured coverage pilot. Milestone 2 captures Crossref references and measures the additional coverage. Milestone 3 evaluates Semantic Scholar and GROBID against the remaining gaps before adding a fallback. + +The initial UI is library-scoped. Database-wide analysis can use an explicit authorized scope in the backfill tool. Graph visualization, citation-context highlighting, OCR, and automatic merging of preprint/published records are deferred. Semantic similarity is not citation evidence. + +## Design decisions + +- Use PostgreSQL tables and transactional functions through the existing Supabase client. No graph database is required. +- Keep citation indexing separate from the capped discovery response in `related_paper_service.py`. Reuse provider request infrastructure only after making it suitable for bulk indexing. +- Resolve exact normalized identifiers automatically. Store title/author/year matches as candidates until validated; never accept the first search result by default. Missing or conflicting identities remain unresolved. +- Resolve the source population before intersecting references with local identities. Keep unresolved references so adding a target paper later can create a link without refetching the source. +- Keep provider assertions distinct from evidence extracted from the actual stored PDF. Preserve version/alias provenance; preprints and published versions can have different bibliographies. +- Use durable database jobs and a separate worker command. FastAPI background tasks may enqueue work, but do not own durable execution. +- An empty provider reference list means no references returned by that provider. It does not establish an empty bibliography. + +## Task 1: Define contracts and representative fixtures + +Dependencies: none. + +Files: create `backend/models/citation.py`, `backend/tests/test_citation_identity.py`, and inline fixtures in the citation test files; update `docs/api/papers.md` with proposed contracts. + +- [x] Define typed provider identities, extracted references, evidence, local citation summaries, and job responses using the existing camel-case model convention. +- [x] Define retrieval states (`pending`, `running`, `succeeded`, `partial`, `not_found`, `failed`) separately from identity resolution (`exact`, `candidate`, `ambiguous`, `unresolved`) and staleness. +- [x] Add minimal sanitized fixtures for A citing B, B not returning references, reciprocal A/B links, an arXiv-only source, duplicate local records, and an ambiguous title. +- [x] Specify direction as `citing_paper_id -> cited_paper_id`; require provider/method provenance and last successful indexing time in responses. + +Acceptance: contracts represent unavailable references, successful empty retrieval, failed retrieval, unresolved targets, and old usable evidence independently. Tests do not require live provider calls. + +## Task 2: Add storage and atomic publication + +Dependencies: Task 1. + +Files: create `backend/migrations/033_paper_citations.sql` (use the next free number when executing), `backend/services/citation_repository.py`, and `backend/tests/sql/paper_citations.sql`; update `docs/database/schema.md` and the fresh-install schema path after checking migration conventions. + +- [x] Add `paper_external_ids`, `paper_reference_runs`, `paper_references`, `paper_citations`, `paper_citation_evidence`, and `paper_citation_jobs`. +- [x] Index provider/external-ID lookup, reference target identifiers, source-paper reference retrieval, job claim/retry fields, and reverse citation lookup. Make `(citing_paper_id, cited_paper_id)` unique and reject self-edges. +- [x] Permit one provider identity to map to multiple local paper copies; retain ambiguity rather than arbitrarily selecting the first copy. Apply library scope when selecting visible targets. +- [x] Record job scope, source fingerprint, attempt count, next retry, lease expiry, and timestamps. Add atomic claim/reclaim functions to prevent competing workers from publishing the same job. +- [x] Stage each provider run and publish its active evidence in one transaction. Fence publication using the job lease/version and current paper fingerprint so stale workers cannot overwrite newer results. +- [x] On successful refresh, replace that provider's active evidence. Keep evidence from other providers. Remove a derived edge only when no active supporting evidence remains. +- [x] Preserve last successful evidence after partial/failed runs; expose the failed refresh separately. Cascade deleted papers' derived records without deleting unrelated evidence. +- [x] Follow the repository's actual access model; validate library scope server-side rather than relying on a caller-supplied ID alone. + +Acceptance: SQL checks cover duplicate edges, reversed queries, deletion, independent provider support, failed refresh preservation, successful evidence removal, and stale publication rejection. Run these against a disposable PostgreSQL/Supabase instance; service mocks cannot validate transaction behavior. + +## Task 3: Implement identity resolution and OpenAlex retrieval + +Dependencies: Tasks 1–2. + +Files: create `backend/services/citation_identity_service.py`, `backend/services/openalex_client.py`, `backend/services/citation_service.py`, and `backend/tests/test_citation_service.py`; adjust `related_paper_service.py` only if extracting shared HTTP helpers. + +- [x] Canonicalize DOI prefixes/casing and arXiv base identifiers while retaining versions. Support legacy arXiv formats and Unicode-preserving title normalization. +- [x] Resolve sources by identifier; verify any provider location/alias used to connect an arXiv record to a published work. Retain available author/year metadata with title candidates; keep them unaccepted until independently validated. +- [x] Persist accepted identities and unresolved/candidate outcomes. Do not reuse discovery's `return exact or results[0]` behavior. +- [x] Retrieve complete outgoing reference arrays and store provider reference IDs even when no local target exists. Keep provider coverage separate from matching coverage. +- [x] Maintain indexed local identities in PostgreSQL and intersect references in library-scoped SQL. Avoid one database request or remote metadata lookup per reference. +- [x] Bound per-source worker concurrency, honor retry headers, and distinguish authentication, rate-limit, not-found, and transient failures. Batch referenced-work retrieval in the pilot; keep credentials out of logs. + +Acceptance: fixtures establish correct direction, no semantic-neighbor edges, no first-search-result matches, no self/known-duplicate-work edges, deterministic duplicate handling, and full reference retrieval without discovery limits. Existing related-paper behavior remains compatible if shared code changes. + +## Task 4: Build resumable jobs and reconciliation + +Dependencies: Tasks 2–3. + +Files: create `backend/services/citation_job_service.py`, `backend/scripts/index_paper_citations.py`, `backend/scripts/citation_worker.py`, and `backend/tests/test_citation_jobs.py`; add a paginated iterator in `backend/services/paper_service.py`. + +- [x] Add stable keyset pagination over paper IDs and persist scope/checkpoints. Never use the existing single `list_papers` response as the entire backfill population. +- [x] Register local identifiers first and reconcile retained references whenever a provider identity arrives, independent of processing order. Capture a bounded initial population and use incremental jobs for later imports. +- [x] Define the backfill CLI: required `--library-id` or explicit `--all-libraries`, optional `--limit`, `--dry-run`, and `--resume` with a persisted batch ID. Dry run reports selection and existing state without provider requests or database writes. +- [x] Implement leased jobs with bounded retries, restart recovery, and deterministic idempotency keys. A worker crash must leave a reclaimable job. +- [x] Add per-paper reindex jobs and transactional reconciliation. Identity changes immediately invalidate or hide affected identity-dependent links; a transient provider failure preserves valid older evidence with a stale indicator. +- [x] Enqueue indexing/reconciliation on all paper creation paths, including identifier, BibTeX, direct creation, and agent imports. Connect source/PDF changes and deletions to invalidation. Trace callers so hooks are not limited to one router. +- [x] Keep indexing independent of `auto_note_enabled`. Enqueue atomically with paper creation; failed provider work must not remove the paper. Recover missed captured payloads by refetch and expose indexing failures. + +Acceptance: a fixture population larger than a database page is fully processed; stopping/restarting produces the same edges as uninterrupted execution. Importing B after A creates A-to-B from retained references. Repeated imports/reindex requests do not duplicate jobs or evidence. + +## Task 5: Expose citation APIs + +Dependencies: Tasks 3–4. + +Files: create `backend/routers/citations.py` and `backend/tests/test_citation_routes.py`; register the router in `backend/app.py`; update `docs/api/papers.md`. + +- [x] Implement `GET /api/papers/{paper_id}/citations?direction=outgoing|incoming&limit=...&offset=...` with deterministic pagination and a total count. +- [x] Include local paper summaries, evidence source/method, last successful index time, latest attempt status, and unresolved-reference counts. Counts must use the same scope as results. +- [x] Implement `POST /api/papers/{paper_id}/citations/reindex` returning HTTP 202 and durable per-provider job summaries, plus `GET /api/citation-jobs/{job_id}` for progress. +- [x] Bind ordinary reads to the source paper's library and verify access to both ends. Do not let a read request trigger provider indexing. +- [x] Match existing not-found/error contracts and keep request/response casing consistent with frontend conventions. + +Acceptance: route tests cover direction, pagination, scope isolation, unknown paper/job, repeated reindex requests, and status reporting after failed refresh with previous data. Incoming links are the reverse query over stored edges, not a second inference. + +## Task 6: Add paper-detail citation lists + +Dependencies: Task 5. + +Files: create `frontend/src/components/PaperCitations.jsx` and its test; update `frontend/src/services/api.js`, its tests, and `frontend/src/pages/Paper.jsx`. Evaluate reuse in `PaperInfoPanel.jsx` after the dedicated view works. + +- [x] Add “References in this library” and “Cited by in this library” lists with counts, paper navigation, pagination, and source evidence. +- [x] Show indexing progress, last indexed time, stale evidence, unresolved references, and retry controls where relevant. Poll queued/running jobs with cleanup when navigating away. +- [x] Use “Not indexed yet,” “Indexing failed,” and “No indexed citations found” as distinct states. Describe unresolved references as not matched locally, not as proof those works are absent from the database. +- [x] Ignore responses from a previously selected paper/library. Keep related-paper recommendations and their import controls separate. + +Acceptance: component tests cover incoming/outgoing navigation, empty versus failed/unindexed states, pagination, refresh with existing data, and navigation during an in-flight request. Keyboard navigation and loading/error states work in a manual browser check. + +## Task 7: Validate and roll out the OpenAlex milestone + +Dependencies: Tasks 1–6. + +Files: add operator instructions in `docs/guides/citation-indexing.md`; create `docs/citation-indexing-pilot-report.md` after the pilot, recording observations rather than predicted coverage. + +- [x] Run focused tests as each service ships; run backend/frontend regression suites and the frontend production build when the milestone is integrated. +- [x] Apply the migration in a disposable environment and verify schema installation, atomic publication, worker restart, and a full API/UI flow with fixed fixtures. +- [x] Select a reproducible pilot of up to 30 papers across DOI, arXiv-only, and PDF-only records, using all available records in small strata. Record the actual selection and date; include known positive pairs where available. +- [x] Resolve target identities across the chosen library so a source-only sample does not create artificial coverage gaps. Limit reference retrieval to pilot sources initially. +- [x] Record identity-resolution rate, provider reference availability, local edges, ambiguities, failures, request volume, and runtime separately. Manually compare up to 20 detected edges with bibliographies and check every selected known positive pair. Document denominators and unavailable PDFs. +- [x] Treat every detected false positive as a release blocker until the matching rule is fixed and checked again. Report recall limits explicitly; do not promise complete coverage from this sample. +- [x] Prepare migration/backfill commands, rollback instructions, and the pilot report. Keep worker failure messages free of credentials and full PDF contents. +- [ ] Apply migration 033 to the intended live database, validate its permissions and concurrent workers, then start the worker and expand the persisted backfill with resumable batches. +- [x] Document rollback by disabling enqueueing, stopping workers, and hiding the feature while retaining additive tables/evidence. Avoid destructive rollback of indexed data. + +Milestone 1 exit: supported citation pairs appear in both directions with inspectable evidence; new imports reconcile; retries and restarts preserve data; coverage limitations are measured and visible. + +## Task 8: Capture Crossref references + +Dependencies: Task 7. + +Files: update `backend/services/import_service.py`; add `backend/services/crossref_reference_service.py` and `backend/tests/test_crossref_references.py`; extend citation integration tests. + +- [x] Retain Crossref `reference` records from DOI lookup in an internal payload validated into the typed citation result by the worker. Pass them to citation storage only after a local paper exists; do not insert provider payload fields into the `papers` row. +- [x] Store reference DOI, bibliographic metadata, and raw text when supplied. Treat absent references separately from successful empty data. +- [x] Resolve DOI targets through the identity layer; retain title-only reference metadata unresolved. Add Crossref backfill for existing DOI papers without captured references. +- [x] Reuse the same evidence publication and job lifecycle. Measure additional edges and unresolved references over the pilot baseline; overlapping sources must not double-count an edge. + +Milestone 2 exit: DOI imports preserve available references without redundant immediate fetches; Crossref evidence coexists with OpenAlex; coverage improvement is reported. + +## Task 9: Evaluate and implement the justified fallback (pending) + +Dependencies: Task 8 and measured unresolved cases. + +Files: proposed `backend/services/semantic_scholar_client.py` and/or `backend/services/pdf_reference_service.py`, with provider/parser fixture tests and deployment documentation. + +- [ ] Evaluate Semantic Scholar on unresolved arXiv/identifier cases. Verify current endpoint fields, pagination, authentication, and limits before implementation. +- [ ] Evaluate GROBID on stored PDFs whose bibliographies are missing from provider results. Read PDFs through the existing managed-storage abstraction, including private GCS objects; do not assume stored URLs are publicly fetchable. +- [ ] Extract from original PDF bytes rather than the truncated Markdown cache. Retain document hash/version, parser version, raw bibliography text, and structured identifiers. Use citation context/page coordinates only when supplied by a suitable parser output. +- [ ] Report scanned/unparseable PDFs as unsupported extraction cases; do not silently produce empty bibliographies. Keep OCR as a separate decision. +- [ ] Compare additional verified local edges, ambiguous matches, failure rate, runtime, and operating requirements. Implement the fallback with demonstrated benefit using the existing jobs/evidence model; record a no-go decision if neither adds sufficient value. + +Milestone 3 exit: remaining gaps have measured causes and either an integrated, tested fallback or an explicit evidence-based deferral. + +## Verification commands + +Run from the repository root after the relevant files exist. Focused tests come first; the full suites below are the integrated milestone gate. SQL transaction checks require the disposable database in Task 2 and are not replaced by pytest mocks. + +```powershell +uv run --directory backend pytest tests/test_citation_identity.py tests/test_citation_service.py tests/test_citation_jobs.py tests/test_citation_routes.py +npm --prefix frontend run test:run -- src/components/PaperCitations.test.jsx src/services/api.test.js +uv run --directory backend pytest +npm --prefix frontend run test:run +npm --prefix frontend run build +git diff --check +``` + +Suggested commit sequence: contracts/schema; provider identity and OpenAlex indexing; durable jobs/reconciliation; citation API; citation UI; pilot/docs; Crossref capture; justified fallback. Keep every implementation checkbox open until its acceptance criteria have been verified. From d8040ad7ae3ca47d1aade6226eb696e6f04a9e57 Mon Sep 17 00:00:00 2001 From: Prannaya Gupta Date: Tue, 3 Nov 2026 19:27:30 -0500 Subject: [PATCH 02/23] feat(db): add durable citation evidence and leased jobs --- .github/workflows/tests.yml | 12 + backend/migrations/033_paper_citations.sql | 330 +++++++++++++++++++++ backend/migrations/schema.sql | 2 + backend/models/citation.py | 65 ++++ backend/tests/run_citation_sql.mjs | 18 ++ backend/tests/sql/paper_citations.sql | 108 +++++++ docs/database/schema.md | 15 + 7 files changed, 550 insertions(+) create mode 100644 backend/migrations/033_paper_citations.sql create mode 100644 backend/models/citation.py create mode 100644 backend/tests/run_citation_sql.mjs create mode 100644 backend/tests/sql/paper_citations.sql diff --git a/.github/workflows/tests.yml b/.github/workflows/tests.yml index ffba26c..90bd5a8 100644 --- a/.github/workflows/tests.yml +++ b/.github/workflows/tests.yml @@ -6,6 +6,18 @@ on: branches: [main] jobs: + citation-database: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: actions/setup-node@v4 + with: + node-version: "20" + - name: Install disposable PostgreSQL engine + run: npm install --prefix backend/data/citation-validation --no-audit --no-fund --ignore-scripts @electric-sql/pglite@0.5.14 + - name: Verify citation migration and transactions + run: node backend/tests/run_citation_sql.mjs + backend: runs-on: ubuntu-latest defaults: diff --git a/backend/migrations/033_paper_citations.sql b/backend/migrations/033_paper_citations.sql new file mode 100644 index 0000000..2c7bdc5 --- /dev/null +++ b/backend/migrations/033_paper_citations.sql @@ -0,0 +1,330 @@ +-- Additive citation index. Apply after the existing paper/library migrations. +BEGIN; + +CREATE TABLE IF NOT EXISTS public.paper_external_ids ( + paper_id TEXT NOT NULL REFERENCES public.papers(id) ON DELETE CASCADE, + provider TEXT NOT NULL, + external_id TEXT NOT NULL, + asserted_by TEXT NOT NULL, + method TEXT NOT NULL DEFAULT 'exact_identifier', + resolved_at TIMESTAMPTZ NOT NULL DEFAULT now(), + PRIMARY KEY (paper_id, provider, external_id, asserted_by) +); +CREATE INDEX IF NOT EXISTS idx_citation_external_id ON public.paper_external_ids(provider, external_id); + +CREATE TABLE IF NOT EXISTS public.paper_reference_runs ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + paper_id TEXT NOT NULL REFERENCES public.papers(id) ON DELETE CASCADE, + provider TEXT NOT NULL CHECK (provider IN ('openalex', 'crossref')), + fingerprint TEXT NOT NULL, + status TEXT NOT NULL CHECK (status IN ('succeeded', 'partial', 'not_found')), + resolution TEXT NOT NULL CHECK (resolution IN ('exact', 'candidate', 'ambiguous', 'unresolved')), + active BOOLEAN NOT NULL DEFAULT false, + version TEXT NOT NULL, + candidates JSONB NOT NULL DEFAULT '[]', + message TEXT, + created_at TIMESTAMPTZ NOT NULL DEFAULT now() +); +CREATE UNIQUE INDEX IF NOT EXISTS idx_citation_active_run ON public.paper_reference_runs(paper_id, provider) WHERE active; +CREATE INDEX IF NOT EXISTS idx_citation_runs ON public.paper_reference_runs(paper_id, created_at DESC); + +CREATE TABLE IF NOT EXISTS public.paper_references ( + id BIGINT GENERATED BY DEFAULT AS IDENTITY PRIMARY KEY, + run_id UUID NOT NULL REFERENCES public.paper_reference_runs(id) ON DELETE CASCADE, + reference_key TEXT NOT NULL, + identifiers JSONB NOT NULL DEFAULT '{}', + raw JSONB NOT NULL DEFAULT '{}', + UNIQUE (run_id, reference_key) +); +CREATE INDEX IF NOT EXISTS idx_reference_identifiers ON public.paper_references USING gin(identifiers); + +CREATE TABLE IF NOT EXISTS public.paper_citations ( + citing_paper_id TEXT NOT NULL REFERENCES public.papers(id) ON DELETE CASCADE, + cited_paper_id TEXT NOT NULL REFERENCES public.papers(id) ON DELETE CASCADE, + PRIMARY KEY (citing_paper_id, cited_paper_id), + CHECK (citing_paper_id <> cited_paper_id) +); +CREATE INDEX IF NOT EXISTS idx_citation_incoming ON public.paper_citations(cited_paper_id, citing_paper_id); + +CREATE TABLE IF NOT EXISTS public.paper_citation_evidence ( + citing_paper_id TEXT NOT NULL, + cited_paper_id TEXT NOT NULL, + reference_id BIGINT NOT NULL REFERENCES public.paper_references(id) ON DELETE CASCADE, + method TEXT NOT NULL DEFAULT 'exact_identifier', + PRIMARY KEY (citing_paper_id, cited_paper_id, reference_id), + FOREIGN KEY (citing_paper_id, cited_paper_id) REFERENCES public.paper_citations ON DELETE CASCADE +); +CREATE INDEX IF NOT EXISTS idx_citation_evidence_reference ON public.paper_citation_evidence(reference_id); + +CREATE TABLE IF NOT EXISTS public.paper_citation_jobs ( + id UUID PRIMARY KEY DEFAULT gen_random_uuid(), + paper_id TEXT NOT NULL REFERENCES public.papers(id) ON DELETE CASCADE, + provider TEXT NOT NULL CHECK (provider IN ('openalex', 'crossref')), + fingerprint TEXT NOT NULL, + status TEXT NOT NULL DEFAULT 'pending' CHECK (status IN ('pending','running','succeeded','partial','not_found','failed')), + attempts INTEGER NOT NULL DEFAULT 0, + lease_token UUID, + lease_until TIMESTAMPTZ, + next_attempt_at TIMESTAMPTZ NOT NULL DEFAULT now(), + error TEXT, + captured_record JSONB, + created_at TIMESTAMPTZ NOT NULL DEFAULT now(), + updated_at TIMESTAMPTZ NOT NULL DEFAULT now(), + UNIQUE (paper_id, provider) +); +CREATE INDEX IF NOT EXISTS idx_citation_claim ON public.paper_citation_jobs(status, next_attempt_at, lease_until); + +-- Match the application's existing server-owned database access model. +ALTER TABLE public.paper_external_ids DISABLE ROW LEVEL SECURITY; +ALTER TABLE public.paper_reference_runs DISABLE ROW LEVEL SECURITY; +ALTER TABLE public.paper_references DISABLE ROW LEVEL SECURITY; +ALTER TABLE public.paper_citations DISABLE ROW LEVEL SECURITY; +ALTER TABLE public.paper_citation_evidence DISABLE ROW LEVEL SECURITY; +ALTER TABLE public.paper_citation_jobs DISABLE ROW LEVEL SECURITY; + +CREATE OR REPLACE FUNCTION public.citation_fingerprint(p public.papers) RETURNS TEXT +LANGUAGE sql IMMUTABLE AS $$ + SELECT md5(jsonb_build_array(p.doi,p.arxiv_id,p.title,p.authors,p.year,p.library_id,p.pdf_url)::text); +$$; + +CREATE OR REPLACE FUNCTION public.citation_local_ids(p public.papers) RETURNS JSONB +LANGUAGE plpgsql IMMUTABLE AS $$ +DECLARE doi_id TEXT; arxiv TEXT; ids JSONB := '{}'; +BEGIN + doi_id := lower(regexp_replace(btrim(coalesce(p.doi,'')), '^(https?://(dx\.)?doi\.org/|doi:\s*)', '', 'i')); + IF doi_id ~ '^10\.\d{4,9}/\S+$' THEN ids := ids || jsonb_build_object('doi',doi_id); END IF; + arxiv := lower(regexp_replace(btrim(coalesce(p.arxiv_id,'')), '^(https?://(www\.|export\.)?arxiv\.org/(abs|pdf)/|arxiv:\s*)', '', 'i')); + arxiv := regexp_replace(regexp_replace(arxiv,'\.pdf$',''),'v\d+$',''); + IF arxiv ~ '^(\d{4}\.\d{4,5}|[a-z][a-z.\-]+/\d{7})$' THEN ids := ids || jsonb_build_object('arxiv',arxiv); END IF; + RETURN ids; +END $$; + +-- Local identifiers are available before a provider succeeds, including unindexed copies. +INSERT INTO paper_external_ids(paper_id,provider,external_id,asserted_by) +SELECT p.id,ident.key,ident.value,'local' FROM papers p +CROSS JOIN LATERAL jsonb_each_text(citation_local_ids(p)) ident ON CONFLICT DO NOTHING; + +CREATE OR REPLACE FUNCTION public.enqueue_paper_citations(p_paper_id TEXT, p_force BOOLEAN DEFAULT false) +RETURNS SETOF public.paper_citation_jobs +LANGUAGE plpgsql SECURITY INVOKER SET search_path = public AS $$ +DECLARE p public.papers; provider_name TEXT; +BEGIN + SELECT * INTO p FROM papers WHERE id = p_paper_id FOR UPDATE; + IF NOT FOUND THEN RETURN; END IF; + INSERT INTO paper_external_ids(paper_id,provider,external_id,asserted_by) + SELECT p.id,key,value,'local' FROM jsonb_each_text(citation_local_ids(p)) ON CONFLICT DO NOTHING; + FOREACH provider_name IN ARRAY ARRAY['openalex','crossref'] LOOP + INSERT INTO paper_citation_jobs(paper_id,provider,fingerprint) + VALUES (p.id,provider_name,citation_fingerprint(p)) + ON CONFLICT (paper_id,provider) DO UPDATE SET + fingerprint=EXCLUDED.fingerprint, status='pending', attempts=0, + lease_token=NULL, lease_until=NULL, next_attempt_at=now(), error=NULL, + captured_record=NULL, updated_at=now() + WHERE paper_citation_jobs.fingerprint <> EXCLUDED.fingerprint + OR (p_force AND paper_citation_jobs.status NOT IN ('pending','running')); + END LOOP; + RETURN QUERY SELECT * FROM paper_citation_jobs WHERE paper_id=p.id ORDER BY provider; +END $$; + +CREATE OR REPLACE FUNCTION public.reconcile_paper_citations(p_library_id TEXT) RETURNS void +LANGUAGE plpgsql SECURITY INVOKER SET search_path = public AS $$ +BEGIN + -- Serialize short publication/reconciliation transactions, not provider requests. + PERFORM pg_advisory_xact_lock(hashtext('researchos-citations')); + DELETE FROM paper_citations c USING papers p + WHERE c.citing_paper_id=p.id AND p.library_id IS NOT DISTINCT FROM p_library_id; + + WITH matches AS ( + SELECT r.id AS ref, run.paper_id AS source, min(e.paper_id) AS target + FROM paper_references r + JOIN paper_reference_runs run ON run.id=r.run_id AND run.active + JOIN papers src ON src.id=run.paper_id + CROSS JOIN LATERAL jsonb_each_text(r.identifiers) ident + JOIN paper_external_ids e ON e.provider=ident.key AND e.external_id=ident.value + JOIN papers dst ON dst.id=e.paper_id AND dst.library_id IS NOT DISTINCT FROM src.library_id + WHERE src.library_id IS NOT DISTINCT FROM p_library_id + GROUP BY r.id, run.paper_id HAVING count(DISTINCT e.paper_id)=1 + ) + INSERT INTO paper_citations + SELECT DISTINCT source,target FROM matches m + WHERE source <> target AND NOT EXISTS ( + SELECT 1 FROM paper_external_ids a JOIN paper_external_ids b + ON a.provider=b.provider AND a.external_id=b.external_id + WHERE a.paper_id=m.source AND b.paper_id=m.target + ) AND NOT EXISTS ( + -- A duplicate can be known by DOI before its OpenAlex identity is resolved. + SELECT 1 FROM paper_external_ids target_id JOIN paper_external_ids duplicate + ON duplicate.provider=target_id.provider AND duplicate.external_id=target_id.external_id + JOIN papers copy ON copy.id=duplicate.paper_id + WHERE target_id.paper_id=m.target AND copy.id<>m.target + AND copy.library_id IS NOT DISTINCT FROM p_library_id + ) ON CONFLICT DO NOTHING; + + INSERT INTO paper_citation_evidence(citing_paper_id,cited_paper_id,reference_id) + SELECT c.citing_paper_id,c.cited_paper_id,r.id + FROM paper_citations c JOIN papers p ON p.id=c.citing_paper_id + JOIN paper_reference_runs run ON run.paper_id=c.citing_paper_id AND run.active + JOIN paper_references r ON r.run_id=run.id + WHERE p.library_id IS NOT DISTINCT FROM p_library_id + AND (SELECT count(DISTINCT ids.paper_id) + FROM jsonb_each_text(r.identifiers) ident + JOIN paper_external_ids ids ON ids.provider=ident.key AND ids.external_id=ident.value + JOIN papers target ON target.id=ids.paper_id + WHERE target.library_id IS NOT DISTINCT FROM p.library_id)=1 + AND EXISTS (SELECT 1 FROM jsonb_each_text(r.identifiers) ident + JOIN paper_external_ids ids ON ids.provider=ident.key AND ids.external_id=ident.value + WHERE ids.paper_id=c.cited_paper_id) + ON CONFLICT DO NOTHING; +END $$; + +CREATE OR REPLACE FUNCTION public.queue_citations_on_paper_change() RETURNS trigger +LANGUAGE plpgsql SECURITY INVOKER SET search_path = public AS $$ +DECLARE captured JSONB; +BEGIN + IF TG_OP='DELETE' THEN + PERFORM reconcile_paper_citations(OLD.library_id); + RETURN OLD; + ELSIF TG_OP='INSERT' THEN + PERFORM enqueue_paper_citations(NEW.id); + PERFORM reconcile_paper_citations(NEW.library_id); + ELSIF citation_fingerprint(OLD) <> citation_fingerprint(NEW) THEN + PERFORM pg_advisory_xact_lock(hashtext('researchos-citations')); + SELECT captured_record INTO captured FROM paper_citation_jobs WHERE paper_id=NEW.id AND provider='crossref'; + DELETE FROM paper_external_ids WHERE paper_id=NEW.id; + UPDATE paper_reference_runs SET active=false WHERE paper_id=NEW.id; + DELETE FROM paper_citations WHERE citing_paper_id=NEW.id OR cited_paper_id=NEW.id; + PERFORM enqueue_paper_citations(NEW.id); + IF OLD.doi IS NOT DISTINCT FROM NEW.doi AND captured IS NOT NULL THEN + UPDATE paper_citation_jobs SET captured_record=captured WHERE paper_id=NEW.id AND provider='crossref'; + END IF; + PERFORM reconcile_paper_citations(OLD.library_id); + IF OLD.library_id IS DISTINCT FROM NEW.library_id THEN + PERFORM reconcile_paper_citations(NEW.library_id); + END IF; + END IF; + RETURN NEW; +END $$; +DROP TRIGGER IF EXISTS paper_citations_changed ON public.papers; +CREATE TRIGGER paper_citations_changed AFTER INSERT OR UPDATE OR DELETE ON public.papers +FOR EACH ROW EXECUTE FUNCTION public.queue_citations_on_paper_change(); + +CREATE OR REPLACE FUNCTION public.claim_citation_job(p_job_id UUID DEFAULT NULL) +RETURNS SETOF public.paper_citation_jobs +LANGUAGE plpgsql SECURITY INVOKER SET search_path = public AS $$ +DECLARE job_id UUID; +BEGIN + UPDATE paper_citation_jobs SET status='failed',error='Worker lease expired repeatedly', + lease_token=NULL,lease_until=NULL,updated_at=now() + WHERE (p_job_id IS NULL OR id=p_job_id) AND status='running' AND lease_until=5; + SELECT id INTO job_id FROM paper_citation_jobs + WHERE (p_job_id IS NULL OR id=p_job_id) + AND ((status='pending' AND next_attempt_at <= now()) OR (status='running' AND lease_until < now())) + ORDER BY next_attempt_at,created_at,id LIMIT 1 FOR UPDATE SKIP LOCKED; + IF job_id IS NULL THEN RETURN; END IF; + RETURN QUERY UPDATE paper_citation_jobs SET status='running',attempts=attempts+1, + lease_token=gen_random_uuid(),lease_until=now()+interval '5 minutes',updated_at=now() + WHERE id=job_id RETURNING *; +END $$; + +CREATE OR REPLACE FUNCTION public.finish_citation_job(p_job_id UUID,p_lease_token UUID,p_result JSONB) +RETURNS BOOLEAN LANGUAGE plpgsql SECURITY INVOKER SET search_path = public AS $$ +DECLARE j paper_citation_jobs; p papers; run_id UUID; ref JSONB; +BEGIN + -- Lock in the same order as paper-change triggers: paper, publication lock, job. + SELECT * INTO p FROM papers WHERE id=(SELECT paper_id FROM paper_citation_jobs WHERE id=p_job_id) FOR UPDATE; + IF NOT FOUND THEN RETURN false; END IF; + PERFORM pg_advisory_xact_lock(hashtext('researchos-citations')); + SELECT * INTO j FROM paper_citation_jobs WHERE id=p_job_id FOR UPDATE; + IF j.status <> 'running' OR j.lease_token IS DISTINCT FROM p_lease_token + OR j.lease_until < now() OR j.fingerprint <> citation_fingerprint(p) THEN RETURN false; END IF; + IF j.provider <> p_result->>'provider' THEN RAISE EXCEPTION 'Provider mismatch'; END IF; + IF p_result->>'status'='succeeded' THEN + UPDATE paper_reference_runs SET active=false WHERE paper_id=p.id AND provider=j.provider AND active; + DELETE FROM paper_external_ids WHERE paper_id=p.id AND asserted_by=j.provider; + END IF; + INSERT INTO paper_reference_runs(paper_id,provider,fingerprint,status,resolution,active,version,candidates,message) + VALUES (p.id,j.provider,j.fingerprint,p_result->>'status',p_result->>'resolution', + p_result->>'status'='succeeded',p_result->>'version',coalesce(p_result->'candidates','[]'),p_result->>'message') + RETURNING id INTO run_id; + INSERT INTO paper_external_ids(paper_id,provider,external_id,asserted_by) + SELECT p.id,key,value,j.provider FROM jsonb_each_text(p_result->'identities') + ON CONFLICT (paper_id,provider,external_id,asserted_by) DO UPDATE SET resolved_at=now(); + FOR ref IN SELECT * FROM jsonb_array_elements(p_result->'references') LOOP + INSERT INTO paper_references(run_id,reference_key,identifiers,raw) + VALUES (run_id,ref->>'key',ref->'identifiers',ref->'raw') ON CONFLICT DO NOTHING; + END LOOP; + UPDATE paper_citation_jobs SET status=p_result->>'status',error=p_result->>'message', + captured_record=NULL,lease_token=NULL,lease_until=NULL,updated_at=now() WHERE id=j.id; + PERFORM reconcile_paper_citations(p.library_id); + RETURN true; +END $$; + +CREATE OR REPLACE FUNCTION public.fail_citation_job(p_job_id UUID,p_lease_token UUID,p_error TEXT, + p_retry BOOLEAN DEFAULT false,p_retry_seconds INTEGER DEFAULT 60) +RETURNS BOOLEAN LANGUAGE plpgsql SECURITY INVOKER SET search_path = public AS $$ +BEGIN + UPDATE paper_citation_jobs SET status=CASE WHEN p_retry AND attempts<5 THEN 'pending' ELSE 'failed' END, + error=left(p_error,300),lease_token=NULL,lease_until=NULL,updated_at=now(), + next_attempt_at=now()+make_interval(secs=>greatest(1,p_retry_seconds)) + WHERE id=p_job_id AND status='running' AND lease_token=p_lease_token AND lease_until >= now(); + RETURN FOUND; +END $$; + +CREATE OR REPLACE FUNCTION public.capture_crossref_references(p_paper_id TEXT,p_record JSONB) +RETURNS BOOLEAN LANGUAGE plpgsql SECURITY INVOKER SET search_path = public AS $$ +BEGIN + -- Only attach to a queued job. A racing worker can safely fetch its own copy. + UPDATE paper_citation_jobs SET captured_record=p_record,updated_at=now() + WHERE paper_id=p_paper_id AND provider='crossref' AND status='pending'; + RETURN FOUND; +END $$; + +CREATE OR REPLACE FUNCTION public.get_paper_citations(p_paper_id TEXT,p_direction TEXT DEFAULT 'outgoing', + p_limit INTEGER DEFAULT 20,p_offset INTEGER DEFAULT 0) +RETURNS JSONB LANGUAGE plpgsql STABLE SECURITY INVOKER SET search_path = public AS $$ +DECLARE lib TEXT; result JSONB; +BEGIN + SELECT library_id INTO lib FROM papers WHERE id=p_paper_id; + IF NOT FOUND THEN RETURN NULL; END IF; + IF p_direction NOT IN ('incoming','outgoing') THEN RAISE EXCEPTION 'Invalid citation direction'; END IF; + WITH edges AS ( + SELECT c.*,CASE WHEN p_direction='outgoing' THEN c.cited_paper_id ELSE c.citing_paper_id END AS target + FROM paper_citations c + JOIN papers a ON a.id=c.citing_paper_id JOIN papers b ON b.id=c.cited_paper_id + WHERE a.library_id IS NOT DISTINCT FROM lib AND b.library_id IS NOT DISTINCT FROM lib + AND CASE WHEN p_direction='outgoing' THEN c.citing_paper_id ELSE c.cited_paper_id END=p_paper_id + ), page AS ( + SELECT e.*,p.title,p.authors,p.year FROM edges e JOIN papers p ON p.id=e.target + ORDER BY p.year DESC,p.title,p.id LIMIT greatest(1,least(p_limit,100)) OFFSET greatest(p_offset,0) + ) + SELECT jsonb_build_object('items',coalesce((SELECT jsonb_agg(jsonb_build_object( + 'id',pg.target,'title',pg.title,'authors',pg.authors,'year',pg.year,'evidence',( + SELECT jsonb_agg(jsonb_build_object('provider',run.provider,'method',ev.method, + 'indexed_at',run.created_at,'reference',r.raw) ORDER BY run.provider,r.id) + FROM paper_citation_evidence ev JOIN paper_references r ON r.id=ev.reference_id + JOIN paper_reference_runs run ON run.id=r.run_id + WHERE ev.citing_paper_id=pg.citing_paper_id AND ev.cited_paper_id=pg.cited_paper_id + )) ORDER BY pg.year DESC,pg.title,pg.target) FROM page pg),'[]'), + 'total',(SELECT count(*) FROM edges),'limit',greatest(1,least(p_limit,100)), + 'offset',greatest(p_offset,0),'direction',p_direction, + 'jobs',coalesce((SELECT jsonb_agg(jsonb_build_object('id',id,'paper_id',paper_id,'provider',provider, + 'status',status,'attempts',attempts,'updated_at',updated_at,'error',error) ORDER BY provider) + FROM paper_citation_jobs WHERE paper_id=p_paper_id),'[]'), + 'runs',coalesce((SELECT jsonb_agg(jsonb_build_object('provider',provider,'status',status, + 'resolution',resolution,'active',active,'indexedAt',created_at,'message',message, + 'candidates',candidates) ORDER BY provider,created_at DESC) FROM ( + SELECT DISTINCT ON (provider,active) * FROM paper_reference_runs + WHERE paper_id=p_paper_id ORDER BY provider,active,created_at DESC,id DESC + ) recent),'[]'), + 'unresolved_count',(SELECT count(*) FROM paper_references r JOIN paper_reference_runs run ON run.id=r.run_id + WHERE run.paper_id=p_paper_id AND run.active AND NOT EXISTS + (SELECT 1 FROM paper_citation_evidence ev WHERE ev.reference_id=r.id)), + 'indexed_papers',(SELECT count(DISTINCT run.paper_id) FROM paper_reference_runs run + JOIN papers p ON p.id=run.paper_id WHERE run.active AND p.library_id IS NOT DISTINCT FROM lib), + 'pending_jobs',(SELECT count(*) FROM paper_citation_jobs j JOIN papers p ON p.id=j.paper_id + WHERE j.status IN ('pending','running') AND p.library_id IS NOT DISTINCT FROM lib), + 'library_papers',(SELECT count(*) FROM papers WHERE library_id IS NOT DISTINCT FROM lib)) INTO result; + RETURN result; +END $$; + +COMMIT; diff --git a/backend/migrations/schema.sql b/backend/migrations/schema.sql index bcffeba..30b0e09 100644 --- a/backend/migrations/schema.sql +++ b/backend/migrations/schema.sql @@ -3,6 +3,8 @@ -- ============================================================================= -- This single file replaces running migrations 001–013 individually. -- New users: paste this into the Supabase SQL Editor and run it once. +-- Then apply numbered migrations 014 onward, including 033_paper_citations.sql +-- for citations; this historical consolidated file only replaces 001-013. -- -- All statements use IF NOT EXISTS / IF NOT EXISTS so the script is safe to -- re-run against a database that already has some tables. diff --git a/backend/models/citation.py b/backend/models/citation.py new file mode 100644 index 0000000..9aa9870 --- /dev/null +++ b/backend/models/citation.py @@ -0,0 +1,65 @@ +"""Citation contracts. Retrieval success is not a claim of complete coverage.""" +from typing import Literal + +from pydantic import Field + +from models.base import CamelModel + +Provider = Literal["openalex", "crossref"] +RunStatus = Literal["succeeded", "partial", "not_found"] + + +class CitationReference(CamelModel): + key: str + identifiers: dict[str, str] = Field(default_factory=dict) + raw: dict = Field(default_factory=dict) + + +class CitationResult(CamelModel): + provider: Provider + status: RunStatus = "succeeded" + resolution: Literal["exact", "candidate", "ambiguous", "unresolved"] = "exact" + identities: dict[str, str] = Field(default_factory=dict) + references: list[CitationReference] = Field(default_factory=list) + candidates: list[dict] = Field(default_factory=list) + message: str | None = None + version: str = "1" + + +class CitationJob(CamelModel): + id: str + paper_id: str + provider: Provider + status: str + attempts: int + updated_at: str + error: str | None = None + + +class CitationEvidence(CamelModel): + provider: str + method: str + indexed_at: str + reference: dict + + +class CitationItem(CamelModel): + id: str + title: str + authors: list[str] + year: int + evidence: list[CitationEvidence] + + +class CitationPage(CamelModel): + items: list[CitationItem] + total: int + limit: int + offset: int + direction: Literal["outgoing", "incoming"] + runs: list[dict] + jobs: list[CitationJob] + unresolved_count: int + indexed_papers: int + library_papers: int + pending_jobs: int = 0 diff --git a/backend/tests/run_citation_sql.mjs b/backend/tests/run_citation_sql.mjs new file mode 100644 index 0000000..e7fbb71 --- /dev/null +++ b/backend/tests/run_citation_sql.mjs @@ -0,0 +1,18 @@ +// Optional disposable PostgreSQL engine: npm install --prefix backend/data/citation-validation @electric-sql/pglite +// Run from any directory: node backend/tests/run_citation_sql.mjs +import { readFile } from 'node:fs/promises' +import { PGlite } from '../data/citation-validation/node_modules/@electric-sql/pglite/dist/index.js' + +const db = new PGlite() +try { + const schema = await readFile(new URL('../migrations/schema.sql', import.meta.url), 'utf8') + // Use the repository's actual library/paper definitions; storage policies need Supabase. + await db.exec(schema.slice(0, schema.indexOf('-- WEBSITES'))) + const migration = await readFile(new URL('../migrations/033_paper_citations.sql', import.meta.url), 'utf8') + await db.exec(migration) + await db.exec(migration) // Migration replay must be safe. + await db.exec(await readFile(new URL('./sql/paper_citations.sql', import.meta.url), 'utf8')) + console.log('Citation SQL transaction checks passed; migration applied twice.') +} finally { + await db.close() +} diff --git a/backend/tests/sql/paper_citations.sql b/backend/tests/sql/paper_citations.sql new file mode 100644 index 0000000..2c31a6d --- /dev/null +++ b/backend/tests/sql/paper_citations.sql @@ -0,0 +1,108 @@ +-- Run after migration 033 on a disposable database. All fixture data rolls back. +BEGIN; +INSERT INTO libraries(id,name,created_at) VALUES ('cite_lib','Citations','2026-11-03'),('cite_other','Other','2026-11-03'); +INSERT INTO papers(id,title,authors,year,venue,status,source,library_id,created_at,doi) VALUES + ('cite_a','Source A','["Author A"]',2025,'','read','human','cite_lib','2026-11-03','10.1234/a'), + ('cite_b','Target B','["Author B"]',2024,'','read','human','cite_lib','2026-11-03','10.1234/b'), + ('cite_other_b','Other library B','[]',2024,'','read','human','cite_other','2026-11-03','10.1234/b'); + +CREATE OR REPLACE FUNCTION pg_temp.publish_citation(p_id TEXT,provider_name TEXT,ids JSONB,refs JSONB, + state TEXT DEFAULT 'succeeded') RETURNS void +LANGUAGE plpgsql AS $$ +DECLARE j public.paper_citation_jobs; +BEGIN + UPDATE public.paper_citation_jobs SET status='pending',next_attempt_at=now() WHERE paper_id=p_id AND provider=provider_name; + SELECT * INTO j FROM public.claim_citation_job((SELECT id FROM public.paper_citation_jobs WHERE paper_id=p_id AND provider=provider_name)); + IF NOT public.finish_citation_job(j.id,j.lease_token,jsonb_build_object( + 'provider',provider_name,'status',state,'resolution','exact','version','test', + 'identities',ids,'references',refs,'candidates','[]'::jsonb)) THEN + RAISE EXCEPTION 'Expected publication'; + END IF; +END $$; + +DO $$ +DECLARE result JSONB; j paper_citation_jobs; old_token UUID; old_id UUID; n INTEGER; +BEGIN + IF (SELECT count(*) FROM paper_citation_jobs WHERE paper_id='cite_a')<>2 THEN + RAISE EXCEPTION 'Insert trigger must enqueue both providers'; + END IF; + PERFORM pg_temp.publish_citation('cite_a','openalex','{"doi":"10.1234/a","openalex":"W1"}', + '[{"key":"W2","identifiers":{"openalex":"W2"},"raw":{"source_work":"W1","target_work":"W2"}}, + {"key":"W1","identifiers":{"openalex":"W1"},"raw":{}}, + {"key":"W999","identifiers":{"openalex":"W999"},"raw":{}}]'); + IF (SELECT count(*) FROM paper_citations WHERE citing_paper_id='cite_a')<>0 THEN + RAISE EXCEPTION 'Unresolved and self references must not create edges'; + END IF; + PERFORM pg_temp.publish_citation('cite_other_b','openalex','{"doi":"10.1234/b","openalex":"W2"}','[]'); + IF (get_paper_citations('cite_a')->>'total')::int<>0 THEN RAISE EXCEPTION 'Cross-library leak'; END IF; + PERFORM pg_temp.publish_citation('cite_b','openalex','{"doi":"10.1234/b","openalex":"W2"}','[]'); + result := get_paper_citations('cite_a'); + IF (result->>'total')::int<>1 OR result#>>'{items,0,id}'<>'cite_b' THEN RAISE EXCEPTION 'Late identity did not reconcile'; END IF; + IF (get_paper_citations('cite_b','incoming')->>'total')::int<>1 + OR (get_paper_citations('cite_b','outgoing')->>'total')::int<>0 THEN RAISE EXCEPTION 'Incorrect direction'; END IF; + IF jsonb_array_length(result#>'{items,0,evidence}')<>1 THEN RAISE EXCEPTION 'Missing evidence'; END IF; + IF (result->>'unresolved_count')::int<>2 THEN RAISE EXCEPTION 'Unresolved count mismatch'; END IF; + IF jsonb_array_length(get_paper_citations('cite_a','outgoing',1,1)->'items')<>0 THEN RAISE EXCEPTION 'Offset ignored'; END IF; + + -- Failure and partial refresh preserve the old published references. + PERFORM enqueue_paper_citations('cite_a',true); + SELECT * INTO j FROM claim_citation_job((SELECT id FROM paper_citation_jobs WHERE paper_id='cite_a' AND provider='openalex')); + IF NOT fail_citation_job(j.id,j.lease_token,'HTTP 429',true,120) THEN RAISE EXCEPTION 'Failure not recorded'; END IF; + IF EXISTS(SELECT 1 FROM claim_citation_job(j.id)) THEN RAISE EXCEPTION 'Retry-After ignored'; END IF; + IF (get_paper_citations('cite_a')->>'total')::int<>1 THEN RAISE EXCEPTION 'Failed refresh destroyed evidence'; END IF; + PERFORM pg_temp.publish_citation('cite_a','openalex','{"doi":"10.1234/a","openalex":"W1"}','[]','partial'); + IF (get_paper_citations('cite_a')->>'total')::int<>1 THEN RAISE EXCEPTION 'Partial refresh destroyed evidence'; END IF; + + -- Multiple providers support one edge, without double counting. + PERFORM pg_temp.publish_citation('cite_a','crossref','{"doi":"10.1234/a"}', + '[{"key":"r1","identifiers":{"doi":"10.1234/b"},"raw":{"DOI":"10.1234/b"}}]'); + result := get_paper_citations('cite_a'); + IF (result->>'total')::int<>1 OR jsonb_array_length(result#>'{items,0,evidence}')<>2 THEN + RAISE EXCEPTION 'Provider evidence was lost or edge counted twice'; + END IF; + PERFORM pg_temp.publish_citation('cite_a','openalex','{"doi":"10.1234/a","openalex":"W1"}','[]'); + IF (get_paper_citations('cite_a')->>'total')::int<>1 THEN RAISE EXCEPTION 'Other provider evidence lost'; END IF; + PERFORM pg_temp.publish_citation('cite_a','crossref','{"doi":"10.1234/a"}','[]'); + IF (get_paper_citations('cite_a')->>'total')::int<>0 THEN RAISE EXCEPTION 'Unsupported edge retained'; END IF; + + -- Expired lease is reclaimed; old worker cannot publish or overwrite new status. + PERFORM enqueue_paper_citations('cite_a',true); + SELECT * INTO j FROM claim_citation_job((SELECT id FROM paper_citation_jobs WHERE paper_id='cite_a' AND provider='openalex')); + old_token:=j.lease_token; old_id:=j.id; + IF EXISTS(SELECT 1 FROM claim_citation_job(j.id)) THEN RAISE EXCEPTION 'Double claim'; END IF; + UPDATE paper_citation_jobs SET lease_until=now()-interval '1 second' WHERE id=j.id; + SELECT * INTO j FROM claim_citation_job(j.id); + IF j.lease_token=old_token OR j.attempts<>2 THEN RAISE EXCEPTION 'Lease not reclaimed'; END IF; + IF finish_citation_job(old_id,old_token,'{}') OR fail_citation_job(old_id,old_token,'old error') THEN + RAISE EXCEPTION 'Old worker was not fenced'; + END IF; + UPDATE papers SET title='Updated Source A' WHERE id='cite_a'; + IF finish_citation_job(j.id,j.lease_token,'{}') THEN RAISE EXCEPTION 'Changed source accepted stale publication'; END IF; + IF EXISTS(SELECT 1 FROM paper_external_ids WHERE paper_id='cite_a' AND asserted_by<>'local') THEN RAISE EXCEPTION 'Identity not invalidated'; END IF; + + -- Ambiguous local duplicate does not choose the first copy. + INSERT INTO papers(id,title,authors,year,venue,status,source,library_id,created_at,doi) + VALUES ('cite_b_duplicate','Target B copy','[]',2024,'','read','human','cite_lib','2026-11-03','10.1234/b'); + PERFORM pg_temp.publish_citation('cite_a','openalex','{"doi":"10.1234/a","openalex":"W1"}', + '[{"key":"W2","identifiers":{"openalex":"W2"},"raw":{}}]'); + IF (get_paper_citations('cite_a')->>'total')::int<>0 THEN + RAISE EXCEPTION 'Unindexed duplicate was ignored by provider-ID matching'; + END IF; + PERFORM pg_temp.publish_citation('cite_b_duplicate','openalex','{"doi":"10.1234/b","openalex":"W2"}','[]'); + PERFORM pg_temp.publish_citation('cite_a','openalex','{"doi":"10.1234/a","openalex":"W1"}', + '[{"key":"W2","identifiers":{"openalex":"W2"},"raw":{}}]'); + IF (get_paper_citations('cite_a')->>'total')::int<>0 THEN RAISE EXCEPTION 'Ambiguous duplicate selected'; END IF; + DELETE FROM papers WHERE id='cite_b_duplicate'; + IF (get_paper_citations('cite_a')->>'total')::int<>1 THEN RAISE EXCEPTION 'Duplicate removal did not reconcile'; END IF; + + -- Reciprocal citations must be independently supported. + PERFORM pg_temp.publish_citation('cite_b','openalex','{"doi":"10.1234/b","openalex":"W2"}', + '[{"key":"W1","identifiers":{"openalex":"W1"},"raw":{}}]'); + IF (get_paper_citations('cite_a','incoming')->>'total')::int<>1 THEN RAISE EXCEPTION 'Reciprocal link missing'; END IF; + DELETE FROM papers WHERE id='cite_b'; + IF EXISTS(SELECT 1 FROM paper_citations WHERE citing_paper_id='cite_b' OR cited_paper_id='cite_b') THEN + RAISE EXCEPTION 'Paper deletion did not cascade'; + END IF; + IF get_paper_citations('missing') IS NOT NULL THEN RAISE EXCEPTION 'Missing paper contract'; END IF; +END $$; +ROLLBACK; diff --git a/docs/database/schema.md b/docs/database/schema.md index 80af3f7..18b77ff 100644 --- a/docs/database/schema.md +++ b/docs/database/schema.md @@ -1,5 +1,20 @@ # Database Schema +## Citation index (migration 033) + +| Table | Purpose | +| --- | --- | +| `paper_external_ids` | Normalized local/provider identifiers with assertion source; IDs can belong to multiple local copies | +| `paper_reference_runs` | Immutable retrieval snapshots, source fingerprint, resolution/status, candidates, and active-run marker | +| `paper_references` | Raw reference evidence and target identifiers, including references with no local match | +| `paper_citations` | Unique directed `(citing_paper_id, cited_paper_id)` edges; self-edges prohibited | +| `paper_citation_evidence` | One or more reference records supporting each edge | +| `paper_citation_jobs` | One durable job per paper/provider, with retry scheduling, leased execution, and optional captured Crossref record | + +Migration 033 provides transaction functions for enqueueing, leasing, publishing, failure recording, reconciliation, and paginated reads. `paper_citations_changed` covers paper inserts, metadata changes, and deletions. Only successful full provider responses replace active evidence; partial/failing attempts retain old evidence. All paper-owned rows cascade on deletion. Reverse citation and identifier lookup indexes support library-scoped matching. RLS follows the repository's existing disabled-RLS model; this feature does not add an authorization boundary. + +See [citation operations](../guides/citation-indexing.md) for installation and worker commands. The historical consolidated `schema.sql` still requires later numbered migrations, including 033. + ## ER Overview ``` From ce5e3d190bfe61cb69e89aa3bd3df4ae31655562 Mon Sep 17 00:00:00 2001 From: Prannaya Gupta Date: Tue, 3 Nov 2026 19:27:38 -0500 Subject: [PATCH 03/23] feat(citations): resolve identities and fetch provider references --- backend/services/citation_identity_service.py | 59 ++++++++++ backend/services/citation_service.py | 102 ++++++++++++++++++ backend/services/openalex_client.py | 55 ++++++++++ backend/tests/test_citation_identity.py | 28 +++++ backend/tests/test_citation_service.py | 98 +++++++++++++++++ 5 files changed, 342 insertions(+) create mode 100644 backend/services/citation_identity_service.py create mode 100644 backend/services/citation_service.py create mode 100644 backend/services/openalex_client.py create mode 100644 backend/tests/test_citation_identity.py create mode 100644 backend/tests/test_citation_service.py diff --git a/backend/services/citation_identity_service.py b/backend/services/citation_identity_service.py new file mode 100644 index 0000000..063128b --- /dev/null +++ b/backend/services/citation_identity_service.py @@ -0,0 +1,59 @@ +"""Conservative identifier handling shared by citation providers.""" +import re +import unicodedata +from urllib.parse import unquote, urlsplit + + +def normalize_doi(value: str | None) -> str | None: + value = unquote(value or "").strip() + value = re.sub(r"^(?:https?://(?:dx\.)?doi\.org/|doi:\s*)", "", value, flags=re.I) + return value.lower() if re.fullmatch(r"10\.\d{4,9}/\S+", value, re.I) else None + + +def normalize_arxiv(value: str | None) -> str | None: + value = (value or "").strip() + if value.startswith(("http://", "https://")): + url = urlsplit(value) + if (url.hostname or "").lower() not in {"arxiv.org", "www.arxiv.org", "export.arxiv.org"}: + return None + value = re.sub(r"^/(?:abs|pdf)/", "", url.path) + value = re.sub(r"^arxiv:\s*", "", value, flags=re.I) + value = re.sub(r"\.pdf$", "", value, flags=re.I) + match = re.fullmatch(r"(\d{4}\.\d{4,5}|[a-z][a-z.\-]+/\d{7})(?:v\d+)?", value, re.I) + return match.group(1).lower() if match else None + + +def normalize_title(value: str) -> str: + value = unicodedata.normalize("NFKC", value).casefold() + return " ".join("".join(c if c.isalnum() or c.isspace() else " " for c in value).split()) + + +def openalex_id(value: str | None) -> str | None: + value = (value or "").removeprefix("https://openalex.org/") + return value if re.fullmatch(r"W\d+", value) else None + + +def local_identifiers(paper: dict) -> dict[str, str]: + return {key: value for key, value in { + "doi": normalize_doi(paper.get("doi")), + "arxiv": normalize_arxiv(paper.get("arxiv_id")), + }.items() if value} + + +def work_identifiers(work: dict) -> dict[str, set[str]]: + ids = work.get("ids") or {} + dois = {normalize_doi(work.get("doi")), normalize_doi(ids.get("doi"))} + arxiv = {normalize_arxiv(ids.get("arxiv"))} + for location in work.get("locations") or []: + dois.add(normalize_doi(location.get("doi"))) + arxiv.update(normalize_arxiv(location.get(key)) for key in ("landing_page_url", "pdf_url")) + return {"doi": dois - {None}, "arxiv": arxiv - {None}} + + +def exact_work_match(paper: dict, work: dict) -> bool: + local = local_identifiers(paper) + remote = work_identifiers(work) + # If both identifiers are present locally, do not accept a contradictory DOI. + if local.get("doi") and remote["doi"] and local["doi"] not in remote["doi"]: + return False + return any(value in remote[key] for key, value in local.items()) diff --git a/backend/services/citation_service.py b/backend/services/citation_service.py new file mode 100644 index 0000000..4751579 --- /dev/null +++ b/backend/services/citation_service.py @@ -0,0 +1,102 @@ +"""Fetch full outgoing reference lists, independently of discovery rankings.""" +import hashlib +import json +from urllib.parse import quote + +import httpx + +from models.citation import CitationReference, CitationResult +from services import openalex_client +from services.citation_identity_service import ( + exact_work_match, local_identifiers, normalize_doi, normalize_title, openalex_id, work_identifiers, +) +from services.openalex_client import CitationProviderError + + +async def fetch_openalex(paper: dict, client: httpx.AsyncClient) -> CitationResult: + ids = local_identifiers(paper) + hits = [] + if ids.get("doi"): + hits = await openalex_client.search(client, filter=f"doi:{ids['doi']}") + exact = [w for w in hits if exact_work_match(paper, w)] + if not exact and ids.get("arxiv"): + hits = await openalex_client.search(client, search=ids["arxiv"]) + exact = [w for w in hits if exact_work_match(paper, w)] + if not exact and paper.get("title"): + hits = await openalex_client.search(client, search=paper["title"]) + exact = [w for w in hits if exact_work_match(paper, w)] + exact = {openalex_id(w.get("id")): w for w in exact if openalex_id(w.get("id"))} + if len(exact) != 1: + candidates = [{"id": w.get("id"), "title": w.get("display_name"), + "year": w.get("publication_year"), + "authors": [a.get("author", {}).get("display_name") for a in w.get("authorships", [])]} + for w in hits if normalize_title(w.get("display_name") or "") == normalize_title(paper.get("title") or "")] + return CitationResult(provider="openalex", status="not_found", identities=ids, + resolution="ambiguous" if len(exact) > 1 else "candidate" if candidates else "unresolved", + candidates=candidates, message="No unique identifier-verified OpenAlex work") + wid, work = next(iter(exact.items())) + resolved_ids = {**ids, "openalex": wid} + # A work verified by one local identifier can supply a unique missing alias. + # Do not choose arbitrarily among multiple preprint/published identifiers. + for kind, values in work_identifiers(work).items(): + if kind not in resolved_ids and len(values) == 1: + resolved_ids[kind] = next(iter(values)) + references = work.get("referenced_works") + if not isinstance(references, list): + return CitationResult(provider="openalex", status="partial", identities=resolved_ids, + message="Provider did not supply a reference list") + result = CitationResult(provider="openalex", identities=resolved_ids) + seen = set() + for target in references: + tid = openalex_id(target) if isinstance(target, str) else None + if not tid: + result.status = "partial" + result.message = "Provider returned an invalid reference identifier" + continue + if tid in seen: + continue + seen.add(tid) + result.references.append(CitationReference(key=tid, identifiers={"openalex": tid}, + raw={"source_work": wid, "target_work": tid})) + return result + + +def parse_crossref(paper: dict, record: dict) -> CitationResult: + ids = local_identifiers(paper) + if not ids.get("doi") or normalize_doi(record.get("DOI")) != ids["doi"]: + raise CitationProviderError("Crossref source DOI did not match") + result = CitationResult(provider="crossref", identities=ids) + references = record.get("reference") + if not isinstance(references, list): + result.status = "partial" + result.message = "Crossref did not supply a reference list" + return result + seen = set() + for ref in references: + if not isinstance(ref, dict): + result.status = "partial" + result.message = "Crossref returned an invalid reference" + continue + key = hashlib.sha256(json.dumps(ref, sort_keys=True).encode()).hexdigest() + if key in seen: + continue + seen.add(key) + doi = normalize_doi(ref.get("DOI")) + result.references.append(CitationReference(key=key, identifiers={"doi": doi} if doi else {}, raw=ref)) + return result + + +async def fetch_crossref(paper: dict, client: httpx.AsyncClient, captured: dict | None = None) -> CitationResult: + doi = normalize_doi(paper.get("doi")) + if not doi: + return CitationResult(provider="crossref", status="not_found", resolution="unresolved", + identities=local_identifiers(paper), message="No DOI available") + if captured is None: + data = await openalex_client.get_json(client, f"https://api.crossref.org/works/{quote(doi, safe='')}") + if data is None: + return CitationResult(provider="crossref", status="not_found", resolution="unresolved", + identities=local_identifiers(paper), message="DOI not found in Crossref") + captured = data.get("message") + if not isinstance(captured, dict): + raise CitationProviderError("Crossref returned no work record") + return parse_crossref(paper, captured) diff --git a/backend/services/openalex_client.py b/backend/services/openalex_client.py new file mode 100644 index 0000000..1324a95 --- /dev/null +++ b/backend/services/openalex_client.py @@ -0,0 +1,55 @@ +"""Bounded provider requests; error messages never include API-key-bearing URLs.""" +import os +from datetime import datetime, timezone +from email.utils import parsedate_to_datetime + +import httpx + + +class CitationProviderError(Exception): + def __init__(self, message: str, *, retryable: bool = False, retry_after: int = 60): + super().__init__(message) + self.retryable = retryable + self.retry_after = retry_after + + +def retry_seconds(value: str | None) -> int: + try: + return max(1, int(value)) + except (ValueError, TypeError): + try: + return max(1, int((parsedate_to_datetime(value) - datetime.now(timezone.utc)).total_seconds())) + except (ValueError, TypeError, OverflowError): + return 60 + + +async def get_json(client: httpx.AsyncClient, url: str, *, params: dict | None = None) -> dict | None: + try: + response = await client.get(url, params=params) + except httpx.RequestError: + raise CitationProviderError("Provider connection failed", retryable=True) from None + if response.status_code == 404: + return None + if response.status_code >= 400: + raise CitationProviderError( + f"Provider returned HTTP {response.status_code}", + retryable=response.status_code == 429 or response.status_code >= 500, + retry_after=retry_seconds(response.headers.get("Retry-After")), + ) + try: + data = response.json() + except ValueError: + raise CitationProviderError("Provider returned invalid JSON", retryable=True) from None + if not isinstance(data, dict): + raise CitationProviderError("Provider returned an invalid response") + return data + + +async def search(client: httpx.AsyncClient, **params) -> list[dict]: + key = os.environ.get("OPENALEX_API_KEY") + if key and key != "*": + params["api_key"] = key + data = await get_json(client, "https://api.openalex.org/works", params={"per_page": 10, **params}) + if data is None or not isinstance(data.get("results"), list): + raise CitationProviderError("OpenAlex returned no results array") + return data["results"] diff --git a/backend/tests/test_citation_identity.py b/backend/tests/test_citation_identity.py new file mode 100644 index 0000000..f7523fc --- /dev/null +++ b/backend/tests/test_citation_identity.py @@ -0,0 +1,28 @@ +import pytest + +from services.citation_identity_service import normalize_doi, normalize_arxiv, normalize_title, exact_work_match + + +@pytest.mark.parametrize('raw,expected', [ + ('https://doi.org/10.1234/ABC', '10.1234/abc'), (' DOI: 10.1234/X ', '10.1234/x'), + ('https://dx.doi.org/10.1234/x', '10.1234/x'), ('not a doi', None), +]) +def test_doi(raw, expected): + assert normalize_doi(raw) == expected + + +@pytest.mark.parametrize('raw,expected', [ + ('https://arxiv.org/pdf/2301.12345v3.pdf', '2301.12345'), + ('arXiv:hep-th/9901001v2', 'hep-th/9901001'), + ('https://example.org/abs/2301.12345', None), +]) +def test_arxiv(raw, expected): + assert normalize_arxiv(raw) == expected + + +def test_unicode_and_conflicting_identifiers(): + assert normalize_title('Énergie: 模型!') == 'énergie 模型' + work = {'doi': 'https://doi.org/10.1234/other', 'locations': [ + {'landing_page_url': 'https://arxiv.org/abs/2301.12345v2'}]} + assert exact_work_match({'arxiv_id': '2301.12345'}, work) + assert not exact_work_match({'doi': '10.1234/local', 'arxiv_id': '2301.12345'}, work) diff --git a/backend/tests/test_citation_service.py b/backend/tests/test_citation_service.py new file mode 100644 index 0000000..3ea5dcc --- /dev/null +++ b/backend/tests/test_citation_service.py @@ -0,0 +1,98 @@ +import asyncio + +import httpx +import pytest + +from services.citation_service import fetch_openalex, fetch_crossref, parse_crossref +from services.openalex_client import CitationProviderError + + +def run_openalex(paper, works): + async def run(): + async with httpx.AsyncClient(transport=httpx.MockTransport( + lambda request: httpx.Response(200, json={'results': works}) + )) as client: + return await fetch_openalex(paper, client) + return asyncio.run(run()) + + +def test_full_references_not_discovery_neighbors(): + work = {'id': 'https://openalex.org/W1', 'doi': 'https://doi.org/10.1234/source', + 'referenced_works': [f'https://openalex.org/W{i}' for i in range(2, 152)], + 'related_works': ['https://openalex.org/W999']} + result = run_openalex({'doi': '10.1234/source'}, [work]) + assert result.status == 'succeeded' + assert len(result.references) == 150 + assert result.references[0].identifiers == {'openalex': 'W2'} + assert all(r.key != 'W999' for r in result.references) + + +def test_title_only_candidate_never_creates_verified_identity(): + result = run_openalex({'title': 'A paper'}, [{'id': 'https://openalex.org/W1', + 'display_name': 'A paper', 'referenced_works': ['https://openalex.org/W2']}]) + assert result.status == 'not_found' + assert result.resolution == 'candidate' + assert result.identities == {} + assert not result.references + + +def test_verified_arxiv_identity_supplies_only_unambiguous_doi_alias(): + work = {'id': 'https://openalex.org/W1', 'doi': 'https://doi.org/10.1234/published', + 'locations': [{'landing_page_url': 'https://arxiv.org/abs/2301.12345v2'}], 'referenced_works': []} + result = run_openalex({'arxiv_id': '2301.12345'}, [work]) + assert result.identities == {'arxiv': '2301.12345', 'doi': '10.1234/published', 'openalex': 'W1'} + work['locations'].append({'doi': 'https://doi.org/10.1234/another-version'}) + assert 'doi' not in run_openalex({'arxiv_id': '2301.12345'}, [work]).identities + + +def test_ambiguous_provider_identity_is_not_accepted(): + result = run_openalex({'doi': '10.1234/x'}, [ + {'id': f'https://openalex.org/W{i}', 'doi': '10.1234/x'} for i in (1, 2)]) + assert result.resolution == 'ambiguous' + assert 'openalex' not in result.identities + + +def test_missing_list_differs_from_empty(): + work = {'id': 'https://openalex.org/W1', 'doi': '10.1234/x'} + assert run_openalex({'doi': '10.1234/x'}, [work]).status == 'partial' + assert run_openalex({'doi': '10.1234/x'}, [{**work, 'referenced_works': []}]).status == 'succeeded' + + +def test_malformed_reference_is_partial_not_retried_as_exception(): + result = run_openalex({'doi': '10.1234/x'}, [{'id': 'https://openalex.org/W1', 'doi': '10.1234/x', + 'referenced_works': ['https://openalex.org/W2', {'unexpected': 'object'}, 'https://openalex.org/W2']}]) + assert result.status == 'partial' + assert len(result.references) == 1 + + +def test_crossref_raw_unresolved_and_deduplication(): + result = parse_crossref({'doi': '10.1234/x'}, {'DOI': '10.1234/x', 'reference': [ + {'DOI': '10.1234/TARGET'}, {'unstructured': 'A bibliography entry'}, {'DOI': '10.1234/TARGET'}]}) + assert len(result.references) == 2 + assert result.references[0].identifiers == {'doi': '10.1234/target'} + assert result.references[1].raw == {'unstructured': 'A bibliography entry'} + assert result.references[1].identifiers == {} + assert parse_crossref({'doi': '10.1234/x'}, {'DOI': '10.1234/x'}).status == 'partial' + + +def test_captured_crossref_avoids_network_and_checks_source(): + async def run(): + async with httpx.AsyncClient(transport=httpx.MockTransport(lambda req: pytest.fail('Unexpected network call'))) as client: + return await fetch_crossref({'doi': '10.1234/x'}, client, {'DOI': '10.1234/x', 'reference': []}) + assert asyncio.run(run()).status == 'succeeded' + with pytest.raises(CitationProviderError, match='did not match'): + parse_crossref({'doi': '10.1234/x'}, {'DOI': '10.1234/wrong'}) + + +def test_rate_limit_retains_retry_after_without_secret(monkeypatch): + monkeypatch.setenv('OPENALEX_API_KEY', 'do-not-leak') + async def run(): + async with httpx.AsyncClient(transport=httpx.MockTransport( + lambda req: httpx.Response(429, headers={'Retry-After': '123'}) + )) as client: + await fetch_openalex({'doi': '10.1234/x'}, client) + with pytest.raises(CitationProviderError) as caught: + asyncio.run(run()) + assert caught.value.retryable + assert caught.value.retry_after == 123 + assert 'do-not-leak' not in str(caught.value) From 71a0dc1cb06ff73196e44b9cc49ce8bb86bc29a6 Mon Sep 17 00:00:00 2001 From: Prannaya Gupta Date: Tue, 3 Nov 2026 19:27:48 -0500 Subject: [PATCH 04/23] feat(citations): add resumable indexing and import capture --- backend/routers/papers.py | 7 +++ backend/scripts/citation_worker.py | 47 +++++++++++++++++ backend/scripts/index_paper_citations.py | 62 ++++++++++++++++++++++ backend/services/citation_job_service.py | 39 ++++++++++++++ backend/services/citation_repository.py | 44 ++++++++++++++++ backend/services/import_service.py | 2 + backend/services/paper_service.py | 28 ++++++++++ backend/tests/test_citation_jobs.py | 65 ++++++++++++++++++++++++ 8 files changed, 294 insertions(+) create mode 100644 backend/scripts/citation_worker.py create mode 100644 backend/scripts/index_paper_citations.py create mode 100644 backend/services/citation_job_service.py create mode 100644 backend/services/citation_repository.py create mode 100644 backend/tests/test_citation_jobs.py diff --git a/backend/routers/papers.py b/backend/routers/papers.py index 5956cb3..9aa9c8c 100644 --- a/backend/routers/papers.py +++ b/backend/routers/papers.py @@ -265,6 +265,13 @@ async def import_paper(data: ImportRequest, background_tasks: BackgroundTasks): library_id=data.library_id, ) paper = paper_service.create_paper(paper_create) + if meta.get("_crossref_record"): + try: + from services.citation_repository import capture_crossref + capture_crossref(paper.id, meta["_crossref_record"]) + except Exception: + # Import succeeds even before migration 033; backfill can recover the references. + logger.warning("Could not capture Crossref references for paper %s", paper.id) logger.info("Imported paper '%s' from identifier '%s'", paper.title, identifier) activity_service.log_activity( type="human", diff --git a/backend/scripts/citation_worker.py b/backend/scripts/citation_worker.py new file mode 100644 index 0000000..ef378c9 --- /dev/null +++ b/backend/scripts/citation_worker.py @@ -0,0 +1,47 @@ +"""Run a durable citation worker. One provider request at a time per process.""" +import argparse +import asyncio +import json +from pathlib import Path +import sys + +BACKEND = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(BACKEND)) +from dotenv import load_dotenv +load_dotenv(BACKEND / ".env") +from services.citation_job_service import run_one + + +async def run(args): + job_ids = None + if args.batch: + manifest = json.loads(args.batch.read_text(encoding="utf-8")) + job_ids = [job for jobs in manifest["jobs"].values() for job in jobs] + while True: + worked = False + for job_id in job_ids if job_ids is not None else [None]: + result = await run_one(job_id) + if result: + worked = True + print(json.dumps(result), flush=True) + if args.once: + return + # Respect provider cooldowns before trying other papers as well. + await asyncio.sleep(max(0, args.interval, result.get("retry_after", 0))) + if not worked: + if args.once or args.drain: + return + await asyncio.sleep(5) + + +if __name__ == "__main__": + parser = argparse.ArgumentParser(description=__doc__) + mode = parser.add_mutually_exclusive_group() + mode.add_argument("--once", action="store_true", help="Process at most one eligible job") + mode.add_argument("--drain", action="store_true", help="Stop when no job is currently eligible; delayed retries remain queued") + parser.add_argument("--batch", type=Path, help="Restrict execution to jobs in a backfill manifest") + parser.add_argument("--interval", type=float, default=1, help="Seconds between jobs (default 1)") + try: + asyncio.run(run(parser.parse_args())) + except KeyboardInterrupt: + pass # An interrupted lease is reclaimed after expiry. diff --git a/backend/scripts/index_paper_citations.py b/backend/scripts/index_paper_citations.py new file mode 100644 index 0000000..a53f2da --- /dev/null +++ b/backend/scripts/index_paper_citations.py @@ -0,0 +1,62 @@ +"""Select a bounded backfill and enqueue durable jobs. No provider calls here.""" +import argparse +import json +import os +from pathlib import Path +import sys +from uuid import uuid4 + +BACKEND = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(BACKEND)) +from dotenv import load_dotenv +load_dotenv(BACKEND / ".env") +from services import citation_repository, paper_service + + +def save_manifest(path: Path, data: dict): + path.parent.mkdir(parents=True, exist_ok=True) + temp = path.with_suffix(".tmp") + temp.write_text(json.dumps(data, indent=2), encoding="utf-8") + os.replace(temp, path) + + +def main(): + parser = argparse.ArgumentParser(description=__doc__) + scope = parser.add_mutually_exclusive_group(required=True) + scope.add_argument("--library-id") + scope.add_argument("--all-libraries", action="store_true") + scope.add_argument("--resume", type=Path, help="Resume an existing batch manifest") + parser.add_argument("--limit", type=int, default=0, help="0 selects all papers in scope") + parser.add_argument("--dry-run", action="store_true") + parser.add_argument("--refresh", action="store_true", help="Also requeue completed jobs") + args = parser.parse_args() + if args.limit < 0: + parser.error("--limit must be nonnegative") + if args.resume: + manifest = args.resume + batch = json.loads(manifest.read_text(encoding="utf-8")) + else: + papers = [] + for paper in paper_service.iter_papers(library_id=args.library_id, all_libraries=args.all_libraries): + papers.append(paper["id"]) + if args.limit and len(papers) >= args.limit: + break + batch = {"id": str(uuid4()), "library_id": args.library_id, "all_libraries": args.all_libraries, + "paper_ids": papers, "jobs": {}, "refresh": args.refresh} + manifest = BACKEND / "data" / "citations" / f"{batch['id']}.json" + if args.dry_run: + print(json.dumps({"papers": len(batch["paper_ids"]), "already_enqueued": len(batch["jobs"]), + "library_id": batch["library_id"], "all_libraries": batch["all_libraries"]})) + return + save_manifest(manifest, batch) + for paper_id in batch["paper_ids"]: + if paper_id in batch["jobs"]: + continue + jobs = citation_repository.enqueue(paper_id, batch["refresh"]) + batch["jobs"][paper_id] = [job["id"] for job in jobs] + save_manifest(manifest, batch) + print(json.dumps({"batch": str(manifest), "papers": len(batch["jobs"])})) + + +if __name__ == "__main__": + main() diff --git a/backend/services/citation_job_service.py b/backend/services/citation_job_service.py new file mode 100644 index 0000000..1444e46 --- /dev/null +++ b/backend/services/citation_job_service.py @@ -0,0 +1,39 @@ +"""One bounded job at a time. Database leases support multiple worker processes.""" +import logging + +import httpx + +from services import citation_repository as repository, paper_service +from services.citation_service import fetch_crossref, fetch_openalex +from services.openalex_client import CitationProviderError + +logger = logging.getLogger(__name__) + + +async def run_one(job_id: str | None = None) -> dict | None: + job = repository.claim(job_id) + if not job: + return None + try: + paper = paper_service.get_paper(job["paper_id"]) + if paper is None: + return {"id": job["id"], "status": "deleted"} + async with httpx.AsyncClient(timeout=30, follow_redirects=True, + headers={"User-Agent": "ResearchOS-Citations/1.0"}) as client: + data = paper.model_dump() + if job["provider"] == "openalex": + result = await fetch_openalex(data, client) + else: + result = await fetch_crossref(data, client, job.get("captured_record")) + published = repository.finish(job, result) + return {"id": job["id"], "status": result.status if published else "superseded"} + except CitationProviderError as exc: + repository.fail(job, str(exc), retry=exc.retryable, + retry_seconds=max(exc.retry_after, 2 ** min(job["attempts"], 10))) + return {"id": job["id"], "status": "retry_pending" if exc.retryable and job["attempts"] < 5 else "failed", + "retry_after": exc.retry_after if exc.retryable else 0} + except Exception as exc: + # Avoid exception strings/traces: HTTP and database errors can contain credentials or payloads. + logger.error("Citation job %s failed (%s)", job["id"], type(exc).__name__) + repository.fail(job, "Internal indexing error; check worker configuration", retry=True) + return {"id": job["id"], "status": "retry_pending" if job["attempts"] < 5 else "failed"} diff --git a/backend/services/citation_repository.py b/backend/services/citation_repository.py new file mode 100644 index 0000000..d10a43a --- /dev/null +++ b/backend/services/citation_repository.py @@ -0,0 +1,44 @@ +"""Database boundary for citation transactions. All mutations run through RPCs.""" +from services.db import get_client + + +def enqueue(paper_id: str, force: bool = False) -> list[dict]: + return get_client().rpc("enqueue_paper_citations", {"p_paper_id": paper_id, "p_force": force}).execute().data or [] + + +def claim(job_id: str | None = None) -> dict | None: + data = get_client().rpc("claim_citation_job", {"p_job_id": job_id}).execute().data + return data[0] if data else None + + +def finish(job: dict, result) -> bool: + return bool(get_client().rpc("finish_citation_job", { + "p_job_id": job["id"], "p_lease_token": job["lease_token"], + "p_result": result.model_dump(), + }).execute().data) + + +def fail(job: dict, message: str, *, retry: bool = False, retry_seconds: int = 60) -> bool: + return bool(get_client().rpc("fail_citation_job", { + "p_job_id": job["id"], "p_lease_token": job["lease_token"], "p_error": message, + "p_retry": retry, "p_retry_seconds": retry_seconds, + }).execute().data) + + +def get_job(job_id: str) -> dict | None: + rows = get_client().table("paper_citation_jobs").select( + "id,paper_id,provider,status,attempts,updated_at,error" + ).eq("id", job_id).execute().data + return rows[0] if rows else None + + +def get_page(paper_id: str, direction: str, limit: int, offset: int) -> dict | None: + return get_client().rpc("get_paper_citations", { + "p_paper_id": paper_id, "p_direction": direction, "p_limit": limit, "p_offset": offset, + }).execute().data + + +def capture_crossref(paper_id: str, record: dict) -> bool: + return bool(get_client().rpc("capture_crossref_references", { + "p_paper_id": paper_id, "p_record": record, + }).execute().data) diff --git a/backend/services/import_service.py b/backend/services/import_service.py index eafd63b..6687f31 100644 --- a/backend/services/import_service.py +++ b/backend/services/import_service.py @@ -152,6 +152,8 @@ async def _fetch_doi(doi: str) -> dict: "arxiv_id": None, "abstract": abstract[:2000] or None, "pdf_url": None, + # Internal provider payload; the import route never inserts this into papers. + "_crossref_record": {key: msg[key] for key in ("DOI", "reference") if key in msg}, } diff --git a/backend/services/paper_service.py b/backend/services/paper_service.py index 76725aa..4aabc28 100644 --- a/backend/services/paper_service.py +++ b/backend/services/paper_service.py @@ -11,6 +11,34 @@ _TABLE = "papers" +def iter_papers(*, library_id: str | None = None, all_libraries: bool = False, + page_size: int = 500): + """Stable keyset traversal, bounded by the initial maximum ID.""" + if not all_libraries and not library_id: + raise ValueError("Select a library or explicitly select all libraries") + if not 1 <= page_size <= 1000: + raise ValueError("page_size must be between 1 and 1000") + + def scoped_query(fields): + query = get_client().table(_TABLE).select(fields) + return query if all_libraries else query.eq("library_id", library_id) + + top = scoped_query("id").order("id", desc=True).limit(1).execute().data + if not top: + return + upper = top[0]["id"] + after = None + while True: + query = scoped_query("*").lte("id", upper).order("id").limit(page_size) + if after is not None: + query = query.gt("id", after) + rows = query.execute().data + if not rows: + return + yield from rows + after = rows[-1]["id"] + + def list_papers( collection_id: Optional[str] = None, status: Optional[str] = None, diff --git a/backend/tests/test_citation_jobs.py b/backend/tests/test_citation_jobs.py new file mode 100644 index 0000000..2afcc60 --- /dev/null +++ b/backend/tests/test_citation_jobs.py @@ -0,0 +1,65 @@ +import asyncio +from types import SimpleNamespace +from unittest.mock import AsyncMock + +from models.citation import CitationResult +from services import citation_job_service as jobs, paper_service +from services.openalex_client import CitationProviderError + + +def setup_job(monkeypatch, provider='openalex'): + job = {'id': 'job1', 'paper_id': 'p1', 'provider': provider, 'lease_token': 'lease', 'attempts': 1} + monkeypatch.setattr(jobs.repository, 'claim', lambda job_id=None: job) + monkeypatch.setattr(jobs.paper_service, 'get_paper', lambda _: SimpleNamespace(model_dump=lambda: {'id': 'p1'})) + return job + + +def test_worker_publish_and_stale_fencing(monkeypatch): + setup_job(monkeypatch) + monkeypatch.setattr(jobs, 'fetch_openalex', AsyncMock(return_value=CitationResult(provider='openalex'))) + monkeypatch.setattr(jobs.repository, 'finish', lambda job, result: False) + assert asyncio.run(jobs.run_one()) == {'id': 'job1', 'status': 'superseded'} + + +def test_retryable_failure_preserves_retry_after(monkeypatch, mocker): + setup_job(monkeypatch) + monkeypatch.setattr(jobs, 'fetch_openalex', AsyncMock(side_effect=CitationProviderError('HTTP 429', retryable=True, retry_after=240))) + failure = mocker.patch.object(jobs.repository, 'fail') + assert asyncio.run(jobs.run_one())['status'] == 'retry_pending' + assert failure.call_args.kwargs == {'retry': True, 'retry_seconds': 240} + + +def test_no_job_does_not_fetch(monkeypatch, mocker): + monkeypatch.setattr(jobs.repository, 'claim', lambda _: None) + fetch = mocker.patch.object(jobs, 'fetch_openalex') + assert asyncio.run(jobs.run_one()) is None + fetch.assert_not_called() + + +def test_crossref_captured_payload_passed_to_fetcher(monkeypatch): + job = setup_job(monkeypatch, 'crossref') + job['captured_record'] = {'DOI': '10.1234/x'} + fetch = AsyncMock(return_value=CitationResult(provider='crossref')) + monkeypatch.setattr(jobs, 'fetch_crossref', fetch) + monkeypatch.setattr(jobs.repository, 'finish', lambda job, result: True) + assert asyncio.run(jobs.run_one())['status'] == 'succeeded' + assert fetch.call_args.args[2] == job['captured_record'] + + +def test_keyset_iterator_handles_server_page_caps(monkeypatch): + rows = [{'id': f'p{i:04}', 'library_id': 'lib'} for i in range(1105)] + class Query: + def __init__(self): + self.after = ''; self.upper = 'z'; self.desc = False; self.size = 500 + def select(self, *_): return self + def eq(self, *_): return self + def gt(self, _, value): self.after = value; return self + def lte(self, _, value): self.upper = value; return self + def order(self, _, desc=False): self.desc = desc; return self + def limit(self, size): self.size = size; return self + def execute(self): + selected = [r for r in rows if self.after < r['id'] <= self.upper] + selected.sort(key=lambda r: r['id'], reverse=self.desc) + return SimpleNamespace(data=selected[:min(self.size, 100)]) + monkeypatch.setattr(paper_service, 'get_client', lambda: SimpleNamespace(table=lambda _: Query())) + assert list(paper_service.iter_papers(library_id='lib')) == rows From ca4b933915e7b6948c3831b8c78f529e6b88ae97 Mon Sep 17 00:00:00 2001 From: Prannaya Gupta Date: Tue, 3 Nov 2026 19:27:56 -0500 Subject: [PATCH 05/23] feat(api): expose local citations and indexing status --- backend/app.py | 2 + backend/routers/citations.py | 44 +++++++++++++++++++++ backend/tests/test_citation_routes.py | 55 +++++++++++++++++++++++++++ docs/api/papers.md | 14 +++++++ 4 files changed, 115 insertions(+) create mode 100644 backend/routers/citations.py create mode 100644 backend/tests/test_citation_routes.py diff --git a/backend/app.py b/backend/app.py index 2ad889f..00d5a7e 100644 --- a/backend/app.py +++ b/backend/app.py @@ -19,6 +19,7 @@ from routers import papers, collections, workflows, runs, proposals, activity, search, libraries, websites, notes, chat, authors, github_repos, settings, notes_copilot, usage, projects, research_questions, experiments, project_notes_copilot, tasks, gap_analysis, batch from services.db import get_client from routers import labs +from routers import citations logging.basicConfig(level=logging.INFO, format="%(levelname)s %(name)s: %(message)s") logger = logging.getLogger(__name__) @@ -46,6 +47,7 @@ async def storage_migration_maintenance(request: Request, call_next): app.include_router(libraries.router) app.include_router(papers.router) +app.include_router(citations.router) app.include_router(websites.router) app.include_router(github_repos.router) app.include_router(collections.router) diff --git a/backend/routers/citations.py b/backend/routers/citations.py new file mode 100644 index 0000000..6ae7719 --- /dev/null +++ b/backend/routers/citations.py @@ -0,0 +1,44 @@ +from typing import Literal +from uuid import UUID + +from fastapi import APIRouter, HTTPException, Query +from postgrest.exceptions import APIError + +from models.citation import CitationJob, CitationPage +from services import citation_repository as repository, paper_service + +router = APIRouter(prefix="/api", tags=["citations"]) + + +def _call(fn, *args): + try: + return fn(*args) + except APIError as exc: + if exc.code in {"42P01", "42883", "PGRST202", "PGRST205"}: + raise HTTPException(503, "Citation indexing is not installed. Apply migration 033.") from None + raise + + +@router.get("/papers/{paper_id}/citations", response_model=CitationPage) +def citations(paper_id: str, direction: Literal["outgoing", "incoming"] = "outgoing", + limit: int = Query(20, ge=1, le=100), offset: int = Query(0, ge=0)): + result = _call(repository.get_page, paper_id, direction, limit, offset) + if result is None: + raise HTTPException(404, {"error": "not_found", "detail": "Paper not found"}) + return CitationPage.model_validate(result) + + +@router.post("/papers/{paper_id}/citations/reindex", status_code=202, response_model=list[CitationJob]) +def reindex(paper_id: str): + jobs = _call(repository.enqueue, paper_id, True) + if not jobs: + raise HTTPException(404, {"error": "not_found", "detail": "Paper not found"}) + return [CitationJob.model_validate(job) for job in jobs] + + +@router.get("/citation-jobs/{job_id}", response_model=CitationJob) +def citation_job(job_id: UUID): + job = _call(repository.get_job, str(job_id)) + if not job or paper_service.get_paper(job["paper_id"]) is None: + raise HTTPException(404, {"error": "not_found", "detail": "Citation job not found"}) + return CitationJob.model_validate(job) diff --git a/backend/tests/test_citation_routes.py b/backend/tests/test_citation_routes.py new file mode 100644 index 0000000..b3965f2 --- /dev/null +++ b/backend/tests/test_citation_routes.py @@ -0,0 +1,55 @@ +from services import citation_repository + + +def page(): + return {'items': [{'id': 'b', 'title': 'B', 'authors': ['Writer'], 'year': 2024, + 'evidence': [{'provider': 'openalex', 'method': 'exact_identifier', + 'indexed_at': '2026-11-03T00:00:00Z', 'reference': {'target_work': 'W2'}}]}], + 'total': 1, 'limit': 20, 'offset': 0, 'direction': 'incoming', 'runs': [], 'jobs': [], + 'unresolved_count': 2, 'indexed_papers': 1, 'library_papers': 2} + + +def test_citation_response_and_direction(client, mocker): + get = mocker.patch.object(citation_repository, 'get_page', return_value=page()) + response = client.get('/api/papers/a/citations?direction=incoming') + assert response.status_code == 200 + assert response.json()['unresolvedCount'] == 2 + assert response.json()['items'][0]['evidence'][0]['indexedAt'] == '2026-11-03T00:00:00Z' + get.assert_called_once_with('a', 'incoming', 20, 0) + + +def test_read_does_not_enqueue_and_missing_paper(client, mocker): + enqueue = mocker.patch.object(citation_repository, 'enqueue') + mocker.patch.object(citation_repository, 'get_page', return_value=None) + assert client.get('/api/papers/missing/citations').json() == {'error': 'not_found', 'detail': 'Paper not found'} + enqueue.assert_not_called() + + +def test_route_validation(client): + assert client.get('/api/papers/a/citations?direction=similar').status_code == 422 + assert client.get('/api/papers/a/citations?limit=1000').status_code == 422 + assert client.get('/api/papers/a/citations?offset=-1').status_code == 422 + assert client.get('/api/citation-jobs/not-a-uuid').status_code == 422 + + +def test_reindex_is_accepted_and_does_not_expose_payload(client, mocker): + mocker.patch.object(citation_repository, 'enqueue', return_value=[{ + 'id': 'job1', 'paper_id': 'a', 'provider': 'openalex', 'status': 'pending', + 'attempts': 0, 'updated_at': '2026-11-03T00:00:00Z', 'captured_record': {'private': 'value'}, + 'lease_token': 'private', + }]) + response = client.post('/api/papers/a/citations/reindex') + assert response.status_code == 202 + assert response.json()[0]['paperId'] == 'a' + assert 'capturedRecord' not in response.json()[0] + assert 'leaseToken' not in response.json()[0] + + +def test_missing_migration_reports_setup_problem(client, mocker): + from postgrest.exceptions import APIError + mocker.patch.object(citation_repository, 'get_page', side_effect=APIError({ + 'code': 'PGRST202', 'message': 'function not found', 'details': '', 'hint': '', + })) + response = client.get('/api/papers/a/citations') + assert response.status_code == 503 + assert 'migration 033' in response.json()['detail'] diff --git a/docs/api/papers.md b/docs/api/papers.md index a91a8ee..b583ccd 100644 --- a/docs/api/papers.md +++ b/docs/api/papers.md @@ -1,5 +1,19 @@ # Papers API +## Citation index + +Requires migration `033_paper_citations.sql` and a separately running citation worker. + +| Endpoint | Behavior | +| --- | --- | +| `GET /api/papers/{id}/citations?direction=outgoing&limit=20&offset=0` | Paginated local references or `direction=incoming` for local citing papers; reads never trigger provider calls | +| `POST /api/papers/{id}/citations/reindex` | HTTP 202 with OpenAlex/Crossref job summaries; pending/running work is reused | +| `GET /api/citation-jobs/{uuid}` | Current status of a durable job; does not expose lease tokens or captured provider payloads | + +The citation page contains `items`, `total`, `limit`, `offset`, `direction`, `runs`, `jobs`, `unresolvedCount`, `indexedPapers`, `libraryPapers`, and `pendingJobs`. Each item contains `id`, `title`, `authors`, `year`, and `evidence`; evidence includes `provider`, `method`, `indexedAt`, and the original provider `reference` object (its keys retain provider casing). Run summaries include `active`, `resolution`, `status`, `indexedAt`, candidate metadata, and messages. + +Both endpoints of a link must belong to the source paper's library. Pagination is ordered by year descending, title, then ID; `limit` is 1–100 and `offset` is nonnegative. Missing papers/jobs return the existing 404 error contract. Missing citation schema returns 503 with installation guidance. Empty results mean no indexed local links, not absence of citations. See [operation and recovery](../guides/citation-indexing.md). + All endpoints are prefixed with `/api/papers`. --- From 6efc1f509f9254d8e54c49a2284dcbf8db0f6339 Mon Sep 17 00:00:00 2001 From: Prannaya Gupta Date: Tue, 3 Nov 2026 19:28:04 -0500 Subject: [PATCH 06/23] feat(ui): show paper references and incoming citations --- frontend/e2e/citations.spec.js | 42 ++++++ frontend/src/components/PaperCitations.jsx | 120 ++++++++++++++++++ .../src/components/PaperCitations.test.jsx | 101 +++++++++++++++ frontend/src/pages/Paper.jsx | 3 + frontend/src/services/api.js | 4 + frontend/src/services/api.test.js | 11 ++ 6 files changed, 281 insertions(+) create mode 100644 frontend/e2e/citations.spec.js create mode 100644 frontend/src/components/PaperCitations.jsx create mode 100644 frontend/src/components/PaperCitations.test.jsx diff --git a/frontend/e2e/citations.spec.js b/frontend/e2e/citations.spec.js new file mode 100644 index 0000000..485ab3f --- /dev/null +++ b/frontend/e2e/citations.spec.js @@ -0,0 +1,42 @@ +import { expect, test } from '@playwright/test' + +test('paper citations show evidence, direction, indexing state, and target navigation', async ({ page }) => { + const paper = id => ({ id, title: id === 'p_a' ? 'Source Paper' : 'Target Paper', + authors: ['Researcher'], year: 2025, venue: 'Journal', status: 'read', source: 'human', + libraryId: 'lib_citations', tags: [], collections: [], links: [], createdAt: '2026-11-03T00:00:00Z' }) + let queued = false + await page.route('**/api/**', async route => { + const url = new URL(route.request().url()) + const path = url.pathname + if (path === '/api/libraries') return route.fulfill({ json: [{ id: 'lib_citations', name: 'Citation Library' }] }) + if (path === '/api/papers/p_a/citations/reindex') { queued = true; return route.fulfill({ status: 202, json: [] }) } + if (/\/papers\/p_[ab]\/citations$/.test(path)) { + const outgoing = url.searchParams.get('direction') === 'outgoing' + const source = path.includes('/p_a/') + const items = source && outgoing ? [{ ...paper('p_b'), evidence: [{ provider: 'openalex', + method: 'exact_identifier', indexedAt: '2026-11-03T00:00:00Z', + reference: { source_work: 'W1', target_work: 'W2' } }] }] : [] + return route.fulfill({ json: { items, total: items.length, limit: 10, offset: 0, + direction: outgoing ? 'outgoing' : 'incoming', unresolvedCount: 1, indexedPapers: 2, libraryPapers: 2, + pendingJobs: queued ? 2 : 0, jobs: queued ? [{ status: 'pending' }] : [], + runs: [{ active: true, provider: 'openalex', indexedAt: '2026-11-03T00:00:00Z' }], + } }) + } + if (/\/papers\/p_[ab]$/.test(path)) return route.fulfill({ json: paper(path.split('/').at(-1)) }) + return route.fulfill({ json: [] }) + }) + await page.goto('/library/paper/p_a') + const citations = page.getByRole('region', { name: 'Paper citations' }) + await expect(citations.getByRole('heading', { name: 'References in this library (1)' })).toBeVisible() + await citations.getByText('Citation evidence').click() + await expect(citations.getByRole('link', { name: 'Source record' })).toHaveAttribute('href', 'https://openalex.org/W1') + await citations.getByRole('button', { name: 'Cited by', exact: true }).click() + await expect(citations.getByText('No indexed citations found.')).toBeVisible() + await citations.getByRole('button', { name: 'References', exact: true }).click() + await citations.getByRole('button', { name: 'Index this paper' }).click() + await expect(citations.getByText('Indexing queued or in progress.')).toBeVisible() + await expect(citations.getByRole('button', { name: 'Indexing…' })).toBeDisabled() + await citations.getByRole('link', { name: 'Target Paper' }).click() + await expect(page).toHaveURL(/\/library\/paper\/p_b$/) + await expect(citations.getByRole('heading', { name: 'References in this library (0)' })).toBeVisible() +}) diff --git a/frontend/src/components/PaperCitations.jsx b/frontend/src/components/PaperCitations.jsx new file mode 100644 index 0000000..aadc61a --- /dev/null +++ b/frontend/src/components/PaperCitations.jsx @@ -0,0 +1,120 @@ +import { useEffect, useRef, useState } from 'react' +import { Link } from 'react-router-dom' +import { papersApi } from '../services/api' + +// Remount on paper/library change so old reads, mutations and polls cannot leak state. +export default function PaperCitations({ paperId, libraryId, sourceVersion = '' }) { + return +} + +function CitationLists({ paperId }) { + const [direction, setDirection] = useState('outgoing') + const [offset, setOffset] = useState(0) + const [data, setData] = useState(null) + const [error, setError] = useState(null) + const [loading, setLoading] = useState(true) + const [queueing, setQueueing] = useState(false) + const [refresh, setRefresh] = useState(0) + const alive = useRef(true) + useEffect(() => { alive.current = true; return () => { alive.current = false } }, []) + + useEffect(() => { + let cancelled = false + let timer + setLoading(true) + setError(null) + async function load() { + try { + const page = await papersApi.citations(paperId, { direction, offset, limit: 10 }) + if (cancelled) return + if (!Array.isArray(page?.items) || !Array.isArray(page?.jobs) || !Array.isArray(page?.runs)) { + throw new Error('Citation data could not be loaded.') + } + setData(page) + setLoading(false) + // Incoming links can arrive when any source in the library is indexed. + if (page.pendingJobs > 0 || page.jobs.some(job => ['pending', 'running'].includes(job.status))) { + timer = setTimeout(load, 10000) + } + } catch (err) { + if (!cancelled) { setError(err.message); setLoading(false) } + } + } + load() + return () => { cancelled = true; clearTimeout(timer) } + }, [paperId, direction, offset, refresh]) + + async function reindex() { + setQueueing(true) + setError(null) + try { + await papersApi.reindexCitations(paperId) + if (alive.current) setRefresh(value => value + 1) + } catch (err) { + if (alive.current) setError(err.message) + } finally { + if (alive.current) setQueueing(false) + } + } + + const jobs = data?.jobs || [] + const busy = queueing || jobs.some(job => ['pending', 'running'].includes(job.status)) + const activeRuns = (data?.runs || []).filter(run => run.active) + const failed = jobs.some(job => job.status === 'failed') + const partial = jobs.some(job => ['partial', 'not_found'].includes(job.status)) + const lastIndexed = activeRuns.map(run => run.indexedAt).sort().at(-1) + const label = direction === 'outgoing' ? 'References in this library' : 'Cited by in this library' + + return ( +
+

Citations

+
+ {['outgoing', 'incoming'].map(value => ( + + ))} +
+

{label}{data ? ` (${data.total})` : ''}

+ {error &&
{error}
} + {loading &&

Loading citations…

} + {data && <> + {busy &&

Indexing queued or in progress.

} + {failed &&

Indexing failed. {activeRuns.length > 0 && 'Showing previously indexed evidence.'}

} + {partial &&

Some references or paper identities are unavailable. Coverage is incomplete.

} + {!activeRuns.length && !busy && !failed && !partial &&

Not indexed yet.

} + {lastIndexed &&

Last indexed: {new Date(lastIndexed).toLocaleString()}

} + {!data.items.length &&

No indexed citations found.

} +
    + {data.items.map(item =>
  • + {item.title} +

    {item.authors.slice(0, 2).join(', ')}{item.year ? ` · ${item.year}` : ''}

    +
    + Citation evidence + {item.evidence.map((evidence, index) =>
    +

    {evidence.provider === 'openalex' ? 'OpenAlex' : 'Crossref'} · identifier match · {new Date(evidence.indexedAt).toLocaleDateString()}

    + {evidence.provider === 'openalex' && /^W\d+$/.test(evidence.reference.source_work) && + Source record} + {evidence.reference.unstructured &&

    {evidence.reference.unstructured}

    } + {evidence.reference.DOI &&

    DOI: {evidence.reference.DOI}

    } + {evidence.reference.target_work &&

    Referenced work: {evidence.reference.target_work}

    } +
    )} +
    +
    • )} +
+ {data.total > 10 &&
+ + {offset + 1}–{Math.min(offset + 10, data.total)} of {data.total} + +
} +

{data.indexedPapers} of {data.libraryPapers} library papers have indexed reference data.

+ {data.unresolvedCount > 0 &&

{data.unresolvedCount} reference records not matched to a unique local citation.

} + } + +
+ ) +} diff --git a/frontend/src/components/PaperCitations.test.jsx b/frontend/src/components/PaperCitations.test.jsx new file mode 100644 index 0000000..52dedc3 --- /dev/null +++ b/frontend/src/components/PaperCitations.test.jsx @@ -0,0 +1,101 @@ +import { act, fireEvent, render, screen, waitFor } from '@testing-library/react' +import { MemoryRouter } from 'react-router-dom' +import { beforeEach, expect, it, vi } from 'vitest' +import PaperCitations from './PaperCitations' +import { papersApi } from '../services/api' + +vi.mock('../services/api', () => ({ papersApi: { citations: vi.fn(), reindexCitations: vi.fn() } })) +beforeEach(() => vi.resetAllMocks()) +const page = (overrides = {}) => ({ items: [], total: 0, limit: 10, offset: 0, direction: 'outgoing', + jobs: [], runs: [], unresolvedCount: 0, indexedPapers: 0, libraryPapers: 2, pendingJobs: 0, ...overrides }) +const target = { id: 'p_b', title: 'Target paper', authors: ['A Writer'], year: 2024, + evidence: [{ provider: 'openalex', method: 'exact_identifier', indexedAt: '2026-11-03T00:00:00Z', + reference: { source_work: 'W1', target_work: 'W2' } }] } +function view(paperId = 'p_a') { return } + +it('shows direction, evidence, and local navigation', async () => { + papersApi.citations.mockResolvedValue(page({ items: [target], total: 1 })) + render(view()) + expect(await screen.findByRole('link', { name: 'Target paper' })).toHaveAttribute('href', '/library/paper/p_b') + expect(screen.getByRole('heading', { name: 'References in this library (1)' })).toBeInTheDocument() + expect(screen.getByRole('link', { name: 'Source record' })).toHaveAttribute('href', 'https://openalex.org/W1') + fireEvent.click(screen.getByRole('button', { name: 'Cited by' })) + await screen.findByRole('heading', { name: 'Cited by in this library (1)' }) + expect(papersApi.citations).toHaveBeenLastCalledWith('p_a', { direction: 'incoming', offset: 0, limit: 10 }) +}) + +it('distinguishes unindexed, failed refresh, and previous evidence', async () => { + papersApi.citations.mockResolvedValueOnce(page()).mockResolvedValue(page({ + items: [target], total: 1, jobs: [{ status: 'failed' }], + runs: [{ active: true, indexedAt: '2026-11-03T00:00:00Z' }], + })) + const result = render(view()) + await screen.findByText('Not indexed yet.') + result.rerender(view('p_other')) + await screen.findByText('Indexing failed. Showing previously indexed evidence.') + expect(screen.getByRole('link', { name: 'Target paper' })).toBeInTheDocument() + expect(screen.queryByText('Not indexed yet.')).not.toBeInTheDocument() +}) + +it('retries read failures without automatically indexing', async () => { + papersApi.citations.mockRejectedValueOnce(new Error('Unavailable')).mockResolvedValue(page()) + render(view()) + expect(await screen.findByRole('alert')).toHaveTextContent('Unavailable') + fireEvent.click(screen.getByRole('button', { name: 'Retry' })) + await screen.findByText('No indexed citations found.') + expect(papersApi.reindexCitations).not.toHaveBeenCalled() +}) + +it('keeps the paper usable when a server returns a malformed citation response', async () => { + papersApi.citations.mockResolvedValue([]) + render(view()) + expect(await screen.findByRole('alert')).toHaveTextContent('Citation data could not be loaded.') + expect(screen.getByRole('heading', { name: 'Citations', exact: true })).toBeInTheDocument() +}) + +it('ignores a response from a previous paper', async () => { + let resolveOld + papersApi.citations.mockReturnValueOnce(new Promise(resolve => { resolveOld = resolve })).mockResolvedValue(page()) + const result = render(view()) + result.rerender(view('p_other')) + await screen.findByText('Not indexed yet.') + await act(async () => resolveOld(page({ items: [target], total: 1 }))) + expect(screen.queryByText('Target paper')).not.toBeInTheDocument() +}) + +it('refreshes after source metadata changes on the same paper', async () => { + papersApi.citations.mockResolvedValueOnce(page({ items: [target], total: 1 })).mockResolvedValue(page({ jobs: [{ status: 'pending' }] })) + const result = render() + await screen.findByText('Target paper') + result.rerender() + await screen.findByText('Indexing queued or in progress.') + expect(screen.queryByText('Target paper')).not.toBeInTheDocument() + expect(papersApi.citations).toHaveBeenCalledTimes(2) +}) + +it('paginates without mixing directions', async () => { + papersApi.citations.mockImplementation((id, { offset }) => Promise.resolve(page({ + total: 11, offset, items: [{ ...target, id: `p_${offset}`, title: `Page ${offset}` }], + }))) + render(view()) + await screen.findByText('Page 0') + fireEvent.click(screen.getByRole('button', { name: 'Next' })) + await screen.findByText('Page 10') + expect(screen.queryByText('Page 0')).not.toBeInTheDocument() + fireEvent.click(screen.getByRole('button', { name: 'Cited by' })) + await waitFor(() => expect(papersApi.citations).toHaveBeenLastCalledWith('p_a', { direction: 'incoming', offset: 0, limit: 10 })) +}) + +it('queues explicitly and preserves visible links while awaiting the worker', async () => { + papersApi.citations.mockResolvedValueOnce(page({ items: [target], total: 1 })).mockResolvedValue(page({ + items: [target], total: 1, jobs: [{ status: 'pending' }], pendingJobs: 1, + })) + papersApi.reindexCitations.mockResolvedValue([{ id: 'job1', status: 'pending' }]) + render(view()) + await screen.findByText('Target paper') + fireEvent.click(screen.getByRole('button', { name: 'Index this paper' })) + await screen.findByText('Indexing queued or in progress.') + expect(papersApi.reindexCitations).toHaveBeenCalledWith('p_a') + expect(screen.getByRole('button', { name: 'Indexing…' })).toBeDisabled() + expect(screen.getByRole('link', { name: 'Target paper' })).toBeInTheDocument() +}) diff --git a/frontend/src/pages/Paper.jsx b/frontend/src/pages/Paper.jsx index ad4af30..543e001 100644 --- a/frontend/src/pages/Paper.jsx +++ b/frontend/src/pages/Paper.jsx @@ -2,6 +2,7 @@ import { useState, useEffect, useRef, useCallback } from 'react' import { useParams, useNavigate, useLocation } from 'react-router-dom' import { papersApi, notesApi, projectsApi, projectPapersApi } from '../services/api' import PaperInfoPanel from '../components/PaperInfoPanel' +import PaperCitations from '../components/PaperCitations' import NotesPanel from '../components/NotesPanel' import CopilotPanel from '../components/CopilotPanel' import { useDragResize } from '../hooks/useDragResize' @@ -522,6 +523,8 @@ export default function Paper() { onStatusChange={(_, newStatus) => setPaper(p => ({ ...p, status: newStatus }))} onPaperUpdate={setPaper} /> + )} {sideTab === 'notes' && ( diff --git a/frontend/src/services/api.js b/frontend/src/services/api.js index 0dc7e56..300a476 100644 --- a/frontend/src/services/api.js +++ b/frontend/src/services/api.js @@ -44,6 +44,10 @@ export const papersApi = { }, get: (id) => apiFetch(`/papers/${id}`), labs: (id) => apiFetch(`/papers/${encodeURIComponent(id)}/labs`), + citations: (id, { direction = 'outgoing', limit = 20, offset = 0 } = {}) => + apiFetch(`/papers/${encodeURIComponent(id)}/citations?${new URLSearchParams({ direction, limit, offset })}`), + reindexCitations: (id) => apiFetch(`/papers/${encodeURIComponent(id)}/citations/reindex`, { method: 'POST' }), + citationJob: (id) => apiFetch(`/citation-jobs/${encodeURIComponent(id)}`), related: (id, { limit = 12 } = {}) => apiFetch(`/papers/${id}/related?limit=${encodeURIComponent(limit)}`), create: (data, { checkDuplicates = false } = {}) => { const qs = checkDuplicates ? '?check_duplicates=true' : '' diff --git a/frontend/src/services/api.test.js b/frontend/src/services/api.test.js index a9c4fac..e566946 100644 --- a/frontend/src/services/api.test.js +++ b/frontend/src/services/api.test.js @@ -252,4 +252,15 @@ describe('api service wrapper', () => { expect(global.fetch.mock.calls[0][0]).toBe('/api/papers/p_42/related?limit=9') }) + + it('citation endpoints encode IDs and preserve direction and pagination', async () => { + global.fetch.mockResolvedValue({ ok: true, status: 200, json: async () => ({ items: [] }) }) + await papersApi.citations('p/a', { direction: 'incoming', offset: 20, limit: 10 }) + await papersApi.reindexCitations('p/a') + await papersApi.citationJob('job-1') + expect(global.fetch.mock.calls[0][0]).toBe('/api/papers/p%2Fa/citations?direction=incoming&limit=10&offset=20') + expect(global.fetch.mock.calls[1][0]).toBe('/api/papers/p%2Fa/citations/reindex') + expect(global.fetch.mock.calls[1][1].method).toBe('POST') + expect(global.fetch.mock.calls[2][0]).toBe('/api/citation-jobs/job-1') + }) }) From 9183413503089fe921b00522b14d8545503b623a Mon Sep 17 00:00:00 2001 From: Prannaya Gupta Date: Tue, 3 Nov 2026 19:28:13 -0500 Subject: [PATCH 07/23] docs(citations): record coverage pilot and operating procedures --- backend/scripts/pilot_paper_citations.py | 139 +++++++++++++++++++++++ docs/citation-indexing-pilot-report.md | 49 ++++++++ docs/guides/citation-indexing.md | 81 +++++++++++++ 3 files changed, 269 insertions(+) create mode 100644 backend/scripts/pilot_paper_citations.py create mode 100644 docs/citation-indexing-pilot-report.md create mode 100644 docs/guides/citation-indexing.md diff --git a/backend/scripts/pilot_paper_citations.py b/backend/scripts/pilot_paper_citations.py new file mode 100644 index 0000000..94b6db8 --- /dev/null +++ b/backend/scripts/pilot_paper_citations.py @@ -0,0 +1,139 @@ +"""Read-only coverage pilot. Fetch public metadata and write a local report; never mutate the database.""" +import argparse +import asyncio +from collections import Counter, defaultdict +from datetime import datetime, timezone +import json +from pathlib import Path +import sys +import time + +BACKEND = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(BACKEND)) +from dotenv import load_dotenv +load_dotenv(BACKEND / '.env') +import httpx +from services import openalex_client, paper_service +from services.citation_service import fetch_crossref, fetch_openalex +from services.citation_identity_service import exact_work_match, local_identifiers, openalex_id, work_identifiers +from services.openalex_client import CitationProviderError + + +def stratum(paper): + ids = local_identifiers(paper) + return 'doi' if ids.get('doi') else 'arxiv' if ids.get('arxiv') else 'pdf_only' if paper.get('pdf_url') else 'metadata_only' + + +async def pilot(args): + start = time.monotonic() + papers = list(paper_service.iter_papers(library_id=args.library_id, all_libraries=args.all_libraries)) + groups = defaultdict(list) + for paper in papers: + groups[stratum(paper)].append(paper) + selected = [p for kind in ('doi', 'arxiv', 'pdf_only') for p in sorted(groups[kind], key=lambda p: p['id'])[:args.per_stratum]] + report = {'created_at': datetime.now(timezone.utc).isoformat(), 'scope': args.library_id or 'all-libraries', + 'population': len(papers), 'strata': {k: len(v) for k,v in groups.items()}, + 'sources': [], 'edges': [], 'errors': [], 'target_resolution_complete': True} + lookup = defaultdict(set) + by_id = {p['id']: p for p in papers} + for paper in papers: + for kind, value in local_identifiers(paper).items(): + lookup[(kind,value)].add(paper['id']) + openalex_local = defaultdict(set) + requested = set() + stopped = set() + requests = 0 + async def count_request(request): + nonlocal requests + requests += 1 + def checkpoint(): + args.output.parent.mkdir(parents=True, exist_ok=True) + args.output.write_text(json.dumps(report, indent=2), encoding='utf-8') + async with httpx.AsyncClient(timeout=30, follow_redirects=True, event_hooks={'request':[count_request]}, + headers={'User-Agent':'ResearchOS-Citation-Pilot/1.0'}) as client: + for paper in selected: + source = {k: paper.get(k) for k in ('id','title','library_id','doi','arxiv_id','pdf_url')} + source['stratum'] = stratum(paper) + source['providers'] = {} + for provider, fetch in (('openalex',fetch_openalex), ('crossref',fetch_crossref)): + if provider in stopped: + source['providers'][provider] = {'status':'skipped', 'reason':'Provider authentication/rate limit stopped this pilot'} + continue + try: + result = await fetch(paper, client) + source['providers'][provider] = result.model_dump() + if result.identities.get('openalex'): + openalex_local[result.identities['openalex']].add(paper['id']) + requested.update(r.identifiers['openalex'] for r in result.references if r.identifiers.get('openalex')) + except CitationProviderError as exc: + source['providers'][provider] = {'status':'failed','reason':str(exc)} + report['errors'].append({'provider':provider,'paper_id':paper['id'],'message':str(exc)}) + if any(code in str(exc) for code in ('401','403','429')): + stopped.add(provider) + await asyncio.sleep(args.interval) + report['sources'].append(source) + checkpoint() + print(json.dumps({'source':paper['id'],'stratum':source['stratum'], + 'statuses':{k:v['status'] for k,v in source['providers'].items()}}),flush=True) + # Resolve reference identities against the whole local population, not only sampled sources. + target_ids = sorted(requested - set(openalex_local)) + for offset in range(0,len(target_ids),100): + if 'openalex' in stopped: + report['target_resolution_complete'] = False + break + try: + works = await openalex_client.search(client, filter='openalex:'+'|'.join(target_ids[offset:offset+100]), + per_page=100, select='id,doi,ids,locations,display_name') + for work in works: + wid = openalex_id(work.get('id')) + for kind,values in work_identifiers(work).items(): + for value in values: + for pid in lookup[(kind,value)]: + if wid and exact_work_match(by_id[pid],work): + openalex_local[wid].add(pid) + except CitationProviderError as exc: + report['errors'].append({'provider':'openalex-targets','message':str(exc)}) + report['target_resolution_complete'] = False + break + await asyncio.sleep(args.interval) + for source in report['sources']: + for provider,data in source['providers'].items(): + if data['status']!='succeeded': + continue + for ref in data['references']: + matches = set() + for kind,value in ref['identifiers'].items(): + candidates = openalex_local[value] if kind=='openalex' else lookup[(kind,value)] + matches.update(pid for pid in candidates if by_id[pid].get('library_id')==source['library_id']) + if len(matches)!=1: + continue + target = by_id[next(iter(matches))] + if target['id']==source['id']: + continue + srcids = local_identifiers(source) + if any(srcids.get(k)==v for k,v in local_identifiers(target).items()): + continue + report['edges'].append({'source_id':source['id'],'source_title':source['title'], + 'target_id':target['id'],'target_title':target['title'],'provider':provider,'reference':ref['raw']}) + report['requests'] = requests + report['elapsed_seconds'] = round(time.monotonic()-start,1) + report['unique_pairs'] = len({(e['source_id'],e['target_id']) for e in report['edges']}) + report['provider_statuses'] = {name:dict(Counter(s['providers'][name]['status'] for s in report['sources'])) for name in ('openalex','crossref')} + checkpoint() + print(json.dumps({k:report[k] for k in ('population','provider_statuses','unique_pairs','requests','elapsed_seconds','errors')})) + + +if __name__=='__main__': + parser = argparse.ArgumentParser(description=__doc__) + scope = parser.add_mutually_exclusive_group(required=True) + scope.add_argument('--library-id') + scope.add_argument('--all-libraries',action='store_true') + parser.add_argument('--per-stratum',type=int,default=10) + parser.add_argument('--interval',type=float,default=1) + parser.add_argument('--output',type=Path,required=True) + args = parser.parse_args() + if args.per_stratum<1 or args.interval<0: + parser.error('Sample size must be positive and interval nonnegative') + if args.output.exists(): + parser.error('Choose a new output path; existing pilot reports are not overwritten') + asyncio.run(pilot(args)) diff --git a/docs/citation-indexing-pilot-report.md b/docs/citation-indexing-pilot-report.md new file mode 100644 index 0000000..9e351fb --- /dev/null +++ b/docs/citation-indexing-pilot-report.md @@ -0,0 +1,49 @@ +# Citation indexing pilot — November 3, 2026 + +The approved read-only pilot found one within-library citation pair, verified in the source PDF: **MAGiC-SLAM: Multi-Agent Gaussian Globally Consistent SLAM cites SplaTAM: Splat, Track & Map 3D Gaussians for Dense RGB-D SLAM**. Page 9, bibliography entry [15], names the target. OpenAlex reported the edge from `W4413157515` to `W4402698355`. + +No migration, backfill, or other live-database write was performed. This pilot used the new provider clients and local matching, not production citation tables. + +## Population and sample + +The inventory contained 391 paper rows in seven libraries: 31 DOI-bearing, 233 additional arXiv-only, 10 PDF-only without DOI/arXiv, and 117 other metadata-only records. Four DOI records also have arXiv IDs, giving 237 arXiv-bearing records overall. There were 254 PDF URLs; availability of every PDF was not checked. + +The sample selected the first ten stable paper IDs in each DOI, arXiv-only, and PDF-only group: 30 sources total. This is a deterministic feasibility sample, not a random estimate of database coverage. Metadata-only records were excluded. Targets were matched against all 391 rows, restricting both ends to the same library. + +| Group | Sources | Exact OpenAlex identities | OpenAlex sources with nonempty references | OpenAlex reference records | Crossref reference records | +| --- | ---: | ---: | ---: | ---: | ---: | +| DOI | 10 | 8 | 7 | 269 | 460 | +| arXiv only | 10 | 10 | 1 | 38 | 0 | +| PDF only | 10 | 0 | 0 | 0 | 0 | + +OpenAlex retrieved records for 18 sources and found no identifier-verified match for 12. Ten of the successful responses had empty reference arrays (one DOI source and nine arXiv-only sources). Crossref returned references for nine DOI sources; one DOI was not found. The remaining twenty sources had no DOI and caused no Crossref network requests. The 460 Crossref references are provider records, not 460 additional unique citations. + +The run made 56 HTTP requests in 80.2 seconds, without provider errors. Referenced-work identity retrieval completed. No OpenAlex key was configured; this does not establish a safe unauthenticated bulk quota. + +## Local edges and verification + +One unique local pair was detected, supported by OpenAlex. Crossref added no local pairs in this sample, although it supplied references for two DOI sources that OpenAlex did not resolve. Crossref DOI references can still miss targets stored only as arXiv records without a verified DOI alias. + +Source paper `p_0246e188` cites target `p_28219176`. The source PDF was read from its existing configured GCS object and inspected using PyMuPDF; entry [15] confirms the target title and authors. This verifies one edge, not a general precision estimate. There was no independently curated set of known positive pairs, so recall was not estimated. + +Sample IDs, metadata, references, candidates, and the detected edge are preserved in ignored `backend/data/citations/pilot-2026-11-03.json`. The verification PDF is ignored at `backend/data/citations/p_0246e188.pdf`. These working artifacts are not committed. + +## Implications + +Provider-only coverage is inadequate for much of this collection. All ten sampled arXiv papers resolved to identities, but nine supplied no references. Successful retrieval must remain distinct from bibliography completeness in API/UI wording. + +PDF-only records need identifier enrichment, bibliography extraction, or reviewed identity matching. The resolver deliberately does not promote title-only results to citation evidence. + +GROBID is the next recommended evaluation for stored PDF bibliographies, but no GROBID service or parser benchmark was configured during this work. Semantic Scholar was not queried with library metadata: the pilot approval covered OpenAlex and Crossref. These fallbacks, OCR, and automated PDF verification remain follow-up work, not completed features. + +## Validation and rollout boundaries + +Migration 033 was applied twice in disposable PGlite PostgreSQL. Transaction fixtures verified direction, late identities, library scope, evidence preservation/replacement, duplicates, stale leases, source edits, reciprocal links, and deletion. Backend and frontend suites, the production build, and the citation browser fixture passed; final counts are tracked in the execution plan. + +Final counts: 340 backend tests and 401 frontend tests passed. The combined citation/library browser run passed 13 of 14 tests, including the citation test. The unchanged row-context-menu test timed out opening/clicking its Copy & export submenu, including on an isolated rerun; that implementation was not modified here. + +The full frontend suite requires `TZ=UTC` for the unchanged Dashboard chart test, whose UTC-midnight fixtures otherwise group into the preceding local date. Citation tests pass in the local timezone as well. + +After the pilot, a tested resolver improvement was added: an identifier-verified OpenAlex work can contribute a unique missing DOI/arXiv alias. Multiple possible aliases remain unaccepted. The pilot was not rerun for this improvement, so the observed edge count above is unchanged and does not predict alias-enriched coverage. + +Production multi-connection contention, Supabase role behavior, worker deployment, and the persisted full-library backfill remain rollout checks. This pilot establishes feasibility and coverage gaps; it is not deployment completion. diff --git a/docs/guides/citation-indexing.md b/docs/guides/citation-indexing.md new file mode 100644 index 0000000..301bb70 --- /dev/null +++ b/docs/guides/citation-indexing.md @@ -0,0 +1,81 @@ +# Citation indexing + +Paper details show outgoing references and incoming citations between papers in the same library, with provider reference evidence. OpenAlex and Crossref indexing runs in a separate durable worker. Reading the page does not contact a provider. + +## Installation + +Apply `backend/migrations/033_paper_citations.sql` after the existing numbered migrations. The historical consolidated `schema.sql` covers migrations 001–013 only and does not replace migration 033. This additive, replayable migration registers existing local DOI/arXiv identifiers but does not enqueue a historical backfill or contact external services. + +Restart the application with this branch. Before migration, citation endpoints return a setup-specific HTTP 503 without preventing ordinary paper operations. Set a valid `OPENALEX_API_KEY` in `backend/.env` for bulk use; credentials stay server-side. Crossref does not require a separate key. + +Start the worker from the repository root: + +```powershell +uv run --directory backend python scripts/citation_worker.py +``` + +The process handles one job at a time with a default one-second interval. Multiple workers can claim distinct jobs, but begin with one to respect provider limits. No daemon or scheduled task is installed automatically. `--once` processes at most one eligible job; `--drain` stops when no jobs are currently eligible, leaving delayed retries queued. + +## Backfill and resume + +Select an explicit scope. Dry run reports selected counts without provider requests or database writes: + +```powershell +uv run --directory backend python scripts/index_paper_citations.py --library-id LIBRARY_ID --dry-run +uv run --directory backend python scripts/index_paper_citations.py --library-id LIBRARY_ID +``` + +The enqueue command prints a manifest under ignored `backend/data/citations`. Keep it to resume enqueueing or restrict a worker to that batch: + +```powershell +uv run --directory backend python scripts/index_paper_citations.py --resume data/citations/BATCH_ID.json +uv run --directory backend python scripts/citation_worker.py --batch data/citations/BATCH_ID.json --drain +``` + +`--all-libraries` selects all visible papers; matching remains separately scoped to each library. `--limit N` bounds the source population. `--refresh` requeues completed jobs. The local manifest freezes selected IDs; leases, retries, status, and evidence live in PostgreSQL. Keyset pagination avoids Supabase response caps. Concurrent inserts are covered by the database trigger even when outside the manifest population. + +All paper insertion paths enqueue providers transactionally, independent of automatic notes. Changes to DOI, arXiv ID, title, authors, year, library, or PDF URL invalidate derived identities/evidence and requeue. DOI imports attach captured Crossref reference records to pending jobs, avoiding a repeat request when the worker has not already claimed the job. Imports remain usable if payload capture fails; backfill can refetch it. Inserts, edits, and deletions reconcile retained references against currently known local identities. + +## Matching and status + +Exact normalized DOI or verified arXiv identifier/location matches establish provider identities. Title-search results alone cannot. Candidate metadata is retained; a candidate-approval UI is deferred. OpenAlex IDs and Crossref reference DOIs match against local identities. Ambiguous duplicate targets remain unresolved. Shared identifiers suppress edges between copies of the same work. Semantic neighbors never create citations. + +`A -> B` means A cites B. Incoming citations query the same edges in reverse. Provider assertions are not labeled PDF-verified. Original identifiers and reference records remain available when a provider relates preprint and published versions. + +Jobs progress from `pending` to `running`, then `succeeded`, `partial`, `not_found`, or `failed`. Transient failures have up to five attempts with provider retry delays. Five-minute leases recover after crashes, and stale workers cannot publish after lease reclamation or source changes. Failed/partial refresh preserves previous good evidence. Successful refresh atomically replaces only that provider's evidence; edges survive while another provider still supports them. + +The UI distinguishes unindexed, failed, and incomplete states. Incoming counts can remain incomplete until other source papers are indexed. Empty provider reference arrays are successful retrievals, not proof of empty bibliographies. Unmatched counts represent reference records, including ambiguous/self-references, rather than missing papers. + +Publication transactions are serialized and reconcile a library at a time; network requests happen outside transactions. Measure multi-worker contention and large-library throughput before increasing worker counts. + +## Coverage pilot + +The pilot reads database metadata, sends selected titles/identifiers to OpenAlex and Crossref, retrieves referenced-work identities, and saves local JSON. It never writes to the database. Use only a scope approved for those external services: + +```powershell +uv run --directory backend python scripts/pilot_paper_citations.py --library-id LIBRARY_ID --per-stratum 10 --output data/citations/pilot.json +``` + +It samples up to ten DOI, ten arXiv-only, and ten PDF-only records by stable ID, matching targets against the full selected scope. Authentication/rate-limit failures stop remaining requests to that provider. Reports include denominators, errors, requests, runtime, candidates, and evidence. They are ignored by Git because they contain library metadata. Review detected pairs against bibliographies before claiming precision; recall requires known positive pairs. + +## Verification + +```powershell +uv run --directory backend pytest tests/test_citation_identity.py tests/test_citation_service.py tests/test_citation_jobs.py tests/test_citation_routes.py +npm --prefix frontend run test:run -- src/components/PaperCitations.test.jsx src/services/api.test.js +``` + +Run `backend/tests/sql/paper_citations.sql` after migration 033 on a disposable database. A local PostgreSQL-compatible option is: + +```powershell +npm install --prefix backend/data/citation-validation --no-audit --no-fund --ignore-scripts @electric-sql/pglite@0.5.14 +node backend/tests/run_citation_sql.mjs +``` + +The runner uses actual repository library/paper definitions, applies migration 033 twice, and rolls back its fixtures. It verifies transaction behavior, not Supabase permissions or production multi-connection contention. Browser coverage lives in `frontend/e2e/citations.spec.js`. + +## Rollback and limitations + +Stop workers and hide/revert the citation UI/API to stop provider traffic. Keep pending jobs and evidence. To stop automatic queueing, disable `paper_citations_changed` in a controlled maintenance step and re-enable it before resuming. No destructive rollback is necessary. + +Citations use the application's existing shared self-hosted access model, with library-scoped queries. They do not add user authentication or tenant isolation. PDF parsing, OCR, Semantic Scholar enrichment, cross-library graph views, and manual candidate approval are deferred. GROBID needs a configured service and a PDF read path supporting private storage. Truncated Markdown and first-page metadata extraction cannot substitute for full bibliographies. From 8fe983c93403cfa4a077cdd16fff657a05158791 Mon Sep 17 00:00:00 2001 From: Prannaya Gupta Date: Tue, 3 Nov 2026 19:40:06 -0500 Subject: [PATCH 08/23] feat(db): version PDF replacements and add opt-in citation jobs --- backend/migrations/034_pdf_citations.sql | 61 ++++++++++++++++++++++++ backend/models/citation.py | 2 +- backend/models/paper.py | 1 + backend/services/citation_repository.py | 4 ++ backend/services/paper_service.py | 2 +- backend/tests/run_citation_sql.mjs | 7 ++- backend/tests/sql/pdf_citations.sql | 46 ++++++++++++++++++ docs/database/schema.md | 2 + 8 files changed, 122 insertions(+), 3 deletions(-) create mode 100644 backend/migrations/034_pdf_citations.sql create mode 100644 backend/tests/sql/pdf_citations.sql diff --git a/backend/migrations/034_pdf_citations.sql b/backend/migrations/034_pdf_citations.sql new file mode 100644 index 0000000..f37dce8 --- /dev/null +++ b/backend/migrations/034_pdf_citations.sql @@ -0,0 +1,61 @@ +-- Optional PDF bibliography indexing. Requires migration 033. +BEGIN; +ALTER TABLE public.papers ADD COLUMN IF NOT EXISTS pdf_revision BIGINT NOT NULL DEFAULT 0; +ALTER TABLE public.paper_reference_runs DROP CONSTRAINT IF EXISTS paper_reference_runs_provider_check; +ALTER TABLE public.paper_reference_runs ADD CONSTRAINT paper_reference_runs_provider_check + CHECK (provider IN ('openalex','crossref','grobid')); +ALTER TABLE public.paper_citation_jobs DROP CONSTRAINT IF EXISTS paper_citation_jobs_provider_check; +ALTER TABLE public.paper_citation_jobs ADD CONSTRAINT paper_citation_jobs_provider_check + CHECK (provider IN ('openalex','crossref','grobid')); + +CREATE OR REPLACE FUNCTION public.citation_fingerprint(p public.papers) RETURNS TEXT +LANGUAGE sql IMMUTABLE AS $$ + SELECT md5(jsonb_build_array(p.doi,p.arxiv_id,p.title,p.authors,p.year,p.library_id,p.pdf_url,p.pdf_revision)::text); +$$; + +CREATE OR REPLACE FUNCTION public.bump_citation_pdf_revision() RETURNS trigger +LANGUAGE plpgsql AS $$ +BEGIN + -- The upload route writes pdf_url even when replacing the same storage key. + NEW.pdf_revision := OLD.pdf_revision + 1; + RETURN NEW; +END $$; +DROP TRIGGER IF EXISTS citation_pdf_revision ON public.papers; +CREATE TRIGGER citation_pdf_revision BEFORE UPDATE OF pdf_url ON public.papers +FOR EACH ROW EXECUTE FUNCTION public.bump_citation_pdf_revision(); + +CREATE OR REPLACE FUNCTION public.enqueue_pdf_citations(p_paper_id TEXT,p_force BOOLEAN DEFAULT false) +RETURNS SETOF public.paper_citation_jobs +LANGUAGE plpgsql SECURITY INVOKER SET search_path=public AS $$ +DECLARE p papers; +BEGIN + SELECT * INTO p FROM papers WHERE id=p_paper_id FOR UPDATE; + IF NOT FOUND THEN RETURN; END IF; + INSERT INTO paper_citation_jobs(paper_id,provider,fingerprint) + VALUES (p.id,'grobid',citation_fingerprint(p)) + ON CONFLICT (paper_id,provider) DO UPDATE SET + fingerprint=EXCLUDED.fingerprint,status='pending',attempts=0,error=NULL, + lease_token=NULL,lease_until=NULL,next_attempt_at=now(),updated_at=now() + WHERE paper_citation_jobs.fingerprint<>EXCLUDED.fingerprint + OR (p_force AND paper_citation_jobs.status NOT IN ('pending','running')); + RETURN QUERY SELECT * FROM paper_citation_jobs WHERE paper_id=p.id AND provider='grobid'; +END $$; + +CREATE OR REPLACE FUNCTION public.refresh_opted_in_pdf_citations() RETURNS trigger +LANGUAGE plpgsql SECURITY INVOKER SET search_path=public AS $$ +BEGIN + IF citation_fingerprint(OLD)<>citation_fingerprint(NEW) + AND EXISTS(SELECT 1 FROM paper_citation_jobs WHERE paper_id=NEW.id AND provider='grobid') THEN + PERFORM enqueue_pdf_citations(NEW.id); + END IF; + RETURN NEW; +END $$; +DROP TRIGGER IF EXISTS paper_pdf_citations_changed ON public.papers; +CREATE TRIGGER paper_pdf_citations_changed AFTER UPDATE ON public.papers +FOR EACH ROW EXECUTE FUNCTION public.refresh_opted_in_pdf_citations(); + +-- Rebase existing queued fingerprints without disturbing published provider evidence. +UPDATE paper_citation_jobs j SET fingerprint=citation_fingerprint(p),status='pending',attempts=0, + lease_token=NULL,lease_until=NULL,next_attempt_at=now(),updated_at=now() +FROM papers p WHERE p.id=j.paper_id AND j.fingerprint<>citation_fingerprint(p); +COMMIT; diff --git a/backend/models/citation.py b/backend/models/citation.py index 9aa9870..cc6d946 100644 --- a/backend/models/citation.py +++ b/backend/models/citation.py @@ -5,7 +5,7 @@ from models.base import CamelModel -Provider = Literal["openalex", "crossref"] +Provider = Literal["openalex", "crossref", "grobid"] RunStatus = Literal["succeeded", "partial", "not_found"] diff --git a/backend/models/paper.py b/backend/models/paper.py index f6e07d8..b64d6c0 100644 --- a/backend/models/paper.py +++ b/backend/models/paper.py @@ -33,6 +33,7 @@ class Paper(CamelModel): rejected: bool = False collections: list[str] = [] pdf_url: Optional[str] = None + pdf_revision: int = 0 github_url: Optional[str] = None website_url: Optional[str] = None links: list[NamedLink] = [] diff --git a/backend/services/citation_repository.py b/backend/services/citation_repository.py index d10a43a..a4b3bc5 100644 --- a/backend/services/citation_repository.py +++ b/backend/services/citation_repository.py @@ -6,6 +6,10 @@ def enqueue(paper_id: str, force: bool = False) -> list[dict]: return get_client().rpc("enqueue_paper_citations", {"p_paper_id": paper_id, "p_force": force}).execute().data or [] +def enqueue_pdf(paper_id: str, force: bool = False) -> list[dict]: + return get_client().rpc("enqueue_pdf_citations", {"p_paper_id": paper_id, "p_force": force}).execute().data or [] + + def claim(job_id: str | None = None) -> dict | None: data = get_client().rpc("claim_citation_job", {"p_job_id": job_id}).execute().data return data[0] if data else None diff --git a/backend/services/paper_service.py b/backend/services/paper_service.py index 4aabc28..841f882 100644 --- a/backend/services/paper_service.py +++ b/backend/services/paper_service.py @@ -84,7 +84,7 @@ def create_paper(data: PaperCreate) -> Paper: created_at=now, **data.model_dump(), ) - get_client().table(_TABLE).insert(paper.model_dump(by_alias=False, exclude={"has_stored_pdf"})).execute() + get_client().table(_TABLE).insert(paper.model_dump(by_alias=False, exclude={"has_stored_pdf", "pdf_revision"})).execute() logger.info("Created paper %s: %s", paper.id, paper.title) return paper diff --git a/backend/tests/run_citation_sql.mjs b/backend/tests/run_citation_sql.mjs index e7fbb71..53d6ebe 100644 --- a/backend/tests/run_citation_sql.mjs +++ b/backend/tests/run_citation_sql.mjs @@ -12,7 +12,12 @@ try { await db.exec(migration) await db.exec(migration) // Migration replay must be safe. await db.exec(await readFile(new URL('./sql/paper_citations.sql', import.meta.url), 'utf8')) - console.log('Citation SQL transaction checks passed; migration applied twice.') + const pdfMigration = await readFile(new URL('../migrations/034_pdf_citations.sql', import.meta.url), 'utf8') + await db.exec(pdfMigration) + await db.exec(pdfMigration) + await db.exec(await readFile(new URL('./sql/paper_citations.sql', import.meta.url), 'utf8')) + await db.exec(await readFile(new URL('./sql/pdf_citations.sql', import.meta.url), 'utf8')) + console.log('Citation and PDF SQL checks passed; migrations 033 and 034 applied twice.') } finally { await db.close() } diff --git a/backend/tests/sql/pdf_citations.sql b/backend/tests/sql/pdf_citations.sql new file mode 100644 index 0000000..23eaffb --- /dev/null +++ b/backend/tests/sql/pdf_citations.sql @@ -0,0 +1,46 @@ +-- Run after 033 and 034 on a disposable database. +BEGIN; +INSERT INTO libraries(id,name,created_at) VALUES ('pdf_lib','PDF citations','2026-11-03'); +INSERT INTO papers(id,title,authors,year,venue,status,source,library_id,created_at,pdf_url,doi) VALUES + ('pdf_source','PDF-only source','[]',2025,'','read','human','pdf_lib','2026-11-03','https://storage.example/p.pdf',NULL), + ('pdf_target','Target','[]',2024,'','read','human','pdf_lib','2026-11-03',NULL,'10.1234/target'); + +DO $$ +DECLARE j paper_citation_jobs; old_token UUID; result JSONB; revision BIGINT; +BEGIN + IF EXISTS(SELECT 1 FROM paper_citation_jobs WHERE provider='grobid' AND paper_id='pdf_source') THEN + RAISE EXCEPTION 'PDF extraction must be opt-in'; + END IF; + PERFORM enqueue_pdf_citations('pdf_source'); + PERFORM enqueue_pdf_citations('pdf_source'); + IF (SELECT count(*) FROM paper_citation_jobs WHERE paper_id='pdf_source' AND provider='grobid')<>1 THEN + RAISE EXCEPTION 'Duplicate PDF jobs'; + END IF; + SELECT * INTO j FROM claim_citation_job((SELECT id FROM paper_citation_jobs WHERE paper_id='pdf_source' AND provider='grobid')); + result := '{"provider":"grobid","status":"succeeded","resolution":"exact","version":"grobid:test;sha256:original","identities":{},"candidates":[],"references":[{"key":"b0","identifiers":{"doi":"10.1234/target"},"raw":{"unstructured":"Original reference","pdf_sha256":"original"}}]}'::jsonb; + IF NOT finish_citation_job(j.id,j.lease_token,result) THEN RAISE EXCEPTION 'PDF publication failed'; END IF; + IF (get_paper_citations('pdf_source')->>'total')::int<>1 THEN RAISE EXCEPTION 'PDF-only source did not match target DOI'; END IF; + IF get_paper_citations('pdf_source')#>>'{items,0,evidence,0,reference,pdf_sha256}'<>'original' THEN + RAISE EXCEPTION 'PDF hash was lost'; + END IF; + PERFORM enqueue_pdf_citations('pdf_source',true); + SELECT * INTO j FROM claim_citation_job(j.id); + -- A parser with no structured output must not erase previous evidence. + IF NOT finish_citation_job(j.id,j.lease_token,jsonb_set(jsonb_set(result,'{status}','"partial"'),'{references}','[]')) THEN + RAISE EXCEPTION 'Partial run failed'; + END IF; + IF (get_paper_citations('pdf_source')->>'total')::int<>1 THEN RAISE EXCEPTION 'Partial extraction erased evidence'; END IF; + PERFORM enqueue_pdf_citations('pdf_source',true); + SELECT * INTO j FROM claim_citation_job(j.id); + SELECT pdf_revision INTO revision FROM papers WHERE id='pdf_source'; + UPDATE papers SET pdf_url=pdf_url WHERE id='pdf_source'; + IF (SELECT pdf_revision FROM papers WHERE id='pdf_source')<>revision+1 THEN RAISE EXCEPTION 'Same-URL upload not versioned'; END IF; + IF finish_citation_job(j.id,j.lease_token,result) THEN RAISE EXCEPTION 'Old PDF worker published after replacement'; END IF; + IF (get_paper_citations('pdf_source')->>'total')::int<>0 THEN RAISE EXCEPTION 'Old PDF evidence still visible'; END IF; + IF (SELECT status FROM paper_citation_jobs WHERE id=j.id)<>'pending' THEN RAISE EXCEPTION 'Opted-in PDF job not requeued'; END IF; + UPDATE papers SET title='Changed target' WHERE id='pdf_target'; + IF EXISTS(SELECT 1 FROM paper_citation_jobs WHERE paper_id='pdf_target' AND provider='grobid') THEN + RAISE EXCEPTION 'Metadata edit enabled PDF extraction without opt-in'; + END IF; +END $$; +ROLLBACK; diff --git a/docs/database/schema.md b/docs/database/schema.md index 18b77ff..996ed74 100644 --- a/docs/database/schema.md +++ b/docs/database/schema.md @@ -15,6 +15,8 @@ Migration 033 provides transaction functions for enqueueing, leasing, publishing See [citation operations](../guides/citation-indexing.md) for installation and worker commands. The historical consolidated `schema.sql` still requires later numbered migrations, including 033. +Migration `034_pdf_citations.sql` adds the opt-in `grobid` provider and `papers.pdf_revision`. A BEFORE trigger increments that revision for every explicit `pdf_url` update, including replacing a file at the same URL. The citation fingerprint includes the revision, so stale extraction workers cannot publish and previous evidence is invalidated. An AFTER trigger requeues PDF extraction only when that paper already has a GROBID job. `enqueue_pdf_citations` explicitly creates that opt-in. Existing retrieval/evidence tables retain the parser version and PDF hash without separate storage tables. + ## ER Overview ``` From f695caf0d7b892fb555a2251912819c7aed6d3fc Mon Sep 17 00:00:00 2001 From: Prannaya Gupta Date: Tue, 3 Nov 2026 19:40:15 -0500 Subject: [PATCH 09/23] feat(citations): extract evidenced PDF references with GROBID --- backend/.env.example | 2 + backend/services/citation_job_service.py | 3 + backend/services/pdf_reference_service.py | 136 ++++++++++++++++++++++ backend/services/pdf_service.py | 23 ++++ backend/tests/test_citation_jobs.py | 10 ++ backend/tests/test_pdf_references.py | 111 ++++++++++++++++++ backend/tests/test_pdf_storage.py | 26 +++++ 7 files changed, 311 insertions(+) create mode 100644 backend/services/pdf_reference_service.py create mode 100644 backend/tests/test_pdf_references.py diff --git a/backend/.env.example b/backend/.env.example index 2cb6856..4337f1b 100644 --- a/backend/.env.example +++ b/backend/.env.example @@ -1,5 +1,7 @@ OPENAI_API_KEY=* OPENALEX_API_KEY=* +# Optional self-hosted bibliography extractor. No PDF is sent until explicitly queued. +GROBID_URL= SUPABASE_URL=* SUPABASE_KEY=* diff --git a/backend/services/citation_job_service.py b/backend/services/citation_job_service.py index 1444e46..8d0e4a5 100644 --- a/backend/services/citation_job_service.py +++ b/backend/services/citation_job_service.py @@ -5,6 +5,7 @@ from services import citation_repository as repository, paper_service from services.citation_service import fetch_crossref, fetch_openalex +from services.pdf_reference_service import fetch_pdf_references from services.openalex_client import CitationProviderError logger = logging.getLogger(__name__) @@ -23,6 +24,8 @@ async def run_one(job_id: str | None = None) -> dict | None: data = paper.model_dump() if job["provider"] == "openalex": result = await fetch_openalex(data, client) + elif job["provider"] == "grobid": + result = await fetch_pdf_references(data, client) else: result = await fetch_crossref(data, client, job.get("captured_record")) published = repository.finish(job, result) diff --git a/backend/services/pdf_reference_service.py b/backend/services/pdf_reference_service.py new file mode 100644 index 0000000..98a5aaf --- /dev/null +++ b/backend/services/pdf_reference_service.py @@ -0,0 +1,136 @@ +"""Opt-in bibliography extraction from managed PDFs through a configured GROBID service.""" +import hashlib +import os +import re +from urllib.parse import urlsplit +import xml.etree.ElementTree as ET + +import httpx + +from models.citation import CitationReference, CitationResult +from services import pdf_service +from services.citation_identity_service import local_identifiers, normalize_arxiv, normalize_doi +from services.openalex_client import CitationProviderError, retry_seconds + +NS = {'tei': 'http://www.tei-c.org/ns/1.0'} +MAX_XML_BYTES = 10 * 1024 * 1024 + + +def configured_url() -> str | None: + value = os.environ.get('GROBID_URL', '').strip().rstrip('/') + if not value: + return None + parsed = urlsplit(value) + if parsed.scheme not in {'http', 'https'} or not parsed.hostname or parsed.username or parsed.password or parsed.query or parsed.fragment: + raise CitationProviderError('GROBID_URL must be an HTTP(S) service URL without credentials, query, or fragment') + return value + + +def _text(node) -> str: + return ' '.join(' '.join(node.itertext()).split()) if node is not None else '' + + +def parse_references(xml: bytes, paper: dict, pdf_hash: str) -> CitationResult: + if len(xml) > MAX_XML_BYTES or b' 1 for v in values.values()), + } + result.references.append(CitationReference(key=key, identifiers=identifiers, raw=raw)) + if not result.references: + result.status = 'partial' + result.message = 'No structured bibliography extracted; scanned or unsupported PDFs may need OCR' + return result + + +async def fetch_pdf_references(paper: dict, client: httpx.AsyncClient) -> CitationResult: + base = configured_url() + if not base: + raise CitationProviderError('PDF indexing requires a configured GROBID_URL') + if not pdf_service.managed_location(paper.get('pdf_url')): + return CitationResult(provider='grobid', status='not_found', resolution='unresolved', + message='Upload or store this PDF before indexing its bibliography') + try: + content = pdf_service.read_managed_pdf(paper['pdf_url']) + except ValueError as exc: + raise CitationProviderError(str(exc)) from None + except Exception: + raise CitationProviderError('Managed PDF could not be read', retryable=True) from None + pdf_hash = hashlib.sha256(content).hexdigest() + try: + response = await client.post(base + '/api/processReferences', + files={'input': ('paper.pdf', content, 'application/pdf')}, + data={'includeRawCitations': '1', 'consolidateCitations': '0'}, + headers={'Accept': 'application/xml'}, timeout=120, follow_redirects=False) + except httpx.RequestError: + raise CitationProviderError('GROBID connection failed', retryable=True) from None + if response.status_code == 204: + return CitationResult(provider='grobid', status='partial', resolution='unresolved', + version=f'grobid:unknown;adapter:1;sha256:{pdf_hash}', + message='No structured bibliography extracted; scanned or unsupported PDFs may need OCR') + if response.status_code != 200: + raise CitationProviderError(f'GROBID returned HTTP {response.status_code}', + retryable=response.status_code == 429 or response.status_code >= 500, + retry_after=retry_seconds(response.headers.get('Retry-After'))) + result = parse_references(response.content, paper, pdf_hash) + # Re-read to catch object replacement during extraction, even at the same URL. + try: + current_hash = hashlib.sha256(pdf_service.read_managed_pdf(paper['pdf_url'])).hexdigest() + except Exception: + raise CitationProviderError('Could not verify PDF after extraction', retryable=True) from None + if current_hash != pdf_hash: + raise CitationProviderError('PDF changed during bibliography extraction', retryable=True, retry_after=1) + return result diff --git a/backend/services/pdf_service.py b/backend/services/pdf_service.py index 89e97bd..754bc19 100644 --- a/backend/services/pdf_service.py +++ b/backend/services/pdf_service.py @@ -100,3 +100,26 @@ def delete_pdf(paper_id: str, pdf_url: str | None) -> None: else: get_client().storage.from_(_BUCKET).remove([path]) logger.info("Deleted PDF for paper %s from %s", paper_id, provider) + + +def read_managed_pdf(pdf_url: str) -> bytes: + """Read configured storage using credentials, with a fixed byte limit and GCS generation guard.""" + location = managed_location(pdf_url) + if location is None: + raise ValueError("PDF reference extraction requires a managed PDF") + limit = 100 * 1024 * 1024 + provider, path = location + if provider == "gcs": + blob = _gcs_bucket().get_blob(path, timeout=30) + if blob is None: + raise ValueError("Stored PDF was not found") + if blob.size is None or int(blob.size) > limit: + raise ValueError("PDF exceeds the 100 MiB extraction limit") + data = blob.download_as_bytes(if_generation_match=int(blob.generation), timeout=60) + else: + data = get_client().storage.from_(_BUCKET).download(path) + if len(data) > limit: + raise ValueError("PDF exceeds the 100 MiB extraction limit") + if not data.startswith(b"%PDF-"): + raise ValueError("Stored file does not have a PDF header") + return data diff --git a/backend/tests/test_citation_jobs.py b/backend/tests/test_citation_jobs.py index 2afcc60..a533d24 100644 --- a/backend/tests/test_citation_jobs.py +++ b/backend/tests/test_citation_jobs.py @@ -46,6 +46,16 @@ def test_crossref_captured_payload_passed_to_fetcher(monkeypatch): assert fetch.call_args.args[2] == job['captured_record'] +def test_pdf_job_uses_pdf_extractor_and_publishes_evidence(monkeypatch): + setup_job(monkeypatch, 'grobid') + fetch = AsyncMock(return_value=CitationResult(provider='grobid', version='grobid:test;sha256:pdf')) + monkeypatch.setattr(jobs, 'fetch_pdf_references', fetch) + monkeypatch.setattr(jobs, 'fetch_crossref', AsyncMock(side_effect=AssertionError('Wrong provider'))) + monkeypatch.setattr(jobs.repository, 'finish', lambda job, result: result.provider == 'grobid') + assert asyncio.run(jobs.run_one())['status'] == 'succeeded' + fetch.assert_awaited_once() + + def test_keyset_iterator_handles_server_page_caps(monkeypatch): rows = [{'id': f'p{i:04}', 'library_id': 'lib'} for i in range(1105)] class Query: diff --git a/backend/tests/test_pdf_references.py b/backend/tests/test_pdf_references.py new file mode 100644 index 0000000..9dfe881 --- /dev/null +++ b/backend/tests/test_pdf_references.py @@ -0,0 +1,111 @@ +import asyncio +import hashlib + +import httpx +import pytest + +from services import pdf_reference_service as service +from services.openalex_client import CitationProviderError + +TEI = b''' + + +Referenced paperAWriter +10.1234/TARGET +A. Writer. Referenced paper. 2024. +Title-only referenceUnresolved book. +hep-th/9901001v2 +''' +PDF = b'%PDF-fixture' +PAPER = {'id': 'p_source', 'pdf_url': 'https://storage.googleapis.com/papers/p_source.pdf'} + + +def test_structured_references_keep_raw_evidence_and_pdf_hash(): + result = service.parse_references(TEI, PAPER, 'abc123') + assert result.provider == 'grobid' + assert result.status == 'succeeded' + assert result.identities == {} # PDF-only sources need no external identity. + assert len(result.references) == 3 + assert result.references[0].identifiers == {'doi': '10.1234/target'} + assert result.references[0].raw['unstructured'] == 'A. Writer. Referenced paper. 2024.' + assert result.references[0].raw['authors'] == ['A Writer'] + assert result.references[0].raw['pdf_sha256'] == 'abc123' + assert result.references[0].raw['parser_version'] == '0.8.test' + assert result.references[1].identifiers == {} + assert result.references[2].identifiers == {'arxiv': 'hep-th/9901001'} + + +def test_conflicting_reference_identifiers_are_not_promoted(): + xml = TEI.replace(b'10.1234/TARGET', + b'10.1234/TARGET10.1234/other') + result = service.parse_references(xml, PAPER, 'hash') + assert result.references[0].identifiers == {} + assert result.references[0].raw['ambiguous_identifiers'] + + +@pytest.mark.parametrize('xml', [b'', b']>']) +def test_invalid_or_unsafe_xml_is_not_empty_success(xml): + with pytest.raises(CitationProviderError): + service.parse_references(xml, PAPER, 'hash') + + +def test_empty_bibliography_is_partial(): + result = service.parse_references(b'', PAPER, 'hash') + assert result.status == 'partial' + assert 'OCR' in result.message + + +def run_fetch(monkeypatch, handler, contents=None): + monkeypatch.setenv('GROBID_URL', 'http://localhost:8070') + monkeypatch.setattr(service.pdf_service, 'managed_location', lambda _: ('gcs', 'p_source.pdf')) + values = iter(contents or [PDF, PDF]) + monkeypatch.setattr(service.pdf_service, 'read_managed_pdf', lambda _: next(values)) + async def run(): + async with httpx.AsyncClient(transport=httpx.MockTransport(handler)) as client: + return await service.fetch_pdf_references(PAPER, client) + return asyncio.run(run()) + + +def test_pdf_request_disables_external_consolidation_and_preserves_hash(monkeypatch): + def handler(request): + assert str(request.url) == 'http://localhost:8070/api/processReferences' + assert b'name="consolidateCitations"\r\n\r\n0' in request.content + assert b'name="includeRawCitations"\r\n\r\n1' in request.content + assert PDF in request.content + return httpx.Response(200, content=TEI) + result = run_fetch(monkeypatch, handler) + assert hashlib.sha256(PDF).hexdigest() in result.version + + +def test_pdf_replacement_during_extraction_is_retryable(monkeypatch): + with pytest.raises(CitationProviderError, match='PDF changed') as exc: + run_fetch(monkeypatch, lambda _: httpx.Response(200, content=TEI), [PDF, b'%PDF-replaced']) + assert exc.value.retryable + + +def test_no_content_is_partial_and_busy_server_honors_delay(monkeypatch): + result = run_fetch(monkeypatch, lambda _: httpx.Response(204)) + assert result.status == 'partial' + with pytest.raises(CitationProviderError) as exc: + run_fetch(monkeypatch, lambda _: httpx.Response(503, headers={'Retry-After':'12'})) + assert exc.value.retryable and exc.value.retry_after == 12 + + +def test_redirect_is_not_followed_with_pdf_data(monkeypatch): + requests = [] + def handler(request): + requests.append(request) + return httpx.Response(307, headers={'Location': 'https://unexpected.example/upload'}) + with pytest.raises(CitationProviderError, match='HTTP 307'): + run_fetch(monkeypatch, handler) + assert len(requests) == 1 + + +def test_missing_configuration_never_reads_or_sends_pdf(monkeypatch): + monkeypatch.delenv('GROBID_URL', raising=False) + monkeypatch.setattr(service.pdf_service, 'read_managed_pdf', lambda _: pytest.fail('Unexpected read')) + async def run(): + async with httpx.AsyncClient(transport=httpx.MockTransport(lambda _: pytest.fail('Unexpected request'))) as client: + await service.fetch_pdf_references(PAPER, client) + with pytest.raises(CitationProviderError, match='GROBID_URL'): + asyncio.run(run()) diff --git a/backend/tests/test_pdf_storage.py b/backend/tests/test_pdf_storage.py index ad60b80..b07ddd9 100644 --- a/backend/tests/test_pdf_storage.py +++ b/backend/tests/test_pdf_storage.py @@ -69,6 +69,32 @@ def test_computed_storage_flag_is_in_api_but_not_database_insert(mocker): assert paper.model_dump(by_alias=True)['hasStoredPdf'] is True assert 'has_stored_pdf' not in query.insert.call_args.args[0] assert 'hasStoredPdf' not in query.insert.call_args.args[0] + assert 'pdf_revision' not in query.insert.call_args.args[0] + + +def test_read_pdf_uses_private_gcs_credentials_and_generation_guard(mocker): + blob = mocker.Mock(size=123, generation='42') + blob.download_as_bytes.return_value = b'%PDF-test' + bucket = mocker.patch.object(pdf_service, '_gcs_bucket').return_value + bucket.get_blob.return_value = blob + assert pdf_service.read_managed_pdf('https://storage.googleapis.com/test-papers/p_1.pdf') == b'%PDF-test' + blob.download_as_bytes.assert_called_once_with(if_generation_match=42, timeout=60) + + +def test_read_pdf_rejects_oversized_object_before_download(mocker): + blob = mocker.Mock(size=101*1024*1024, generation='42') + mocker.patch.object(pdf_service, '_gcs_bucket').return_value.get_blob.return_value = blob + with pytest.raises(ValueError, match='100 MiB'): + pdf_service.read_managed_pdf('https://storage.googleapis.com/test-papers/p_1.pdf') + blob.download_as_bytes.assert_not_called() + + +def test_read_pdf_uses_supabase_storage_and_rejects_arbitrary_urls(mocker): + db = mocker.patch.object(pdf_service, 'get_client').return_value + db.storage.from_.return_value.download.return_value = b'%PDF-supabase' + assert pdf_service.read_managed_pdf('https://source.supabase.co/storage/v1/object/public/pdfs/p_1.pdf') == b'%PDF-supabase' + with pytest.raises(ValueError, match='managed PDF'): + pdf_service.read_managed_pdf('https://example.org/private.pdf') def test_fetch_rejects_gcs_managed_pdf(client, mocker): From 5afdbe57b89fcfe63a642970870aff87edee0ba4 Mon Sep 17 00:00:00 2001 From: Prannaya Gupta Date: Tue, 3 Nov 2026 19:40:25 -0500 Subject: [PATCH 10/23] feat(citations): expose opt-in PDF indexing and pilot controls --- backend/routers/citations.py | 27 ++++++++ backend/scripts/index_paper_citations.py | 15 +++- backend/scripts/pilot_pdf_citations.py | 68 +++++++++++++++++++ backend/tests/test_citation_routes.py | 43 ++++++++++++ docs/api/papers.md | 3 + docs/citation-indexing-pilot-report.md | 4 ++ docs/guides/citation-indexing.md | 31 ++++++++- .../2026-11-02-paper-citation-indexing.md | 18 +++-- frontend/src/components/PaperCitations.jsx | 21 ++++-- .../src/components/PaperCitations.test.jsx | 27 +++++++- frontend/src/pages/Paper.jsx | 3 +- frontend/src/services/api.js | 1 + frontend/src/services/api.test.js | 3 + 13 files changed, 247 insertions(+), 17 deletions(-) create mode 100644 backend/scripts/pilot_pdf_citations.py diff --git a/backend/routers/citations.py b/backend/routers/citations.py index 6ae7719..d035f13 100644 --- a/backend/routers/citations.py +++ b/backend/routers/citations.py @@ -6,6 +6,9 @@ from models.citation import CitationJob, CitationPage from services import citation_repository as repository, paper_service +from services.pdf_reference_service import configured_url +from services.pdf_service import managed_location +from services.openalex_client import CitationProviderError router = APIRouter(prefix="/api", tags=["citations"]) @@ -42,3 +45,27 @@ def citation_job(job_id: UUID): if not job or paper_service.get_paper(job["paper_id"]) is None: raise HTTPException(404, {"error": "not_found", "detail": "Citation job not found"}) return CitationJob.model_validate(job) + + +@router.post("/papers/{paper_id}/citations/pdf/reindex", status_code=202, response_model=list[CitationJob]) +def reindex_pdf(paper_id: str): + paper = paper_service.get_paper(paper_id) + if paper is None: + raise HTTPException(404, {"error": "not_found", "detail": "Paper not found"}) + try: + url = configured_url() + except CitationProviderError as exc: + raise HTTPException(503, str(exc)) from None + if not url: + raise HTTPException(503, "PDF bibliography indexing requires a configured GROBID service.") + if not managed_location(paper.pdf_url): + raise HTTPException(422, "Upload or store this PDF before indexing its bibliography.") + try: + jobs = repository.enqueue_pdf(paper_id, True) + except APIError as exc: + if exc.code in {"42P01", "42883", "PGRST202", "PGRST205"}: + raise HTTPException(503, "PDF bibliography indexing is not installed. Apply migration 034.") from None + raise + if not jobs: + raise HTTPException(404, {"error": "not_found", "detail": "Paper not found"}) + return [CitationJob.model_validate(job) for job in jobs] diff --git a/backend/scripts/index_paper_citations.py b/backend/scripts/index_paper_citations.py index a53f2da..8d71b37 100644 --- a/backend/scripts/index_paper_citations.py +++ b/backend/scripts/index_paper_citations.py @@ -11,6 +11,8 @@ from dotenv import load_dotenv load_dotenv(BACKEND / ".env") from services import citation_repository, paper_service +from services.pdf_reference_service import configured_url +from services.pdf_service import managed_location def save_manifest(path: Path, data: dict): @@ -29,12 +31,15 @@ def main(): parser.add_argument("--limit", type=int, default=0, help="0 selects all papers in scope") parser.add_argument("--dry-run", action="store_true") parser.add_argument("--refresh", action="store_true", help="Also requeue completed jobs") + parser.add_argument("--include-pdf", action="store_true", help="Also enqueue managed PDFs for the configured GROBID service") args = parser.parse_args() if args.limit < 0: parser.error("--limit must be nonnegative") if args.resume: manifest = args.resume batch = json.loads(manifest.read_text(encoding="utf-8")) + if args.include_pdf and not batch.get("include_pdf"): + parser.error("Create a new --include-pdf batch; an existing manifest keeps its original scope") else: papers = [] for paper in paper_service.iter_papers(library_id=args.library_id, all_libraries=args.all_libraries): @@ -42,18 +47,24 @@ def main(): if args.limit and len(papers) >= args.limit: break batch = {"id": str(uuid4()), "library_id": args.library_id, "all_libraries": args.all_libraries, - "paper_ids": papers, "jobs": {}, "refresh": args.refresh} + "paper_ids": papers, "jobs": {}, "refresh": args.refresh, "include_pdf": args.include_pdf} manifest = BACKEND / "data" / "citations" / f"{batch['id']}.json" if args.dry_run: print(json.dumps({"papers": len(batch["paper_ids"]), "already_enqueued": len(batch["jobs"]), "library_id": batch["library_id"], "all_libraries": batch["all_libraries"]})) return + if batch.get("include_pdf") and not configured_url(): + parser.error("--include-pdf requires GROBID_URL; it sends managed PDFs to that configured service") save_manifest(manifest, batch) for paper_id in batch["paper_ids"]: if paper_id in batch["jobs"]: continue jobs = citation_repository.enqueue(paper_id, batch["refresh"]) - batch["jobs"][paper_id] = [job["id"] for job in jobs] + if batch.get("include_pdf"): + paper = paper_service.get_paper(paper_id) + if paper and managed_location(paper.pdf_url): + jobs.extend(citation_repository.enqueue_pdf(paper_id, batch["refresh"])) + batch["jobs"][paper_id] = list(dict.fromkeys(job["id"] for job in jobs)) save_manifest(manifest, batch) print(json.dumps({"batch": str(manifest), "papers": len(batch["jobs"])})) diff --git a/backend/scripts/pilot_pdf_citations.py b/backend/scripts/pilot_pdf_citations.py new file mode 100644 index 0000000..27e81f2 --- /dev/null +++ b/backend/scripts/pilot_pdf_citations.py @@ -0,0 +1,68 @@ +"""Extract one managed PDF through configured GROBID without changing the database.""" +import argparse +import asyncio +from collections import defaultdict +from datetime import datetime, timezone +import json +from pathlib import Path +import sys + +BACKEND = Path(__file__).resolve().parents[1] +sys.path.insert(0, str(BACKEND)) +from dotenv import load_dotenv +load_dotenv(BACKEND / '.env') +import httpx +from services import paper_service +from services.citation_identity_service import local_identifiers +from services.pdf_reference_service import configured_url, fetch_pdf_references +from services.openalex_client import CitationProviderError + + +async def run(args): + paper = paper_service.get_paper(args.paper_id) + if paper is None: + raise ValueError('Paper not found') + async with httpx.AsyncClient(timeout=30) as client: + result = await fetch_pdf_references(paper.model_dump(), client) + identities = defaultdict(set) + targets = {} + # A missing library ID is a distinct scope, never an implicit cross-library match. + for target in paper_service.iter_papers(library_id=paper.library_id, all_libraries=paper.library_id is None): + if target.get('library_id') != paper.library_id: + continue + targets[target['id']] = target + for kind, value in local_identifiers(target).items(): + identities[(kind, value)].add(target['id']) + matches = [] + for ref in result.references: + ids = set() + for kind, value in ref.identifiers.items(): + ids.update(identities[(kind, value)]) + if len(ids) == 1: + target = targets[next(iter(ids))] + source_ids = local_identifiers(paper.model_dump()) + if target['id'] != paper.id and not any(source_ids.get(k) == v for k,v in local_identifiers(target).items()): + matches.append({'reference_key':ref.key, 'paper_id':target['id'], 'title':target['title']}) + report = {'created_at':datetime.now(timezone.utc).isoformat(), 'paper_id':paper.id, + 'library_id':paper.library_id, 'extraction':result.model_dump(), 'local_matches':matches, + 'matching_note':'Exact local DOI/arXiv identifiers only; provider aliases and title candidates are excluded.'} + args.output.parent.mkdir(parents=True, exist_ok=True) + with args.output.open('x', encoding='utf-8') as handle: + json.dump(report, handle, indent=2) + print(json.dumps({'status':result.status, 'references':len(result.references), + 'matched_reference_records':len(matches), 'output':str(args.output)})) + + +if __name__ == '__main__': + parser = argparse.ArgumentParser(description=__doc__) + parser.add_argument('--paper-id', required=True) + parser.add_argument('--output', type=Path, required=True) + args = parser.parse_args() + if args.output.exists(): + parser.error('Choose a new output path; existing pilot reports are not overwritten') + try: + if not configured_url(): + parser.error('Configure a self-hosted GROBID_URL before sending a PDF for extraction') + asyncio.run(run(args)) + except (ValueError, CitationProviderError) as exc: + parser.exit(1, f'{exc}\n') diff --git a/backend/tests/test_citation_routes.py b/backend/tests/test_citation_routes.py index b3965f2..6d362d3 100644 --- a/backend/tests/test_citation_routes.py +++ b/backend/tests/test_citation_routes.py @@ -53,3 +53,46 @@ def test_missing_migration_reports_setup_problem(client, mocker): response = client.get('/api/papers/a/citations') assert response.status_code == 503 assert 'migration 033' in response.json()['detail'] + + +def test_pdf_indexing_requires_service_and_never_queues_implicitly(client, mocker, monkeypatch): + from types import SimpleNamespace + monkeypatch.delenv('GROBID_URL', raising=False) + mocker.patch('routers.citations.paper_service.get_paper', return_value=SimpleNamespace(pdf_url='stored')) + queue = mocker.patch.object(citation_repository, 'enqueue_pdf') + response = client.post('/api/papers/a/citations/pdf/reindex') + assert response.status_code == 503 + assert 'GROBID' in response.json()['detail'] + queue.assert_not_called() + + +def test_pdf_indexing_checks_managed_storage_and_accepts_explicit_job(client, mocker, monkeypatch): + from types import SimpleNamespace + monkeypatch.setenv('GROBID_URL', 'http://localhost:8070') + mocker.patch('routers.citations.paper_service.get_paper', return_value=SimpleNamespace(pdf_url='stored')) + location = mocker.patch('routers.citations.managed_location', return_value=None) + queue = mocker.patch.object(citation_repository, 'enqueue_pdf', return_value=[{ + 'id':'pdf-job', 'paper_id':'a', 'provider':'grobid', 'status':'pending', 'attempts':0, + 'updated_at':'2026-11-03T00:00:00Z', + }]) + assert client.post('/api/papers/a/citations/pdf/reindex').status_code == 422 + queue.assert_not_called() + location.return_value = ('gcs','a.pdf') + response = client.post('/api/papers/a/citations/pdf/reindex') + assert response.status_code == 202 + assert response.json()[0]['provider'] == 'grobid' + queue.assert_called_once_with('a', True) + + +def test_pdf_indexing_missing_migration_has_specific_guidance(client, mocker, monkeypatch): + from types import SimpleNamespace + from postgrest.exceptions import APIError + monkeypatch.setenv('GROBID_URL', 'http://localhost:8070') + mocker.patch('routers.citations.paper_service.get_paper', return_value=SimpleNamespace(pdf_url='stored')) + mocker.patch('routers.citations.managed_location', return_value=('gcs','a.pdf')) + mocker.patch.object(citation_repository, 'enqueue_pdf', side_effect=APIError({ + 'code':'PGRST202', 'message':'missing', 'details':'', 'hint':'', + })) + response = client.post('/api/papers/a/citations/pdf/reindex') + assert response.status_code == 503 + assert 'migration 034' in response.json()['detail'] diff --git a/docs/api/papers.md b/docs/api/papers.md index b583ccd..247cd15 100644 --- a/docs/api/papers.md +++ b/docs/api/papers.md @@ -8,12 +8,15 @@ Requires migration `033_paper_citations.sql` and a separately running citation w | --- | --- | | `GET /api/papers/{id}/citations?direction=outgoing&limit=20&offset=0` | Paginated local references or `direction=incoming` for local citing papers; reads never trigger provider calls | | `POST /api/papers/{id}/citations/reindex` | HTTP 202 with OpenAlex/Crossref job summaries; pending/running work is reused | +| `POST /api/papers/{id}/citations/pdf/reindex` | Explicit opt-in for a managed PDF; requires migration 034 and `GROBID_URL`; HTTP 202 with its GROBID job summary | | `GET /api/citation-jobs/{uuid}` | Current status of a durable job; does not expose lease tokens or captured provider payloads | The citation page contains `items`, `total`, `limit`, `offset`, `direction`, `runs`, `jobs`, `unresolvedCount`, `indexedPapers`, `libraryPapers`, and `pendingJobs`. Each item contains `id`, `title`, `authors`, `year`, and `evidence`; evidence includes `provider`, `method`, `indexedAt`, and the original provider `reference` object (its keys retain provider casing). Run summaries include `active`, `resolution`, `status`, `indexedAt`, candidate metadata, and messages. Both endpoints of a link must belong to the source paper's library. Pagination is ordered by year descending, title, then ID; `limit` is 1–100 and `offset` is nonnegative. Missing papers/jobs return the existing 404 error contract. Missing citation schema returns 503 with installation guidance. Empty results mean no indexed local links, not absence of citations. See [operation and recovery](../guides/citation-indexing.md). +PDF extraction returns 503 when its service/migration is unavailable and 422 for an unmanaged PDF. Its evidence provider is `grobid`; the original reference includes `unstructured`, `title`, `authors`, `date`, `pdf_sha256`, `parser_version`, and `tei`. Paper responses expose `pdfRevision` for same-URL replacement invalidation. Raw evidence keeps provider/parser field casing. + All endpoints are prefixed with `/api/papers`. --- diff --git a/docs/citation-indexing-pilot-report.md b/docs/citation-indexing-pilot-report.md index 9e351fb..2c237b3 100644 --- a/docs/citation-indexing-pilot-report.md +++ b/docs/citation-indexing-pilot-report.md @@ -47,3 +47,7 @@ The full frontend suite requires `TZ=UTC` for the unchanged Dashboard chart test After the pilot, a tested resolver improvement was added: an identifier-verified OpenAlex work can contribute a unique missing DOI/arXiv alias. Multiple possible aliases remain unaccepted. The pilot was not rerun for this improvement, so the observed edge count above is unchanged and does not predict alias-enriched coverage. Production multi-connection contention, Supabase role behavior, worker deployment, and the persisted full-library backfill remain rollout checks. This pilot establishes feasibility and coverage gaps; it is not deployment completion. + +## Subsequent PDF adapter work + +An opt-in GROBID adapter, managed-storage reader, file revision safeguards, and read-only PDF pilot command have since been implemented and tested using controlled TEI/HTTP fixtures. The user confirmed there is no GROBID service available yet. Therefore the provider pilot numbers above remain the only measured coverage results; parser fixtures do not establish PDF extraction accuracy or added real-world citation coverage. diff --git a/docs/guides/citation-indexing.md b/docs/guides/citation-indexing.md index 301bb70..549ef2b 100644 --- a/docs/guides/citation-indexing.md +++ b/docs/guides/citation-indexing.md @@ -8,6 +8,33 @@ Apply `backend/migrations/033_paper_citations.sql` after the existing numbered m Restart the application with this branch. Before migration, citation endpoints return a setup-specific HTTP 503 without preventing ordinary paper operations. Set a valid `OPENALEX_API_KEY` in `backend/.env` for bulk use; credentials stay server-side. Crossref does not require a separate key. +## Optional PDF bibliography extraction + +Apply `034_pdf_citations.sql` after 033, and configure `GROBID_URL` with a self-hosted service base URL (for example, `http://localhost:8070`). See the [official GROBID service documentation](https://grobid.readthedocs.io/en/latest/Grobid-service/) for running the service. No GROBID service is currently configured for this workspace, so real extraction coverage has not been benchmarked. + +For a paper with a managed PDF, click **Index PDF bibliography**. This explicitly opts the paper into sending its stored PDF to the configured service. Existing metadata indexing does not enable PDF extraction automatically. The worker calls `processReferences` with `includeRawCitations=1` and `consolidateCitations=0`, disabling GROBID's external metadata consolidation. HTTP redirects are not followed with the PDF payload. + +For a batch, add `--include-pdf` to a new backfill command: + +```powershell +uv run --directory backend python scripts/index_paper_citations.py --library-id LIBRARY_ID --include-pdf --dry-run +uv run --directory backend python scripts/index_paper_citations.py --library-id LIBRARY_ID --include-pdf +``` + +The execute command requires `GROBID_URL` and queues only PDFs recognized as managed by the configured GCS/Supabase storage. The same worker and resume manifest handle these jobs. Credentials read private storage directly; arbitrary external PDF URLs must first be stored through the existing application import/upload path. The extraction size limit is 100 MiB. + +Each reference retains its raw bibliography text, parsed title/authors/date, TEI record, parser version, and PDF SHA-256. Matching uses only explicit DOI/arXiv identifiers; title-only and contradictory identifiers remain unresolved. No reference output (including HTTP 204) is a partial extraction that preserves prior evidence, not proof that the PDF has no bibliography. Scanned PDFs may require OCR, which is not implemented. + +Once a service is available, evaluate one existing managed PDF without applying migration 034 or writing citation tables: + +```powershell +uv run --directory backend python scripts/pilot_pdf_citations.py --paper-id PAPER_ID --output data/citations/pdf-pilot.json +``` + +This command sends the full PDF to the configured service and writes a local report containing raw reference evidence and exact local DOI/arXiv matches. It does not use title matching or provider alias enrichment and should be followed by bibliography review. No real GROBID pilot has been run yet. + +Migration 034 versions every explicit PDF URL update, including uploads replacing the same storage key. Old evidence is invalidated and previously opted-in PDF jobs requeue; unselected papers stay opted out. The worker also re-reads the PDF after extraction to detect content replacement during processing. Existing provider jobs are requeued once when upgrading their fingerprint format; published evidence is retained until a successful refresh. Deploy 034 before enabling PDF indexing and avoid replaying 033 after 034. + Start the worker from the repository root: ```powershell @@ -72,10 +99,10 @@ npm install --prefix backend/data/citation-validation --no-audit --no-fund --ign node backend/tests/run_citation_sql.mjs ``` -The runner uses actual repository library/paper definitions, applies migration 033 twice, and rolls back its fixtures. It verifies transaction behavior, not Supabase permissions or production multi-connection contention. Browser coverage lives in `frontend/e2e/citations.spec.js`. +The runner uses actual repository library/paper definitions, applies migrations 033 and 034 twice, and rolls back its fixtures. It verifies transaction behavior, including same-URL PDF replacement, not Supabase permissions or production multi-connection contention. Browser coverage lives in `frontend/e2e/citations.spec.js`. PDF parser/storage fixtures are in `backend/tests/test_pdf_references.py` and `backend/tests/test_pdf_storage.py`. ## Rollback and limitations Stop workers and hide/revert the citation UI/API to stop provider traffic. Keep pending jobs and evidence. To stop automatic queueing, disable `paper_citations_changed` in a controlled maintenance step and re-enable it before resuming. No destructive rollback is necessary. -Citations use the application's existing shared self-hosted access model, with library-scoped queries. They do not add user authentication or tenant isolation. PDF parsing, OCR, Semantic Scholar enrichment, cross-library graph views, and manual candidate approval are deferred. GROBID needs a configured service and a PDF read path supporting private storage. Truncated Markdown and first-page metadata extraction cannot substitute for full bibliographies. +Citations use the application's existing shared self-hosted access model, with library-scoped queries. They do not add user authentication or tenant isolation. OCR, Semantic Scholar enrichment, cross-library graph views, and manual candidate approval remain deferred. The GROBID adapter is implemented and fixture-tested; a real service and coverage pilot are still needed. It reads full PDF bytes rather than truncated Markdown or first-page metadata. diff --git a/docs/superpowers/plans/2026-11-02-paper-citation-indexing.md b/docs/superpowers/plans/2026-11-02-paper-citation-indexing.md index 9fdcf33..2864668 100644 --- a/docs/superpowers/plans/2026-11-02-paper-citation-indexing.md +++ b/docs/superpowers/plans/2026-11-02-paper-citation-indexing.md @@ -2,7 +2,7 @@ Date: November 2, 2026 Branch: `feat/paper-citation-indexing` -Status: OpenAlex/Crossref implementation locally validated; production rollout and PDF fallback pending +Status: OpenAlex/Crossref and opt-in PDF adapter implemented; live rollout and GROBID coverage pilot pending Basis: [citation indexing investigation](../../citation-indexing-investigation.md) ## Execution record - November 3, 2026 @@ -24,6 +24,16 @@ No migration or backfill has been applied to the live database. Production permi Validation: 340 backend tests and 401 frontend tests passed; the full frontend suite used `TZ=UTC` for the existing Dashboard fixture's date assumptions. Migration replay and SQL transaction checks passed. The production build passed with the existing bundle-size warning. The citation browser fixture passed. Combined citation/library browser checks passed 13 of 14 tests: the unchanged row-context-menu test timed out opening/clicking its Copy & export submenu and also timed out when rerun alone. No context-menu implementation was changed. A 391-paper dry-run inventory completed without enqueueing jobs. +## PDF fallback development + +The existing implementation was committed in seven strategic groups: plan, schema, provider resolution, jobs/import capture, API, UI, and pilot/operations. Development then continued with an optional GROBID adapter and migration 034. + +Implemented: credentialed managed-PDF reads, GROBID `processReferences` with consolidation disabled, raw bibliography/TEI evidence, parser versions and content hashes, conservative DOI/arXiv matching, partial-output preservation, explicit API/UI/backfill opt-in, and a read-only PDF pilot command. PDF revision tracking and a content recheck fence results from replaced files even when the URL is unchanged. PDF jobs refresh automatically only for previously opted-in papers. Title-only references remain unresolved. + +The user confirmed no GROBID service is available yet. Extraction remains disabled until `GROBID_URL` is configured; no private PDF has been sent to GROBID and no real parser coverage gain is claimed. Semantic Scholar remains unqueried. The next operational step is to configure a self-hosted service and run `pilot_pdf_citations.py` before rolling out PDF extraction. + +Validation for this phase: 358 backend tests and 403 frontend tests passed, including parser output, private-storage access, request opt-in, file replacement, evidence display, and missing-service behavior. Both migrations replay successfully and SQL fixtures pass. The citation browser test and production frontend build also passed. The earlier context-menu browser timeout and UTC-dependent Dashboard fixture remain documented above. + ## Outcome and scope Let a user open a paper and see which papers in the current library it cites and which library papers cite it. Each connection must have retrievable evidence, a reliable direction, and an indexing status that distinguishes missing data from a failed lookup. @@ -143,7 +153,7 @@ Files: add operator instructions in `docs/guides/citation-indexing.md`; create ` - [x] Record identity-resolution rate, provider reference availability, local edges, ambiguities, failures, request volume, and runtime separately. Manually compare up to 20 detected edges with bibliographies and check every selected known positive pair. Document denominators and unavailable PDFs. - [x] Treat every detected false positive as a release blocker until the matching rule is fixed and checked again. Report recall limits explicitly; do not promise complete coverage from this sample. - [x] Prepare migration/backfill commands, rollback instructions, and the pilot report. Keep worker failure messages free of credentials and full PDF contents. -- [ ] Apply migration 033 to the intended live database, validate its permissions and concurrent workers, then start the worker and expand the persisted backfill with resumable batches. +- [ ] Apply migration 033 (and 034 for PDF extraction) to the intended live database, validate permissions and concurrent workers, then start the worker and expand the persisted backfill with resumable batches. - [x] Document rollback by disabling enqueueing, stopping workers, and hiding the feature while retaining additive tables/evidence. Avoid destructive rollback of indexed data. Milestone 1 exit: supported citation pairs appear in both directions with inspectable evidence; new imports reconcile; retries and restarts preserve data; coverage limitations are measured and visible. @@ -169,8 +179,8 @@ Files: proposed `backend/services/semantic_scholar_client.py` and/or `backend/se - [ ] Evaluate Semantic Scholar on unresolved arXiv/identifier cases. Verify current endpoint fields, pagination, authentication, and limits before implementation. - [ ] Evaluate GROBID on stored PDFs whose bibliographies are missing from provider results. Read PDFs through the existing managed-storage abstraction, including private GCS objects; do not assume stored URLs are publicly fetchable. -- [ ] Extract from original PDF bytes rather than the truncated Markdown cache. Retain document hash/version, parser version, raw bibliography text, and structured identifiers. Use citation context/page coordinates only when supplied by a suitable parser output. -- [ ] Report scanned/unparseable PDFs as unsupported extraction cases; do not silently produce empty bibliographies. Keep OCR as a separate decision. +- [x] Extract from original PDF bytes rather than the truncated Markdown cache. Retain document hash/version, parser version, raw bibliography text, and structured identifiers. Use citation context/page coordinates only when supplied by a suitable parser output. +- [x] Report empty or unparseable extraction as partial/failure, preserving old evidence and explaining that scanned PDFs may need OCR. Keep OCR as a separate decision. - [ ] Compare additional verified local edges, ambiguous matches, failure rate, runtime, and operating requirements. Implement the fallback with demonstrated benefit using the existing jobs/evidence model; record a no-go decision if neither adds sufficient value. Milestone 3 exit: remaining gaps have measured causes and either an integrated, tested fallback or an explicit evidence-based deferral. diff --git a/frontend/src/components/PaperCitations.jsx b/frontend/src/components/PaperCitations.jsx index aadc61a..4f19b03 100644 --- a/frontend/src/components/PaperCitations.jsx +++ b/frontend/src/components/PaperCitations.jsx @@ -3,11 +3,11 @@ import { Link } from 'react-router-dom' import { papersApi } from '../services/api' // Remount on paper/library change so old reads, mutations and polls cannot leak state. -export default function PaperCitations({ paperId, libraryId, sourceVersion = '' }) { - return +export default function PaperCitations({ paperId, libraryId, sourceVersion = '', hasStoredPdf = false }) { + return } -function CitationLists({ paperId }) { +function CitationLists({ paperId, hasStoredPdf }) { const [direction, setDirection] = useState('outgoing') const [offset, setOffset] = useState(0) const [data, setData] = useState(null) @@ -44,11 +44,12 @@ function CitationLists({ paperId }) { return () => { cancelled = true; clearTimeout(timer) } }, [paperId, direction, offset, refresh]) - async function reindex() { + async function reindex(pdf = false) { setQueueing(true) setError(null) try { - await papersApi.reindexCitations(paperId) + if (pdf) await papersApi.reindexPdfCitations(paperId) + else await papersApi.reindexCitations(paperId) if (alive.current) setRefresh(value => value + 1) } catch (err) { if (alive.current) setError(err.message) @@ -94,12 +95,14 @@ function CitationLists({ paperId }) {
Citation evidence {item.evidence.map((evidence, index) =>
-

{evidence.provider === 'openalex' ? 'OpenAlex' : 'Crossref'} · identifier match · {new Date(evidence.indexedAt).toLocaleDateString()}

+

{{ openalex: 'OpenAlex', crossref: 'Crossref', grobid: 'PDF bibliography (GROBID)' }[evidence.provider] || evidence.provider} · identifier match · {new Date(evidence.indexedAt).toLocaleDateString()}

{evidence.provider === 'openalex' && /^W\d+$/.test(evidence.reference.source_work) && Source record} {evidence.reference.unstructured &&

{evidence.reference.unstructured}

} {evidence.reference.DOI &&

DOI: {evidence.reference.DOI}

} {evidence.reference.target_work &&

Referenced work: {evidence.reference.target_work}

} + {evidence.provider === 'grobid' && evidence.reference.title &&

{evidence.reference.title}

} + {evidence.provider === 'grobid' && evidence.reference.pdf_sha256 &&

PDF fingerprint: {evidence.reference.pdf_sha256.slice(0, 12)}…

}
)}
)} @@ -112,9 +115,13 @@ function CitationLists({ paperId }) {

{data.indexedPapers} of {data.libraryPapers} library papers have indexed reference data.

{data.unresolvedCount > 0 &&

{data.unresolvedCount} reference records not matched to a unique local citation.

} } - + {hasStoredPdf &&
+ +

Uses the configured PDF extraction service to read the stored PDF.

+
} ) } diff --git a/frontend/src/components/PaperCitations.test.jsx b/frontend/src/components/PaperCitations.test.jsx index 52dedc3..1fb1087 100644 --- a/frontend/src/components/PaperCitations.test.jsx +++ b/frontend/src/components/PaperCitations.test.jsx @@ -4,7 +4,7 @@ import { beforeEach, expect, it, vi } from 'vitest' import PaperCitations from './PaperCitations' import { papersApi } from '../services/api' -vi.mock('../services/api', () => ({ papersApi: { citations: vi.fn(), reindexCitations: vi.fn() } })) +vi.mock('../services/api', () => ({ papersApi: { citations: vi.fn(), reindexCitations: vi.fn(), reindexPdfCitations: vi.fn() } })) beforeEach(() => vi.resetAllMocks()) const page = (overrides = {}) => ({ items: [], total: 0, limit: 10, offset: 0, direction: 'outgoing', jobs: [], runs: [], unresolvedCount: 0, indexedPapers: 0, libraryPapers: 2, pendingJobs: 0, ...overrides }) @@ -99,3 +99,28 @@ it('queues explicitly and preserves visible links while awaiting the worker', as expect(screen.getByRole('button', { name: 'Indexing…' })).toBeDisabled() expect(screen.getByRole('link', { name: 'Target paper' })).toBeInTheDocument() }) + +it('PDF extraction is an explicit action and reports missing service configuration', async () => { + papersApi.citations.mockResolvedValue(page()) + papersApi.reindexPdfCitations.mockRejectedValue(new Error('Configure GROBID first')) + render() + await screen.findByText('Not indexed yet.') + expect(papersApi.reindexPdfCitations).not.toHaveBeenCalled() + fireEvent.click(screen.getByRole('button', { name: 'Index PDF bibliography' })) + expect(await screen.findByRole('alert')).toHaveTextContent('Configure GROBID first') + expect(papersApi.reindexPdfCitations).toHaveBeenCalledWith('p_a') + expect(papersApi.reindexCitations).not.toHaveBeenCalled() +}) + +it('shows PDF bibliography evidence with the original reference and file fingerprint', async () => { + papersApi.citations.mockResolvedValue(page({ items: [{ ...target, evidence: [{ provider: 'grobid', + method: 'exact_identifier', indexedAt: '2026-11-03T00:00:00Z', + reference: { unstructured: 'A. Writer, Target paper, 2024.', pdf_sha256: 'a'.repeat(64) }, + }] }], total: 1 })) + render(view()) + await screen.findByText('Target paper') + expect(screen.getByText(/PDF bibliography \(GROBID\)/)).toBeInTheDocument() + expect(screen.getByText('A. Writer, Target paper, 2024.')).toBeInTheDocument() + expect(screen.getByTitle('a'.repeat(64))).toBeInTheDocument() + expect(screen.queryByRole('button', { name: 'Index PDF bibliography' })).not.toBeInTheDocument() +}) diff --git a/frontend/src/pages/Paper.jsx b/frontend/src/pages/Paper.jsx index 543e001..72213b8 100644 --- a/frontend/src/pages/Paper.jsx +++ b/frontend/src/pages/Paper.jsx @@ -524,7 +524,8 @@ export default function Paper() { onPaperUpdate={setPaper} /> + hasStoredPdf={paper.hasStoredPdf} + sourceVersion={JSON.stringify([paper.doi, paper.arxivId, paper.title, paper.authors, paper.year, paper.pdfUrl, paper.pdfRevision])} /> )} {sideTab === 'notes' && ( diff --git a/frontend/src/services/api.js b/frontend/src/services/api.js index 300a476..be88ddc 100644 --- a/frontend/src/services/api.js +++ b/frontend/src/services/api.js @@ -47,6 +47,7 @@ export const papersApi = { citations: (id, { direction = 'outgoing', limit = 20, offset = 0 } = {}) => apiFetch(`/papers/${encodeURIComponent(id)}/citations?${new URLSearchParams({ direction, limit, offset })}`), reindexCitations: (id) => apiFetch(`/papers/${encodeURIComponent(id)}/citations/reindex`, { method: 'POST' }), + reindexPdfCitations: (id) => apiFetch(`/papers/${encodeURIComponent(id)}/citations/pdf/reindex`, { method: 'POST' }), citationJob: (id) => apiFetch(`/citation-jobs/${encodeURIComponent(id)}`), related: (id, { limit = 12 } = {}) => apiFetch(`/papers/${id}/related?limit=${encodeURIComponent(limit)}`), create: (data, { checkDuplicates = false } = {}) => { diff --git a/frontend/src/services/api.test.js b/frontend/src/services/api.test.js index e566946..1c768e4 100644 --- a/frontend/src/services/api.test.js +++ b/frontend/src/services/api.test.js @@ -258,9 +258,12 @@ describe('api service wrapper', () => { await papersApi.citations('p/a', { direction: 'incoming', offset: 20, limit: 10 }) await papersApi.reindexCitations('p/a') await papersApi.citationJob('job-1') + await papersApi.reindexPdfCitations('p/a') expect(global.fetch.mock.calls[0][0]).toBe('/api/papers/p%2Fa/citations?direction=incoming&limit=10&offset=20') expect(global.fetch.mock.calls[1][0]).toBe('/api/papers/p%2Fa/citations/reindex') expect(global.fetch.mock.calls[1][1].method).toBe('POST') expect(global.fetch.mock.calls[2][0]).toBe('/api/citation-jobs/job-1') + expect(global.fetch.mock.calls[3][0]).toBe('/api/papers/p%2Fa/citations/pdf/reindex') + expect(global.fetch.mock.calls[3][1].method).toBe('POST') }) }) From 7f5c86ee3bb4180911ee9d123d7c98a2131d68e6 Mon Sep 17 00:00:00 2001 From: Prannaya Gupta Date: Wed, 4 Nov 2026 00:53:09 -0500 Subject: [PATCH 11/23] feat(db): add independently opted-in local PDF citation jobs --- .../migrations/035_local_pdf_citations.sql | 37 +++++++++++++++ backend/models/citation.py | 2 +- backend/tests/run_citation_sql.mjs | 7 ++- backend/tests/sql/local_pdf_citations.sql | 46 +++++++++++++++++++ 4 files changed, 90 insertions(+), 2 deletions(-) create mode 100644 backend/migrations/035_local_pdf_citations.sql create mode 100644 backend/tests/sql/local_pdf_citations.sql diff --git a/backend/migrations/035_local_pdf_citations.sql b/backend/migrations/035_local_pdf_citations.sql new file mode 100644 index 0000000..d18a5b9 --- /dev/null +++ b/backend/migrations/035_local_pdf_citations.sql @@ -0,0 +1,37 @@ +-- Local PDF extraction, opt-in independently of GROBID. Requires 033 and 034. +BEGIN; +ALTER TABLE public.paper_reference_runs DROP CONSTRAINT IF EXISTS paper_reference_runs_provider_check; +ALTER TABLE public.paper_reference_runs ADD CONSTRAINT paper_reference_runs_provider_check CHECK (provider IN ('openalex','crossref','grobid','pymupdf')); +ALTER TABLE public.paper_citation_jobs DROP CONSTRAINT IF EXISTS paper_citation_jobs_provider_check; +ALTER TABLE public.paper_citation_jobs ADD CONSTRAINT paper_citation_jobs_provider_check CHECK (provider IN ('openalex','crossref','grobid','pymupdf')); +CREATE OR REPLACE FUNCTION public.enqueue_local_pdf_citations(p_paper_id TEXT,p_force BOOLEAN DEFAULT false) +RETURNS SETOF public.paper_citation_jobs +LANGUAGE plpgsql SECURITY INVOKER SET search_path=public AS $$ +DECLARE p papers; +BEGIN + SELECT * INTO p FROM papers WHERE id=p_paper_id FOR UPDATE; + IF NOT FOUND THEN RETURN; END IF; + INSERT INTO paper_citation_jobs(paper_id,provider,fingerprint) + VALUES (p.id,'pymupdf',citation_fingerprint(p)) + ON CONFLICT (paper_id,provider) DO UPDATE SET + fingerprint=EXCLUDED.fingerprint,status='pending',attempts=0,error=NULL, + lease_token=NULL,lease_until=NULL,next_attempt_at=now(),updated_at=now() + WHERE paper_citation_jobs.fingerprint<>EXCLUDED.fingerprint + OR (p_force AND paper_citation_jobs.status NOT IN ('pending','running')); + RETURN QUERY SELECT * FROM paper_citation_jobs WHERE paper_id=p.id AND provider='pymupdf'; +END $$; + +CREATE OR REPLACE FUNCTION public.refresh_opted_in_pdf_citations() RETURNS trigger +LANGUAGE plpgsql SECURITY INVOKER SET search_path=public AS $$ +BEGIN + IF citation_fingerprint(OLD)<>citation_fingerprint(NEW) THEN + IF EXISTS(SELECT 1 FROM paper_citation_jobs WHERE paper_id=NEW.id AND provider='grobid') THEN + PERFORM enqueue_pdf_citations(NEW.id); + END IF; + IF EXISTS(SELECT 1 FROM paper_citation_jobs WHERE paper_id=NEW.id AND provider='pymupdf') THEN + PERFORM enqueue_local_pdf_citations(NEW.id); + END IF; + END IF; + RETURN NEW; +END $$; +COMMIT; diff --git a/backend/models/citation.py b/backend/models/citation.py index cc6d946..f516cd3 100644 --- a/backend/models/citation.py +++ b/backend/models/citation.py @@ -5,7 +5,7 @@ from models.base import CamelModel -Provider = Literal["openalex", "crossref", "grobid"] +Provider = Literal["openalex", "crossref", "grobid", "pymupdf"] RunStatus = Literal["succeeded", "partial", "not_found"] diff --git a/backend/tests/run_citation_sql.mjs b/backend/tests/run_citation_sql.mjs index 53d6ebe..148fed4 100644 --- a/backend/tests/run_citation_sql.mjs +++ b/backend/tests/run_citation_sql.mjs @@ -17,7 +17,12 @@ try { await db.exec(pdfMigration) await db.exec(await readFile(new URL('./sql/paper_citations.sql', import.meta.url), 'utf8')) await db.exec(await readFile(new URL('./sql/pdf_citations.sql', import.meta.url), 'utf8')) - console.log('Citation and PDF SQL checks passed; migrations 033 and 034 applied twice.') + const localMigration = await readFile(new URL('../migrations/035_local_pdf_citations.sql', import.meta.url), 'utf8') + await db.exec(localMigration) + await db.exec(localMigration) + await db.exec(await readFile(new URL('./sql/pdf_citations.sql', import.meta.url), 'utf8')) + await db.exec(await readFile(new URL('./sql/local_pdf_citations.sql', import.meta.url), 'utf8')) + console.log('Citation SQL checks passed; migrations 033, 034 and 035 applied twice.') } finally { await db.close() } diff --git a/backend/tests/sql/local_pdf_citations.sql b/backend/tests/sql/local_pdf_citations.sql new file mode 100644 index 0000000..891f71e --- /dev/null +++ b/backend/tests/sql/local_pdf_citations.sql @@ -0,0 +1,46 @@ +-- Run after 033, 034 and 035 on a disposable database. +BEGIN; +INSERT INTO libraries(id,name,created_at) VALUES ('pdf_lib','PDF citations','2026-11-03'); +INSERT INTO papers(id,title,authors,year,venue,status,source,library_id,created_at,pdf_url,doi) VALUES + ('pdf_source','PDF-only source','[]',2025,'','read','human','pdf_lib','2026-11-03','https://storage.example/p.pdf',NULL), + ('pdf_target','Target','[]',2024,'','read','human','pdf_lib','2026-11-03',NULL,'10.1234/target'); + +DO $$ +DECLARE j paper_citation_jobs; old_token UUID; result JSONB; revision BIGINT; +BEGIN + IF EXISTS(SELECT 1 FROM paper_citation_jobs WHERE provider='pymupdf' AND paper_id='pdf_source') THEN + RAISE EXCEPTION 'PDF extraction must be opt-in'; + END IF; + PERFORM enqueue_local_pdf_citations('pdf_source'); + PERFORM enqueue_local_pdf_citations('pdf_source'); + IF (SELECT count(*) FROM paper_citation_jobs WHERE paper_id='pdf_source' AND provider='pymupdf')<>1 THEN + RAISE EXCEPTION 'Duplicate PDF jobs'; + END IF; + SELECT * INTO j FROM claim_citation_job((SELECT id FROM paper_citation_jobs WHERE paper_id='pdf_source' AND provider='pymupdf')); + result := '{"provider":"pymupdf","status":"succeeded","resolution":"exact","version":"pymupdf:test;sha256:original","identities":{},"candidates":[],"references":[{"key":"b0","identifiers":{"doi":"10.1234/target"},"raw":{"unstructured":"Original reference","pdf_sha256":"original"}}]}'::jsonb; + IF NOT finish_citation_job(j.id,j.lease_token,result) THEN RAISE EXCEPTION 'PDF publication failed'; END IF; + IF (get_paper_citations('pdf_source')->>'total')::int<>1 THEN RAISE EXCEPTION 'PDF-only source did not match target DOI'; END IF; + IF get_paper_citations('pdf_source')#>>'{items,0,evidence,0,reference,pdf_sha256}'<>'original' THEN + RAISE EXCEPTION 'PDF hash was lost'; + END IF; + PERFORM enqueue_local_pdf_citations('pdf_source',true); + SELECT * INTO j FROM claim_citation_job(j.id); + -- A parser with no structured output must not erase previous evidence. + IF NOT finish_citation_job(j.id,j.lease_token,jsonb_set(jsonb_set(result,'{status}','"partial"'),'{references}','[]')) THEN + RAISE EXCEPTION 'Partial run failed'; + END IF; + IF (get_paper_citations('pdf_source')->>'total')::int<>1 THEN RAISE EXCEPTION 'Partial extraction erased evidence'; END IF; + PERFORM enqueue_local_pdf_citations('pdf_source',true); + SELECT * INTO j FROM claim_citation_job(j.id); + SELECT pdf_revision INTO revision FROM papers WHERE id='pdf_source'; + UPDATE papers SET pdf_url=pdf_url WHERE id='pdf_source'; + IF (SELECT pdf_revision FROM papers WHERE id='pdf_source')<>revision+1 THEN RAISE EXCEPTION 'Same-URL upload not versioned'; END IF; + IF finish_citation_job(j.id,j.lease_token,result) THEN RAISE EXCEPTION 'Old PDF worker published after replacement'; END IF; + IF (get_paper_citations('pdf_source')->>'total')::int<>0 THEN RAISE EXCEPTION 'Old PDF evidence still visible'; END IF; + IF (SELECT status FROM paper_citation_jobs WHERE id=j.id)<>'pending' THEN RAISE EXCEPTION 'Opted-in PDF job not requeued'; END IF; + UPDATE papers SET title='Changed target' WHERE id='pdf_target'; + IF EXISTS(SELECT 1 FROM paper_citation_jobs WHERE paper_id='pdf_target' AND provider='pymupdf') THEN + RAISE EXCEPTION 'Metadata edit enabled PDF extraction without opt-in'; + END IF; +END $$; +ROLLBACK; From 6928e9b5b23c615c0825899b13ada0c29fd0dc8f Mon Sep 17 00:00:00 2001 From: Prannaya Gupta Date: Wed, 4 Nov 2026 00:53:18 -0500 Subject: [PATCH 12/23] feat(citations): extract numbered PDF bibliographies locally --- backend/.env.example | 3 +- backend/routers/citations.py | 19 +-- backend/scripts/index_paper_citations.py | 9 +- backend/scripts/pilot_pdf_citations.py | 9 +- backend/services/citation_job_service.py | 7 +- backend/services/citation_repository.py | 7 +- .../services/local_pdf_reference_service.py | 149 ++++++++++++++++++ backend/tests/test_citation_jobs.py | 10 ++ backend/tests/test_citation_routes.py | 16 +- backend/tests/test_local_pdf_references.py | 108 +++++++++++++ frontend/src/components/PaperCitations.jsx | 8 +- .../src/components/PaperCitations.test.jsx | 10 +- 12 files changed, 318 insertions(+), 37 deletions(-) create mode 100644 backend/services/local_pdf_reference_service.py create mode 100644 backend/tests/test_local_pdf_references.py diff --git a/backend/.env.example b/backend/.env.example index 4337f1b..d2f9f0b 100644 --- a/backend/.env.example +++ b/backend/.env.example @@ -1,6 +1,7 @@ OPENAI_API_KEY=* OPENALEX_API_KEY=* -# Optional self-hosted bibliography extractor. No PDF is sent until explicitly queued. +# Optional alternate extractor; local PyMuPDF indexing needs no service. +# PDFs are sent here only for explicitly selected GROBID jobs. GROBID_URL= SUPABASE_URL=* SUPABASE_KEY=* diff --git a/backend/routers/citations.py b/backend/routers/citations.py index d035f13..3d3b882 100644 --- a/backend/routers/citations.py +++ b/backend/routers/citations.py @@ -48,23 +48,24 @@ def citation_job(job_id: UUID): @router.post("/papers/{paper_id}/citations/pdf/reindex", status_code=202, response_model=list[CitationJob]) -def reindex_pdf(paper_id: str): +def reindex_pdf(paper_id: str, engine: Literal["pymupdf", "grobid"] = "pymupdf"): paper = paper_service.get_paper(paper_id) if paper is None: raise HTTPException(404, {"error": "not_found", "detail": "Paper not found"}) - try: - url = configured_url() - except CitationProviderError as exc: - raise HTTPException(503, str(exc)) from None - if not url: - raise HTTPException(503, "PDF bibliography indexing requires a configured GROBID service.") + if engine == "grobid": + try: + url = configured_url() + except CitationProviderError as exc: + raise HTTPException(503, str(exc)) from None + if not url: + raise HTTPException(503, "PDF bibliography indexing requires a configured GROBID service.") if not managed_location(paper.pdf_url): raise HTTPException(422, "Upload or store this PDF before indexing its bibliography.") try: - jobs = repository.enqueue_pdf(paper_id, True) + jobs = repository.enqueue_pdf(paper_id, True, engine) except APIError as exc: if exc.code in {"42P01", "42883", "PGRST202", "PGRST205"}: - raise HTTPException(503, "PDF bibliography indexing is not installed. Apply migration 034.") from None + raise HTTPException(503, "PDF bibliography indexing is not installed. Apply migrations 034 and 035.") from None raise if not jobs: raise HTTPException(404, {"error": "not_found", "detail": "Paper not found"}) diff --git a/backend/scripts/index_paper_citations.py b/backend/scripts/index_paper_citations.py index 8d71b37..c84cd7e 100644 --- a/backend/scripts/index_paper_citations.py +++ b/backend/scripts/index_paper_citations.py @@ -31,7 +31,7 @@ def main(): parser.add_argument("--limit", type=int, default=0, help="0 selects all papers in scope") parser.add_argument("--dry-run", action="store_true") parser.add_argument("--refresh", action="store_true", help="Also requeue completed jobs") - parser.add_argument("--include-pdf", action="store_true", help="Also enqueue managed PDFs for the configured GROBID service") + parser.add_argument("--include-pdf", action="store_true", help="Also enqueue managed PDFs for local bibliography extraction") args = parser.parse_args() if args.limit < 0: parser.error("--limit must be nonnegative") @@ -47,13 +47,14 @@ def main(): if args.limit and len(papers) >= args.limit: break batch = {"id": str(uuid4()), "library_id": args.library_id, "all_libraries": args.all_libraries, - "paper_ids": papers, "jobs": {}, "refresh": args.refresh, "include_pdf": args.include_pdf} + "paper_ids": papers, "jobs": {}, "refresh": args.refresh, "include_pdf": args.include_pdf, "pdf_engine": "pymupdf"} manifest = BACKEND / "data" / "citations" / f"{batch['id']}.json" if args.dry_run: print(json.dumps({"papers": len(batch["paper_ids"]), "already_enqueued": len(batch["jobs"]), "library_id": batch["library_id"], "all_libraries": batch["all_libraries"]})) return - if batch.get("include_pdf") and not configured_url(): + engine = batch.get("pdf_engine", "grobid") # Preserve the scope of older manifests. + if batch.get("include_pdf") and engine == "grobid" and not configured_url(): parser.error("--include-pdf requires GROBID_URL; it sends managed PDFs to that configured service") save_manifest(manifest, batch) for paper_id in batch["paper_ids"]: @@ -63,7 +64,7 @@ def main(): if batch.get("include_pdf"): paper = paper_service.get_paper(paper_id) if paper and managed_location(paper.pdf_url): - jobs.extend(citation_repository.enqueue_pdf(paper_id, batch["refresh"])) + jobs.extend(citation_repository.enqueue_pdf(paper_id, batch["refresh"], engine)) batch["jobs"][paper_id] = list(dict.fromkeys(job["id"] for job in jobs)) save_manifest(manifest, batch) print(json.dumps({"batch": str(manifest), "papers": len(batch["jobs"])})) diff --git a/backend/scripts/pilot_pdf_citations.py b/backend/scripts/pilot_pdf_citations.py index 27e81f2..f157a6e 100644 --- a/backend/scripts/pilot_pdf_citations.py +++ b/backend/scripts/pilot_pdf_citations.py @@ -1,4 +1,4 @@ -"""Extract one managed PDF through configured GROBID without changing the database.""" +"""Extract one managed PDF locally (or explicitly through GROBID) without changing the database.""" import argparse import asyncio from collections import defaultdict @@ -16,6 +16,7 @@ from services.citation_identity_service import local_identifiers from services.pdf_reference_service import configured_url, fetch_pdf_references from services.openalex_client import CitationProviderError +from services.local_pdf_reference_service import fetch_local_pdf_references async def run(args): @@ -23,7 +24,8 @@ async def run(args): if paper is None: raise ValueError('Paper not found') async with httpx.AsyncClient(timeout=30) as client: - result = await fetch_pdf_references(paper.model_dump(), client) + result = (await fetch_pdf_references(paper.model_dump(), client) if args.engine == "grobid" + else await fetch_local_pdf_references(paper.model_dump())) identities = defaultdict(set) targets = {} # A missing library ID is a distinct scope, never an implicit cross-library match. @@ -57,11 +59,12 @@ async def run(args): parser = argparse.ArgumentParser(description=__doc__) parser.add_argument('--paper-id', required=True) parser.add_argument('--output', type=Path, required=True) + parser.add_argument('--engine', choices=['pymupdf', 'grobid'], default='pymupdf') args = parser.parse_args() if args.output.exists(): parser.error('Choose a new output path; existing pilot reports are not overwritten') try: - if not configured_url(): + if args.engine == "grobid" and not configured_url(): parser.error('Configure a self-hosted GROBID_URL before sending a PDF for extraction') asyncio.run(run(args)) except (ValueError, CitationProviderError) as exc: diff --git a/backend/services/citation_job_service.py b/backend/services/citation_job_service.py index 8d0e4a5..af8fa8a 100644 --- a/backend/services/citation_job_service.py +++ b/backend/services/citation_job_service.py @@ -6,6 +6,7 @@ from services import citation_repository as repository, paper_service from services.citation_service import fetch_crossref, fetch_openalex from services.pdf_reference_service import fetch_pdf_references +from services.local_pdf_reference_service import fetch_local_pdf_references from services.openalex_client import CitationProviderError logger = logging.getLogger(__name__) @@ -26,8 +27,12 @@ async def run_one(job_id: str | None = None) -> dict | None: result = await fetch_openalex(data, client) elif job["provider"] == "grobid": result = await fetch_pdf_references(data, client) - else: + elif job["provider"] == "pymupdf": + result = await fetch_local_pdf_references(data) + elif job["provider"] == "crossref": result = await fetch_crossref(data, client, job.get("captured_record")) + else: + raise CitationProviderError("Unsupported citation provider") published = repository.finish(job, result) return {"id": job["id"], "status": result.status if published else "superseded"} except CitationProviderError as exc: diff --git a/backend/services/citation_repository.py b/backend/services/citation_repository.py index a4b3bc5..d450207 100644 --- a/backend/services/citation_repository.py +++ b/backend/services/citation_repository.py @@ -6,8 +6,11 @@ def enqueue(paper_id: str, force: bool = False) -> list[dict]: return get_client().rpc("enqueue_paper_citations", {"p_paper_id": paper_id, "p_force": force}).execute().data or [] -def enqueue_pdf(paper_id: str, force: bool = False) -> list[dict]: - return get_client().rpc("enqueue_pdf_citations", {"p_paper_id": paper_id, "p_force": force}).execute().data or [] +def enqueue_pdf(paper_id: str, force: bool = False, engine: str = "pymupdf") -> list[dict]: + if engine not in {"pymupdf", "grobid"}: + raise ValueError("Unsupported PDF engine") + rpc = "enqueue_local_pdf_citations" if engine == "pymupdf" else "enqueue_pdf_citations" + return get_client().rpc(rpc, {"p_paper_id": paper_id, "p_force": force}).execute().data or [] def claim(job_id: str | None = None) -> dict | None: diff --git a/backend/services/local_pdf_reference_service.py b/backend/services/local_pdf_reference_service.py new file mode 100644 index 0000000..d215023 --- /dev/null +++ b/backend/services/local_pdf_reference_service.py @@ -0,0 +1,149 @@ +"""Conservative, local extraction of numbered bibliographies; no OCR or remote parser.""" +import hashlib +import re + +import fitz + +from models.citation import CitationReference, CitationResult +from services import pdf_service +from services.citation_identity_service import local_identifiers, normalize_arxiv, normalize_doi +from services.openalex_client import CitationProviderError + +HEADING = re.compile(r'^(?:\d+[.\s]+)?(?:references|bibliography|works cited)\s*$', re.I) +STOP = re.compile(r'^(?:[A-Z\d]+[.\s]+)?(?:appendix|appendices|supplementary|supplemental|acknowledg\w*|biograph\w*)\b', re.I) +LABEL = re.compile(r'^\s*(?:\[(\d{1,3})\]|(\d{1,3})\.)\s+(.+)') + + +def _lines(page): + blocks = [b for b in page.get_text('dict', flags=fitz.TEXTFLAGS_DICT & ~fitz.TEXT_PRESERVE_IMAGES)['blocks'] if b['type'] == 0] + middle = page.rect.width / 2 + left = [b for b in blocks if b['bbox'][2] < middle + 12 and b['bbox'][0] < middle - 40] + right = [b for b in blocks if b['bbox'][0] > middle - 12 and b['bbox'][2] > middle + 40] + columns = bool(left and right) + blocks.sort(key=lambda b: (0 if not columns or b['bbox'][0] < middle - 12 else 1, b['bbox'][1], b['bbox'][0])) + for block in blocks: + column = int(columns and block['bbox'][0] > middle - 12) + for line in block['lines']: + text = ''.join(span['text'] for span in line['spans']).strip() + y0, y1 = line['bbox'][1], line['bbox'][3] + # Running headers and footers are not bibliography continuation. + if (y0 < page.rect.height * .06 or y1 > page.rect.height * .94) and not HEADING.fullmatch(text): + continue + if text and not text.isdigit(): + size = max(span['size'] for span in line['spans']) + yield text, size, column, y0, y1 + + +def explicit_identifiers(text): + values = {'doi': set(), 'arxiv': set()} + for value in re.findall(r'\b10\.\d{4,9}/[^\s<>]+', text, re.I): + value = value.rstrip('.,;:') + while value.endswith(')') and value.count(')') > value.count('('): + value = value[:-1] + value = normalize_doi(value) + if value: + values['doi'].add(value) + pattern = r'(?:arxiv\s*:\s*|arxiv\.org/(?:abs|pdf)/)(\d{4}\.\d{4,5}(?:v\d+)?|[a-z.-]+/\d{7}(?:v\d+)?)' + for value in re.findall(pattern, text, re.I): + value = normalize_arxiv(value) + if value: + values['arxiv'].add(value) + ambiguous = any(len(v) > 1 for v in values.values()) + return ({} if ambiguous else {k: next(iter(v)) for k, v in values.items() if v}), ambiguous + + +def extract_references(content: bytes, paper: dict) -> CitationResult: + digest = hashlib.sha256(content).hexdigest() + result = CitationResult(provider='pymupdf', identities=local_identifiers(paper), + version=f'pymupdf:{fitz.VersionBind};adapter:1;sha256:{digest}') + entries, current, found, stopped = [], None, False, False + label_style = None + reference_size, previous, uncertain_boundary = None, None, False + unsupported = [] + try: + with fitz.open(stream=content, filetype='pdf') as document: + if document.needs_pass or len(document) > 1000: + raise CitationProviderError('Encrypted or oversized PDF cannot be indexed locally') + for page_number, page in enumerate(document, 1): + lines = list(_lines(page)) + if entries and any(STOP.match(line[0]) for line in lines[:4]): + break + previous = None + for line, size, column, y0, y1 in lines: + if not found: + if HEADING.fullmatch(line): + found = True + continue + if STOP.match(line): + stopped = True + break + if HEADING.fullmatch(line): + continue + match = LABEL.match(line) + style = ('bracket' if match[1] else 'dot') if match else None + if match and label_style and style != label_style: + match = None + if current and not match: + if size > reference_size * 1.18: + stopped = True + break + if previous and previous[0] == column and y0 - previous[1] > max(30, reference_size * 3): + uncertain_boundary = True + stopped = True + break + previous = (column, y1) + if match: + reference_size = size + label_style = style + current = {'label': int(match[1] or match[2]), 'lines': [line], 'pages': [page_number]} + entries.append(current) + elif current: + current['lines'].append(line) + if page_number not in current['pages']: + current['pages'].append(page_number) + else: + unsupported.append({'page': page_number, 'text': line}) + if stopped: + break + except CitationProviderError: + raise + except Exception: + raise CitationProviderError('PDF text could not be extracted locally') from None + complete = bool(entries) and not unsupported and not uncertain_boundary and [e['label'] for e in entries] == list(range(1, len(entries) + 1)) + for entry in entries: + text = '\n'.join(entry['lines']) + identifiers, ambiguous = explicit_identifiers(text) + result.references.append(CitationReference( + key=hashlib.sha256(f"{entry['label']}:{text}".encode()).hexdigest(), + identifiers=identifiers if complete else {}, + raw={'unstructured': text, 'pages': entry['pages'], 'reference_label': str(entry['label']), + 'pdf_sha256': digest, 'parser': 'PyMuPDF', 'parser_version': fitz.VersionBind, + 'ambiguous_identifiers': ambiguous})) + if not complete: + result.status = 'partial' + result.resolution = 'unresolved' + result.message = 'No complete numbered bibliography recognized; manual review or another parser is needed' + # Preserve unnumbered text for inspection without claiming citation identities. + if unsupported: + result.candidates.append({'bibliography_lines': unsupported, 'pdf_sha256': digest}) + else: + result.message = 'Numbered bibliography extracted locally; identifier-only matching, no OCR or title matching' + return result + + +async def fetch_local_pdf_references(paper: dict) -> CitationResult: + if not pdf_service.managed_location(paper.get('pdf_url')): + return CitationResult(provider='pymupdf', status='not_found', resolution='unresolved', + message='Upload or store this PDF before indexing its bibliography') + try: + content = pdf_service.read_managed_pdf(paper['pdf_url']) + result = extract_references(content, paper) + if hashlib.sha256(pdf_service.read_managed_pdf(paper['pdf_url'])).digest() != hashlib.sha256(content).digest(): + raise CitationProviderError('PDF changed during bibliography extraction', retryable=True, retry_after=1) + return result + except CitationProviderError: + raise + except ValueError as exc: + raise CitationProviderError(str(exc)) from None + except Exception: + raise CitationProviderError('Managed PDF could not be read', retryable=True) from None diff --git a/backend/tests/test_citation_jobs.py b/backend/tests/test_citation_jobs.py index a533d24..e25a503 100644 --- a/backend/tests/test_citation_jobs.py +++ b/backend/tests/test_citation_jobs.py @@ -73,3 +73,13 @@ def execute(self): return SimpleNamespace(data=selected[:min(self.size, 100)]) monkeypatch.setattr(paper_service, 'get_client', lambda: SimpleNamespace(table=lambda _: Query())) assert list(paper_service.iter_papers(library_id='lib')) == rows + + +def test_local_pdf_job_uses_local_extractor_and_publishes_evidence(monkeypatch): + setup_job(monkeypatch, 'pymupdf') + fetch = AsyncMock(return_value=CitationResult(provider='pymupdf', version='pymupdf:test;sha256:pdf')) + monkeypatch.setattr(jobs, 'fetch_local_pdf_references', fetch) + monkeypatch.setattr(jobs, 'fetch_crossref', AsyncMock(side_effect=AssertionError('Wrong provider'))) + monkeypatch.setattr(jobs.repository, 'finish', lambda job, result: result.provider == 'pymupdf') + assert asyncio.run(jobs.run_one())['status'] == 'succeeded' + fetch.assert_awaited_once() diff --git a/backend/tests/test_citation_routes.py b/backend/tests/test_citation_routes.py index 6d362d3..1f58ffb 100644 --- a/backend/tests/test_citation_routes.py +++ b/backend/tests/test_citation_routes.py @@ -55,12 +55,12 @@ def test_missing_migration_reports_setup_problem(client, mocker): assert 'migration 033' in response.json()['detail'] -def test_pdf_indexing_requires_service_and_never_queues_implicitly(client, mocker, monkeypatch): +def test_explicit_grobid_indexing_requires_service(client, mocker, monkeypatch): from types import SimpleNamespace monkeypatch.delenv('GROBID_URL', raising=False) mocker.patch('routers.citations.paper_service.get_paper', return_value=SimpleNamespace(pdf_url='stored')) queue = mocker.patch.object(citation_repository, 'enqueue_pdf') - response = client.post('/api/papers/a/citations/pdf/reindex') + response = client.post('/api/papers/a/citations/pdf/reindex?engine=grobid') assert response.status_code == 503 assert 'GROBID' in response.json()['detail'] queue.assert_not_called() @@ -68,11 +68,11 @@ def test_pdf_indexing_requires_service_and_never_queues_implicitly(client, mocke def test_pdf_indexing_checks_managed_storage_and_accepts_explicit_job(client, mocker, monkeypatch): from types import SimpleNamespace - monkeypatch.setenv('GROBID_URL', 'http://localhost:8070') + monkeypatch.delenv('GROBID_URL', raising=False) mocker.patch('routers.citations.paper_service.get_paper', return_value=SimpleNamespace(pdf_url='stored')) location = mocker.patch('routers.citations.managed_location', return_value=None) queue = mocker.patch.object(citation_repository, 'enqueue_pdf', return_value=[{ - 'id':'pdf-job', 'paper_id':'a', 'provider':'grobid', 'status':'pending', 'attempts':0, + 'id':'pdf-job', 'paper_id':'a', 'provider':'pymupdf', 'status':'pending', 'attempts':0, 'updated_at':'2026-11-03T00:00:00Z', }]) assert client.post('/api/papers/a/citations/pdf/reindex').status_code == 422 @@ -80,14 +80,14 @@ def test_pdf_indexing_checks_managed_storage_and_accepts_explicit_job(client, mo location.return_value = ('gcs','a.pdf') response = client.post('/api/papers/a/citations/pdf/reindex') assert response.status_code == 202 - assert response.json()[0]['provider'] == 'grobid' - queue.assert_called_once_with('a', True) + assert response.json()[0]['provider'] == 'pymupdf' + queue.assert_called_once_with('a', True, 'pymupdf') def test_pdf_indexing_missing_migration_has_specific_guidance(client, mocker, monkeypatch): from types import SimpleNamespace from postgrest.exceptions import APIError - monkeypatch.setenv('GROBID_URL', 'http://localhost:8070') + monkeypatch.delenv('GROBID_URL', raising=False) mocker.patch('routers.citations.paper_service.get_paper', return_value=SimpleNamespace(pdf_url='stored')) mocker.patch('routers.citations.managed_location', return_value=('gcs','a.pdf')) mocker.patch.object(citation_repository, 'enqueue_pdf', side_effect=APIError({ @@ -95,4 +95,4 @@ def test_pdf_indexing_missing_migration_has_specific_guidance(client, mocker, mo })) response = client.post('/api/papers/a/citations/pdf/reindex') assert response.status_code == 503 - assert 'migration 034' in response.json()['detail'] + assert 'migrations 034 and 035' in response.json()['detail'] diff --git a/backend/tests/test_local_pdf_references.py b/backend/tests/test_local_pdf_references.py new file mode 100644 index 0000000..bfe1561 --- /dev/null +++ b/backend/tests/test_local_pdf_references.py @@ -0,0 +1,108 @@ +import asyncio +import hashlib + +import fitz +import pytest + +from services import local_pdf_reference_service as service +from services.openalex_client import CitationProviderError + + +def pdf(*pages): + with fitz.open() as document: + for text in pages: + page = document.new_page() + page.insert_text((50, 60), text) + return document.tobytes() + + +def test_only_bibliography_identifiers_and_raw_evidence(): + content = pdf('Body DOI: 10.1234/body\nReferences\n[1] Author. Title. doi:10.1234/target.\n[2] Other. arXiv:2401.12345v2.\n[3] Title only.') + result = service.extract_references(content, {}) + assert result.status == 'succeeded' + assert [r.identifiers for r in result.references] == [{'doi': '10.1234/target'}, {'arxiv': '2401.12345'}, {}] + assert result.references[0].raw['pages'] == [1] + assert result.references[0].raw['pdf_sha256'] == hashlib.sha256(content).hexdigest() + assert 'Author. Title.' in result.references[0].raw['unstructured'] + + +def test_columns_years_page_continuations_and_appendix_boundary(): + with fitz.open() as document: + page = document.new_page(width=600, height=800) + page.insert_text((50, 60), 'References\n[1] First author.\n2024. 4\n[2] Second author.') + page.insert_text((320, 60), '[3] Third author.\ndoi:10.1234/third') + page = document.new_page(width=600, height=800) + page.insert_text((50, 60), 'continued text.\n[4] Fourth author.') + page = document.new_page() + page.insert_text((50, 60), 'Paper title\nSupplementary Material\ndoi:10.1234/not-a-reference') + content = document.tobytes() + result = service.extract_references(content, {}) + assert result.status == 'succeeded' + assert len(result.references) == 4 + assert result.references[2].raw['pages'] == [1, 2] + assert '2024. 4' in result.references[0].raw['unstructured'] + assert 'Paper title' not in result.references[-1].raw['unstructured'] + + +@pytest.mark.parametrize('text', [ + 'Body text DOI:10.1234/body', + 'References\nSmith. An unnumbered work. DOI:10.1234/work', + 'References\n[1] First DOI:10.1234/first\n[3] Missing entry.', + 'References\n[1] First.\n[1] Duplicate label.', + 'References\n[2] Missing beginning.', + '', +]) +def test_unsupported_or_incomplete_bibliographies_do_not_publish_identifiers(text): + result = service.extract_references(pdf(text), {}) + assert result.status == 'partial' + assert all(not r.identifiers for r in result.references) + + +def test_dotted_labels_and_conflicting_identifiers(): + result = service.extract_references(pdf('Bibliography\n1. doi:10.1234/a doi:10.1234/b\n2. arXiv:hep-th/9901001v3.'), {}) + assert result.status == 'succeeded' + assert result.references[0].identifiers == {} + assert result.references[0].raw['ambiguous_identifiers'] + assert result.references[1].identifiers == {'arxiv': 'hep-th/9901001'} + + +def test_large_appendix_heading_ends_bibliography(): + with fitz.open() as document: + page = document.new_page() + page.insert_text((50, 60), 'References\n[1] Work. doi:10.1234/reference', fontsize=10) + page.insert_text((50, 110), 'A', fontsize=15) + page.insert_text((80, 110), 'Implementation details', fontsize=15) + page.insert_text((50, 140), 'doi:10.1234/body', fontsize=10) + result = service.extract_references(document.tobytes(), {}) + assert result.status == 'succeeded' + assert result.references[0].identifiers == {'doi': '10.1234/reference'} + assert 'Implementation' not in result.references[0].raw['unstructured'] + + +def test_unheaded_biography_gap_is_incomplete_and_not_reference_text(): + with fitz.open() as document: + page = document.new_page() + page.insert_text((50, 60), 'References\n[1] Work. doi:10.1234/reference', fontsize=10) + page.insert_text((50, 250), 'Author received a degree. doi:10.1234/body', fontsize=10) + result = service.extract_references(document.tobytes(), {}) + assert result.status == 'partial' + assert result.references[0].identifiers == {} + assert 'received' not in result.references[0].raw['unstructured'] + + +def test_replacement_during_extraction_is_not_published(mocker): + mocker.patch.object(service.pdf_service, 'managed_location', return_value=('gcs', 'a.pdf')) + mocker.patch.object(service.pdf_service, 'read_managed_pdf', side_effect=[pdf('References\n[1] Work.'), pdf('replacement')]) + with pytest.raises(CitationProviderError, match='changed') as error: + asyncio.run(service.fetch_local_pdf_references({'pdf_url': 'managed'})) + assert error.value.retryable + + +def test_local_extraction_needs_no_grobid_or_http(mocker, monkeypatch): + monkeypatch.delenv('GROBID_URL', raising=False) + mocker.patch.object(service.pdf_service, 'managed_location', return_value=('gcs', 'a.pdf')) + read = mocker.patch.object(service.pdf_service, 'read_managed_pdf', return_value=pdf('References\n[1] Work.')) + result = asyncio.run(service.fetch_local_pdf_references({'pdf_url': 'managed'})) + assert result.provider == 'pymupdf' + assert result.status == 'succeeded' + assert read.call_count == 2 diff --git a/frontend/src/components/PaperCitations.jsx b/frontend/src/components/PaperCitations.jsx index 4f19b03..08cfb60 100644 --- a/frontend/src/components/PaperCitations.jsx +++ b/frontend/src/components/PaperCitations.jsx @@ -95,14 +95,14 @@ function CitationLists({ paperId, hasStoredPdf }) {
Citation evidence {item.evidence.map((evidence, index) =>
-

{{ openalex: 'OpenAlex', crossref: 'Crossref', grobid: 'PDF bibliography (GROBID)' }[evidence.provider] || evidence.provider} · identifier match · {new Date(evidence.indexedAt).toLocaleDateString()}

+

{{ openalex: 'OpenAlex', crossref: 'Crossref', grobid: 'PDF bibliography (GROBID)', pymupdf: 'PDF bibliography (local)' }[evidence.provider] || evidence.provider} · identifier match · {new Date(evidence.indexedAt).toLocaleDateString()}

{evidence.provider === 'openalex' && /^W\d+$/.test(evidence.reference.source_work) && Source record} {evidence.reference.unstructured &&

{evidence.reference.unstructured}

} {evidence.reference.DOI &&

DOI: {evidence.reference.DOI}

} {evidence.reference.target_work &&

Referenced work: {evidence.reference.target_work}

} - {evidence.provider === 'grobid' && evidence.reference.title &&

{evidence.reference.title}

} - {evidence.provider === 'grobid' && evidence.reference.pdf_sha256 &&

PDF fingerprint: {evidence.reference.pdf_sha256.slice(0, 12)}…

} + {['grobid', 'pymupdf'].includes(evidence.provider) && evidence.reference.title &&

{evidence.reference.title}

} + {['grobid', 'pymupdf'].includes(evidence.provider) && evidence.reference.pdf_sha256 &&

PDF fingerprint: {evidence.reference.pdf_sha256.slice(0, 12)}…

}
)}
)} @@ -120,7 +120,7 @@ function CitationLists({ paperId, hasStoredPdf }) { {hasStoredPdf &&
-

Uses the configured PDF extraction service to read the stored PDF.

+

Reads numbered bibliographies locally. Matches explicit DOI/arXiv identifiers; no GROBID setup needed.

} ) diff --git a/frontend/src/components/PaperCitations.test.jsx b/frontend/src/components/PaperCitations.test.jsx index 1fb1087..9b28176 100644 --- a/frontend/src/components/PaperCitations.test.jsx +++ b/frontend/src/components/PaperCitations.test.jsx @@ -100,26 +100,26 @@ it('queues explicitly and preserves visible links while awaiting the worker', as expect(screen.getByRole('link', { name: 'Target paper' })).toBeInTheDocument() }) -it('PDF extraction is an explicit action and reports missing service configuration', async () => { +it('PDF extraction is an explicit action and reports queue failures', async () => { papersApi.citations.mockResolvedValue(page()) - papersApi.reindexPdfCitations.mockRejectedValue(new Error('Configure GROBID first')) + papersApi.reindexPdfCitations.mockRejectedValue(new Error('PDF indexing unavailable')) render() await screen.findByText('Not indexed yet.') expect(papersApi.reindexPdfCitations).not.toHaveBeenCalled() fireEvent.click(screen.getByRole('button', { name: 'Index PDF bibliography' })) - expect(await screen.findByRole('alert')).toHaveTextContent('Configure GROBID first') + expect(await screen.findByRole('alert')).toHaveTextContent('PDF indexing unavailable') expect(papersApi.reindexPdfCitations).toHaveBeenCalledWith('p_a') expect(papersApi.reindexCitations).not.toHaveBeenCalled() }) it('shows PDF bibliography evidence with the original reference and file fingerprint', async () => { - papersApi.citations.mockResolvedValue(page({ items: [{ ...target, evidence: [{ provider: 'grobid', + papersApi.citations.mockResolvedValue(page({ items: [{ ...target, evidence: [{ provider: 'pymupdf', method: 'exact_identifier', indexedAt: '2026-11-03T00:00:00Z', reference: { unstructured: 'A. Writer, Target paper, 2024.', pdf_sha256: 'a'.repeat(64) }, }] }], total: 1 })) render(view()) await screen.findByText('Target paper') - expect(screen.getByText(/PDF bibliography \(GROBID\)/)).toBeInTheDocument() + expect(screen.getByText(/PDF bibliography \(local\)/)).toBeInTheDocument() expect(screen.getByText('A. Writer, Target paper, 2024.')).toBeInTheDocument() expect(screen.getByTitle('a'.repeat(64))).toBeInTheDocument() expect(screen.queryByRole('button', { name: 'Index PDF bibliography' })).not.toBeInTheDocument() From e69b2f815ad5eee9b161e4853305a570cca47ecb Mon Sep 17 00:00:00 2001 From: Prannaya Gupta Date: Wed, 4 Nov 2026 00:53:27 -0500 Subject: [PATCH 13/23] docs(citations): record local PDF pilot and revise rollout plan --- docs/api/papers.md | 2 +- docs/database/schema.md | 2 ++ docs/guides/citation-indexing.md | 20 ++++++++--------- docs/local-pdf-citation-pilot-report.md | 22 +++++++++++++++++++ .../2026-11-02-paper-citation-indexing.md | 19 +++++++++++++++- 5 files changed, 52 insertions(+), 13 deletions(-) create mode 100644 docs/local-pdf-citation-pilot-report.md diff --git a/docs/api/papers.md b/docs/api/papers.md index 247cd15..93c63e1 100644 --- a/docs/api/papers.md +++ b/docs/api/papers.md @@ -8,7 +8,7 @@ Requires migration `033_paper_citations.sql` and a separately running citation w | --- | --- | | `GET /api/papers/{id}/citations?direction=outgoing&limit=20&offset=0` | Paginated local references or `direction=incoming` for local citing papers; reads never trigger provider calls | | `POST /api/papers/{id}/citations/reindex` | HTTP 202 with OpenAlex/Crossref job summaries; pending/running work is reused | -| `POST /api/papers/{id}/citations/pdf/reindex` | Explicit opt-in for a managed PDF; requires migration 034 and `GROBID_URL`; HTTP 202 with its GROBID job summary | +| `POST /api/papers/{id}/citations/pdf/reindex` | Explicit opt-in for a managed PDF; defaults to local PyMuPDF with migrations 034 and 035; HTTP 202 with its job summary. Optional `?engine=grobid` requires `GROBID_URL` | | `GET /api/citation-jobs/{uuid}` | Current status of a durable job; does not expose lease tokens or captured provider payloads | The citation page contains `items`, `total`, `limit`, `offset`, `direction`, `runs`, `jobs`, `unresolvedCount`, `indexedPapers`, `libraryPapers`, and `pendingJobs`. Each item contains `id`, `title`, `authors`, `year`, and `evidence`; evidence includes `provider`, `method`, `indexedAt`, and the original provider `reference` object (its keys retain provider casing). Run summaries include `active`, `resolution`, `status`, `indexedAt`, candidate metadata, and messages. diff --git a/docs/database/schema.md b/docs/database/schema.md index 996ed74..b1dc6d9 100644 --- a/docs/database/schema.md +++ b/docs/database/schema.md @@ -562,3 +562,5 @@ RLS: disabled. - **Policies**: `pdfs public read`, `pdfs backend write`, `pdfs backend delete`, `pdfs backend update` (all granted to `public` role) PDF public URL pattern: `{SUPABASE_URL}/storage/v1/object/public/pdfs/{paper_id}.pdf` + +Migration `035_local_pdf_citations.sql` adds the opt-in `pymupdf` provider and `enqueue_local_pdf_citations` RPC without changing existing GROBID jobs. Source changes requeue each previously selected PDF engine independently. The local API default uses this RPC. Apply migrations in sequence; replaying 034 after 035 would restore older provider constraints. diff --git a/docs/guides/citation-indexing.md b/docs/guides/citation-indexing.md index 549ef2b..8f67716 100644 --- a/docs/guides/citation-indexing.md +++ b/docs/guides/citation-indexing.md @@ -10,30 +10,28 @@ Restart the application with this branch. Before migration, citation endpoints r ## Optional PDF bibliography extraction -Apply `034_pdf_citations.sql` after 033, and configure `GROBID_URL` with a self-hosted service base URL (for example, `http://localhost:8070`). See the [official GROBID service documentation](https://grobid.readthedocs.io/en/latest/Grobid-service/) for running the service. No GROBID service is currently configured for this workspace, so real extraction coverage has not been benchmarked. - -For a paper with a managed PDF, click **Index PDF bibliography**. This explicitly opts the paper into sending its stored PDF to the configured service. Existing metadata indexing does not enable PDF extraction automatically. The worker calls `processReferences` with `includeRawCitations=1` and `consolidateCitations=0`, disabling GROBID's external metadata consolidation. HTTP redirects are not followed with the PDF payload. - -For a batch, add `--include-pdf` to a new backfill command: +Apply `034_pdf_citations.sql` and then `035_local_pdf_citations.sql` after 033. The default extractor uses the existing PyMuPDF dependency locally; no GROBID service or new API key is required. Click **Index PDF bibliography** for a managed PDF, or add `--include-pdf` to a new backfill: ```powershell uv run --directory backend python scripts/index_paper_citations.py --library-id LIBRARY_ID --include-pdf --dry-run uv run --directory backend python scripts/index_paper_citations.py --library-id LIBRARY_ID --include-pdf ``` -The execute command requires `GROBID_URL` and queues only PDFs recognized as managed by the configured GCS/Supabase storage. The same worker and resume manifest handle these jobs. Credentials read private storage directly; arbitrary external PDF URLs must first be stored through the existing application import/upload path. The extraction size limit is 100 MiB. +PDF indexing remains opt-in. New batches record `pdf_engine: pymupdf`; old manifests without that field retain their original GROBID scope and service requirement. Managed GCS/Supabase credentials read private files directly. Arbitrary external PDF URLs must first be stored through the existing import/upload path. The size limit is 100 MiB; local extraction also rejects encrypted documents and documents over 1,000 pages. -Each reference retains its raw bibliography text, parsed title/authors/date, TEI record, parser version, and PDF SHA-256. Matching uses only explicit DOI/arXiv identifiers; title-only and contradictory identifiers remain unresolved. No reference output (including HTTP 204) is a partial extraction that preserves prior evidence, not proof that the PDF has no bibliography. Scanned PDFs may require OCR, which is not implemented. +The local extractor recognizes numbered bibliographies beneath References/Bibliography/Works Cited headings, orders one- or two-column text, and retains raw text, page numbers, parser version and PDF SHA-256. It uses only explicit DOI/arXiv identifiers. Title-only references and conflicting identifiers remain unresolved. Nonsequential numbering, unrecognized bibliographies and uncertain boundaries return partial results without publishing new identifiers or erasing previous successful evidence. Scans, unnumbered references, unusual layouts and line-wrapped identifiers remain coverage limitations. No OCR or fuzzy title linking is performed. Extraction success does not guarantee that every bibliography entry was found. -Once a service is available, evaluate one existing managed PDF without applying migration 034 or writing citation tables: +Run a read-only pilot without applying migrations or writing citation tables: ```powershell uv run --directory backend python scripts/pilot_pdf_citations.py --paper-id PAPER_ID --output data/citations/pdf-pilot.json ``` -This command sends the full PDF to the configured service and writes a local report containing raw reference evidence and exact local DOI/arXiv matches. It does not use title matching or provider alias enrichment and should be followed by bibliography review. No real GROBID pilot has been run yet. +This downloads the managed PDF, parses it locally, and saves raw evidence and exact same-library matches. Reports must use a new output path. The five-PDF local pilot retained 203 reference records: three successful extractions (119 records) and two partial results (84 records). It found **zero additional exact local links**. See `docs/local-pdf-citation-pilot-report.md`. Keep this feature opt-in; the evidence does not justify a library-wide automatic PDF backfill. + +GROBID remains an optional alternate parser. Configure `GROBID_URL` and explicitly use `POST /api/papers/{id}/citations/pdf/reindex?engine=grobid` or add `--engine grobid` to the pilot command to send a PDF to that service. The adapter disables external metadata consolidation and HTTP redirects. No real GROBID pilot has been run. Leaving `GROBID_URL` blank does not affect local extraction or OpenAlex/Crossref indexing. -Migration 034 versions every explicit PDF URL update, including uploads replacing the same storage key. Old evidence is invalidated and previously opted-in PDF jobs requeue; unselected papers stay opted out. The worker also re-reads the PDF after extraction to detect content replacement during processing. Existing provider jobs are requeued once when upgrading their fingerprint format; published evidence is retained until a successful refresh. Deploy 034 before enabling PDF indexing and avoid replaying 033 after 034. +Migration 034 versions every explicit PDF URL update, including replacing the same storage key. Migration 035 preserves independent opt-in for each PDF engine: source changes requeue only engines previously selected. Stale workers cannot publish, and extraction re-reads the file to detect replacement during processing. Apply migrations in order; do not replay 033/034 after 035. Start the worker from the repository root: @@ -99,7 +97,7 @@ npm install --prefix backend/data/citation-validation --no-audit --no-fund --ign node backend/tests/run_citation_sql.mjs ``` -The runner uses actual repository library/paper definitions, applies migrations 033 and 034 twice, and rolls back its fixtures. It verifies transaction behavior, including same-URL PDF replacement, not Supabase permissions or production multi-connection contention. Browser coverage lives in `frontend/e2e/citations.spec.js`. PDF parser/storage fixtures are in `backend/tests/test_pdf_references.py` and `backend/tests/test_pdf_storage.py`. +The runner uses actual repository library/paper definitions, applies migrations 033, 034 and 035 twice, and rolls back its fixtures. It verifies transaction behavior, including same-URL PDF replacement, not Supabase permissions or production multi-connection contention. Browser coverage lives in `frontend/e2e/citations.spec.js`. PDF parser/storage fixtures are in `backend/tests/test_local_pdf_references.py`, `backend/tests/test_pdf_references.py` and `backend/tests/test_pdf_storage.py`. ## Rollback and limitations diff --git a/docs/local-pdf-citation-pilot-report.md b/docs/local-pdf-citation-pilot-report.md new file mode 100644 index 0000000..1f05ceb --- /dev/null +++ b/docs/local-pdf-citation-pilot-report.md @@ -0,0 +1,22 @@ +# Local PDF citation pilot — 2026-11-04 + +The user approved a local PyMuPDF fallback because no GROBID service is available. This pilot read five managed PDFs and same-library paper metadata, parsed PDFs locally, and saved reports under ignored `backend/data/citations`. It made no database changes and sent no PDF to an external extraction service. + +Selection: MAGiC-SLAM (`p_0246e188`), the previously checked positive citation example, plus the first four other managed PDFs in the saved 30-paper provider sample. This is a small convenience sample, not a representative coverage estimate. Local reports ending `-2026-11-04-v2.json` supersede the initial exploratory reports. + +| Paper ID | Result | Retained references | References with accepted explicit identifiers | Exact same-library matches | +| --- | --- | ---: | ---: | ---: | +| p_0246e188 | Successful extraction | 47 | 3 | 0 | +| p_00b3abe6 | Successful extraction | 46 | 6 | 0 | +| p_1fd97cce | Successful extraction | 26 | 19 | 0 | +| p_23ef2062 | Partial: bibliography not recognized | 0 | 0 | 0 | +| p_3c3dad3b | Partial: uncertain boundary before biographies | 84 | 0 | 0 | +| **Total** | **3 successful, 2 partial** | **203** | **28** | **0** | + +The initial parser mistakenly included appendix text and author biographies in two final reference records. Boundary inspection exposed both problems. Font-size transitions now stop appendix headings, and unexplained large gaps stop extraction with a partial result. Column ordering, year lines mistaken for labels, page continuations, section boundaries and partial-result suppression have regression fixtures. The partial 84-record result retains text but emits no matching identifiers. + +MAGiC-SLAM's reference 15 names SplaTAM but contains no explicit DOI/arXiv identifier. Its text and page number are retained, and the local parser correctly leaves it unresolved. The earlier provider pilot supplies the evidence for that pair. This local pilot does not establish new-edge precision or overall recall: there were no newly detected edges to verify, and bibliography review was focused on boundaries and the known positive rather than exhaustive transcription of every reference. + +The implemented parser is deliberately limited to numbered bibliographies with recognized headings. Scanned PDFs, unnumbered entries, unusual layouts, clipped margins and line-wrapped identifiers can be missed. Success means the supported layout was recognized, not a guarantee of complete bibliography coverage. Matching uses local DOI/arXiv identities only, without provider alias enrichment or title matching. + +Decision: retain local extraction as an **opt-in aid**, but do not automatically backfill every PDF. It adds inspectable bibliography records without requiring GROBID, while this sample demonstrated no additional exact local links. OpenAlex/Crossref remain the primary indexing path. A reviewed title-resolution workflow is a separate future step; GROBID and OCR remain deferred. diff --git a/docs/superpowers/plans/2026-11-02-paper-citation-indexing.md b/docs/superpowers/plans/2026-11-02-paper-citation-indexing.md index 2864668..47e1c0b 100644 --- a/docs/superpowers/plans/2026-11-02-paper-citation-indexing.md +++ b/docs/superpowers/plans/2026-11-02-paper-citation-indexing.md @@ -153,7 +153,7 @@ Files: add operator instructions in `docs/guides/citation-indexing.md`; create ` - [x] Record identity-resolution rate, provider reference availability, local edges, ambiguities, failures, request volume, and runtime separately. Manually compare up to 20 detected edges with bibliographies and check every selected known positive pair. Document denominators and unavailable PDFs. - [x] Treat every detected false positive as a release blocker until the matching rule is fixed and checked again. Report recall limits explicitly; do not promise complete coverage from this sample. - [x] Prepare migration/backfill commands, rollback instructions, and the pilot report. Keep worker failure messages free of credentials and full PDF contents. -- [ ] Apply migration 033 (and 034 for PDF extraction) to the intended live database, validate permissions and concurrent workers, then start the worker and expand the persisted backfill with resumable batches. +- [ ] Apply migration 033 (and 034/035 for PDF extraction) to the intended live database, validate permissions and concurrent workers, then start the worker and expand the persisted backfill with resumable batches. - [x] Document rollback by disabling enqueueing, stopping workers, and hiding the feature while retaining additive tables/evidence. Avoid destructive rollback of indexed data. Milestone 1 exit: supported citation pairs appear in both directions with inspectable evidence; new imports reconcile; retries and restarts preserve data; coverage limitations are measured and visible. @@ -199,3 +199,20 @@ git diff --check ``` Suggested commit sequence: contracts/schema; provider identity and OpenAlex indexing; durable jobs/reconciliation; citation API; citation UI; pilot/docs; Crossref capture; justified fallback. Keep every implementation checkbox open until its acceptance criteria have been verified. + + +## Revised fallback decision ? 2026-11-04 + +The user cannot run GROBID locally and approved proceeding with an existing-dependency fallback. OpenAlex/Crossref remain the primary path; GROBID is optional and disabled unless explicitly configured and selected. + +- [x] Add a local PyMuPDF parser for numbered bibliographies; preserve raw text, pages, hash and parser version. +- [x] Match only explicit DOI/arXiv identifiers; retain title-only entries unresolved. Return partial for unsupported layouts and uncertain boundaries. +- [x] Add migration 035, independent local PDF opt-in, worker dispatch, default local API/UI action and local CLI pilot. Preserve GROBID jobs and older batch scopes. +- [x] Run a five-PDF read-only pilot from the previously approved sample plus the known MAGiC-SLAM positive. Review bibliography boundaries and fix appendix/biography contamination before acceptance. +- [x] Record measured results in `docs/local-pdf-citation-pilot-report.md`: three successful and two partial extractions; 203 retained records; zero additional exact local links. +- [ ] Apply migrations to the intended live database and start a bounded provider indexing rollout. No live migration or backfill has been performed in this development session. +- [ ] Consider a separate reviewed title-matching workflow if increased local citation coverage is needed. Do not turn title similarity into automatic citation evidence. + +Decision: ship the local fallback as an opt-in aid, with a no-go for automatic library-wide PDF extraction based on this pilot. Defer GROBID/OCR and keep the provider milestone independent of local parser coverage. + +Validation for this revision: 372 backend tests and 403 frontend tests passed; migrations 033/034/035 each applied twice in disposable PGlite with lifecycle fixtures; citation browser workflow and production build passed. The build retains the existing large-bundle warning. Live Supabase permissions and concurrent deployment behavior remain rollout checks. From 4d9e04676181ed8b8a34f6021f3d7eb600074b72 Mon Sep 17 00:00:00 2001 From: Prannaya Gupta Date: Wed, 4 Nov 2026 10:09:23 -0500 Subject: [PATCH 14/23] fix(ui): remove citation implementation notes --- frontend/src/components/PaperCitations.jsx | 21 ++++++++----------- .../src/components/PaperCitations.test.jsx | 12 +++++------ 2 files changed, 15 insertions(+), 18 deletions(-) diff --git a/frontend/src/components/PaperCitations.jsx b/frontend/src/components/PaperCitations.jsx index 08cfb60..fad13eb 100644 --- a/frontend/src/components/PaperCitations.jsx +++ b/frontend/src/components/PaperCitations.jsx @@ -36,8 +36,8 @@ function CitationLists({ paperId, hasStoredPdf }) { if (page.pendingJobs > 0 || page.jobs.some(job => ['pending', 'running'].includes(job.status))) { timer = setTimeout(load, 10000) } - } catch (err) { - if (!cancelled) { setError(err.message); setLoading(false) } + } catch { + if (!cancelled) { setError('Citation data could not be loaded.'); setLoading(false) } } } load() @@ -51,8 +51,8 @@ function CitationLists({ paperId, hasStoredPdf }) { if (pdf) await papersApi.reindexPdfCitations(paperId) else await papersApi.reindexCitations(paperId) if (alive.current) setRefresh(value => value + 1) - } catch (err) { - if (alive.current) setError(err.message) + } catch { + if (alive.current) setError('Could not start indexing. Please try again.') } finally { if (alive.current) setQueueing(false) } @@ -83,8 +83,8 @@ function CitationLists({ paperId, hasStoredPdf }) { {loading &&

Loading citations…

} {data && <> {busy &&

Indexing queued or in progress.

} - {failed &&

Indexing failed. {activeRuns.length > 0 && 'Showing previously indexed evidence.'}

} - {partial &&

Some references or paper identities are unavailable. Coverage is incomplete.

} + {failed &&

Indexing failed. {activeRuns.length > 0 && 'Showing previous results.'}

} + {partial &&

Some references could not be found.

} {!activeRuns.length && !busy && !failed && !partial &&

Not indexed yet.

} {lastIndexed &&

Last indexed: {new Date(lastIndexed).toLocaleString()}

} {!data.items.length &&

No indexed citations found.

} @@ -95,14 +95,12 @@ function CitationLists({ paperId, hasStoredPdf }) {
Citation evidence {item.evidence.map((evidence, index) =>
-

{{ openalex: 'OpenAlex', crossref: 'Crossref', grobid: 'PDF bibliography (GROBID)', pymupdf: 'PDF bibliography (local)' }[evidence.provider] || evidence.provider} · identifier match · {new Date(evidence.indexedAt).toLocaleDateString()}

+

{{ openalex: 'OpenAlex', crossref: 'Crossref', grobid: 'PDF', pymupdf: 'PDF' }[evidence.provider] || 'Source'} · {new Date(evidence.indexedAt).toLocaleDateString()}

{evidence.provider === 'openalex' && /^W\d+$/.test(evidence.reference.source_work) && Source record} {evidence.reference.unstructured &&

{evidence.reference.unstructured}

} {evidence.reference.DOI &&

DOI: {evidence.reference.DOI}

} - {evidence.reference.target_work &&

Referenced work: {evidence.reference.target_work}

} {['grobid', 'pymupdf'].includes(evidence.provider) && evidence.reference.title &&

{evidence.reference.title}

} - {['grobid', 'pymupdf'].includes(evidence.provider) && evidence.reference.pdf_sha256 &&

PDF fingerprint: {evidence.reference.pdf_sha256.slice(0, 12)}…

}
)}
)} @@ -112,15 +110,14 @@ function CitationLists({ paperId, hasStoredPdf }) { {offset + 1}–{Math.min(offset + 10, data.total)} of {data.total} } -

{data.indexedPapers} of {data.libraryPapers} library papers have indexed reference data.

- {data.unresolvedCount > 0 &&

{data.unresolvedCount} reference records not matched to a unique local citation.

} +

{data.indexedPapers} of {data.libraryPapers} papers indexed.

+ {data.unresolvedCount > 0 &&

{data.unresolvedCount} references not linked to a paper in this library.

} } {hasStoredPdf &&
-

Reads numbered bibliographies locally. Matches explicit DOI/arXiv identifiers; no GROBID setup needed.

} ) diff --git a/frontend/src/components/PaperCitations.test.jsx b/frontend/src/components/PaperCitations.test.jsx index 9b28176..9c50187 100644 --- a/frontend/src/components/PaperCitations.test.jsx +++ b/frontend/src/components/PaperCitations.test.jsx @@ -32,7 +32,7 @@ it('distinguishes unindexed, failed refresh, and previous evidence', async () => const result = render(view()) await screen.findByText('Not indexed yet.') result.rerender(view('p_other')) - await screen.findByText('Indexing failed. Showing previously indexed evidence.') + await screen.findByText('Indexing failed. Showing previous results.') expect(screen.getByRole('link', { name: 'Target paper' })).toBeInTheDocument() expect(screen.queryByText('Not indexed yet.')).not.toBeInTheDocument() }) @@ -40,7 +40,7 @@ it('distinguishes unindexed, failed refresh, and previous evidence', async () => it('retries read failures without automatically indexing', async () => { papersApi.citations.mockRejectedValueOnce(new Error('Unavailable')).mockResolvedValue(page()) render(view()) - expect(await screen.findByRole('alert')).toHaveTextContent('Unavailable') + expect(await screen.findByRole('alert')).toHaveTextContent('Citation data could not be loaded.') fireEvent.click(screen.getByRole('button', { name: 'Retry' })) await screen.findByText('No indexed citations found.') expect(papersApi.reindexCitations).not.toHaveBeenCalled() @@ -107,20 +107,20 @@ it('PDF extraction is an explicit action and reports queue failures', async () = await screen.findByText('Not indexed yet.') expect(papersApi.reindexPdfCitations).not.toHaveBeenCalled() fireEvent.click(screen.getByRole('button', { name: 'Index PDF bibliography' })) - expect(await screen.findByRole('alert')).toHaveTextContent('PDF indexing unavailable') + expect(await screen.findByRole('alert')).toHaveTextContent('Could not start indexing. Please try again.') expect(papersApi.reindexPdfCitations).toHaveBeenCalledWith('p_a') expect(papersApi.reindexCitations).not.toHaveBeenCalled() }) -it('shows PDF bibliography evidence with the original reference and file fingerprint', async () => { +it('shows the PDF source and original reference without implementation details', async () => { papersApi.citations.mockResolvedValue(page({ items: [{ ...target, evidence: [{ provider: 'pymupdf', method: 'exact_identifier', indexedAt: '2026-11-03T00:00:00Z', reference: { unstructured: 'A. Writer, Target paper, 2024.', pdf_sha256: 'a'.repeat(64) }, }] }], total: 1 })) render(view()) await screen.findByText('Target paper') - expect(screen.getByText(/PDF bibliography \(local\)/)).toBeInTheDocument() + expect(screen.getByText(/^PDF\b/)).toBeInTheDocument() expect(screen.getByText('A. Writer, Target paper, 2024.')).toBeInTheDocument() - expect(screen.getByTitle('a'.repeat(64))).toBeInTheDocument() + expect(screen.queryByTitle('a'.repeat(64))).not.toBeInTheDocument() expect(screen.queryByRole('button', { name: 'Index PDF bibliography' })).not.toBeInTheDocument() }) From e779930a304fdca8b7496c0bc2b4afa30273fa08 Mon Sep 17 00:00:00 2001 From: Prannaya Gupta Date: Wed, 4 Nov 2026 10:16:35 -0500 Subject: [PATCH 15/23] fix(citations): distinguish retrieved references from library links --- frontend/e2e/citations.spec.js | 2 +- frontend/src/components/PaperCitations.jsx | 12 +++++++++--- frontend/src/components/PaperCitations.test.jsx | 16 +++++++++++++++- 3 files changed, 25 insertions(+), 5 deletions(-) diff --git a/frontend/e2e/citations.spec.js b/frontend/e2e/citations.spec.js index 485ab3f..32efd30 100644 --- a/frontend/e2e/citations.spec.js +++ b/frontend/e2e/citations.spec.js @@ -31,7 +31,7 @@ test('paper citations show evidence, direction, indexing state, and target navig await citations.getByText('Citation evidence').click() await expect(citations.getByRole('link', { name: 'Source record' })).toHaveAttribute('href', 'https://openalex.org/W1') await citations.getByRole('button', { name: 'Cited by', exact: true }).click() - await expect(citations.getByText('No indexed citations found.')).toBeVisible() + await expect(citations.getByText('No papers in this library are linked as citing this paper yet.')).toBeVisible() await citations.getByRole('button', { name: 'References', exact: true }).click() await citations.getByRole('button', { name: 'Index this paper' }).click() await expect(citations.getByText('Indexing queued or in progress.')).toBeVisible() diff --git a/frontend/src/components/PaperCitations.jsx b/frontend/src/components/PaperCitations.jsx index fad13eb..82c78ea 100644 --- a/frontend/src/components/PaperCitations.jsx +++ b/frontend/src/components/PaperCitations.jsx @@ -62,9 +62,15 @@ function CitationLists({ paperId, hasStoredPdf }) { const busy = queueing || jobs.some(job => ['pending', 'running'].includes(job.status)) const activeRuns = (data?.runs || []).filter(run => run.active) const failed = jobs.some(job => job.status === 'failed') - const partial = jobs.some(job => ['partial', 'not_found'].includes(job.status)) + const partial = jobs.some(job => ['partial', 'not_found'].includes(job.status) + && !(job.provider === 'crossref' && job.error === 'No DOI available')) const lastIndexed = activeRuns.map(run => run.indexedAt).sort().at(-1) const label = direction === 'outgoing' ? 'References in this library' : 'Cited by in this library' + const emptyMessage = direction === 'incoming' + ? 'No papers in this library are linked as citing this paper yet.' + : data?.unresolvedCount > 0 + ? 'References were found, but none are linked to papers in this library yet.' + : 'No linked references in this library yet.' return (
@@ -87,7 +93,7 @@ function CitationLists({ paperId, hasStoredPdf }) { {partial &&

Some references could not be found.

} {!activeRuns.length && !busy && !failed && !partial &&

Not indexed yet.

} {lastIndexed &&

Last indexed: {new Date(lastIndexed).toLocaleString()}

} - {!data.items.length &&

No indexed citations found.

} + {!data.items.length &&

{emptyMessage}

}
    {data.items.map(item =>
  • {item.title} @@ -111,7 +117,7 @@ function CitationLists({ paperId, hasStoredPdf }) { }

    {data.indexedPapers} of {data.libraryPapers} papers indexed.

    - {data.unresolvedCount > 0 &&

    {data.unresolvedCount} references not linked to a paper in this library.

    } + {direction === 'outgoing' && data.unresolvedCount > 0 &&

    {data.unresolvedCount} references not linked to a paper in this library.

    } } } -

    {data.indexedPapers} of {data.libraryPapers} papers indexed.

    {direction === 'outgoing' && data.unresolvedCount > 0 &&

    {data.unresolvedCount} references not linked to a paper in this library.

    } } - {hasStoredPdf &&
    - -
    } + {lastIndexed &&

    Last indexed: {new Date(lastIndexed).toLocaleString()}

    }
) } diff --git a/frontend/src/components/PaperCitations.test.jsx b/frontend/src/components/PaperCitations.test.jsx index b01f013..8f80a11 100644 --- a/frontend/src/components/PaperCitations.test.jsx +++ b/frontend/src/components/PaperCitations.test.jsx @@ -107,23 +107,23 @@ it('queues explicitly and preserves visible links while awaiting the worker', as papersApi.reindexCitations.mockResolvedValue([{ id: 'job1', status: 'pending' }]) render(view()) await screen.findByText('Target paper') - fireEvent.click(screen.getByRole('button', { name: 'Index this paper' })) + fireEvent.click(screen.getByRole('button', { name: 'Index', exact: true })) await screen.findByText('Indexing queued or in progress.') expect(papersApi.reindexCitations).toHaveBeenCalledWith('p_a') expect(screen.getByRole('button', { name: 'Indexing…' })).toBeDisabled() expect(screen.getByRole('link', { name: 'Target paper' })).toBeInTheDocument() }) -it('PDF extraction is an explicit action and reports queue failures', async () => { +it('the single index action starts providers and reports queue failures', async () => { papersApi.citations.mockResolvedValue(page()) - papersApi.reindexPdfCitations.mockRejectedValue(new Error('PDF indexing unavailable')) - render() + papersApi.reindexCitations.mockRejectedValue(new Error('Indexing unavailable')) + render() await screen.findByText('Not indexed yet.') expect(papersApi.reindexPdfCitations).not.toHaveBeenCalled() - fireEvent.click(screen.getByRole('button', { name: 'Index PDF bibliography' })) + fireEvent.click(screen.getByRole('button', { name: 'Index', exact: true })) expect(await screen.findByRole('alert')).toHaveTextContent('Could not start indexing. Please try again.') - expect(papersApi.reindexPdfCitations).toHaveBeenCalledWith('p_a') - expect(papersApi.reindexCitations).not.toHaveBeenCalled() + expect(papersApi.reindexCitations).toHaveBeenCalledWith('p_a') + expect(papersApi.reindexPdfCitations).not.toHaveBeenCalled() }) it('shows the PDF source and original reference without implementation details', async () => { diff --git a/frontend/src/pages/Paper.jsx b/frontend/src/pages/Paper.jsx index 72213b8..fd043fb 100644 --- a/frontend/src/pages/Paper.jsx +++ b/frontend/src/pages/Paper.jsx @@ -524,7 +524,6 @@ export default function Paper() { onPaperUpdate={setPaper} /> )} From 5d8911f5e783f46a38baf8895e8e33ed09c1a455 Mon Sep 17 00:00:00 2001 From: Prannaya Gupta Date: Wed, 4 Nov 2026 11:39:25 -0500 Subject: [PATCH 20/23] style(citations): align Index with header and lab button styling --- frontend/src/components/PaperCitations.jsx | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/frontend/src/components/PaperCitations.jsx b/frontend/src/components/PaperCitations.jsx index 14d1b55..73f13c0 100644 --- a/frontend/src/components/PaperCitations.jsx +++ b/frontend/src/components/PaperCitations.jsx @@ -73,7 +73,13 @@ function CitationLists({ paperId }) { return (
-

Citations

+
+

Citations

+ +
{['outgoing', 'incoming'].map(value => ( {lastIndexed &&

Last indexed: {new Date(lastIndexed).toLocaleString()}

}
) From 9b337ee67beeb2c5e2b80c398883cfb899818368 Mon Sep 17 00:00:00 2001 From: Prannaya Gupta Date: Wed, 4 Nov 2026 11:40:07 -0500 Subject: [PATCH 21/23] style(citations): reduce note and paper title text size --- frontend/src/components/PaperCitations.jsx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/frontend/src/components/PaperCitations.jsx b/frontend/src/components/PaperCitations.jsx index 73f13c0..510678a 100644 --- a/frontend/src/components/PaperCitations.jsx +++ b/frontend/src/components/PaperCitations.jsx @@ -72,9 +72,9 @@ function CitationLists({ paperId }) { : 'No linked references in this library yet.' return ( -
+
-

Citations

+

Citations

} - {direction === 'outgoing' && data.unresolvedCount > 0 &&

{data.unresolvedCount} references not linked to a paper in this library.

} + {direction === 'outgoing' && data.unresolvedCount > 0 &&

{data.unresolvedCount} references not linked to a paper in this library.

} } {lastIndexed &&

Last indexed: {new Date(lastIndexed).toLocaleString()}

}
diff --git a/frontend/src/components/PaperCitations.test.jsx b/frontend/src/components/PaperCitations.test.jsx index 8f80a11..56d4036 100644 --- a/frontend/src/components/PaperCitations.test.jsx +++ b/frontend/src/components/PaperCitations.test.jsx @@ -13,12 +13,11 @@ const target = { id: 'p_b', title: 'Target paper', authors: ['A Writer'], year: reference: { source_work: 'W1', target_work: 'W2' } }] } function view(paperId = 'p_a') { return } -it('shows direction, evidence, and local navigation', async () => { +it('shows direction and local navigation', async () => { papersApi.citations.mockResolvedValue(page({ items: [target], total: 1 })) render(view()) expect(await screen.findByRole('link', { name: 'Target paper' })).toHaveAttribute('href', '/library/paper/p_b') expect(screen.getByRole('heading', { name: 'References in this library (1)' })).toBeInTheDocument() - expect(screen.getByRole('link', { name: 'Source record' })).toHaveAttribute('href', 'https://openalex.org/W1') fireEvent.click(screen.getByRole('button', { name: 'Cited by' })) await screen.findByRole('heading', { name: 'Cited by in this library (1)' }) expect(papersApi.citations).toHaveBeenLastCalledWith('p_a', { direction: 'incoming', offset: 0, limit: 10 }) @@ -126,15 +125,15 @@ it('the single index action starts providers and reports queue failures', async expect(papersApi.reindexPdfCitations).not.toHaveBeenCalled() }) -it('shows the PDF source and original reference without implementation details', async () => { +it('shows PDF-linked papers without an evidence dropdown', async () => { papersApi.citations.mockResolvedValue(page({ items: [{ ...target, evidence: [{ provider: 'pymupdf', method: 'exact_identifier', indexedAt: '2026-11-03T00:00:00Z', reference: { unstructured: 'A. Writer, Target paper, 2024.', pdf_sha256: 'a'.repeat(64) }, }] }], total: 1 })) render(view()) await screen.findByText('Target paper') - expect(screen.getByText(/^PDF\b/)).toBeInTheDocument() - expect(screen.getByText('A. Writer, Target paper, 2024.')).toBeInTheDocument() + expect(screen.queryByText('Citation evidence')).not.toBeInTheDocument() + expect(screen.queryByText('A. Writer, Target paper, 2024.')).not.toBeInTheDocument() expect(screen.queryByTitle('a'.repeat(64))).not.toBeInTheDocument() expect(screen.queryByRole('button', { name: 'Index PDF bibliography' })).not.toBeInTheDocument() }) From ff388023c3f02ea52675ef2c7bc68faa725dc037 Mon Sep 17 00:00:00 2001 From: Prannaya Gupta Date: Wed, 4 Nov 2026 11:43:23 -0500 Subject: [PATCH 23/23] style(citations): match detail headings and remove divider --- frontend/src/components/PaperCitations.jsx | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/frontend/src/components/PaperCitations.jsx b/frontend/src/components/PaperCitations.jsx index 516c748..4085e52 100644 --- a/frontend/src/components/PaperCitations.jsx +++ b/frontend/src/components/PaperCitations.jsx @@ -72,9 +72,9 @@ function CitationLists({ paperId }) { : 'No linked references in this library yet.' return ( -
+
-

Citations

+

Citations