Parse PDFs into searchable LiteParse sidecar notes

[phernandez]

Summary

Add a focused implementation/spike for parsing PDFs into searchable markdown sidecar notes using LiteParse.

Related umbrella issues: #542 and #543. This issue narrows the scope to a local, LiteParse-backed PDF -> markdown sidecar flow rather than the broader documents/images/OCR/vision feature set.

Core model:

• Store the original PDF as a file entity (content_type: application/pdf) with checksum, size, mtime, and stable read_content access.
• Do not body-index the PDF itself for search. If needed, add an explicit search-index opt-out for the PDF file row.
• Parse the PDF into markdown with LiteParse.
• Store the parsed markdown in a sidecar note that references the source PDF and indexes as a normal note/search document.

Example layout:

docs/report.pdf
docs/report.pdf.md          # or docs/report.extract.md

Why LiteParse

LiteParse looks like a good fit for a first local implementation:

• Open source and local-first: Apache-2.0, no API keys, no cloud dependency, no LLM dependency.
• Python package is compatible with BM's Python floor: PyPI lists Requires: Python >=3.10; BM requires Python >=3.12.
• As of the 2026-06-19 investigation, PyPI listed liteparse 2.1.1, released 2026-06-18.
• Supports Python, Node/TypeScript, Rust, and WASM.
• Can emit markdown, text, JSON with bounding boxes, and screenshots.
• Markdown mode reconstructs headings, tables, lists, images, and links.
• Quality is heuristic/rule-based, so dense tables, scans, charts, and complex layouts may need a stronger parser later.

Relevant docs:

• https://www.llamaindex.ai/blog/markdown-comes-to-liteparse?utm_medium=socials&utm_source=twitter&utm_campaign=2026-jun-
• https://developers.llamaindex.ai/liteparse/
• https://developers.llamaindex.ai/liteparse/guides/markdown/
• https://developers.llamaindex.ai/liteparse/guides/library-usage/
• https://pypi.org/project/liteparse/

LiteParse examples from the docs:

lit parse document.pdf --format markdown -o output.md

from liteparse import LiteParse

parser = LiteParse(
    output_format="markdown",
    image_mode="off",          # or "placeholder" / "embed"
    extract_links=True,
)
result = parser.parse("document.pdf")
markdown = result.text

Local smoke test in this repo succeeded:

uv run --with liteparse python - <<'PY'
from pathlib import Path
from liteparse import LiteParse

pdf_path = Path('tests/Non-MarkdownFileSupport.pdf')
parser = LiteParse(output_format='markdown', image_mode='off', extract_links=True, quiet=True)
result = parser.parse(str(pdf_path))
print('text_length', len(result.text))
print('pages', len(result.pages))
print(result.text[:1000])
PY

Result:

text_length 2728
pages 3

Current BM fit

The repo already has most of the underlying shape:

• Entity supports any file via content_type, file_path, checksum, mtime, and size. Markdown-only permalink uniqueness is already scoped to content_type = 'text/markdown'.
• Regular files are already represented as note_type="file" in the batch indexer path.
• Search indexing already treats markdown and non-markdown differently: markdown gets content/observations/relations; regular files get a lightweight entity row with no body content.
• read_content already returns PDFs as base64 document resources and has a PDF test fixture/path.
• Link resolution can resolve exact titles and file paths, so a sidecar can intentionally reference the PDF file entity by path, e.g. [[docs/report.pdf]].

Important implementation wrinkle:

• The batch indexer can handle regular files, but the single-file observed-object path is still markdown-oriented (index_markdown_file). PDF support should generalize that interface or route PDF sidecar generation through a separate command/service.

Proposed sidecar metadata

The sidecar note should make provenance and idempotency explicit:

---
title: Report
type: pdf_extract
source_path: docs/report.pdf
source_entity_id: <pdf entity external_id>
source_checksum: <pdf checksum>
source_content_type: application/pdf
parser: liteparse
parser_version: 2.1.1
parsed_at: 2026-06-19T00:00:00Z
parser_options:
  output_format: markdown
  image_mode: off
  extract_links: true
  ocr_enabled: true
---

The parsed body should remain editable markdown, but the generated/source metadata gives us a way to skip reparsing when the PDF checksum and parser options are unchanged.

Search-only sidecar mode

LiteParse returns ordinary markdown. BM currently parses markdown body content for semantic observations and relations, including:

• - [category] content
• - relation [[Target]]
• inline [[Target]]

A raw PDF extract can easily contain those shapes accidentally. The sidecar should not mint arbitrary graph observations/relations from extracted text by default.

Recommended approach:

• Add a search_only / bm_parse_semantics: false / similar frontmatter flag for generated extraction sidecars.
• Index the body text for FTS/semantic search.
• Skip observation/relation extraction for that note.
• Add only controlled provenance relations, such as extracted_from [[docs/report.pdf]], through explicit sidecar metadata or a safe generated section.

Suggested UX

Start with an explicit CLI command rather than automatic parse-on-sync:

bm import pdf docs/report.pdf \
  --sidecar docs/report.pdf.md \
  --image-mode off \
  --ocr \
  --target-pages "1-10"

Possible defaults:

• sidecar path: <source>.md, e.g. report.pdf.md
• image_mode=off or placeholder for the first cut
• extract_links=true
• parse only on command, not during normal sync/watch
• skip reparse if source checksum, LiteParse version, and parser options match sidecar metadata

Later phases can add directory import, watcher support, OCR tuning, image extraction, and agent-generated summaries/observations.

Dependency strategy

Because LiteParse is new and marked beta on PyPI, prefer a lazy/optional dependency first:

• Do not import LiteParse at CLI startup.
• Add an optional extra such as basic-memory[pdf], or provide an actionable install hint when the command is used without LiteParse installed.
• Keep parser calls behind a small adapter/service so tests can inject a fake parser.
• If we decide to make it a default dependency later, update pyproject.toml, uv.lock, docs, and package checks accordingly.

Acceptance criteria

• A PDF can be indexed as a file entity and remains accessible via read_content.
• A CLI command can parse a PDF with LiteParse and create/update a markdown sidecar note.
• The sidecar stores source path, source entity external_id, source checksum, parser name/version, parsed timestamp, and parser options.
• The sidecar is searchable through existing text/hybrid/semantic search paths.
• The original PDF content is not body-indexed; search hits should come from the sidecar, not raw PDF bytes.
• Generated sidecars do not accidentally create observations/relations from arbitrary extracted markdown syntax.
• The implementation is idempotent: unchanged PDF checksum + unchanged parser options should skip reparse unless forced.
• Missing LiteParse dependency yields an actionable error.
• Tests cover:
  • parser adapter with injected fake parser
  • fixture PDF -> sidecar metadata + content
  • source PDF entity is readable through read_content
  • sidecar links back to the file entity
  • unchanged checksum/options skip reparse
  • search-only mode prevents accidental observation/relation extraction from extracted - [foo] and [[bar]] text

Open questions

• Should the sidecar path convention be <source>.md, <stem>.extract.md, or configurable only?
• Should the PDF file row still be discoverable by filename/title search, or should it be completely absent from search results?
• Should generated sidecars sync to cloud while binary sources remain local-only?
• Should cloud ever parse PDFs, or should this remain local-only unless a separate cloud document-processing product path exists?
• How should manual edits to generated sidecars interact with forced reparse?
• What max-pages / file-size guardrails should the CLI enforce by default?

[claude]

Claude finished @phernandez's task in 1m 34s —— View job

---

Issue Triage

• Read and analyze the issue
• Explore codebase for context (importers, related files)
• Check for duplicate issues
• Apply labels (enhancement label already present; component/priority/complexity labels not yet created in repo)
• Post triage summary

---

Triage Summary

Type: Feature — new functionality (PDF → markdown sidecar via LiteParse)

Priority: Medium — valuable capability enhancement, no breakage or data loss involved

Component: CLI (primary UX), Database/Sync (search-only flag, entity indexing), Importer (new PdfImporter class)

Complexity: Complex — spans multiple subsystems with new patterns

---

Analysis

This is a well-scoped, narrowed sub-issue of the umbrella #542 / #543 ("Add support for documents, PDFs, and images"). Rather than duplicating those, it deliberately constrains scope to a local, LiteParse-backed PDF → markdown sidecar flow and provides a thorough spec, which makes it immediately actionable.

Not a duplicate. Issues #542 and #543 are broader; #543 is itself marked duplicate of #542. This issue is the focused first implementation step.

Key design decisions already made in the issue:

• bm import pdf <file> CLI command (not watcher/auto-sync)
• LiteParse as optional dep (basic-memory[pdf] extra), with actionable error on missing
• Sidecar stores provenance frontmatter; idempotent on checksum + parser options
• search_only frontmatter flag to prevent FTS-extracted text from minting accidental observations/relations
• Only controlled extracted_from [[docs/report.pdf]] relation is added

What exists in the codebase today (good news):

• Importer base class in src/basic_memory/importers/base.py — new PdfImporter slots in cleanly
• Entity already supports content_type, checksum, mtime, size for non-markdown files
• read_content already handles PDFs as base64 document resources (and has a test fixture)
• Search already treats markdown vs non-markdown files differently

What needs to be built:

1. PdfImporter class — wraps LiteParse adapter, writes sidecar .md with provenance frontmatter
2. LiteParse adapter/service (thin wrapper; injected fake in tests)
3. search_only frontmatter flag in markdown parser — skip observation/relation extraction when set
4. bm import pdf CLI subcommand with --sidecar, --image-mode, --ocr, --target-pages, --force
5. Idempotency check (compare source_checksum + parser_options in existing sidecar)
6. basic-memory[pdf] optional extra in pyproject.toml
7. Tests covering all acceptance criteria from the issue

Open questions worth resolving before implementation:

• Sidecar naming: <source>.md vs <stem>.extract.md — recommend defaulting to <source>.md (e.g. report.pdf.md) with --sidecar override
• Whether manual edits to sidecars survive forced reparse (recommend: preserve on reparse unless --overwrite passed)
• Max file-size default (reasonable starting point: 50 MB)

Suggested labels to add (once created in the repo):

• feature, medium, cli, complex

[phernandez]

also consider https://docs.kreuzberg.dev for parsing different doc types. Compare to LiteParse.