Source definitions for Lightnovel Crawler.
A source tells the crawler how to read one website: where the title is, how to find the
chapter list, which element holds the chapter text. Here a source is data, not code. One
YAML file per host, validated against a published schema, interpreted at runtime by
sourcelib (on PyPI as lncrawl-sourcelib).
That means a broken site can be fixed by editing a few selectors, in an editor with autocomplete, without writing Python and without waiting for an app release.
| Path | What lives there |
|---|---|
specs/<host>.yaml |
One live source per host. The filename is the host. |
disabled/<host>.yaml |
Hosts that are down, blocking, or were never implemented. Each carries a disabled: reason. |
base/<name>.yaml |
Shared definitions other specs extend. No base_url, so never registered as a source. |
hooks/ |
The escape hatch, for the rare site that cannot be described as data. sites/ per host, shared/ across hosts, lib/ for helpers. |
fixtures/<host>/ |
Recorded pages, so a spec can be tested offline. |
schema/ |
The JSON Schema, generated from the model. Editors read this. |
docs/ |
The normative format definition. |
Finding a source is always the same rule: specs/ plus the host, lowercased, without
www. or a scheme.
Nothing generated is committed, with one exception. The manifest that installed apps poll is
published rather than stored here, but schema/source.v1.json is committed because editors read
it from the working tree: without it in-tree, autocomplete needs a network round trip, and
autocomplete is most of what makes editing a spec possible without knowing the format.
It is generated by sourcelib, so it is treated as a vendored artifact from a pinned
dependency, the way a lockfile is. The file records the interpreter version that produced it in
x-generator, and CI installs exactly that version to regenerate and compare. A format change
therefore arrives as a visible bump to this file rather than as an unexplained difference in an
unrelated pull request.
pip install lncrawl-sourcelib # or use `lncrawl dev …` if you already have the crawler
sourcelib explain https://example.com/novel/some-book # what the page looks like
$EDITOR specs/example.com.yaml
sourcelib try specs/example.com.yaml https://example.com/novel/some-bookexplain prints a short structural summary of a page, including candidate selectors and how
many elements each one matches, which is usually enough to write the spec. try runs it and
reports what each field produced, or which field matched nothing and where.
Open a pull request with one host per PR. See CONTRIBUTING.md.
RFC-0001 is the normative definition: every field, the evaluation rules, the transform steps, and the hook contract. It lives with the interpreter, because a grammar version covers the model, the step registry and the hook points together and all three are implemented there.
Read it when the answer has to be exact. For getting a source written, docs/ is shorter and worked through examples.
A minimal source is two meaningful lines, because it inherits everything else:
spec: 1
extends: base/wordpress.yaml
base_url: https://example.com/Apache-2.0, matching the scraper package. See NOTICE for why the definitions here
are independent of the crawler's own GPL-3.0-or-later code.