Skip to content

validation.md

Latest commit

 

History

History
396 lines (364 loc) · 24 KB

validation.md

File metadata and controls

396 lines (364 loc) · 24 KB
Raw

Validation

Solid Memo checks that the data it reads from a Pod, and every change it writes, has the form the app expects. This page says what is checked and when, what happens to data that does not fit, and how it can be repaired.

Solid Memo checks the data it works on against its shapes and the DCAT-AP profile, in the browser:

  • Every instance is checked when it is opened, and what happens with data that does not conform is the user's invalid data policy. What another app wrote is only ever warned about (below).
  • Every write is checked before it is saved, so the app never adds invalid data to a pod (write check).
  • What can be repaired is repaired on request; what cannot is shown, and can be removed (repair).
  • The same report is a developer tool too: #/validate?instance=…, linked as "Validate this instance" at the bottom of the workspace while developer mode (a per-instance preference) is on, shows it in full.
  • The Studio's health screen shows what the check finds to every user, of the instance or of one deck (checkDeck, its entry and documents only), with the repairs, beside what else is wrong with a deck.

Text in Markdown is never invalid: CommonMark has no invalid document, so no shape constrains it, and the app shows any text safely (markdown.md). A library release is held to more when it is published, by npm run library:check's Markdown rules, which are about how its text shows, not whether it conforms.

Flow

flowchart LR
    accTitle: Validating an instance
    accDescr: The validation screen lists the instance's documents and checks each against its published shape with the SHACL engine.
    ui["ValidationContainer<br/>#/validate"] --> uc["validateInstance"]
    uc --> docs["instanceDocumentUrls<br/>meta, preferences, palettes, catalog,<br/>every deck's cards + reviews,<br/>every draft's documents"]
    docs --> port["ShapeValidator.validateDocument"]
    port --> pod["pod document<br/>(authenticated fetch)"]
    port --> shapes["ns/shapes/&lt;class&gt;/v&lt;N&gt;.ttl<br/>(published with the site)"]
    port --> engine["rdf-validate-shacl<br/>(lazy chunk)"]
Loading
  • validateInstance (application) lists the decks, names every document the instance may hold, the answer log's month documents included, and the documents of the drafts its catalog links, and asks the ShapeValidator port about each, saying where it is: a draft's document is checked as a draft's ("draft", below), against the draft shapes and without DCAT-AP; summarize (domain) counts the violations. It reads only.
  • Opening an instance runs checkInstance instead: the same check, but a document still at a version the instance's digest says conformed, by the same rules, is neither downloaded nor checked again (validateDocumentSince: a 304 is enough). A document that conforms is given a receipt. The rules are named by a hash of every shape and vendored profile file, computed when the site is built (siteDefines in packages/site/src/siteBuild.ts, which apps/web/vite.config.ts uses), so a site with other shapes checks everything again. It leaves out the answer log, which grows every session: answers are checked as they are written, and in the full check. The developer report always runs the full validateInstance.
  • The format update and a library upgrade check what they write as every save does: each subject a write touches, against its shape, before the write is sent (the write check). A document such a write would make fail is not written (dataNotConforming): it stays as it was, named with why; the format update goes on with the other documents, and a library upgrade stops there, as at any failure. What another app wrote has warnings, not violations, so it never stops a write.
  • shaclShapeValidator.ts fetches the document, converts it to an RDF/JS dataset (toRdfJsDataset) and, for every subject, picks the shape by class and stored version exactly as the mappers do (pickShape), and by the values a shape fixes (an FSRS-6 review state's sm:scheduler, shapes.md); a subject with a Solid Memo class is checked against that class's shapes only, so a deck group, also a dcat:Catalog, is never checked as the catalog. A subject is then checked (with its violations), newer (a format this app does not know: skipped, reported) or untyped (no Solid Memo class: listed so strays are visible). A deck's document's own subject, when all it says is which deck the document is part of (isDeckBackLink in ownership.ts), is Solid Memo's and not listed. A document that does not exist is missing, which is normal, not a problem. A subject another app wrote is marked foreign and its results are warnings (below).
  • Where a deck is is checked beside its shape, in an instance's catalog: an entry that is another document's subject, or a cards or reviews document at an address no deck's document may have (one a request or a server would take elsewhere, another resource's access control or metadata, one of the instance's own documents; data-model.md), is a violation of the entry (DeckPlace, placeViolations in shaclShapeValidator.ts). Which folder the documents are in, and their names, are free. The app does not use such a deck at all, so it is never set aside: the notice names the problem, and removing the entry is all that is offered. A draft's deck is a draft's, and not held to it.
  • A deck whose documents are unclaimed (they do not say they are the deck's, and are outside the instance's own folder: data-model.md) is named in the report's unclaimed (DeckRepository.unclaimedDocuments), with the documents' addresses. It is no violation: it blocks and sets aside nothing, and the deck is read and studied from. Those documents are not checked, as a repair of what a check finds writes to the document. Solid Memo's data check and the Studio's Health screen show them (UnclaimedDecksNotice) with "These are this deck's documents", which, once the user confirms, claims them (claimDeckDocuments); the next check checks them.
  • The shapes are fetched at their IRIs, under https://solid-memo.com/ns/shapes/ (SHAPES_BASE), not bundled: they are published with the site anyway, and the document the browser checks against is the one CI validated. siteFetch in appUseCases.ts reads them from wherever the site is served, so npm run dev and npm start use the repository's ns/. Each shape document is fetched once per session.
  • The SHACL engine (engine.ts, the only module that imports rdf-validate-shacl) is loaded with a dynamic import, so the library is a separate chunk fetched only when a validation is asked for. It validates one focus node against one node shape (validateNode), so no sh:targetClass is needed.
  • One engine is made per shape document and shared by every document checked against it. An engine runs one check at a time: the library keeps a single report per validator, so two checks at once (two documents checked together) would mix up their results.

Profiles: DCAT-AP and SKOS

Besides Solid Memo's own shapes, data is held to two published profiles, vendored verbatim under packages/vocab/vendor/README.md (pinned by commit and sha256 in packages/vocab/vendor/manifest.json, which a test checks) and published with the site at /vendor/:

Profile Shapes Applies to
dcat-ap DCAT-AP 3.0.1 (SEMIC) catalogs, decks and deck releases, distributions, agents
skos SkoHub skos.shacl.ttl + skos.bestPractice.shacl.ttl, for SKOS the concept schemes in ns/vocab/
  • profiles.ts names each profile's files. Their shapes pick their own targets (sh:targetClass), so a profile checks a whole graph with the engine's validate, not one subject with validateNode.
  • The engine is SHACL Core only: coreOnly drops the SPARQL-based constraints (SkoHub has some) when the shapes are loaded. The CI cross-check runs them in full (below).
  • DCAT-AP's class checks (dcat:theme must be a skos:Concept, dcterms:language a dcterms:LinguisticSystem, …) look for the value's type in the data graph, so the reference data in ns/vocab/external.ttl (the EU authority-table entries and media types Solid Memo uses) is loaded next to the data being checked. Add a term there before data uses it.
  • A member a catalog or deck group lists from another document (dcat:dataset, dcat:catalog), such as a dataset another app added to the catalog, is described in that document, which a check of this one cannot see. In the browser (isMemberElsewhere in shaclShapeValidator.ts) DCAT-AP's class check on such a link says nothing: neither the data check nor the write check holds it to its class. A member of the same document still must be a dcat:Dataset or dcat:Catalog.
  • In node (npm run library, the tests) validateProfile (packages/shacl/node/shacl.ts) fails on any violation about a subject of the document. Warnings (a profile's recommendations) fail too for Solid Memo's own concept schemes, which are held to SKOS best practice.
  • Fixtures under packages/vocab/fixtures/profile/<profile>/{valid,invalid}/ pin every property DCAT-AP makes mandatory on the classes Solid Memo writes: title and description on a catalog, dataset and dataset series, a publisher on a catalog, an access URL on a distribution, a name on an agent, and the class of dcterms:source, dcat:theme and dcterms:language values, and of the members a catalog or deck group lists: a deck (dcat:dataset) must be a dcat:Dataset and a group (dcat:catalog) a dcat:Catalog, so a member the document does not describe fails.

The report

Per document, its URL and status; per subject, "conforms to deck format 2", "deck format 3 is newer than this app knows; skipped", "not a Solid Memo subject", a subject only DCAT-AP has something to say about (a license, say), "written by another app; Solid Memo only warns about it", or a table of severity, property, message and value (DCAT-AP's results marked as such). The summary line says "All 9 documents conform" or "3 violations in 2 documents". A subject is checked in its stored format, so outdated but valid data passes: the format update is a separate matter.

The invalid data policy

validateInstance runs when an instance is opened (Workspace, query ["validation", instance], shared with the developer view) and again after a repair or a format update. What happens when the report does not conform is a preference, sm:invalidDataPolicy, a concept of sm:InvalidDataPolicies (vocab.md):

Policy What the app does
Block the instance (sm:blockInstance) The workspace waits for the check, then shows the Data check notice with the repair in place of every screen. Preferences and the validation view stay reachable, so the policy can be changed.
Set invalid data aside (sm:blockSubject, the default) A deck whose entry, cards or review states do not conform is set aside (setAsideDecks): listed as "Set aside", not offered for study, its pages saying so. The catalog itself or a deck group that does not conform sets the arrangement of the deck list aside (arrangementSetAside): the list shows every deck, but cannot be rearranged, nor its groups renamed or deleted, until it is repaired, as a list a newer version arranged cannot. Any other subject that does not conform (a deck's agents or distribution, the catalog's publisher, the preferences, the instance record) sets nothing aside: the notice reports it, and the write check still refuses a write that would leave it not conforming. Everything else works. The notice names the decks set aside, and says when the list cannot be rearranged.
Warn only (sm:warnOnly) The notice, and nothing else.
  • The default is a choice of the app, not of the data. Preferences format 3 and later state a policy, so the default applies only where none is stated: an instance whose preferences were never saved, and preferences format 2, which the step to format 3 gives the default. Preferences saved with "Block the instance" keep it. In the vocabulary the default concept's definition says "The default.", as sm:systemTheme's does.
  • Until the preferences are read, the workspace waits for the check as under "Block the instance", so a user who chose it never sees data before it is checked. Preferences that cannot be read leave the default.
  • Under "Set invalid data aside", nothing is used before the check says what is set aside: until it is done, a deck's pages (its browser, study, preferences and course) wait for it, and the deck list shows but cannot be rearranged, so no write lands on data the check would set aside (a review state a reader drops would be studied as new, and the first answer would write over its history). A check that fails sets nothing aside.
  • A check that cannot run (the shapes unreachable, say) is a warning, not a block: the app does not lock a user out of their data over its own trouble.
  • The Studio holds to the same check and policy (useDataCheck in ui, which Solid Memo's workspace uses too): it shows a deck set aside, or every deck of an instance blocked, read-only, leaves it out of every bulk action, and changes nothing until the check is done (studio.md).

Data another app wrote

The pod is shared: another app may list a dataset in the catalog, or add its own subjects to a cards document. A subject is Solid Memo's when it has a Solid Memo class (sm:Deck, sm:Card, …), carries sm:formatVersion, the stamp Solid Memo writes on every subject it writes, is the catalog (catalog.ttl#catalog), or is named by one of Solid Memo's subjects as its dcterms:creator, dcterms:publisher or dcat:distribution: the catalog, a deck's agents and its distribution have no Solid Memo class, and they stay Solid Memo's when another app's rewrite has dropped their stamp (ownership.ts). A review state of another scheduler (any value of sm:scheduler but the sm:sm2 and sm:fsrs6 IRIs, a literal too) is never Solid Memo's, whatever its class or stamp: its fields are another algorithm's, which need not fit the review-state shape, so it is only warned about (data-model.md). An FSRS-6 state, naming sm:fsrs6, is Solid Memo's own: it is checked against FSRS review state 1, the shape that value picks, and a violation of it counts as any other, a half-written snapshot repaired by dropping it, as an SM-2 state's is. Any other subject is another app's, and shaclShapeValidator.ts marks it foreign: it is still checked, against the shape its class picks (a foaf:Agent against AgentV1) and DCAT-AP, but every result about it, DCAT-AP's included, is a warning. So under every policy it sets no deck aside, leaves the arrangement as it is and blocks no instance, and the document holding it still conforms (and gets its receipt). The developer report shows its results; the notice, which is about violations, does not, and the repair offers nothing for it: a warning is no problem. A subject another app wrote with a Solid Memo class is held to that class's shape, as Solid Memo's own are. The same rule says what a removal takes out of a document: deleting an instance, a deck or a draft removes from each document only Solid Memo's subjects of that document's kind, and deletes the document only when nothing else is left in it (data-model.md). A subject of Solid Memo's class that does not conform to its shape is removed with the rest; another app's subject, and anything whose owner is unclear, is kept, and the document with it, which the result names as kept.

A catalog's or deck group's link to a member another app wrote is that app's data too. When the member is a subject of the same document that is not Solid Memo's (a schema:Dataset, say, which fails DCAT-AP's class check) or a blank node (which fails the shape's sh:nodeKind sh:IRI), the result about the link is a warning, in the check and in the write check alike: the catalog still conforms, the arrangement stays editable, and no repair unlinks the member. The drop-dangling-members repair, and writing the catalog whole, keep such a link; a link to a subject of the document that nothing describes is still the catalog's own problem.

The write check

Every repository write (a deck, its agents and distribution, the catalog, cards, review states, preferences, a palette and its color sets, the instance record) is checked before it is saved: the subjects the write touches against their shapes, and — in a document with DCAT subjects — against DCAT-AP (checkSubjects in shaclShapeValidator.ts, wired into the repositories as checkWrite in appUseCases.ts). A write that would not conform is refused with every problem named, and nothing is saved. Only what the write touches is checked, so a document with an old problem elsewhere can still be written to.

Every caller names where the write goes (WriteContext in writeCheck.ts), and that picks the shapes, as pickShape picks them for a read:

  • "pod": an instance's documents, checked against the pod's shapes and, where they have DCAT subjects, DCAT-AP. Every write the app makes is one, but a draft's.
  • "draft": a release's draft. Its deck, chapters and steps are checked against the draft shapes (DraftDeckV1, DraftChapterV1, DraftStepV1, shapes.md), which do not yet ask what a release needs; its cards, distractors and agents against the shapes every context shares. DCAT-AP is not run: a draft may still lack what a published dataset needs, such as a description. The release check holds the draft to all of it before it is published: validateRelease checks the release it will be, moved to its address, against the library shapes ("library"), DCAT-AP and SKOS, with a library's index beside it for the library's policy. Its publisher and series are often in another document, a library's index. For the library, the index is beside the release, and DCAT-AP holds them to their class there. In a pod, nothing else is read: a release there is whole on its own (deck-library.md), so a publisher or series it does not describe fails DCAT-AP's class check. The next version of a library release describes them, as the index does (nextVersionDraft). Every write of a draft is one (solidReleaseDraftRepository.ts), and so is the check of a draft's documents.

A published release is never written, so no write is checked as one.

A release a learner adds from a link is read, not written, and checked as a document in the "library" context (validateDocument(url, "library")): each subject against its library shape, no DCAT-AP. It is read as anyone reads it, with the fetch the shapes are read with: no login goes to its host. A violation refuses the release; a warning does not.

Repair

planRepair (domain/repair.ts) turns a report into repairs for the problems with one safe answer, written in the subject's own stored format:

Problem Repair
A deck without a description The default description, Flashcards: <title>.
A deck without a study direction, or an unknown one Front to back, as format 1 studied every deck
A review state with half an undo snapshot (an SM-2 or an FSRS-6 state's) The snapshot dropped (as the review-state 1 → 2 migration does), the four terms of an FSRS-6 state's or the five of an SM-2 state's
A malformed due day Recomputed from the last review and the interval
An agent without a name Named after its IRI
A catalog or deck group listing a deck or group the document does not describe (DCAT-AP's class check on dcat:dataset or dcat:catalog) Those links dropped (a member typed sm:Deck or sm:DeckGroup alone is kept, and so is every link to another document)

Every other problem is listed with its document linked, to be fixed there or removed (after a confirmation). Removing a subject also drops the catalog's and the deck groups' membership links to it (dcat:dataset, dcat:catalog), and only those: who else names it, a deck its creator or the catalog its publisher, is left as it is. applyRepairs (solidRepairRepository.ts) reads each document once, applies its repairs, writes it once, and the instance is checked again.

The CI cross-check

CI checks what Solid Memo publishes again with an independent SHACL engine, pySHACL (pinned in scripts/requirements-ci.txt), which also runs SPARQL-based constraints: scripts/shacl_crosscheck.py holds the deck library's index and every version, and the pod catalog documents of the deck format 4, 5 and 6 fixtures (format 5 titles a deck in any language, English or not; format 6 tags its keywords) and of the deck group 1 fixtures (nested groups), to DCAT-AP, and the vocabulary's concept schemes to SkoHub's SKOS shapes, best practice included. It also holds a release published in a pod (the library deck 6 fixtures, which describe their series and publisher themselves) to DCAT-AP with no index beside it, and that release, the draft fixtures (draft deck, chapter and step 1) and the review-state fixtures (review state 2 and FSRS review state 1, side by side in one reviews document) to Solid Memo's own shapes: the script picks each subject's shape by class, format version, context and the values a shape fixes as pickShape does, and targets it with sh:targetNode. Each version is checked beside the index, but only for the subjects it states something about, targeted the same way: DCAT-AP asks a node only about its own statements and the class of what they point at, so a version cannot break an index node it says nothing about, and the index has a check of its own. The script stops if DCAT-AP starts asking more, or if any document states rdfs:subClassOf, since either would make that untrue. The checks run in parallel, one process per CPU. A disagreement between the engines, or a constraint the browser's engine cannot run, fails CI. It reads ns/ and decks/ from the repository, each file at the IRI the site publishes it under, and needs neither a build nor the network, so it runs as its own job beside the build. Locally: pip install -r scripts/requirements-ci.txt, then npm run crosscheck (which runs python3 scripts/shacl_crosscheck.py, and says what to install if pySHACL is missing).