Add LinkedDataHub docker-compose deployment#5
Merged
Conversation
Deploy the datasets locally at https://localhost:4443/ with `make up`, mirroring the linkeddatahub.com / Homepage compose layout (nginx TLS + client-cert proxy, stock atomgraph/linkeddatahub:5.6.0 — no WAR overlay, two atomgraph/fuseki:6.1.0 stores, three Varnish caches, SMTP relay). Unlike those projects, RDF data is not created document-by-document over HTTP via the LDH CLI — the ~926k-quad datasets/current/*/*.trig files are bulk-loaded straight into the fuseki-end-user TDB2 store by `make load` via a one-off `tdb-loader` compose service (profiles: ["load"]) that runs tdb2.tdbloader from the Jena CLI bundled inside the fuseki image jar. Triplestore ports are never published to the host; SPARQL is served at https://localhost:4443/sparql. `make load` ends with `make public`, the direct-to-triplestore equivalent of LDH CLI's make-public.sh (anonymous read works for the untyped ETL documents because LDH's ACL query leaves $Type unbound when a document has no rdf:type). Also flip the default ETL base URI to https://localhost:4443/ so a plain `make -C etl` produces data loadable into the local instance; committed datasets/current/ remain generated with BASE=https://linkeddata.lt/. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Every entity/concept document is now typed dh:Item with sioc:has_container
pointing at its container; taxonomy scheme documents are dh:Containers with
sioc:has_parent <{base}taxonomies/>. A new etl/containers/ domain generates
the nine top-level dh:Container documents (sioc:has_parent = base URI) into
datasets/current/containers/containers.trig. This is what LinkedDataHub's
container pages use to list children (sioc:has_parent/has_container UNION),
so the loaded data is now browsable, not just dereferenceable.
validate.sh now waives foaf:primaryTopic for dh:Container documents;
graphify.sh accepts mappings with no input data (VALUES-driven, bash 3.2
empty-array fix); SHACL shapes gained Item/Container document shapes.
Committed datasets regenerated with BASE=https://linkeddata.lt/.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…trigs The bulk-loaded container documents rendered no children listings: LDH only renders one when the container document itself carries an rdf:_1 block pointing at an ldh:Object with rdf:value ldh:ChildrenView. Scaffolding is now a LinkedDataHub-Apps-style app/ tree — one Turtle file per container (root + 9 top-level containers + 7 taxonomy scheme containers, schemes keeping their skos:ConceptScheme as #this) — PUT over HTTP by `make install` via the LDH CLI (put.sh + Jena turtle from a ../LinkedDataHub checkout, LDH_HOME-overridable). LDH adds sioc:has_parent, dct:created/modified and acl:owner server-side and strips any client-sent sioc triples. ETL datasets now contain only dh:Item documents, still bulk-loaded with `make load`, attaching to the scaffolding via unchanged sioc:has_container URIs: the etl/containers/ domain is gone and the 7 taxonomy mappings no longer emit scheme graphs. validate.sh is strict again (dct:title + foaf:primaryTopic on every graph); SHACL keeps only the :ItemShape as LDH-specific, and :ConceptShape's scheme references switch from sh:class to host-agnostic IRI patterns since the scheme entity left the files. Also fix a stale-base trap: static-CSV taxonomy outputs kept the previous BASE because make skips missing intermediate cache/*.nt when the .trig looks up to date (this had committed localhost-based trigs for the 4 static schemes). A cache/.base stamp now invalidates the chain when BASE changes. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Deploys the LTLOD datasets locally on LinkedDataHub at https://localhost:4443/ with a docker-compose stack mirroring the
linkeddatahub.com/Homepagelayout — but with datasets bulk-loaded directly into thefuseki-end-userTDB2 store instead of the LDH CLI's document-by-document HTTP approach (too slow at ~926k quads; the direct load takes ~8 s).Usage
Design notes
atomgraph/linkeddatahub:5.6.0image — no custom Java/WAR overlay.expose: 3030; SPARQL is served through LDH athttps://localhost:4443/sparql.tdb2.tdbloadervia a one-offtdb-loadercompose service (profiles: ["load"]), using the Jena CLI bundled inside theatomgraph/fuseki:6.1.0image jar — no extra images.make loadstopsfuseki-end-user, loads, restarts it and the Varnish caches. Load is append-only; clean rebuild:make down && rm -rf fuseki/end-user && make up && make load.bin/make-public.shreplicates LDH CLI'smake-public.shINSERT directly againstfuseki-admininside the docker network (no owner WebID cert needed). Anonymous read works for the untyped ETL documents because LDH's ACL query leaves$Typeunbound when a document has nordf:type.https://localhost:4443/; committeddatasets/current/remain generated withBASE=https://linkeddata.lt/.Test plan (all verified locally)
make up: stack healthy on first boot;docker compose psshows only nginx publishing ports (81/4443/5443), no 3030 on hostmake load: 925,748 quads in 8.4 s; re-run is idempotent (unchanged count)https://localhost:4443/→ 200 (RDF + HTML content negotiation)https://localhost:4443/sparql= 925,781 (matches internal endpoint)curl -k -H "Accept: text/turtle" https://localhost:4443/admin-units/12/→ Birštono savivaldybė graphDocument hierarchy
Container scaffolding (root + 9 top-level containers + 7 taxonomy scheme containers) lives in
app/as per-container Turtle files, PUT over HTTP by the one-timemake installvia the LDH CLI (LinkedDataHub-Apps style; needs a../LinkedDataHubcheckout,LDH_HOME-overridable) — so each container carries theldh:ChildrenViewblock that makes LDH render children listings, plus server-managedsioc:has_parent/acl:owner. ETL datasets contain onlydh:Itemdocuments, bulk-loaded bymake load, attached viasioc:has_container. Verified: root lists 9 containers,admin-units/has 26,883 children,streets/61,149,taxonomies/7 schemes, children listings render.🤖 Generated with Claude Code