diff --git a/PLAN.md b/PLAN.md index 795e8c1..854575c 100644 --- a/PLAN.md +++ b/PLAN.md @@ -100,7 +100,7 @@ Cost is one extra PR per set and a slower orphan cleanup; the benefit is that no | `lecture-python-intro` | `publish*` tag (manual) | **yes** | | `lecture-wasm` | push to `main` | no — self-heals on merge | -A repo that publishes on push needs no split. Neither does deleting a copy that **no lecture reads in either repo** — typically one a repo committed alongside its mirrored sources while the lecture itself fetches the *other* repo's copy by URL (a *mirror-orphan*); `lecture-wasm` holds five of these (Track X, below). +A repo that publishes on push needs no split. Neither does deleting a copy that **no lecture reads in either repo** — typically one a repo committed alongside its mirrored sources while the lecture itself fetches the *other* repo's copy by URL (a *mirror-orphan*); `lecture-wasm` held five of these, deleted under Track X (below). ### 4. A migration moves bytes; it does not update them @@ -218,14 +218,14 @@ The remaining work decomposes by **consuming series** rather than by hosting pat | **C — `advanced.myst`** | 6, **all done**. **C1**: the `bbh` pair and `hansen_jagannathan_1991_data.json` landed in [#92](https://github.com/QuantEcon/data-lectures/pull/92), flipped in [#95](https://github.com/QuantEcon/data-lectures/pull/95), validated in [#96](https://github.com/QuantEcon/data-lectures/issues/96). **C2**: `fred_data.csv`, `acs_data_summary.csv` and `dataBHS.mat` (converted to `dataBHS.csv`) landed in [#98](https://github.com/QuantEcon/data-lectures/pull/98), flipped in [#99](https://github.com/QuantEcon/data-lectures/pull/99), validated in [#100](https://github.com/QuantEcon/data-lectures/issues/100) | none by URL — but six other org repos hold byte-identical copies of `acs_data_summary.csv` and `dataBHS.mat`, and `lecture-tools-techniques` publishes its own read of the latter, so acceptance was scoped to advanced's own URLs | — | | **D — `programming`** | 1, **done**: `test_pwt.csv` rode with wave C2 ([#98](https://github.com/QuantEcon/data-lectures/pull/98), [#99](https://github.com/QuantEcon/data-lectures/pull/99)); its four consuming repos are recorded in [#101](https://github.com/QuantEcon/data-lectures/pull/101) | none | — | | **E — dynamic / live-API** | the UNRATE twin, then the 15 incidental API lectures | wasm is the forcing customer | [#14](https://github.com/QuantEcon/data-lectures/issues/14) schema decisions, [#26](https://github.com/QuantEcon/data-lectures/issues/26) fetch layer | -| **X — orphan sweep** | 24 committed orphans across 6 repos (`audit.json`, 2026-08-31) — dp 10, programming 5, intro 4, python.myst 2, wasm 2, `continuous_time_mcs` 1; tracked at [QuantEcon/workspace-lectures#57](https://github.com/QuantEcon/workspace-lectures/issues/57) | per repo | nothing — every repo's repoints have landed; `graph.txt` needs its own per-repo reader sweep (below) | +| **X — orphan sweep** | **done 2026-09-01** — 23 of the 24 `audit.json` orphans deleted (dp 10, programming 4, intro 4, python.myst 2, wasm 2, `continuous_time_mcs` 1), `python_advanced_features/test_table.csv` kept as an exercise download, plus 46 translation copies the audit cannot see (`lecture-python.zh-cn` 15, `lecture-python-programming.{zh-cn,fr,fa}` 9 each, `.ml` 6, `lecture-intro.zh-cn` 4); twelve PRs, ledger at [QuantEcon/workspace-lectures#57](https://github.com/QuantEcon/workspace-lectures/issues/57) | — | site clearance rides the settle policy, verified at [QuantEcon/workspace-lectures#40](https://github.com/QuantEcon/workspace-lectures/issues/40) | | **Y — consumer interface (`qeld`)** | the `qeld` package, Q1–Q7 of `PLAN-QELD-PACKAGE.md` — audit support, the package, pilots, then adoption by win; QEP graduation stays | — | nothing — re-scoped 2026-08-12 (D11): the DNS → custom domain → URL-sweep sequence this row used to carry is retired | -**`graph.txt` was closed out as a non-migration (2026-08-12).** It is synthetic teaching data — `provenance: toy`, null in every real provenance field — and the shortest-path exercise teaches its format by quoting the first line, so the data has to stay visible on the page. Hosting it here would have put a toy in a registry that exists to carry provenance. Instead `lecture-wasm` stopped fetching intro's committed copy over the network and embeds it with `%%file` like every sibling ([QuantEcon/lecture-wasm#63](https://github.com/QuantEcon/lecture-wasm/pull/63)), which retired the last cross-repo read of that blob anywhere in the organisation. `graph.txt` consequently no longer appears as a scanned dataset at all. Four repos embed it via `%%file` — intro, dp, jax and wasm — and two of those (intro, dp) also commit a copy the cell overwrites before reading, so those two are shadowed orphans; jax and wasm commit none, which is the cleaner shape. The remaining committed copies (`lecture-intro.zh-cn`, the canary, `lecture-python.zh-cn`, `lecture-dp.monorepo`, `ipynb_pdf_constructor`) are read by nothing. Intro's committed copy is now deletable as Track X — but the same blob sits in **7** repos byte-identically (a further two, `QuantEcon.jl` and `QuantEcon.lectures.code`, hold a 4,692-byte variant differing by one trailing space) and is regenerated at 17 `%%file` sites, including archived `.rst` ancestors that `gh search code` cannot see, so that deletion needs its own per-repo reader sweep rather than an org-wide sweep. +**`graph.txt` was closed out as a non-migration (2026-08-12).** It is synthetic teaching data — `provenance: toy`, null in every real provenance field — and the shortest-path exercise teaches its format by quoting the first line, so the data has to stay visible on the page. Hosting it here would have put a toy in a registry that exists to carry provenance. Instead `lecture-wasm` stopped fetching intro's committed copy over the network and embeds it with `%%file` like every sibling ([QuantEcon/lecture-wasm#63](https://github.com/QuantEcon/lecture-wasm/pull/63)), which retired the last cross-repo read of that blob anywhere in the organisation. `graph.txt` consequently no longer appears as a scanned dataset at all. Four repos embed it via `%%file` — intro, dp, jax and wasm — and two of those (intro, dp) also commit a copy the cell overwrites before reading, so those two are shadowed orphans; jax and wasm commit none, which is the cleaner shape. The remaining committed copies (`lecture-intro.zh-cn`, the canary, `lecture-python.zh-cn`, `lecture-dp.monorepo`, `ipynb_pdf_constructor`) are read by nothing. Intro's committed copy is now deletable as Track X — but the same blob sits in **7** repos byte-identically (a further two, `QuantEcon.jl` and `QuantEcon.lectures.code`, hold a 4,692-byte variant differing by one trailing space) and is regenerated at 17 `%%file` sites, including archived `.rst` ancestors that `gh search code` cannot see, so that deletion needed its own per-repo reader sweep rather than an org-wide sweep. **Deleted 2026-09-01** from intro, dp, `lecture-intro.zh-cn` and `lecture-python.zh-cn` (Track X); the one outside-org reader, `devopseng99/project.lecture-wasm`, was recorded and accepted on QuantEcon/workspace-lectures#57. The copies in the canary, `lecture-dp.monorepo`, `ipynb_pdf_constructor`, `QuantEcon.jl` and `QuantEcon.lectures.code` are read by nothing and stay. -`lecture-dp`, `lecture-jax` and `continuous_time_mcs` are **not data consumers** — dp's 10 committed files are inherited orphans, jax embeds `graph.txt` via `%%file`, and continuous_time_mcs has one orphan scratch file. They appear only in Track X. +`lecture-dp`, `lecture-jax` and `continuous_time_mcs` are **not data consumers** — dp's 10 committed files were inherited orphans and continuous_time_mcs had one orphan scratch file (all deleted 2026-09-01, Track X); jax embeds `graph.txt` via `%%file` and commits nothing. -**Tracks A–D are complete (2026-08-18); E, X and Y remain.** They were independent of each other and ran in the order A → B → C+D. The only hard dependencies left in the programme are: Track E's rollout needs its own template proven first; Track X's `graph.txt` deletion needs a per-repo reader sweep; and Track Y's adoption sweep (qeld Q7) is last. +**Tracks A–D are complete (2026-08-18) and Track X (2026-09-01); E and Y remain.** They were independent of each other and ran in the order A → B → C+D → X. The only hard dependencies left in the programme are: Track E's rollout needs its own template proven first, and Track Y's adoption sweep (qeld Q7) is last. **Track Y was re-scoped 2026-08-12: invest in `qeld`, defer the custom domain indefinitely** (`PLAN-QELD-PACKAGE.md` D11, where the reasoning is recorded in full). The short form: the package delivers everything the domain would have — a stable interface point that survives backend rework — plus tidier lectures and call-site metadata, without the forever-promise of a branded public host, and so without [#35](https://github.com/QuantEcon/data-lectures/issues/35)'s promotion gate ever coming due. It is a commitment decision, not an effort one: the DNS record remains two actions QuantEcon controls (the stale A record was deleted; the name is NXDOMAIN as of 2026-08-10) and [#37](https://github.com/QuantEcon/data-lectures/issues/37) stays open as deferred-not-dead, reopenable at any time because qeld's base URL stays on the raw forms (never `quantecon.github.io` — D11's redirect trap). The classifier constraint this paragraph used to carry moves with the re-scope: `classify_url` must learn the `qeld.url('X')` pattern **before** any consumer adopts it (qeld Q1), for the same structural reason it would have had to learn the canonical host before a URL sweep — otherwise every migrated read classifies as broken and the dashboard inverts. @@ -347,7 +347,7 @@ The first end-to-end deployment: one dataset per hosting pattern, each the harde ### Phase 9 — Adoption (broad sweep — the step that stalled in Feb 2025) - [x] Repoint the remaining consuming lectures as datasets land here (data#4) — **done 2026-08-18**: all 40 static datasets in the corpus are migrated and `repointed` (tracks A–D above; the last four landed in [#98](https://github.com/QuantEcon/data-lectures/pull/98) and flipped in [#99](https://github.com/QuantEcon/data-lectures/pull/99)). The "Repoint rules" stay binding on any future wave: repoint all consumers of a dataset together, and never delete a copy a sibling repo reads -- [ ] Remove lecture repos' duplicate copies as each repoint merges (tracked at [QuantEcon/workspace-lectures#57](https://github.com/QuantEcon/workspace-lectures/issues/57) now that meta#337 is closed) — 24 orphans today (`audit.json`, 2026-08-31), Track X. Note the wasm mirror copies are only safe to delete **after** wasm reads data-lectures directly, not before +- [x] Remove lecture repos' duplicate copies as each repoint merges — **done 2026-09-01** (Track X, [QuantEcon/workspace-lectures#57](https://github.com/QuantEcon/workspace-lectures/issues/57)): 23 audit orphans plus 46 translation copies deleted across twelve repos, one PR each; the wasm mirror copies went only after wasm read data-lectures directly - [ ] Intake rule for migrations: constructed datasets arrive **with their builders**; of the 5 known constructed-but-unscripted files, three arrived with recovered builders in Track C (`fred_data.csv` and the two `bbh` extracts — see `builders/README.md`); `hansen_jagannathan_1991_data.json` and `acs_data_summary.csv` still have none — recorded as QEP follow-ups per meta#338 - [ ] Graduate the convention to a QEP and merge manual#108, with the remaining sweep as its rollout checklist diff --git a/scripts/audit_annotations.yml b/scripts/audit_annotations.yml index 863eac9..ad47c3d 100644 --- a/scripts/audit_annotations.yml +++ b/scripts/audit_annotations.yml @@ -174,83 +174,9 @@ api: # exercise-download (prose link target — referenced, just not by a code cell) # --------------------------------------------------------------------------- committed_unreferenced: - lecture-python-intro:lectures/datasets/GDP_per_capita_world_bank.csv: - kind: orphan - note: no reference anywhere; the QuantEcon/data copy was dropped at Phase 2 - lecture-python-intro:lectures/datasets/Metadata_Country_API_NY.GDP.PCAP.CD_DS2_en_csv_v2_4770417.csv: - kind: orphan - note: metadata twin of the World Bank GDP-per-capita orphan - lecture-python-intro:lectures/datasets/fig_3.ods: - kind: orphan - note: source-format twin of fig_3.xlsx; referenced by nothing - lecture-python-programming:lectures/_static/lecture_specific/pandas/data/ticker_data.csv: - kind: orphan - note: no reference anywhere - lecture-python-programming:lectures/_static/lecture_specific/python_advanced_features/numbers.txt: - kind: shadowed - note: debugging lecture writes it with %%file lecture-python-programming:lectures/_static/lecture_specific/python_advanced_features/test_table.csv: kind: exercise-download note: prose exercise link tells the reader to download this exact URL — keep - lecture-python-programming:lectures/_static/lecture_specific/python_foundations/test_table.csv: - kind: orphan - note: duplicate of the python_advanced_features copy - lecture-python-programming:lectures/_static/lecture_specific/python_foundations/us_cities.txt: - kind: shadowed - note: python_essentials writes it with %%writefile - lecture-python.myst:lectures/_static/lecture_specific/finite_markov/web_graph_data.txt: - kind: shadowed - note: finite_markov writes it with %%file - lecture-python.myst:lectures/web_graph_data.txt: - kind: shadowed - note: shadowed duplicate at lectures/ root - lecture-python-intro:lectures/graph.txt: - kind: shadowed - note: > - short_path regenerates it via %%file before reading it, so the committed - bytes are never consumed — the build prints "Overwriting graph.txt". It - became an orphan when QuantEcon/lecture-wasm#63 removed the only URL read - of it in the organisation; deletable as Track X - lecture-dp:lectures/graph.txt: - kind: shadowed - note: short_path regenerates it via %%file - lecture-dp:lectures/_static/lecture_specific/finite_markov/web_graph_data.txt: - kind: orphan - note: inherited from python.myst; no finite_markov data reference in this repo - lecture-dp:lectures/_static/lecture_specific/match_transport/acs_data_summary.csv: - kind: orphan - note: inherited copy, no consuming lecture in this repo - lecture-dp:lectures/_static/lecture_specific/mle/fp.dta: - kind: orphan - note: inherited from python.myst, no consuming lecture - lecture-dp:lectures/_static/lecture_specific/ols/maketable1.dta: - kind: orphan - note: inherited from python.myst, no consuming lecture - lecture-dp:lectures/_static/lecture_specific/ols/maketable2.dta: - kind: orphan - note: inherited from python.myst, no consuming lecture - lecture-dp:lectures/_static/lecture_specific/ols/maketable4.dta: - kind: orphan - note: inherited from python.myst, no consuming lecture - lecture-dp:lectures/_static/lecture_specific/pandas_panel/countries.csv: - kind: orphan - note: inherited copy; the consuming lectures live in other repos and now read data-lectures - lecture-dp:lectures/_static/lecture_specific/pandas_panel/employ.csv: - kind: orphan - note: inherited copy; the consuming lectures live in other repos and now read data-lectures - lecture-dp:lectures/_static/lecture_specific/pandas_panel/realwage.csv: - kind: orphan - note: inherited copy; the consuming lectures live in other repos and now read data-lectures - lecture-wasm:lectures/datasets/GDP_per_capita_world_bank.csv: - kind: mirror-orphan - note: orphan in intro, mirrored into wasm - lecture-wasm:lectures/datasets/Metadata_Country_API_NY.GDP.PCAP.CD_DS2_en_csv_v2_4770417.csv: - kind: mirror-orphan - note: orphan in intro, mirrored into wasm - continuous_time_mcs:lectures/old_stuff.txt: - kind: orphan - note: scratch file at lectures/ root; referenced by nothing - # Scanner-visible targets that are not datasets (tutorial artifacts the # lecture itself creates, code files, etc.) — suppressed from the audit. ignore_targets: []