Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions PLAN.md
Original file line number Diff line number Diff line change
Expand Up @@ -100,7 +100,7 @@ Cost is one extra PR per set and a slower orphan cleanup; the benefit is that no
| `lecture-python-intro` | `publish*` tag (manual) | **yes** |
| `lecture-wasm` | push to `main` | no — self-heals on merge |

A repo that publishes on push needs no split. Neither does deleting a copy that **no lecture reads in either repo** — typically one a repo committed alongside its mirrored sources while the lecture itself fetches the *other* repo's copy by URL (a *mirror-orphan*); `lecture-wasm` holds five of these (Track X, below).
A repo that publishes on push needs no split. Neither does deleting a copy that **no lecture reads in either repo** — typically one a repo committed alongside its mirrored sources while the lecture itself fetches the *other* repo's copy by URL (a *mirror-orphan*); `lecture-wasm` held five of these, deleted under Track X (below).

### 4. A migration moves bytes; it does not update them

Expand Down Expand Up @@ -218,14 +218,14 @@ The remaining work decomposes by **consuming series** rather than by hosting pat
| **C — `advanced.myst`** | 6, **all done**. **C1**: the `bbh` pair and `hansen_jagannathan_1991_data.json` landed in [#92](https://github.com/QuantEcon/data-lectures/pull/92), flipped in [#95](https://github.com/QuantEcon/data-lectures/pull/95), validated in [#96](https://github.com/QuantEcon/data-lectures/issues/96). **C2**: `fred_data.csv`, `acs_data_summary.csv` and `dataBHS.mat` (converted to `dataBHS.csv`) landed in [#98](https://github.com/QuantEcon/data-lectures/pull/98), flipped in [#99](https://github.com/QuantEcon/data-lectures/pull/99), validated in [#100](https://github.com/QuantEcon/data-lectures/issues/100) | none by URL — but six other org repos hold byte-identical copies of `acs_data_summary.csv` and `dataBHS.mat`, and `lecture-tools-techniques` publishes its own read of the latter, so acceptance was scoped to advanced's own URLs | — |
| **D — `programming`** | 1, **done**: `test_pwt.csv` rode with wave C2 ([#98](https://github.com/QuantEcon/data-lectures/pull/98), [#99](https://github.com/QuantEcon/data-lectures/pull/99)); its four consuming repos are recorded in [#101](https://github.com/QuantEcon/data-lectures/pull/101) | none | — |
| **E — dynamic / live-API** | the UNRATE twin, then the 15 incidental API lectures | wasm is the forcing customer | [#14](https://github.com/QuantEcon/data-lectures/issues/14) schema decisions, [#26](https://github.com/QuantEcon/data-lectures/issues/26) fetch layer |
| **X — orphan sweep** | 24 committed orphans across 6 repos (`audit.json`, 2026-08-31) — dp 10, programming 5, intro 4, python.myst 2, wasm 2, `continuous_time_mcs` 1; tracked at [QuantEcon/workspace-lectures#57](https://github.com/QuantEcon/workspace-lectures/issues/57) | per repo | nothing — every repo's repoints have landed; `graph.txt` needs its own per-repo reader sweep (below) |
| **X — orphan sweep** | **done 2026-09-01** — 23 of the 24 `audit.json` orphans deleted (dp 10, programming 4, intro 4, python.myst 2, wasm 2, `continuous_time_mcs` 1), `python_advanced_features/test_table.csv` kept as an exercise download, plus 46 translation copies the audit cannot see (`lecture-python.zh-cn` 15, `lecture-python-programming.{zh-cn,fr,fa}` 9 each, `.ml` 6, `lecture-intro.zh-cn` 4); twelve PRs, ledger at [QuantEcon/workspace-lectures#57](https://github.com/QuantEcon/workspace-lectures/issues/57) | — | site clearance rides the settle policy, verified at [QuantEcon/workspace-lectures#40](https://github.com/QuantEcon/workspace-lectures/issues/40) |
| **Y — consumer interface (`qeld`)** | the `qeld` package, Q1–Q7 of `PLAN-QELD-PACKAGE.md` — audit support, the package, pilots, then adoption by win; QEP graduation stays | — | nothing — re-scoped 2026-08-12 (D11): the DNS → custom domain → URL-sweep sequence this row used to carry is retired |

**`graph.txt` was closed out as a non-migration (2026-08-12).** It is synthetic teaching data — `provenance: toy`, null in every real provenance field — and the shortest-path exercise teaches its format by quoting the first line, so the data has to stay visible on the page. Hosting it here would have put a toy in a registry that exists to carry provenance. Instead `lecture-wasm` stopped fetching intro's committed copy over the network and embeds it with `%%file` like every sibling ([QuantEcon/lecture-wasm#63](https://github.com/QuantEcon/lecture-wasm/pull/63)), which retired the last cross-repo read of that blob anywhere in the organisation. `graph.txt` consequently no longer appears as a scanned dataset at all. Four repos embed it via `%%file` — intro, dp, jax and wasm — and two of those (intro, dp) also commit a copy the cell overwrites before reading, so those two are shadowed orphans; jax and wasm commit none, which is the cleaner shape. The remaining committed copies (`lecture-intro.zh-cn`, the canary, `lecture-python.zh-cn`, `lecture-dp.monorepo`, `ipynb_pdf_constructor`) are read by nothing. Intro's committed copy is now deletable as Track X — but the same blob sits in **7** repos byte-identically (a further two, `QuantEcon.jl` and `QuantEcon.lectures.code`, hold a 4,692-byte variant differing by one trailing space) and is regenerated at 17 `%%file` sites, including archived `.rst` ancestors that `gh search code` cannot see, so that deletion needs its own per-repo reader sweep rather than an org-wide sweep.
**`graph.txt` was closed out as a non-migration (2026-08-12).** It is synthetic teaching data — `provenance: toy`, null in every real provenance field — and the shortest-path exercise teaches its format by quoting the first line, so the data has to stay visible on the page. Hosting it here would have put a toy in a registry that exists to carry provenance. Instead `lecture-wasm` stopped fetching intro's committed copy over the network and embeds it with `%%file` like every sibling ([QuantEcon/lecture-wasm#63](https://github.com/QuantEcon/lecture-wasm/pull/63)), which retired the last cross-repo read of that blob anywhere in the organisation. `graph.txt` consequently no longer appears as a scanned dataset at all. Four repos embed it via `%%file` — intro, dp, jax and wasm — and two of those (intro, dp) also commit a copy the cell overwrites before reading, so those two are shadowed orphans; jax and wasm commit none, which is the cleaner shape. The remaining committed copies (`lecture-intro.zh-cn`, the canary, `lecture-python.zh-cn`, `lecture-dp.monorepo`, `ipynb_pdf_constructor`) are read by nothing. Intro's committed copy is now deletable as Track X — but the same blob sits in **7** repos byte-identically (a further two, `QuantEcon.jl` and `QuantEcon.lectures.code`, hold a 4,692-byte variant differing by one trailing space) and is regenerated at 17 `%%file` sites, including archived `.rst` ancestors that `gh search code` cannot see, so that deletion needed its own per-repo reader sweep rather than an org-wide sweep. **Deleted 2026-09-01** from intro, dp, `lecture-intro.zh-cn` and `lecture-python.zh-cn` (Track X); the one outside-org reader, `devopseng99/project.lecture-wasm`, was recorded and accepted on QuantEcon/workspace-lectures#57. The copies in the canary, `lecture-dp.monorepo`, `ipynb_pdf_constructor`, `QuantEcon.jl` and `QuantEcon.lectures.code` are read by nothing and stay.

`lecture-dp`, `lecture-jax` and `continuous_time_mcs` are **not data consumers** — dp's 10 committed files are inherited orphans, jax embeds `graph.txt` via `%%file`, and continuous_time_mcs has one orphan scratch file. They appear only in Track X.
`lecture-dp`, `lecture-jax` and `continuous_time_mcs` are **not data consumers** — dp's 10 committed files were inherited orphans and continuous_time_mcs had one orphan scratch file (all deleted 2026-09-01, Track X); jax embeds `graph.txt` via `%%file` and commits nothing.

**Tracks A–D are complete (2026-08-18); E, X and Y remain.** They were independent of each other and ran in the order A → B → C+D. The only hard dependencies left in the programme are: Track E's rollout needs its own template proven first; Track X's `graph.txt` deletion needs a per-repo reader sweep; and Track Y's adoption sweep (qeld Q7) is last.
**Tracks A–D are complete (2026-08-18) and Track X (2026-09-01); E and Y remain.** They were independent of each other and ran in the order A → B → C+D → X. The only hard dependencies left in the programme are: Track E's rollout needs its own template proven first, and Track Y's adoption sweep (qeld Q7) is last.

**Track Y was re-scoped 2026-08-12: invest in `qeld`, defer the custom domain indefinitely** (`PLAN-QELD-PACKAGE.md` D11, where the reasoning is recorded in full). The short form: the package delivers everything the domain would have — a stable interface point that survives backend rework — plus tidier lectures and call-site metadata, without the forever-promise of a branded public host, and so without [#35](https://github.com/QuantEcon/data-lectures/issues/35)'s promotion gate ever coming due. It is a commitment decision, not an effort one: the DNS record remains two actions QuantEcon controls (the stale A record was deleted; the name is NXDOMAIN as of 2026-08-10) and [#37](https://github.com/QuantEcon/data-lectures/issues/37) stays open as deferred-not-dead, reopenable at any time because qeld's base URL stays on the raw forms (never `quantecon.github.io` — D11's redirect trap). The classifier constraint this paragraph used to carry moves with the re-scope: `classify_url` must learn the `qeld.url('X')` pattern **before** any consumer adopts it (qeld Q1), for the same structural reason it would have had to learn the canonical host before a URL sweep — otherwise every migrated read classifies as broken and the dashboard inverts.

Expand Down Expand Up @@ -347,7 +347,7 @@ The first end-to-end deployment: one dataset per hosting pattern, each the harde
### Phase 9 — Adoption (broad sweep — the step that stalled in Feb 2025)

- [x] Repoint the remaining consuming lectures as datasets land here (data#4) — **done 2026-08-18**: all 40 static datasets in the corpus are migrated and `repointed` (tracks A–D above; the last four landed in [#98](https://github.com/QuantEcon/data-lectures/pull/98) and flipped in [#99](https://github.com/QuantEcon/data-lectures/pull/99)). The "Repoint rules" stay binding on any future wave: repoint all consumers of a dataset together, and never delete a copy a sibling repo reads
- [ ] Remove lecture repos' duplicate copies as each repoint merges (tracked at [QuantEcon/workspace-lectures#57](https://github.com/QuantEcon/workspace-lectures/issues/57) now that meta#337 is closed) — 24 orphans today (`audit.json`, 2026-08-31), Track X. Note the wasm mirror copies are only safe to delete **after** wasm reads data-lectures directly, not before
- [x] Remove lecture repos' duplicate copies as each repoint merges — **done 2026-09-01** (Track X, [QuantEcon/workspace-lectures#57](https://github.com/QuantEcon/workspace-lectures/issues/57)): 23 audit orphans plus 46 translation copies deleted across twelve repos, one PR each; the wasm mirror copies went only after wasm read data-lectures directly
- [ ] Intake rule for migrations: constructed datasets arrive **with their builders**; of the 5 known constructed-but-unscripted files, three arrived with recovered builders in Track C (`fred_data.csv` and the two `bbh` extracts — see `builders/README.md`); `hansen_jagannathan_1991_data.json` and `acs_data_summary.csv` still have none — recorded as QEP follow-ups per meta#338
- [ ] Graduate the convention to a QEP and merge manual#108, with the remaining sweep as its rollout checklist

Expand Down
74 changes: 0 additions & 74 deletions scripts/audit_annotations.yml
Original file line number Diff line number Diff line change
Expand Up @@ -174,83 +174,9 @@ api:
# exercise-download (prose link target — referenced, just not by a code cell)
# ---------------------------------------------------------------------------
committed_unreferenced:
lecture-python-intro:lectures/datasets/GDP_per_capita_world_bank.csv:
kind: orphan
note: no reference anywhere; the QuantEcon/data copy was dropped at Phase 2
lecture-python-intro:lectures/datasets/Metadata_Country_API_NY.GDP.PCAP.CD_DS2_en_csv_v2_4770417.csv:
kind: orphan
note: metadata twin of the World Bank GDP-per-capita orphan
lecture-python-intro:lectures/datasets/fig_3.ods:
kind: orphan
note: source-format twin of fig_3.xlsx; referenced by nothing
lecture-python-programming:lectures/_static/lecture_specific/pandas/data/ticker_data.csv:
kind: orphan
note: no reference anywhere
lecture-python-programming:lectures/_static/lecture_specific/python_advanced_features/numbers.txt:
kind: shadowed
note: debugging lecture writes it with %%file
lecture-python-programming:lectures/_static/lecture_specific/python_advanced_features/test_table.csv:
kind: exercise-download
note: prose exercise link tells the reader to download this exact URL — keep
lecture-python-programming:lectures/_static/lecture_specific/python_foundations/test_table.csv:
kind: orphan
note: duplicate of the python_advanced_features copy
lecture-python-programming:lectures/_static/lecture_specific/python_foundations/us_cities.txt:
kind: shadowed
note: python_essentials writes it with %%writefile
lecture-python.myst:lectures/_static/lecture_specific/finite_markov/web_graph_data.txt:
kind: shadowed
note: finite_markov writes it with %%file
lecture-python.myst:lectures/web_graph_data.txt:
kind: shadowed
note: shadowed duplicate at lectures/ root
lecture-python-intro:lectures/graph.txt:
kind: shadowed
note: >
short_path regenerates it via %%file before reading it, so the committed
bytes are never consumed — the build prints "Overwriting graph.txt". It
became an orphan when QuantEcon/lecture-wasm#63 removed the only URL read
of it in the organisation; deletable as Track X
lecture-dp:lectures/graph.txt:
kind: shadowed
note: short_path regenerates it via %%file
lecture-dp:lectures/_static/lecture_specific/finite_markov/web_graph_data.txt:
kind: orphan
note: inherited from python.myst; no finite_markov data reference in this repo
lecture-dp:lectures/_static/lecture_specific/match_transport/acs_data_summary.csv:
kind: orphan
note: inherited copy, no consuming lecture in this repo
lecture-dp:lectures/_static/lecture_specific/mle/fp.dta:
kind: orphan
note: inherited from python.myst, no consuming lecture
lecture-dp:lectures/_static/lecture_specific/ols/maketable1.dta:
kind: orphan
note: inherited from python.myst, no consuming lecture
lecture-dp:lectures/_static/lecture_specific/ols/maketable2.dta:
kind: orphan
note: inherited from python.myst, no consuming lecture
lecture-dp:lectures/_static/lecture_specific/ols/maketable4.dta:
kind: orphan
note: inherited from python.myst, no consuming lecture
lecture-dp:lectures/_static/lecture_specific/pandas_panel/countries.csv:
kind: orphan
note: inherited copy; the consuming lectures live in other repos and now read data-lectures
lecture-dp:lectures/_static/lecture_specific/pandas_panel/employ.csv:
kind: orphan
note: inherited copy; the consuming lectures live in other repos and now read data-lectures
lecture-dp:lectures/_static/lecture_specific/pandas_panel/realwage.csv:
kind: orphan
note: inherited copy; the consuming lectures live in other repos and now read data-lectures
lecture-wasm:lectures/datasets/GDP_per_capita_world_bank.csv:
kind: mirror-orphan
note: orphan in intro, mirrored into wasm
lecture-wasm:lectures/datasets/Metadata_Country_API_NY.GDP.PCAP.CD_DS2_en_csv_v2_4770417.csv:
kind: mirror-orphan
note: orphan in intro, mirrored into wasm
continuous_time_mcs:lectures/old_stuff.txt:
kind: orphan
note: scratch file at lectures/ root; referenced by nothing

# Scanner-visible targets that are not datasets (tutorial artifacts the
# lecture itself creates, code files, etc.) — suppressed from the audit.
ignore_targets: []
Expand Down
Loading