Skip to content

content: curated resources for every topic (spec 002) - #34

Merged
matbrgz merged 22 commits into
masterfrom
content/curated-resources
Oct 11, 2026
Merged

content: curated resources for every topic (spec 002)#34
matbrgz merged 22 commits into
masterfrom
content/curated-resources

Conversation

@matbrgz

@matbrgz matbrgz commented Oct 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Member

Spec 002 replaces the generated example.com placeholders with real, checked resources for every subcategory: 2,210 in en-US and 2,025 in pt-BR.

Status

Every category in both editions is curated. ruby test/resources_data_test.rb passes: zero placeholders, all entries well formed, no duplicates.

Also merged here: #33, 115 verified resources imported from a knowledge list and a personal-wiki thread.

How each category was checked

  1. Curation: agy filled _data/<lang>/<category>/<subcategory>/<type>.yaml. pt-BR favours Brazilian institutions and Portuguese resources, with descriptions in Portuguese and a language field.
  2. Format: every entry has name, url (https) and description, with no placeholders and no duplicates within a file. The placeholder test now compares URL hosts (code scanning alert 142).
  3. Links: scripts/check_resource_links.rb ran per category and then over all data (2,515 unique URLs).
    • Failures were checked again. Entries failing twice with 404/410 or an unknown host were removed: 50 in total, mostly moved LibreTexts, MIT OCW and SciELO pages, SBC journal URLs and dead domains.
    • The checker now sends Accept: text/html; without it some sites answer 404 (crates.io, PDET, BCG, CDC). The two entries removed by mistake before the fix were restored.
  4. Spot review of random entries per category for accuracy and language.

Kept despite a transient failure (check by hand)

27 URLs failed on the check date with a timeout, a 5xx error, a TLS problem or a bot block, and were kept:

  • Fail repeatedly from the checker's network: McKinsey (4 pages), Redfin, Salone del Mobile, Unreal Engine, dataintensive.net (2 entries), Effie, SAP Community, OpenCores, EEVblog, Project Gutenberg (504), Institute for PR (2), CCI Self-Esteem.
  • pt-BR government and institutional sites: Periódicos Capes, Brasiliana USP, Planalto Legislação, SGS Banco Central, Portal da Indústria, ONTL Infra S.A., Painel de Obras.
  • Events and others: gamescom latam, QCon SP, POPL, SBC, Prêmio Aberje, HubSpot Brasil, The Animator's Survival Kit, Júlio de Lima.

Before merging

  • All categories pushed; ruby test/resources_data_test.rb passes.
  • ruby scripts/check_resource_links.rb run on the full data.
  • Spot-check the transient links above by hand.

🤖 Generated with Claude Code

matbrgz and others added 15 commits October 10, 2026 17:42
@matbrgz @claude
docs: add spec 002 curated resources with data test and link checker
f4f81ba 
1,272 resource files hold only a generated placeholder. Spec 002 defines
3-8 real, verified resources per subcategory and type, both languages.
- test/resources_data_test.rb: no placeholders, required fields, https,
  no duplicates (fails until curation is done).
- scripts/check_resource_links.rb: HTTP reachability check (network).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@matbrgz @claude
content: curate en-US Academics resources (spec 002 pilot)
91672bb 
42 files (7 subcategories x 6 types), 165 resources researched by agy
(Antigravity CLI) and reviewed by Claude: format validated, every URL
checked over HTTP (150 unique, 0 failing after fixing 2 moved event
pages and dropping 1 dead domain), relevance spot-checked.
The link checker now uses curl (Net::HTTP aborted reads and reported
reachable sites such as pubmed and nature as failures).

Co-Authored-By: Antigravity (agy) <noreply@google.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@matbrgz @claude
content: curate en-US resources (marketing, development, computing, d…
0eac5c2 
…esign and more)

Batches researched by agy (Antigravity CLI) and reviewed by Claude: 400
more files curated in en-US (442/1272 overall). Every URL checked over
HTTP: 1,117 unique, 19 entries removed (dead domains, invalid
certificates, moved deep links) plus one duplicate; one CDC page kept
(403 bot block, page exists). pt-BR batches pending (agy quota).

Co-Authored-By: Antigravity (agy) <noreply@google.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@matbrgz @claude
content: import 115 verified resources from a knowledge list and a pe…
08bf043 
…rsonal-wiki thread

Links come from a public personal knowledge repository and a Hacker News
thread about personal wikis. Of 478 links, 393 remained after removing
duplicates, social and link-shortener URLs. 279 answered HTTP 200; pages
behind bot challenges were left out because their content could not be
checked. Each kept item was classified into the IPL taxonomy with a
description written for IPL (the sources have no licence, so no text was
copied), checked again at its final URL, and de-duplicated against existing
IPL data:
- 115 added to 41 en-US files;
- 158 skipped (personal notes and profiles, dead or repurposed projects,
  duplicates, thin or marketing pages);
- 5 already in IPL.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@matbrgz
Merge pull request #33 from InternetPublicLibrary/content/knowledge-i…
1e8c07a 
…mport

content: import 115 verified resources from a knowledge list and a personal-wiki thread
@matbrgz @claude
content: curate en-US/business resources
c386ab7 
89 files curated with agy (spec 002), checked by Claude. Links checked; entries whose URL failed twice with 404/410 or an unknown host were removed. Kept despite a transient failure (timeout, 5xx, TLS) on the check date: see the PR.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@matbrgz @claude
content: curate en-US/computing resources
18e5409 
54 files curated with agy (spec 002), checked by Claude. Links checked; entries whose URL failed twice with 404/410 or an unknown host were removed. Kept despite a transient failure (timeout, 5xx, TLS) on the check date: see the PR.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@matbrgz @claude
content: curate en-US/design resources
378587b 
27 files curated with agy (spec 002), checked by Claude. Links checked; entries whose URL failed twice with 404/410 or an unknown host were removed. Kept despite a transient failure (timeout, 5xx, TLS) on the check date: see the PR.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@matbrgz @claude
content: curate en-US/personal-development resources
0e8264d 
22 files curated with agy (spec 002), checked by Claude. Links checked; entries whose URL failed twice with 404/410 or an unknown host were removed. Kept despite a transient failure (timeout, 5xx, TLS) on the check date: see the PR.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@matbrgz @claude
content: curate en-US/photography resources
bb9d1b6 
2 files curated with agy (spec 002), checked by Claude. Links checked; entries whose URL failed twice with 404/410 or an unknown host were removed. Kept despite a transient failure (timeout, 5xx, TLS) on the check date: see the PR.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@matbrgz @claude
content: curate pt-BR/academics resources
b0d4b37 
42 files curated with agy (spec 002), checked by Claude. Links checked; entries whose URL failed twice with 404/410 or an unknown host were removed. Kept despite a transient failure (timeout, 5xx, TLS) on the check date: see the PR.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@matbrgz @claude
content: curate pt-BR/business resources
b9e1a3f 
90 files curated with agy (spec 002), checked by Claude. Links checked; entries whose URL failed twice with 404/410 or an unknown host were removed. Kept despite a transient failure (timeout, 5xx, TLS) on the check date: see the PR.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@matbrgz @claude
content: curate pt-BR/design resources
9ae0da8 
60 files curated with agy (spec 002), checked by Claude. Links checked; entries whose URL failed twice with 404/410 or an unknown host were removed. Kept despite a transient failure (timeout, 5xx, TLS) on the check date: see the PR.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@matbrgz
Merge remote-tracking branch 'origin/content/curated-resources' into …
5ef2008 
…content/curated-resources

# Conflicts:
#	_data/en-US/business/communications/learn.yaml
#	_data/en-US/computing/hardware/databases.yaml
#	_data/en-US/computing/machine-learning/learn.yaml
#	_data/en-US/computing/machine-learning/publications.yaml
#	_data/en-US/computing/natural-language-processing/learn.yaml
#	_data/en-US/computing/natural-language-processing/publications.yaml
#	_data/en-US/computing/natural-language-processing/tools.yaml
#	_data/en-US/computing/networking/learn.yaml
#	_data/en-US/computing/networking/publications.yaml
#	_data/en-US/computing/operating-systems/blogs-and-sites.yaml
#	_data/en-US/computing/operating-systems/learn.yaml
#	_data/en-US/design/graphic-design/blogs-and-sites.yaml
#	_data/en-US/design/graphic-design/learn.yaml
#	_data/en-US/design/graphic-design/tools.yaml
@matbrgz @claude
content: curate pt-BR/computing resources
84a07e6 
108 files curated with agy (spec 002), checked by Claude. Links checked; 3 entries whose URL failed twice with 404 were removed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
github-advanced-security[bot]
github-advanced-security Bot found potential problems Oct 11, 2026
View reviewed changes
Comment thread test/resources_data_test.rb Fixed
@matbrgz @claude
fix: detect placeholder resources by URL host, not by substring
c8149a8 
Code scanning alert 142 (rb/incomplete-url-substring-sanitization): the test
looked for "example.com" anywhere in a file. It now parses each entry's URL
and matches the host against the reserved example domains. A real URL that
mentions example.com in its path no longer counts as a placeholder, and
example.org/.net placeholders are now caught.

The well-formedness test also found data problems, fixed here:
- EMNLP, NAACL and COLING shared the ACL portal URL; each now points to its
  ACL Anthology venue page.
- OSDI and USENIX ATC shared one URL; each now has its USENIX page.
- Paul Graham's essays use https.
- Thinking with Type is removed: it has no working https site.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@cloudflare-workers-and-pages

cloudflare-workers-and-pages Bot commented Oct 11, 2026 •
edited
Loading

Copy link
Copy Markdown

Deploying iplibrary-website with  Cloudflare Pages  Cloudflare Pages

Latest commit: 8731560
Status: ✅  Deploy successful!
Preview URL: https://514b5e39.iplibrary-website.pages.dev
Branch Preview URL: https://content-curated-resources.iplibrary-website.pages.dev

View logs

matbrgz and others added 6 commits October 11, 2026 17:44
@matbrgz @claude
fix: link checker sends Accept: text/html; restore PDET
edde982 
Some sites (crates.io, pdet.mte.gov.br) answer 404 to requests without an
Accept header, so the checker reported live sites as dead. With the header,
only these two of the 28 URLs removed so far turned out to be alive. PDET
is restored here; crates.io returns with pt-BR/development.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@matbrgz @claude
content: curate pt-BR/development resources
daf7a5b 
84 files curated with agy (spec 002), checked by Claude. Links checked; entries failing twice with 404 or an unknown host were removed (dropsdejogos, roadsec.com.br, an old SICP mirror), as was Vimcasts (no https). Kept despite a transient failure: dataintensive.net, gamescomlatam.com, unrealengine.com, qconsp.com, popl.sigplan.org, sbc.org.br, juliodelima.com.br.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@matbrgz @claude
content: curate pt-BR/photography resources
4ea849d 
78 files curated with agy (spec 002), checked by Claude. Links checked; one entry with an unknown host (resumofotografico.com.br) was removed.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@matbrgz @claude
content: curate pt-BR/personal-development resources
110ac24 
90 files curated with agy (spec 002), checked by Claude. Links checked; 8 entries that failed twice with 404 or an unknown host were removed (three SciELO journal pages no longer on SciELO, two gov.br pages, two dead domains, one moved article). Kept despite a TLS error from the checker: mckinsey.com.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@matbrgz @claude
content: curate pt-BR/marketing resources
b67bb4d 
84 files curated with agy (spec 002), checked by Claude. Links checked; 3 entries that failed twice with 404 or an unknown host were removed (an Intercom page no longer on SciELO, an RD Station subdomain, VidCon Brasil). Kept despite a transient failure: hubspot.com.br (timeout), premioaberje.com.br (TLS).

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@matbrgz @claude
content: remove four dead pt-BR links found by the full link check
8731560 
hbrbr.uol.com.br and universidade.rdstation.com.br no longer resolve; the Intercom page on SciELO and vidcon.com.br answer 404. Each failed the check twice.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@matbrgz
matbrgz marked this pull request as ready for review October 11, 2026 21:05
@matbrgz
matbrgz merged commit 4b1a38a into master Oct 11, 2026
10 of 13 checks passed
@matbrgz matbrgz mentioned this pull request Oct 11, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

1 more reviewer
@github-advanced-security github-advanced-security[bot] github-advanced-security[bot] left review comments
Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@matbrgz @github-advanced-security