Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 5 additions & 3 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -147,7 +147,7 @@ No application code changes required.

## Catalog

The catalog currently contains 100 operational datasets spanning natural
The catalog currently contains 141 operational datasets spanning natural
hazards, weather, water, climate, flood risk, drought, space weather, global
disaster alerts, public health, clinical research, cybersecurity, package
graphs, legislation, sanctions screening, government spending and procurement,
Expand All @@ -158,8 +158,10 @@ broadband, bridges, EV charging, consumer finance, education, K-12 directories,
housing prices, rents, food and product recalls, elections, European statistics,
live transit feeds, aviation, provider directories, drinking water, preprints,
pageviews, agriculture, tropical cyclones, crime, companies, occupations,
global forecasts, live OSM, food products, nursing homes, LEI, and euro-area
statistics. The YAML files in
global forecasts, live OSM, food products, nursing homes, LEI, euro-area
statistics, patents, vital statistics, mortgages, OECD and national statistics,
Mauna Loa CO2, sea ice, caselaw, lobbying, Certificate Transparency, and
public GitHub activity. The YAML files in
[`data/datasets`](data/datasets) are the source of truth for the current list.

## Deploy to Vercel
Expand Down
79 changes: 79 additions & 0 deletions data/datasets/cdc-uscs-cancer-statistics.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,79 @@
id: cdc-uscs-cancer-statistics
name: U.S. Cancer Statistics
description: >
Official federal cancer incidence and mortality statistics for building
state comparison and cancer-burden dashboards.
theme: Health, Food & Safety
url: https://www.cdc.gov/united-states-cancer-statistics/index.html
access_type:
- download
- api
api_key_required: false
free_to_access: true
size_gb_min: 0
size_gb_max: 0.2
formats:
- CSV
- JSON
- Excel
license: U.S. Public Domain with CDC attribution and use requirements
license_url: https://www.cdc.gov/other/agencymaterials.html
url_checks:
source_marker: United States Cancer Statistics
license_marker: public domain
domains:
- Public Health
- Epidemiology
data_types:
- Tabular
- Aggregated Data
tasks:
- Incidence Comparison
- Trend Analysis
- Public Health Planning
difficulty: intermediate
geography:
- United States
temporal_coverage: USCS incidence and mortality years published by CDC and NCI
update_frequency: annual
provider: Centers for Disease Control and Prevention
source_type: government
last_verified: 2026-08-18
getting_started:
overview: >
USCS is the official federal combination of NPCR and SEER incidence with
NVSS mortality. Start with a public state-level incidence table, not the
SEER*Stat research file. Rates are age-adjusted to a standard population
and are not individual case listings.
prerequisites:
- Python 3.10 or newer
- A notebook environment such as Jupyter or Google Colab
- An internet connection
access_steps:
- Open the USCS page and locate Data Visualizations or public-use tables.
- Download or query one recent state incidence file.
- Keep state, cancer site, year, and age-adjusted rate.
python:
packages:
- pandas
- requests
code: |
import pandas as pd
import requests

response = requests.get(
"https://data.cdc.gov/resource/5kmb-vjq2.json",
params={"$limit": 50, "$order": "year DESC"},
timeout=30,
)
response.raise_for_status()
stats = pd.DataFrame(response.json())
stats["retrieved_at_utc"] = pd.Timestamp.now(tz="UTC")
print(stats.head())
first_project:
title: Compare state cancer incidence rows
goal: Test whether a USCS public table can power a bounded state-burden sketch.
steps:
- Keep geography, site, year, and age-adjusted rate from the extract.
- Rank a single site-year by rate and flag missing geographies.
- Explain that USCS public tables are aggregated rates, not SEER research microdata, and cannot identify individuals.
73 changes: 73 additions & 0 deletions data/datasets/census-lehd-lodes.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,73 @@
id: census-lehd-lodes
name: Census LEHD Origin-Destination Employment Statistics
description: >
Block-level origin-destination jobs data for building commute, workforce, and
local employment-map prototypes.
theme: Geospatial & Infrastructure
url: https://lehd.ces.census.gov/data/
access_type:
- download
api_key_required: false
free_to_access: true
size_gb_min: 0.01
size_gb_max: 50
formats:
- CSV
- GZIP
license: U.S. Census Bureau data-use terms
license_url: https://www.census.gov/data/developers/about/terms-of-service.html
url_checks:
source_marker: LEHD Origin-Destination Employment Statistics
license_marker: Terms of Service
domains:
- Labor Economics
- Local Economics
- Transportation
data_types:
- Tabular
- Geospatial
- Origin-Destination
tasks:
- Commute Analysis
- Workforce Mapping
- Site Selection
difficulty: intermediate
geography:
- United States
temporal_coverage: annual LODES vintages with state-specific start years
update_frequency: annual
provider: U.S. Census Bureau
source_type: government
last_verified: 2026-08-18
getting_started:
overview: >
LODES Origin-Destination files count jobs connecting home and work census
blocks. Start with one small state's OD main file. Counts are synthesised
from administrative records, some states are missing in some years, and
block-to-block files are large.
prerequisites:
- Python 3.10 or newer
- A notebook environment such as Jupyter or Google Colab
- An internet connection
access_steps:
- Open the LEHD data page and choose LODES OD for one small state.
- Download the JT00 OD main gzip for the latest vintage listed.
- Keep workplace block, residence block, and total jobs.
python:
packages:
- pandas
code: |
import pandas as pd

jobs = pd.read_csv(
"https://lehd.ces.census.gov/data/lodes/LODES8/vt/od/vt_od_main_JT00_2022.csv.gz",
nrows=200,
)
print(jobs.head())
first_project:
title: Inspect Vermont LODES job flows
goal: Test whether a one-state OD extract can power a bounded commute sketch.
steps:
- Keep workplace geocode, residence geocode, and total jobs from the first rows.
- Sum jobs by workplace census tract prefix and list empty geocodes.
- Explain that LODES is synthesised administrative data, some states are omitted, and a sample of rows is not a statewide commute census.
77 changes: 77 additions & 0 deletions data/datasets/certificate-transparency-crtsh.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,77 @@
id: certificate-transparency-crtsh
name: Certificate Transparency Search
description: >
Issued TLS certificates indexed from public CT logs for building domain
certificate monitors and issuance alerts.
theme: Technology & Cybersecurity
url: https://certificate.transparency.dev/
access_type:
- api
api_key_required: false
free_to_access: true
size_gb_min: 0
size_gb_max: 0.05
formats:
- JSON
license: Certificate Transparency log public data / crt.sh use
license_url: https://certificate.transparency.dev/
url_checks:
source_marker: Certificate Transparency
license_marker: Certificate Transparency
domains:
- Cybersecurity
- Internet Infrastructure
data_types:
- Event Data
- Registry Data
- Tabular
tasks:
- Certificate Monitoring
- Alerting
- Asset Discovery
difficulty: beginner
geography:
- Global
temporal_coverage: certificates submitted to participating CT logs
update_frequency: continuous
provider: Sectigo crt.sh
source_type: company
last_verified: 2026-08-18
getting_started:
overview: >
crt.sh queries public Certificate Transparency logs. Start with one
registered domain and a JSON search. CT logs are the origin; crt.sh is the
query surface. Logs can lag, expired certificates remain, and this is not a
complete inventory of every private PKI.
prerequisites:
- Python 3.10 or newer
- A notebook environment such as Jupyter or Google Colab
- A domain you are authorized to inspect
access_steps:
- Open crt.sh and try one Identity search in the browser.
- Request JSON for that domain with output=json.
- Keep issuer name, common name, and not-after date.
python:
packages:
- pandas
- requests
code: |
import pandas as pd
import requests

response = requests.get(
"https://crt.sh/",
params={"q": "example.com", "output": "json"},
timeout=60,
)
response.raise_for_status()
certificates = pd.DataFrame(response.json())
certificates["retrieved_at_utc"] = pd.Timestamp.now(tz="UTC")
print(certificates[["issuer_name", "common_name", "not_after"]].head())
first_project:
title: List certificates for one domain
goal: Test whether a CT search can power a bounded issuance monitor.
steps:
- Deduplicate certificate ids and keep issuer, common name, and expiry.
- Flag certificates whose not-after date is in the past.
- Explain that CT coverage depends on participating logs and is not a private-PKI inventory.
76 changes: 76 additions & 0 deletions data/datasets/chrome-ux-report.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
id: chrome-ux-report
name: Chrome UX Report API
description: >
Real-user Chrome experience metrics for building Core Web Vitals monitors
from origin-level CrUX records.
theme: Technology & Cybersecurity
url: https://developer.chrome.com/docs/crux
access_type:
- api
api_key_required: true
free_to_access: true
size_gb_min: 0
size_gb_max: 0.01
formats:
- JSON
license: Google APIs Terms of Service
license_url: https://developers.google.com/terms
url_checks:
source_marker: Chrome UX Report
license_marker: Google APIs Terms of Service
domains:
- Web Performance
- Software
data_types:
- Aggregated Data
- Time Series
tasks:
- Performance Monitoring
- Origin Comparison
- Alerting
difficulty: beginner
geography:
- Global
temporal_coverage: rolling 28-day CrUX collection windows
update_frequency: daily
provider: Google
source_type: company
last_verified: 2026-08-18
getting_started:
overview: >
The CrUX API returns origin-level p75 metrics such as LCP and INP. Start
with one origin you are authorized to monitor. CrUX covers opted-in Chrome
users, not synthetic tests, and small origins can be missing.
prerequisites:
- Python 3.10 or newer
- A notebook environment such as Jupyter or Google Colab
- A Google API key saved in the CRUX_API_KEY environment variable
access_steps:
- Enable the Chrome UX Report API in Google Cloud and save CRUX_API_KEY.
- Read the query endpoint documentation.
- Request metrics for one HTTPS origin.
python:
packages:
- pandas
- requests
code: |
import os
import pandas as pd
import requests

response = requests.post(
"https://chromeuxreport.googleapis.com/v1/records:queryRecord",
params={"key": os.environ["CRUX_API_KEY"]},
json={"origin": "https://www.example.com"},
timeout=30,
)
response.raise_for_status()
record = pd.json_normalize(response.json())
print(record)
first_project:
title: Read one origin's Core Web Vitals
goal: Test whether CrUX can power a bounded real-user performance card.
steps:
- Confirm p75 LCP, INP, and CLS fields returned for the origin.
- Compare those p75 values to the published "good" thresholds.
- Explain that CrUX is opted-in Chrome field data and that small origins can be absent.
Loading
Loading