Repository navigation
Canonicalise country, platform, sensor_modality and license in hf_datasets.json - #89
Merged
Merged
Conversation
…asets.json Output of `python src/registry_vocabulary.py static/data/hf_datasets.json` from AgML-Journal-Ingestion PR #10, which extends the registry vocabulary from crop_types/agricultural_task to the remaining categorical fields. 221 records change: 57 country, 55 platform, 58 sensor_modality, 51 license. - country: USA / United States of America -> United States (46 datasets), UK -> United Kingdom, Perú -> Peru, México -> Mexico; datasets spanning several countries become a list instead of a ", "/"; " joined string, so the Location filter lists each country once. 92 -> 80 distinct options. - platform: drone -> uav, aircraft/airborne -> aerial, smartphone/mobile -> handheld, ground_fixed/time-lapse/phenocam -> fixed, spaceborne -> satellite, "handheld/ground" -> "handheld, ground", lists -> comma-joined, typo fixed. 23 -> 9 distinct Platform filter options. - sensor_modality: RGB -> rgb, thermal-rgb -> "rgb, thermal", lists -> comma-joined. 12 -> 9 distinct. - license: CC BY 4.0 -> cc-by-4.0, Apache 2.0 -> apache-2.0, MIT -> mit, cc-0 / cc0 1.0 / "CC0: Public Domain" -> cc0-1.0, gpl-3.0+ -> gpl-3.0, cc-by-Nnc-nd-4.0 -> cc-by-nc-nd-4.0. 35 -> 19 distinct. embeddings/meta.json + vectors.bin regenerated (npm run generate-embeddings) since the embedded text includes these fields; global.json is unchanged. No site code changes: Dataset.country is already string | string[] and parsePlatform / display_modality already split on commas. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
js2552
approved these changes
Oct 20, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Data-only companion to AgML-Journal-Ingestion #10, same shape as #87: the ingestion repo's
python src/registry_vocabulary.py static/data/hf_datasets.jsonrun over the current file, plus regenerated embeddings. No site code changes.What changes (221 records)
countryUSA/United States of America→United States(46 datasets);UK→United Kingdom;Perú→Peru,México→Mexico; 21 multi-country strings ("Slovakia, Czech Republic","Canada; United Kingdom; United States") → lists. Location filter: 92 → 80 optionsplatformdrone→uav;aircraft/airborne→aerial;smartphone/mobile→handheld;ground_fixed/time-lapse/phenocam→fixed;spaceborne→satellite;handheld/ground→handheld, ground; 10 list values → comma-joined;handhledtypo. Platform filter: 23 → 9 options (aerial, fixed, ground, handheld, mixed, robot, satellite, tripod, uav)sensor_modalityRGB→rgb(50);thermal-rgb→rgb, thermal; 6 list values → comma-joined. 12 → 9 distinctlicenseCC BY 4.0→cc-by-4.0,Apache 2.0→apache-2.0,MIT→mit,cc-0/cc0 1.0/CC0: Public Domain→cc0-1.0,gpl-3.0+→gpl-3.0,cc-by-Nnc-nd-4.0→cc-by-nc-nd-4.0. 35 → 19 distinctMulti-country datasets become a list rather than one joined string because
Dataset.countryis already typedstring | string[],useDatasetOptionsflattens lists, and 3 records already used that form, so each country gets its own Location option.parsePlatformanddisplay_modalityalready split on commas.environment(9greenhousevs the LLM path'sfield) is intentionally not touched; see the ingestion PR for why.Verified locally
npm run generate-datasetsandnpm run generate-embeddings(6647 records);global.jsonunchanged.npm run test:unit: 23/23 pass.npm run build: success; builthf_datasets.jsonhas noUSA,drone,RGB,CC BY 4.0orhandheld/groundvalues left.🤖 Generated with Claude Code