Automate external reference data download for the Nextflow pipeline - #8
Draft
m-crown wants to merge 8 commits into
Draft
Automate external reference data download for the Nextflow pipeline#8m-crown wants to merge 8 commits into
m-crown wants to merge 8 commits into
Conversation
Implements docs/reference_data_download_plan.md: a manifest (nextflow/bin/reference_data_manifest.yaml) and script (nextflow/bin/download_reference_data.py) that fetch, validate, and where needed reformat the ~20 external reference files the pipeline needs, replacing the manual download table in docs/installation.md. Tested end-to-end against live sources during implementation (not just written from the plan), which surfaced several corrections the plan didn't anticipate: - ChEBI_Results.tsv derivation: real flat-file columns are lowercase, KEGG cross-references live under type=="MANUAL_X_REF" filtered by source_id (not a dedicated type value), structures.tsv.gz has a direct `smiles` column rather than a melted TYPE/STRUCTURE format, and the derived ID column must be CHEBI:-prefixed and KEGG-COMPOUND-only to match what get_ec_information.py actually merges on and its ChEBI has_role cofactor regex expects. Verified by diffing output against the repo's existing manually-exported data_files/ChEBI_Results.tsv. - SIFTS's only live source is a comma-separated, uncompressed .csv, not the tab-separated gzipped .tsv.gz the pipeline reads - added a csv_to_tsv_gz post-process step. - SCOP/SCOP2/CCD/RHEA URLs could not be confirmed live; marked confidence: unverified in the manifest with notes, rather than silently presented as equally solid as the verified entries. - Pfam's clan_membership.txt.gz/clan.txt.gz no longer exist upstream (consolidated into Pfam-A.clans.tsv.gz); downloaded but flagged needs_code_update since utils.get_pfam_annotations doesn't parse the new format yet - separate follow-up work, not silently patched here. Also: added pyyaml to nextflow/envs/procoggraph.yaml, and fixed two small pre-existing doc bugs adjacent to the section rewritten (conda env filename, assemblies_data.csv extension, and a wrong step cross-reference for params.manifest). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
User pointed at https://www.ebi.ac.uk/pdbe/scop/download. The page is a JS-rendered SPA so a plain fetch returned no content; rendering it in a browser surfaced the real link (scop-cla-latest.txt), and a HEAD request confirmed the sibling scop-des-latest.txt at the same path. Note the upstream filenames have no "2" (scop-cla-latest.txt / scop-des-latest.txt) despite being the SCOP2 dataset - target_filename keeps the local scop2-*-latest.txt names to match nextflow.config's params.scop2_domains_info_file/scop2_descriptions_file. Verified end-to-end: downloaded both files via download_reference_data.py and parsed them with the exact logic from utils.get_scop2_domains_info (pd.read_csv sep=" " comment="#" for the classification file, the NODE_ID/NODE_NAME regex for the descriptions file) - 36,900 domain classification rows and 10,347 description rows, correct shape and content, no parsing errors. Marked confidence: verified in the manifest. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ifest User pointed at https://scop.berkeley.edu/downloads/ver=1.75 and https://ftp.ebi.ac.uk/pub/databases/msd/pdbechem_v2/ccd/components.cif. SCOP: real upstream filenames use a period (dir.cla.scop.1.75.txt / dir.des.scop.1.75.txt) not the underscore this manifest originally guessed (dir.cla.scop.1_75.txt 404s) - same "download name differs from local target name" pattern already seen with the SCOP2 files. target_filename kept as the underscore form to match params.scop_domains_info_file/scop_descriptions_file. CCD: real file is already uncompressed (856,048,364 bytes / ~816M), unlike the previously-guessed files.wwpdb.org/.../components.cif.gz path - dropped the gunzip post_process and fixed min_size_bytes accordingly. Verified end-to-end: - SCOP: downloaded both files via download_reference_data.py and ran the exact parsing logic from utils.get_scop_domains_info/ clean_and_merge_scop_col against them - all 7 SCOP hierarchy levels (cl/cf/sf/fa/dm/sp/px) merged with zero unmatched rows across 110,800 domains (the real code's internal assertion would have failed otherwise). - CCD: full 816M file not downloaded for verification given its size, but a partial byte-range fetch confirmed genuine, well-formed mmCIF content (data_000, _chem_comp.* fields) at the new URL. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
User confirmed rhea2ec.tsv/rhea-directions.tsv/rhea-reaction-smiles.tsv at https://ftp.expasy.org/databases/rhea/tsv/ - matched the manifest's existing guesses exactly, marked confidence: verified. For preprocess_rhea.py --rd_dir (a directory of ~37,100 individual per-reaction .rd files), found that Rhea publishes a bulk rhea-rd.tar.gz archive (24.8M) alongside the individual-file directory - its own README explicitly says "please avoid browsing the subdirectories because they contain a large number of files," confirming the tarball is the intended route rather than downloading files one by one. Added directory-target support to download_reference_data.py to handle this, since every other entry so far has been a single file: - new extract_tar_gz post_process, registered alongside gunzip/csv_to_tsv_gz. - check_size() now branches on dest_path.is_dir() - directory entries check a new min_file_count manifest field instead of byte size. - Caught by inspecting the tarball before writing the extraction code: it already contains a top-level rd/ folder (rd/10001.rd, not bare 10001.rd), so extracting *into* a directory named "rd" would have double-nested to rd/rd/10001.rd, which preprocess_rhea.py's flat os.listdir(rd_dir) wouldn't find. Extraction target is dest_path's parent instead, letting the tarball's own top-level folder become dest_path, with a post-extraction check that this actually happened. Verified end-to-end via the real script: extraction produces 37,100 flat .rd files at the correct directory level, confirmed genuine RD-format content (RHEA:release=141 reaction records with real CHEBI molecule blocks). Also confirmed skip-existing correctly handles a directory target on re-run. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Pfam replaced pfamA.txt.gz + clan_membership.txt.gz + clan.txt.gz with a single Pfam-A.clans.tsv.gz (pfam_accession, clan_acc, clan_id, pfam_name, pfam_description per row, no header). This was flagged but deliberately left undone in the reference-data-download work (docs/reference_data_download_plan.md #2) since it needed a real code change, not just a new URL. Implementing it now: - utils.get_pfam_annotations(pfam_clans_file): rewritten to take the one consolidated file instead of three, reproducing the same 8-column output contract (pfam_accession/pfam_name/pfam_description/clan/pfam/ clan_acc/clan_description/clan_comment) that process_all_pdb_contacts.py and produce_neo4j_files.py already depend on downstream - so nothing else needed to change in produce_neo4j_files.py. - Schema gap: the consolidated file has no equivalent of the old clan.txt.gz's free-text clan_comment field (confirmed no supplementary clan-detail file exists on Pfam's FTP anymore). clan_description is now the short clan id (e.g. "GPCR_A"); clan_comment is left null rather than duplicating clan_description into it and pretending it's real data. - process_all_pdb_contacts.py: --pfam_a_file/--pfam_clan_rels/--pfam_clans collapsed into a single --pfam_clans_file. - nextflow.config: pfam_clans/pfam_clan_rels/pfam_a_file params collapsed into pfam_clans_file. - main.nf: PROCESS_ALL_CONTACTS input/script call/workflow invocation updated to match (three inputs -> one). - reference_data_manifest.yaml: pfam_a entry removed (nothing consumes pfamA.txt.gz anymore), pfam_clans_consolidated promoted from needs_code_update to a normal direct_url entry, now included in the "ready to run pipeline" set. - docs/installation.md updated to match. Verified end-to-end: downloaded the real Pfam-A.clans.tsv.gz via the script and ran the exact new parsing logic against it - output column set matches process_all_pdb_contacts.py's pfam_cols exactly, a known clan member (PF00001/GPCR_A) and a known non-clan family (PF00015) both produce correct rows, and a simulated version of the actual downstream left-merge (including an unmatched accession) behaves correctly. 13,933 clan-member / 16,201 non-clan rows, consistent with real Pfam statistics. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
…ers file User found https://ftp.ncbi.nlm.nih.gov/pubchem/Compound/Extras/CID-Identifiers.tsv.gz - NCBI's bulk PubChem CID-to-external-identifier cross-reference table (12.7M rows: CID, identifier, identifier_type). Filtering to identifier_type == "KEGG ID" gives exactly the CID<->KEGG pairs get_ec_information.py needs (26,933 of 12.7M rows), closing the last remaining manual-only gap in the reference data manifest. - new pubchem_cid_identifiers direct_url manifest entry (98M compressed). - pubchem_substance_mapping changed from source_type: manual to derived, via a new derive_pubchem_kegg_mapping function that writes one "CID:<n> KEGG:<code>" line per pair, matching the exact hand-rolled "KEY:VALUE" line format get_ec_information.py's parser (nextflow/bin/get_ec_information.py:556-570) expects. - KEGG ID rows mix KEGG COMPOUND (C#####) and KEGG DRUG (D#####) codes; kept both rather than filtering to compounds only, since neither the original manual export's description nor the downstream code distinguishes them, and KEGG's REST API handles both uniformly. - docs/installation.md updated - every reference data file is now fetched automatically, no manual steps remain. Verified end-to-end: downloaded the real 98M file via the script, ran the derivation, then fed the output through the exact real parsing logic from get_ec_information.py (the "KEY:VALUE KEY:VALUE" line splitter) and its downstream CID-matching logic - both work correctly against the derived file. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
nextflow/bin/reference_data_manifest.yaml+nextflow/bin/download_reference_data.pyto automate fetching all external reference data the pipeline needs, replacing manual download steps.direct_url/derivedwithconfidence: verified— tested end-to-end against live sources, zero remainingmanualentries.utils.get_pfam_annotations, a SIFTS format mismatch, and SCOP/SCOP2/CCD/RHEA URL corrections (including RHEA'srd/directory needing bulk-tarball extraction support).Stacked on #7 (fix/pipeline-wiring-bugs) — this PR's diff is scoped to just the reference-data-download work.
Test plan
download_reference_data.pyagainst the manifest and confirm every entry resolvesget_pfam_annotations🤖 Generated with Claude Code