Skip to content

Commit aac3979

Browse files
committed
sqlite: add per-entity public modules (BioProject, Study, BioSample, Experiment, Run)
Each module exposes insert / tryGet / delete / listAccessions and deconstructs the full F# record into its row family in one transaction. - BioProject: Collaborators + RelatedProjects (PARENT/CHILD/PEER DU) + FirstPublic date. - Study: takes the parent bioProjectAccession explicitly (the F# type doesn't carry it); rebuilds StudyDescriptor with its existing-type enum, nullable ProjectId, and RelatedStudies XRef + IsPrimary. - BioSample: handles the biosample_name child table. - Experiment: serializes StudyRef + Library.SampleDescriptor via the shared References helper; library strategy/source/selection enums, paired-end nominal length/sdev, targeted loci, spot descriptor. - Run: ExperimentRef, RunDate (ISO-8601), RunCenter, DataBlock.MemberName, spot descriptor (Files deferred per plan section D4). Extracts the Platform DU dispatch into a new Internal/Platforms.fs so Experiment and Run share one ~170-line helper instead of duplicating it. Expands the schema's experiment_platform / run_platform CHECK constraint from 11 to all 18 Platform DU cases — the generated F# Platform type has 18, the original constraint would have rejected 7 (ION_TORRENT, ELEMENT, ULTIMA, VELA_DIAGNOSTICS, GENAPSYS, GENEMIND, TAPESTRI). Verified via fsi: every entity round-trips with all populated collections, FK enforcement raises on a bogus parent, cascade delete honors the schema's ON DELETE SET NULL on study.bioproject_accession.
1 parent d1c34c6 commit aac3979

10 files changed

Lines changed: 1250 additions & 12 deletions

File tree

‎plans/sqlite-store.md‎

Lines changed: 14 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -343,14 +343,20 @@ O(entity_count) queries per reconstruction; with FK indexes this stays cheap.
343343

344344
**Result:** Clean build, zero warnings. Smoke test through `dotnet fsi`: schema loads (42 tables), all 5 identifier kinds round-trip (PRIMARY/SECONDARY/EXTERNAL/SUBMITTER/UUID) with label/namespace preserved, attributes round-trip with NULL value/units preserved, all 3 Link DU cases (URL/XREF/ENTREZ with nullable int64 Id) round-trip cleanly.
345345

346-
### Chunk 4 — Per-entity public modules
347-
348-
- [ ] `Schema.fs` — public `Schema.init : SqliteConnection -> unit`
349-
- [ ] `BioProject.fs` — `insert`, `tryGet`, `delete`, `listAccessions`
350-
- [ ] `Study.fs` — same surface; handles `study_descriptor` denormalization
351-
- [ ] `BioSample.fs` — same surface; handles `biosample_name`
352-
- [ ] `Experiment.fs` — same surface; uses `References` for `StudyRef` and `Library.SampleDescriptor`; serializes `Platform` DU into `experiment_platform` + `_params`
353-
- [ ] `Run.fs` — same surface; uses `References` for `ExperimentRef`; serializes optional `Platform` DU
346+
### Chunk 4 — Per-entity public modules ✅
347+
348+
- [x] `Schema.fs` — public `Schema.init : SqliteConnection -> unit`
349+
- [x] `BioProject.fs` — `insert`, `tryGet`, `delete`, `listAccessions`; covers `Collaborators` and `RelatedProjects` (PARENT/CHILD/PEER DU)
350+
- [x] `Study.fs` — same surface; reconstructs the full `StudyDescriptor` (existing-type enum, NullableProjectId, RelatedStudies with XRef). Takes parent `bioProjectAccession` (NULL ok) since the F# type doesn't model the parent FK.
351+
- [x] `BioSample.fs` — same surface; handles `biosample_name`
352+
- [x] `Experiment.fs` — same surface; uses `References` for `StudyRef` and `Library.SampleDescriptor`; serializes `Platform` DU via the new shared `Internal/Platforms.fs` helper; round-trips library strategy/source/selection enums, paired-end nominal length/sdev, targeted loci, spot descriptor read specs
353+
- [x] `Run.fs` — same surface; uses `References` for `ExperimentRef`; reuses `Platforms` helper; persists `RunDate` (ISO-8601), `RunCenter`, `DataBlock.MemberName`, spot descriptor
354+
355+
**Extras landed:**
356+
- Schema's platform `CHECK (kind IN (...))` constraint expanded from 11 to all 18 DU cases (the F# `Platform` type has 18, the original constraint would have rejected 7).
357+
- `Internal/Platforms.fs` extracted as a sibling to `Identifiers/Attributes/Links/References` — both Experiment and Run share it.
358+
359+
**Result:** fsi round-trip test exercises every entity (and through them every helper). Outputs match inputs for: identifiers (all 5 kinds), attributes, URL/XREF/ENTREZ links, collaborators, related-projects (PARENT case), `FirstPublic` date, `StudyDescriptor` with enum + nullable long + RelatedStudies, library strategy/source/selection enums, paired layout, Platform `ILLUMINA → IlluminaNovaSeq6000`, `RunDate` timestamp, ExperimentRef. FK enforcement verified (bogus parent rejected). `listAccessions` works for all five entities. Cascade delete behaves correctly with `bioproject → study` being `ON DELETE SET NULL` (so dropping a BioProject leaves orphan studies, by design).
354360

355361
### Chunk 5 — Tests
356362

‎src/BioFSharp.INSDC.SQLite/BioFSharp.INSDC.SQLite.fsproj‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -14,6 +14,13 @@
1414
<Compile Include="Internal\Attributes.fs" />
1515
<Compile Include="Internal\Links.fs" />
1616
<Compile Include="Internal\References.fs" />
17+
<Compile Include="Internal\Platforms.fs" />
18+
<Compile Include="Schema.fs" />
19+
<Compile Include="BioProject.fs" />
20+
<Compile Include="Study.fs" />
21+
<Compile Include="BioSample.fs" />
22+
<Compile Include="Experiment.fs" />
23+
<Compile Include="Run.fs" />
1724
<EmbeddedResource Include="schema\insdc_schema.sql" />
1825
<None Include="..\..\docs\img\icon.png" Pack="true" PackagePath="" />
1926
<None Include="..\..\README.md" Pack="true" PackagePath="" />
Lines changed: 178 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,178 @@
1+
namespace BioFSharp.INSDC.SQLite
2+
3+
open System
4+
open Microsoft.Data.Sqlite
5+
open BioFSharp.FileFormats.INSDC
6+
open BioFSharp.INSDC.SQLite.Internal
7+
8+
/// SQLite persistence for `BioProject`. Each public function maps a single
9+
/// in-memory record to the full set of rows it spans (core row + identifiers
10+
/// + attributes + links + collaborators + related-projects).
11+
module BioProject =
12+
13+
[<Literal>]
14+
let private accessionColumn = "bioproject_accession"
15+
16+
let private identifierOwner accession : IdentifierOwner =
17+
{ Table = "bioproject_identifiers"; AccessionColumn = accessionColumn; Accession = accession }
18+
19+
let private attributeOwner accession : AttributeOwner =
20+
{ Table = "bioproject_attributes"; AccessionColumn = accessionColumn; Accession = accession }
21+
22+
let private linkOwner accession : LinkOwner =
23+
{ Table = "bioproject_links"; AccessionColumn = accessionColumn; Accession = accession }
24+
25+
let private formatDate (date: Nullable<DateTime>) : obj =
26+
if date.HasValue then box (date.Value.ToString("yyyy-MM-dd")) else null
27+
28+
let private parseDate (text: string) : Nullable<DateTime> =
29+
if isNull text then Nullable() else Nullable(DateTime.Parse(text))
30+
31+
let private insertCore (connection: SqliteConnection) (project: BioProject) : unit =
32+
Sql.execNonQuery
33+
connection
34+
"INSERT INTO bioproject (accession, alias, center_name, broker_name, name, title, description, first_public) \
35+
VALUES (@acc, @alias, @cn, @bn, @name, @title, @desc, @fp);"
36+
[
37+
"@acc", box project.Accession
38+
"@alias", box project.Alias
39+
"@cn", box project.CenterName
40+
"@bn", box project.BrokerName
41+
"@name", box project.Name
42+
"@title", box project.Title
43+
"@desc", box project.Description
44+
"@fp", formatDate project.FirstPublic
45+
]
46+
|> ignore
47+
48+
let private insertCollaborators (connection: SqliteConnection) (project: BioProject) : unit =
49+
if isNull project.Collaborators then () else
50+
project.Collaborators
51+
|> Seq.iteri (fun i name ->
52+
Sql.execNonQuery
53+
connection
54+
"INSERT INTO bioproject_collaborators (bioproject_accession, ordinal, name) VALUES (@acc, @ordinal, @name);"
55+
[
56+
"@acc", box project.Accession
57+
"@ordinal", box i
58+
"@name", box name
59+
]
60+
|> ignore)
61+
62+
let private insertRelatedProjects (connection: SqliteConnection) (project: BioProject) : unit =
63+
if isNull project.RelatedProjects then () else
64+
project.RelatedProjects
65+
|> Seq.iteri (fun i related ->
66+
let kind, target =
67+
if not (isNull related.ParentProject) then "PARENT", related.ParentProject.Accession
68+
elif not (isNull related.ChildProject) then "CHILD", related.ChildProject.Accession
69+
elif not (isNull related.PeerProject) then "PEER", related.PeerProject.Accession
70+
else
71+
failwithf
72+
"BioProject %s related-project at ordinal %d has none of PARENT/CHILD/PEER set"
73+
project.Accession i
74+
Sql.execNonQuery
75+
connection
76+
"INSERT INTO bioproject_related_projects (bioproject_accession, ordinal, kind, related_accession) \
77+
VALUES (@acc, @ordinal, @kind, @target);"
78+
[
79+
"@acc", box project.Accession
80+
"@ordinal", box i
81+
"@kind", box kind
82+
"@target", box target
83+
]
84+
|> ignore)
85+
86+
/// Persists `project` and every row it deconstructs into. Wraps all
87+
/// inserts in a single transaction so FK or constraint violations leave
88+
/// the database untouched.
89+
let insert (connection: SqliteConnection) (project: BioProject) : unit =
90+
Sql.withTransaction connection (fun _tx ->
91+
insertCore connection project
92+
Identifiers.write connection (identifierOwner project.Accession) project.Identifiers
93+
Attributes.write connection (attributeOwner project.Accession) project.ProjectAttributes
94+
Links.write connection (linkOwner project.Accession) project.ProjectLinks
95+
insertCollaborators connection project
96+
insertRelatedProjects connection project)
97+
98+
/// Reconstructs a `BioProject` from its accession by joining all owner
99+
/// tables. Returns `None` when no core row exists.
100+
let tryGet (connection: SqliteConnection) (accession: string) : BioProject option =
101+
let core =
102+
Sql.tryQueryOne
103+
connection
104+
"SELECT alias, center_name, broker_name, name, title, description, first_public \
105+
FROM bioproject WHERE accession = @acc;"
106+
[ "@acc", box accession ]
107+
(fun reader ->
108+
let alias = Sql.readStringOrNull reader 0
109+
let centerName = Sql.readStringOrNull reader 1
110+
let brokerName = Sql.readStringOrNull reader 2
111+
let name = Sql.readStringOrNull reader 3
112+
let title = Sql.readStringOrNull reader 4
113+
let description = Sql.readStringOrNull reader 5
114+
let firstPublic = Sql.readStringOrNull reader 6
115+
alias, centerName, brokerName, name, title, description, firstPublic)
116+
match core with
117+
| None -> None
118+
| Some (alias, centerName, brokerName, name, title, description, firstPublic) ->
119+
let project = BioProject()
120+
project.Accession <- accession
121+
project.Alias <- alias
122+
project.CenterName <- centerName
123+
project.BrokerName <- brokerName
124+
project.Name <- name
125+
project.Title <- title
126+
project.Description <- description
127+
project.FirstPublic <- parseDate firstPublic
128+
project.Identifiers <- Identifiers.read connection (identifierOwner accession)
129+
for attr in Attributes.read connection (attributeOwner accession) do
130+
project.ProjectAttributes.Add(attr)
131+
for link in Links.read connection (linkOwner accession) do
132+
project.ProjectLinks.Add(link)
133+
Sql.queryAll
134+
connection
135+
"SELECT name FROM bioproject_collaborators WHERE bioproject_accession = @acc ORDER BY ordinal;"
136+
[ "@acc", box accession ]
137+
(fun reader -> reader.GetString(0))
138+
|> List.iter project.Collaborators.Add
139+
Sql.queryAll
140+
connection
141+
"SELECT kind, related_accession FROM bioproject_related_projects \
142+
WHERE bioproject_accession = @acc ORDER BY ordinal;"
143+
[ "@acc", box accession ]
144+
(fun reader ->
145+
let kind = reader.GetString(0)
146+
let target = reader.GetString(1)
147+
let related = BioProjectRelatedProjectsRelatedProject()
148+
match kind with
149+
| "PARENT" ->
150+
related.ParentProject <- BioProjectRelatedProjectsRelatedProjectParentProject(Accession = target)
151+
| "CHILD" ->
152+
related.ChildProject <- BioProjectRelatedProjectsRelatedProjectChildProject(Accession = target)
153+
| "PEER" ->
154+
related.PeerProject <- BioProjectRelatedProjectsRelatedProjectPeerProject(Accession = target)
155+
| other ->
156+
failwithf "Unexpected related-project kind '%s' for bioproject '%s'" other accession
157+
related)
158+
|> List.iter project.RelatedProjects.Add
159+
Some project
160+
161+
/// Removes the row from `bioproject`; every owned table's rows are deleted
162+
/// via the schema's ON DELETE CASCADE.
163+
let delete (connection: SqliteConnection) (accession: string) : unit =
164+
Sql.execNonQuery
165+
connection
166+
"DELETE FROM bioproject WHERE accession = @acc;"
167+
[ "@acc", box accession ]
168+
|> ignore
169+
170+
/// Lists every BioProject accession currently stored, in lexicographic
171+
/// order. Suitable for crawler resume / dedup bookkeeping.
172+
let listAccessions (connection: SqliteConnection) : string seq =
173+
Sql.queryAll
174+
connection
175+
"SELECT accession FROM bioproject ORDER BY accession;"
176+
[]
177+
(fun reader -> reader.GetString(0))
178+
:> string seq
Lines changed: 127 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,127 @@
1+
namespace BioFSharp.INSDC.SQLite
2+
3+
open Microsoft.Data.Sqlite
4+
open BioFSharp.FileFormats.INSDC
5+
open BioFSharp.INSDC.SQLite.Internal
6+
7+
/// SQLite persistence for `BioSample`. The taxonomic identity sits in its own
8+
/// child table (`biosample_name`); everything else follows the standard
9+
/// identifiers/attributes/links pattern.
10+
module BioSample =
11+
12+
[<Literal>]
13+
let private accessionColumn = "biosample_accession"
14+
15+
let private identifierOwner accession : IdentifierOwner =
16+
{ Table = "biosample_identifiers"; AccessionColumn = accessionColumn; Accession = accession }
17+
18+
let private attributeOwner accession : AttributeOwner =
19+
{ Table = "biosample_attributes"; AccessionColumn = accessionColumn; Accession = accession }
20+
21+
let private linkOwner accession : LinkOwner =
22+
{ Table = "biosample_links"; AccessionColumn = accessionColumn; Accession = accession }
23+
24+
let private insertCore (connection: SqliteConnection) (sample: BioSample) : unit =
25+
Sql.execNonQuery
26+
connection
27+
"INSERT INTO biosample (accession, alias, center_name, broker_name, title, description) \
28+
VALUES (@acc, @alias, @cn, @bn, @title, @desc);"
29+
[
30+
"@acc", box sample.Accession
31+
"@alias", box sample.Alias
32+
"@cn", box sample.CenterName
33+
"@bn", box sample.BrokerName
34+
"@title", box sample.Title
35+
"@desc", box sample.Description
36+
]
37+
|> ignore
38+
39+
let private insertSampleName (connection: SqliteConnection) (sample: BioSample) : unit =
40+
match sample.SampleName with
41+
| null -> ()
42+
| name ->
43+
Sql.execNonQuery
44+
connection
45+
"INSERT INTO biosample_name (biosample_accession, taxon_id, scientific_name, common_name, display_name) \
46+
VALUES (@acc, @tid, @sci, @common, @display);"
47+
[
48+
"@acc", box sample.Accession
49+
"@tid", box name.TaxonId
50+
"@sci", box name.ScientificName
51+
"@common", box name.CommonName
52+
"@display", box name.DisplayName
53+
]
54+
|> ignore
55+
56+
/// Persists `sample` and every row it deconstructs into.
57+
let insert (connection: SqliteConnection) (sample: BioSample) : unit =
58+
Sql.withTransaction connection (fun _tx ->
59+
insertCore connection sample
60+
insertSampleName connection sample
61+
Identifiers.write connection (identifierOwner sample.Accession) sample.Identifiers
62+
Attributes.write connection (attributeOwner sample.Accession) sample.SampleAttributes
63+
Links.write connection (linkOwner sample.Accession) sample.SampleLinks)
64+
65+
let private readSampleName (connection: SqliteConnection) (accession: string) : BioSampleName =
66+
Sql.tryQueryOne
67+
connection
68+
"SELECT taxon_id, scientific_name, common_name, display_name \
69+
FROM biosample_name WHERE biosample_accession = @acc;"
70+
[ "@acc", box accession ]
71+
(fun reader ->
72+
BioSampleName(
73+
TaxonId = reader.GetInt32(0),
74+
ScientificName = Sql.readStringOrNull reader 1,
75+
CommonName = Sql.readStringOrNull reader 2,
76+
DisplayName = Sql.readStringOrNull reader 3))
77+
|> Option.defaultValue null
78+
79+
/// Reconstructs a `BioSample` from its accession by joining every owner
80+
/// table. Returns `None` when no core row exists.
81+
let tryGet (connection: SqliteConnection) (accession: string) : BioSample option =
82+
let core =
83+
Sql.tryQueryOne
84+
connection
85+
"SELECT alias, center_name, broker_name, title, description \
86+
FROM biosample WHERE accession = @acc;"
87+
[ "@acc", box accession ]
88+
(fun reader ->
89+
Sql.readStringOrNull reader 0,
90+
Sql.readStringOrNull reader 1,
91+
Sql.readStringOrNull reader 2,
92+
Sql.readStringOrNull reader 3,
93+
Sql.readStringOrNull reader 4)
94+
match core with
95+
| None -> None
96+
| Some (alias, centerName, brokerName, title, description) ->
97+
let sample = BioSample()
98+
sample.Accession <- accession
99+
sample.Alias <- alias
100+
sample.CenterName <- centerName
101+
sample.BrokerName <- brokerName
102+
sample.Title <- title
103+
sample.Description <- description
104+
sample.SampleName <- readSampleName connection accession
105+
sample.Identifiers <- Identifiers.read connection (identifierOwner accession)
106+
for attr in Attributes.read connection (attributeOwner accession) do
107+
sample.SampleAttributes.Add(attr)
108+
for link in Links.read connection (linkOwner accession) do
109+
sample.SampleLinks.Add(link)
110+
Some sample
111+
112+
/// Removes the row from `biosample`; cascades through every owned table.
113+
let delete (connection: SqliteConnection) (accession: string) : unit =
114+
Sql.execNonQuery
115+
connection
116+
"DELETE FROM biosample WHERE accession = @acc;"
117+
[ "@acc", box accession ]
118+
|> ignore
119+
120+
/// Lists every BioSample accession in the database, lexicographically.
121+
let listAccessions (connection: SqliteConnection) : string seq =
122+
Sql.queryAll
123+
connection
124+
"SELECT accession FROM biosample ORDER BY accession;"
125+
[]
126+
(fun reader -> reader.GetString(0))
127+
:> string seq

0 commit comments

Comments
 (0)