Add SEC Form 13F institutional holdings as the first Whales dataset - #15
Merged
Merged
Conversation
JosueNina
force-pushed
the
feature-sec-whales
branch
from
September 18, 2026 20:34
638e433 to
9e6a36d
Compare
…publishing every position as its manager filed it
JosueNina
force-pushed
the
feature-sec-whales
branch
from
September 18, 2026 20:38
9e6a36d to
2acf863
Compare
The EDGAR day archive lacked the cover page and four columns the processor reads, so every daily run threw. An incremental run published each touched ticker's zip with that day's entry alone. Both now have end to end tests. Processor - Read the cover page and the missing line columns from EDGAR filings, and list each filing once per day - Add an incremental run's dates to a copy of the published zip - Drop the per ticker date index; the history guard counts zips - Leave a field the filing left empty empty, rather than writing a zero - Join OtherManager with single semicolons - Hold only share lines against the close when deciding the value unit - Give an option line that names no side the one its CUSIP states, and resolve options on fund families by the price of the line - Resolve a CUSIP no ISIN can hold to nothing instead of stopping the run - Delete staging files as they are consumed, which halves peak disk use - Split unresolved CUSIPs into foreign (CINS) and domestic in the summary - Share the environment and config handling between both datasets Data class - Add ManagerName, read from managers.csv by SEC13FManagerNameProvider - Remove protobuf, which this dataset does not use Also removes the unused 13D/13G resolver and fixtures, follows Pershing Square across its two CIKs in the demonstration algorithms, replaces the samples with the data they read, and documents the data folder dependencies in the README. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…reads The published manager names could not be read back as a CSV, an incremental run stepped over the days an outage had missed and reported success, and the daily path took the filer from the index rather than from the filing. All three now have tests. Processor - Take the commas out of a manager name where the cover page is read, and again where managers.csv is written, since the names folded in were published carrying theirs - Read the filer's CIK from the primary document, falling back to the index, so the daily path and the data sets name the same manager - Reach back to the day after the last one folded in, not only the ten day lookback, so the days a run missed while EDGAR blocked it are caught up rather than skipped - Keep the manager name of the latest filing rather than of the last row read, so a day whose index came late does not roll the name back - Let an option line resolved by price take that price's unit, the way a share line takes its own: its filing has no share line to read when every line is an option - Stop taking a managers.csv that could not be read for the day's answer, which left every name null until midnight over one missed fetch Tests - Fold a real EDGAR day into a real published history, since every other test builds the day's archive with this repository's own writer and reads it back the same way Also drops comments left behind by the universe this PR removed and a byte order mark on the processor that no other file carries, and rewrites the published sample the way the processor now writes it.
JosueNina
force-pushed
the
feature-sec-whales
branch
from
September 21, 2026 17:14
cd4c2e6 to
e464735
Compare
A second review round found five defects, four of them in the commit before this one. One filing could state its option lines in two units, the manager name still rolled back across runs, the catch-up had no ceiling, a day that staged no row lost the names it read, and a test answered differently depending on the order NUnit ran it in. Processor - Let an option line resolved by price keep its filing's unit. The close names the fund and nothing else: an option line has no price of its own, so reading the unit off the close it matched published one manager's option book a thousandfold apart, since a line on a single-issue CUSIP never sees a close. Reverts that part of the commit before this one and restores the one policy, that only a share line is held against a close - Keep the published manager name when this run folds in a day older than the last one an earlier run folded in. managers.csv carries no filing date, so a day whose index came late would otherwise overwrite the newer name already published, and it would stay wrong until the manager filed again - Read at most five EDGAR days in a daily run. Nothing is published until every day of the gap is read, so an outage wider than the run's hour failed the run, lost its downloads and left the next run one day further behind, for good. The rebuild is not capped: it states in edgar-days.txt the day it reaches - Write managers.csv on a day whose filings all failed to resolve. The day is recorded as folded in whatever came out of it, so no later run reads its cover pages again and a name skipped here is lost until that manager files a line that resolves Data class - Drop the loaded day in SEC13FManagerNameProvider.Reset() along with the names, so a test is not answered with the day an earlier test in the same process stamped Tests - A gap wider than one run is closed over the runs that follow, and the rebuild still reads every weekday of its window - The published name survives a late day folded in after a newer one, and a day that staged no row still publishes the managers it read - The provider test stamps the day itself rather than relying on the order it is run in The published sample is unchanged: the option unit read off a close is only ever taken on a fund family's CUSIP, and every ticker in the sample is a single-issue issuer.
… fetches A third review round found both fixes of the commit before this one. The guard against a rolled back manager name held every manager against a single run wide date, so it threw the name away whenever the late day carried that manager's latest filing, which on a late deadline day is most of them. The catch up was bounded by a count of days, but a day costs a hundredfold more than another, so five days can be the three heaviest of the quarter. Processor - Take the date of a published manager name from the published rows rather than from the last day an earlier run folded in. The rows carry the CIK and sit in an entry named by their filing date, so a manager that filed again between the filing this run read and what is published keeps its published name, and every other manager is named by the filing this run read. The scan is reached only by a run folding in a day older than the last one folded in, which is a day whose index EDGAR published late - Bound a daily run by the holdings filings it fetches rather than by a count of days. Each filing is its own round trip, and over the June to August 2026 quarter a day carries 65 at the median, 312 at the ninetieth percentile and 1,835 on the 45 day deadline of 2026-08-14, so a count of days bounds nothing. The budget is that heaviest day, which an ordinary run already carries every quarter - Weigh a day before fetching any of it: its index is one round trip and names every filing it holds, so a day that does not fit what the run has left is left whole for the next run instead of read in half. A run always reads its first day, since a day cannot be split and one heavier than the budget would otherwise never be read at all Tests - A manager that filed again keeps its published name and one that did not takes the name of the late day; the second fails on the commit before this one - A manager is named by its latest filing when two late days are folded in at once, which is the only case where the date read off the published rows decides rather than their mere presence: with a one day window the two tests above pass on a scan that keeps no dates - A gap wider than one run closes over the runs that follow with none of them fetching more than the budget or its first day, whichever is heavier, walked with the filings per day measured over the quarter. The looser bound this asserted first was passed by a run that reached the first day exemption twice - A day that does not fit is left whole and not one of its filings is fetched The published sample is unchanged: a rebuild reaches neither the scan nor the budget.
…ing ending the run An EDGAR daily index can name a filing whose own date is an earlier day: one 13F-HR across the 2026 Q2 and Q3 indexes, filed 2026-04-27 and listed on 04-28. Reading that day, the run had rows for 04-27, deleted the published entry for it and wrote back the one manager it had just read, so every other manager's positions of 04-27 were lost from every ticker that bank reports, and the run returned success. A full rebuild never hit it, since it writes each zip once with its rows already grouped by date. Processor - Add this run's rows to a published date rather than replacing the date. What is published for a date can come from filings this read does not carry: the daily index and the quarterly data sets do not name the same filings for a day. Only the lines of the filings being written are dropped, so a filing read twice is published once and a day re-read after a publish that failed half way is still idempotent - Stamp the rows of a daily archive with the day its index named them, not with the date the filing states. A back-dated filing was not public until the index carried it, so taking its own date would let a backtest read it a day early. The cost is that a rebuild reads that accession from the data sets, which state the earlier date, and republishes the row a day before the daily path put it - Skip a filing this reader cannot read and carry on with the day. Thrown, it failed the day, the day was never recorded and every run after it met the same filing: the data set stopped until someone shipped code. A day whose filings are unreadable beyond one and five percent is a layout change and still throws, and a network failure is not caught, so the day is still retried - Format a whole VALUE rather than casting it to a long, which threw on the nonsense filers type into that column and stopped the run over one line - Collapse the whitespace a sanitised name is left with, so a manager who writes "Pershing Square Capital Management, L.P." is not published with two spaces Data class and demonstration algorithms - Treat a name the managers stopped reporting as a report of no shares. A position sold out of has no line in the new quarter, so its newest period stays behind the newest the manager reported anywhere; read as the name's own latest quarter, it went on being compared with the quarter before it and the algorithm held the name forever - Guard the share change against a quarter that reported none, which divided by zero and ended the algorithm in C# - Correct the comment on Reader: LEAN takes an exception out of it as a reader error and drops the line, so throwing would turn a silent skip into a logged one and nothing more Tests - An incremental run keeps the published rows of a date it writes into, and a filing read twice is published once; both fail on the commit before this one - A daily archive stamps its rows with the day it was read - A filing that cannot be read is skipped, and a day of them still throws - A VALUE past long.MaxValue is published rather than thrown on The sample under output/alternative/sec/13f is regenerated: 47 of its 74,309 rows carried a title with two spaces, and managers.csv is cut to the CIKs its zips reference.
Sanitize collapses the whitespace its own replacements leave, so the committed sample no longer matched what the processor writes. Regenerated from a full rebuild, over the window the sample has always carried, 2026-03-02 to 2026-09-18. - 47 of the sample's 74,309 rows carried a title with two spaces, and managers.csv 6,251 of its 13,278 names - One title carried a non-breaking space rather than a run of plain ones, CL<U+00A0>A, which collapsing normalises to a plain space. It is one row in the 863,538 these seven tickers hold over the whole history, and a column read as text is better off without it - managers.csv is cut to the 7,057 CIKs the sample's zips reference, down from 13,278, which takes the sample from 2.1 MB to 1.8 MB Nothing else moved: entry for entry and line for line, every other row is the one published before, so the sample still shows what a rebuild produces and not a hand-edited file.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
First source of the Whales product. Publishes every position an institutional manager reports on Form 13F, as filed: one record per line of an information table, with the manager, the quarter it reports and the position. No holder counts or share totals, because no filing states them; counting is left to the algorithm.
A point is a SEC13FHoldings collection holding every position reported for a security on one filing date. One zip per security holds one entry per date, under alternative/sec/13f, which keeps the dataset at a file per security instead of the eight and a half million a loose file per date would take. A managers.csv beside it names each CIK, which is why no published field may carry a comma.
Checked against the SEC tables by an independent reader that imports nothing from the processor: 2,082,893 published rows over the December 2020 quarter, across 8,115 securities, no differences. 96.6 percent of the reported lines resolved to a security and were published; the rest are mostly foreign issuers LEAN does not list.
The run reads three things from the LEAN data folder and downloads none of them: the map files, the security database, and the coarse universe files, of which it needs only the last trading day of each quarter, about four files a year. Without the closes it still publishes, falling back to the SEC's unit rule for each filing date, and coverage falls.
Schedules 13D and 13G follow in their own PR once this merges, since they build on the shared EDGAR code this adds.