Conversation
JosueNina
force-pushed
the
feature-sec-13f-institutional-holdings
branch
from
September 10, 2026 15:06
f28d740 to
bff67db
Compare
…ale universe rows
…os and the incremental run
…nd the crosswalk rename gap
…h them and check units and crosswalk matches against the close
… be checked and regenerate the samples
…w filers' listed filings answered 404 for over a minute
… one trading under its own ticker, since the database repeats identifiers across a company's listings
…t, since the self-hosted runners keep it between runs
…dds the holders that reported old CUSIPs the crosswalk does not know
…n one point per quarter LEAN hands an algorithm a single data point per security per timestamp, so a release that restated several quarters reached it as one of them and the rest were dropped. Measured over the processed history, 7% of the 8.5M (security, release) points carry more than one quarter, up to 75, which is 1.7M of the 10.2M rows that `slice[symbol]`, `slice.Get<T>()` and the typed `History<T>()` never showed. The universe had the same shape: a security with two live quarters occupied two records, so the pandas frame kept the first of them and the security cache the last. - `SEC13F` replaces `SEC13FHoldings` as the per security point and carries `Holdings`, one `SEC13FHolding` per restated quarter, oldest first. `MostReported` is the quarter the most managers have reported, which is the reading the DataFrame carries as columns and the safer default for ranking while a new quarter fills in; `Latest` is the newest and stays out of the frame. Each quarter exposes `Quarter`, a label such as 2020Q2 derived from `PeriodEnd`. - The point enumerates its quarters, so `history(..., flatten=True)` expands it into one row per quarter, and `[ProtoContract(IgnoreListHandling = true)]` keeps it serializable as a protobuf sub type, which a `BaseDataCollection` cannot be. The quarter is deliberately not a `BaseData` for the same reason. - `SEC13FUniverse` replaces `SEC13FHoldingsUniverse` and holds one `SEC13F` per security, so the selection function, the security cache and the flattened frame all see the same record. - A published line now reads `stamp,quarterCount,<10 columns per quarter>`, and a universe line `securityIdentifier,ticker,quarterCount,<10 columns per quarter>`. The reader derives the group width from the count, so a measure appended later leaves every existing file readable. The staging files the processor merges through are unchanged. - Demonstration algorithms, the listing documentation and the sample output follow. The documentation no longer claims that History returns every quarter, which was true of the DataFrame and never of the typed History, and no longer tells the reader to iterate AllData. - The security database is assumed to be in place, as the map files are, rather than checked for at startup: the processor reads it through TryRead and carries on with an empty list when it is missing, which is what lets the tests run without it.
Contributor
Author
|
Superseded by #15, which redesigns the dataset to publish each manager's positions as filed instead of summed totals, and ships it as the first Whales dataset. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This is a first version of the SEC Form 13F institutional holdings dataset, the quarterly record of how much of a company institutions hold and how many of them hold it.
One BaseData class, SEC13FHoldings, plus its universe class. Eight measures per security per publication day: Holders, Shares, HoldingValue (also the data point's Value), CallShares, PutShares, PrincipalValue, VotingSole and VotingShared, plus a ConfidentialOmitted flag. Coverage is 6,616 US Equities from May 2013, 6,740,981 rows.
The data comes from the SEC's own Form 13F structured data sets, which are TSV tables the SEC extracts from the filings, so nothing here parses XML. The measures are cumulative per quarter and the reported quarter travels in PeriodEnd, since the managers of one quarter file across roughly fifty days and one publication day carries several quarters. The processor reads dataset-name from the config so the two SEC datasets can live side by side.
Holders is a distinct count of managers, so unlike the summed measures it cannot be rebuilt from the published totals. The run publishes the filers it counted in a filers.zip beside the security files and reads it back on the next one, 290 MB against 2.3 GB of data. A nightly rebuild from the SEC archives would drop it, but /raw is not restored between runs, so that would mean refetching about 5 GB every night.
It also ships demo algorithms and universe demos in C# and Python, plus unit tests. I ran the four of them in LEAN over the shipped sample and they pull the data and run through a backtest fine.