diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..2106871 --- /dev/null +++ b/.gitignore @@ -0,0 +1,7 @@ +# runtime state - the database, logs and tickerplant logs are all regenerated +var/ + +# editor / tooling +.vscode/ +.claude/ +*.swp diff --git a/README.md b/README.md index 6c19c87..58d7cc0 100644 --- a/README.md +++ b/README.md @@ -1,2 +1,261 @@ -# TorQ-Virtual-Tables -Utilising KDB-X Virtual Tables to build different format of data capture +# TorQ Virtual-Table Capture Pack + +An application overlay for [TorQ](https://github.com/DataIntellectTech/TorQ) that captures +data partitioned by **date and instrument**, and serves it through kdb-x virtual tables. + +There is no RDB, no HDB, no gateway and no sort process. The writer writes where the readers +read, and nothing moves at end of day. + +## Why + +A normal kdb+ stack partitions by date and applies a `p#` attribute to `sym` overnight so +that selective lookups are fast. That attribute cannot be maintained on a partition being +appended to, so the live day is always un-indexed and a query like ``where sym=`AMD`` +scans it. + +This pack makes the instrument a *directory* instead. A selective lookup becomes a directory +lookup rather than a scan, it needs no index, and there is nothing to rebuild at end of day. + +See `docs/virtual-table-capture-pack.md` — start with §0, which lists each design decision +alongside the alternative that was tried and rejected. + +## Requirements + +- A TorQ checkout (5.2.x) +- kdb-x, with the `kx.pq.t` virtual-table module on `QPATH`. Not kdb+ 4.x: the reader + binds `mkP` with ``use`kx.pq.t``, and `use` is a kdb-x keyword + +## Setup + +Point `TORQHOME` at your TorQ checkout, either in the environment: + +```sh +export TORQHOME=/path/to/TorQ +``` + +or by filling in the one line in `vt-env.sh` that is deliberately left empty: + +```sh +export TORQHOME="${TORQHOME:-}" # -> ${TORQHOME:-/path/to/TorQ} +``` + +Everything else derives from that and from the location of the pack, so it can be cloned +anywhere. `QHOME`, `QLIC` and `QPATH` are defaulted to the usual kdb-x locations under +`~/.kx` and can be overridden the same way. + +## Run + +```sh +./start.sh # discovery + tickerplant + WDB + IDB + feed +./stop.sh +``` + +Compression is a separate, occasional job — it exits when it finishes, so cron it for a quiet +window rather than running it under the stack. Two gates decide what it touches: an age tier +(`minage` in `appconfig/compressionconfig.csv`, 7 days) so recent data stays uncompressed and +fast to query, and a size gate (`.cmp.minfilesize` in `appconfig/settings/compression.q`) that +skips column files too small to free a filesystem block: + +```sh +./compress.sh --dry-run # what it would touch, and the ceiling on what it can free +./compress.sh # compress everything older than minage +``` + +Then watch it work: + +```sh +find var/db -mindepth 3 -maxdepth 3 -type d | head +tail -f var/logs/out_wdb1.log +``` + +And query it: + +```q +h:hopen `::6030:idb:pass +h"select n:count i by sym from trade" +h"select from trade where sym=`AMD, date=.vtidb.current" +``` + +The partition column is exposed under the name in `partitioncol` (`appconfig/settings/idb.q`), +set here to `sym` so queries read the same as against a conventional database. + +`.vtidb.current` is the partition the writer is filling. Prefer it to `.z.D` in examples: +if `rolltimeoffset` is set, the business day ends somewhere other than midnight and the two +disagree for part of every day. + +Ask twice a few seconds apart and the counts move. Nothing was reloaded — each partition is +opened as a live view, so rows the writer appends are visible immediately. + +## Self test + +```sh +./selftest.sh +``` + +Publishes a brand new instrument to the tickerplant and checks it travels the whole chain: +the writer creates a partition directory, notifies, the reader rebuilds, the rows come back +through the virtual table, the partition column is *not* in the files, and plain appends need +no rebuild. Exits non-zero on failure, so it can be wired into a smoke test. + +## Regression + +```sh +./regress.sh # every test the environment allows +./regress.sh --quick # self-contained tests only +./regress.sh --no-mutate # skip the one test that rewrites files in var/db +``` + +Runs the sixteen assertion tests in `testfiles/` and summarises them. Ten work in a scratch +directory and never write to `var/db`, so they are safe to run at any time — though four of +them seed that scratch copy from a partition in `var/db`, and one loads TorQ's `timezone.q` +and `eodtime.q`, so the stack needs to have run at least once and `TORQHOME` must be set. The +other six need the stack up, and are skipped with a note if nothing is listening on the IDB +port (`KDBBASEPORT`+30, so 6030 by default). Exits non-zero if any test fails, and prints the +log path for each failure. + +A test that *cannot run yet* exits **77** and is reported as `SKIP` rather than `FAIL`: the +database has no partitions, too few instruments, or only one date. On a clone whose stack has +been up for a few minutes that is the expected state of `vt-compress-test`, which needs the +stack to have crossed a day boundary. Anything reported as `FAIL` is a real failure. + +Two of them need explaining. `vt-compress-test` goes through `./compress.sh --test`, which +swaps in a 1-day age tier — and it leaves those partitions compressed, which is the one thing +in the suite that changes `var/db`. Re-running is still safe: `vt-damage-test` copies a +partition and truncates a column in it, and a compressed column raises where an uncompressed +one short-reads, so it forces its copy back to uncompressed first and asserts the same thing +on every run. `vt-compare-kdb` is checked against a recorded baseline of +**16 matched, 3 differed** rather than its exit code, because it exits non-zero whenever any +difference exists and three are expected (see §9 of the doc). + +## Layout + +``` +vt-env.sh the only file to edit: TORQHOME, then everything derives +start.sh / stop.sh bring the stack up and down. stop.sh is scoped by path, so it + leaves other TorQ stacks on the machine alone even though they + share the default procnames +selftest.sh end-to-end smoke test (code/selftest.q) +regress.sh runs the sixteen assertion tests in testfiles/ +loadtest.sh throughput run on a clean stack (code/loadtest.q). DESTRUCTIVE: + it does rm -rf var to start from a known state, so run it on a + throwaway copy unless you mean to lose the database +compress.sh the weekend compression job; --dry-run and --test +database.q the schema the tickerplant loads + +appconfig/ + process.csv the process list + sort.csv declares the partition column (sym) + compressionconfig.csv the age tier: how old a partition must be before compression + compressionconfig-test.csv a 1-day copy, used by ./compress.sh --test + passwords/ accesslist.txt and feed.txt - the stock TorQ demo credentials + settings/default.q settings shared by every process + settings/wdb.q WDB config, including symdomain (see §8.3.1 for multi-stack) + settings/idb.q IDB config + settings/compression.q the size gate (.cmp.minfilesize) + settings/feed.q demo feed config + settings/segmentedtickerplant.q + +code/ + wdb/vtwrite.q the writer overrides this design needs (see §4 of the doc) + processes/vtidb.q the IDB reader (see §5 of the doc) + processes/vtcompress.q the compression job (see §7 of the doc) + processes/vtcompress-report.q the --dry-run report, loaded by vtcompress.q + tick/feed.q demo feed, FSP trade/quote generator + tick/loadfeed.q the load-test feed: no timers, driven as fast as it will go + selftest.q the end-to-end check run by ./selftest.sh + loadtest.q the load driver and measurement run by ./loadtest.sh + +docs/ the architecture document and the status report +testfiles/ tests and evidence scripts - see below +var/ created at runtime: db/, logs/, tplogs/ +``` + +`testfiles/` holds two different kinds of script, and only the first kind asserts: + +``` +assertions (16) run by ./regress.sh, each exits non-zero on failure + self-contained (10) vt-partition vt-rollover vt-newtable vt-restart vt-inflight + vt-damage vt-multistack vt-wdbrestart vt-compat vt-diskfull + need the stack (4) vt-replay vt-symdomain vt-collision vt-tprestart + special (2) vt-compress-test (via ./compress.sh --test) + vt-compare-kdb.sh (drives vt-kdb-prep.q and vt-kdb-compare.q) + +evidence (9) measurements and probes, read for their output, not pass/fail + vt-compress-ab before/after on the same partition + vt-compress-ratio compression ratio by column + vt-compress-sizes file-size distribution against the filesystem block + vt-scale-test partition count against mapping cost + vt-sym-concurrency two writers against one sym file + vt-gap-test what a missing partition directory actually does + vt-limitations what virtual tables do not support + vt-probe an annotated tour of the on-disk structures + vt-sample-legacy the legacy single-directory sample, kept for comparison +``` + +`var/db` is the whole database — live and historical data in one directory. + +## What is on disk + +``` +var/db/ + sym + 2026.08.17/ + trade/ + AMD/ time price size stop cond ex side + AAPL/ time price size stop cond ex side + quote/ + AMD/ time bid ask bsize asize mode ex src +``` + +Note that `sym` is **not** among the columns. It is carried by the directory name. +That is deliberate and load-bearing: a column stored inside the files can never be used to +skip directories, so leaving it in would make every query on it scan the whole database. + +## Evidence + +The scripts under `testfiles/` are runnable and reproduce the findings behind the design: + +```sh +q testfiles/vt-probe.q # how the query engine routes conditions +q testfiles/vt-limitations.q # what works and what does not, presentable output +q testfiles/vt-sample-legacy.q # attaching an existing date-partitioned HDB +q testfiles/vt-compress-ratio.q # where compression's saving lands, by original file size +q testfiles/vt-compress-sizes.q # how that saving scales with rows per instrument per day +q testfiles/vt-compress-ab.q # uncompressed vs gated vs ungated: disk and latency +q testfiles/vt-rollover-test.q # end of day keeps the date it just closed, without rescanning +q testfiles/vt-multistack-test.q # one reader over two capture stacks, all domain configurations +q testfiles/vt-replay-test.q # the overrides are in force during tickerplant log replay +q testfiles/vt-symdomain-test.q # rows arrive live; a new symbol value resolves within seconds +q testfiles/vt-collision-test.q # two instruments sharing a sanitised directory name +q testfiles/vt-damage-test.q # truncated / .d-less / corrupt partitions, and the blast radius +q testfiles/vt-newtable-test.q # a table appearing mid-life +q testfiles/vt-tprestart-test.q # liveness: subscribed, growing, no cached feed handle +q testfiles/vt-wdbrestart-test.q # the writer deletes the live partition, then replays it back +q testfiles/vt-restart-test.q # a reader restarting mid-flush, and where it takes the live date from +q testfiles/vt-inflight-test.q # querying a partition while it is being written +q testfiles/vt-diskfull-test.q # ENOSPC: what survives, what duplicates, what fails loudly +q testfiles/vt-partition-test.q # the partition column is in the name, not the files +q testfiles/vt-compat-test.q # 38 client operations: direct, wrapped, or unreachable +q testfiles/vt-scale-test.q # rebuild and query cost against partition count +q testfiles/vt-gap-test.q # a table missing a date the others have +q testfiles/vt-sym-concurrency.q # concurrent enumeration against one shared domain file +./testfiles/vt-compare-kdb.sh # same bytes, two databases: this layout vs stock kdb+ +./compress.sh --test # compression underneath a live reader: ratio and read cost +``` + +## Status + +The capture layer, the reader, compression, end of day and multi-stack read are built and +verified end to end: a new instrument published to the tickerplant is queryable through the +IDB 2 ms later, and the writer's appends need no reload at all. + +Results are checked against the same bytes loaded into a stock kdb+ database: **16 of 19 +queries identical, three raise an error, none silently different** +(`testfiles/vt-compare-kdb.sh`). Memory mappings are not a limit — the reader opens partitions +with a trailing slash, which does not memory-map, so the count stays flat however much history +is attached (§8.2). + +One item remains open and cannot be resolved inside the pack: a single virtual table cannot +span both the new format and existing date-partitioned history, because the column list is +taken from the first directory only. See §10 of the architecture document, and +`docs/status-report.md` for the ticket-level state. diff --git a/appconfig/compressionconfig-test.csv b/appconfig/compressionconfig-test.csv new file mode 100644 index 0000000..1d76e35 --- /dev/null +++ b/appconfig/compressionconfig-test.csv @@ -0,0 +1,2 @@ +table,minage,column,calgo,cblocksize,clevel +default,1,default,2,16,9 diff --git a/appconfig/compressionconfig.csv b/appconfig/compressionconfig.csv new file mode 100644 index 0000000..159d08f --- /dev/null +++ b/appconfig/compressionconfig.csv @@ -0,0 +1,2 @@ +table,minage,column,calgo,cblocksize,clevel +default,7,default,2,16,9 diff --git a/appconfig/passwords/accesslist.txt b/appconfig/passwords/accesslist.txt new file mode 100644 index 0000000..fce4efa --- /dev/null +++ b/appconfig/passwords/accesslist.txt @@ -0,0 +1,26 @@ +discovery:pass +feed:pass +gateway:pass +hdb:pass +housekeeping:pass +kill:pass +monitor:pass +rdb:pass +reporter:pass +sort:pass +tickerplant:pass +wdb:pass +chainedtp:pass +sortworker:pass +metrics:pass +vwapsub:pass +dqc:pass +dqcdb:pass +dqe:pass +dqedb:pass +segmentedtickerplant:pass +filealerter:pass +admin:admin +idb:pass +positions:pass +torquser:pass diff --git a/appconfig/passwords/feed.txt b/appconfig/passwords/feed.txt new file mode 100644 index 0000000..ee3b766 --- /dev/null +++ b/appconfig/passwords/feed.txt @@ -0,0 +1 @@ +feed:pass diff --git a/appconfig/process.csv b/appconfig/process.csv new file mode 100644 index 0000000..406585c --- /dev/null +++ b/appconfig/process.csv @@ -0,0 +1,7 @@ +host,port,proctype,procname,U,localtime,g,T,w,load,startwithall,extras,qcmd +localhost,{KDBBASEPORT}+1,discovery,discovery1,${TORQAPPHOME}/appconfig/passwords/accesslist.txt,1,0,,,${KDBCODE}/processes/discovery.q,1,,q +localhost,{KDBBASEPORT},segmentedtickerplant,stp1,${TORQAPPHOME}/appconfig/passwords/accesslist.txt,1,0,,,${KDBCODE}/processes/segmentedtickerplant.q,1,-schemafile ${TORQAPPHOME}/database.q -tplogdir ${KDBTPLOG},q +localhost,{KDBBASEPORT}+5,wdb,wdb1,${TORQAPPHOME}/appconfig/passwords/accesslist.txt,1,1,,,${KDBCODE}/processes/wdb.q,1,,q +localhost,{KDBBASEPORT}+14,feed,feed1,,1,0,,,${KDBAPPCODE}/tick/feed.q,1,,q +localhost,{KDBBASEPORT}+30,idb,idb1,${TORQAPPHOME}/appconfig/passwords/accesslist.txt,1,1,60,4000,${KDBAPPCODE}/processes/vtidb.q,1,-s 4,q +localhost,{KDBBASEPORT}+40,compression,cmp1,${TORQAPPHOME}/appconfig/passwords/accesslist.txt,1,0,,,${KDBAPPCODE}/processes/vtcompress.q,0,,q diff --git a/appconfig/settings/compression.q b/appconfig/settings/compression.q new file mode 100644 index 0000000..23d96e5 --- /dev/null +++ b/appconfig/settings/compression.q @@ -0,0 +1,24 @@ +// Virtual-table capture pack : compression config +// see docs/virtual-table-capture-pack.md §4.4 and §7 + +\d .cmp +hdbpath:hsym`$getenv`KDBHDB // one database root - the writer writes where the + // readers read, so this is the capture tree itself +maxage:365 // oldest partition to consider. the lower bound is + // minage in compressionconfig.csv - the age tier - + // and must stay >0 so the live partition is untouched + +minfilesize:4096 // the size gate: skip any column file this size or + // smaller. a file is allocated in whole filesystem + // blocks, so one that already fits in a block frees + // nothing and only adds work to every read. measured + // at 400 of 750 files - roughly half of + // them - on the reference partition (doc §7.3). + // set 0 to compress everything, as stock TorQ does + +// the hdbstructure override that teaches compression to see this layout is NOT here, even +// though it is configuration in spirit. settings files load ~13 ms BEFORE code/common/ +// compress.q, so anything defined here is silently overwritten by the stock definition and +// the job compresses nothing. it lives in code/processes/vtcompress.q instead, which is +// loaded after common code. +\d . diff --git a/appconfig/settings/default.q b/appconfig/settings/default.q new file mode 100644 index 0000000..cbc926f --- /dev/null +++ b/appconfig/settings/default.q @@ -0,0 +1,24 @@ +system"c 23 2000" + +// --------------------------------------------------------------------------- +// End-of-day roll time, applied to every process. +// +// The stack's business day is GMT (appconfig/settings/segmentedtickerplant.q), so with the +// default offset of zero the day rolls at midnight UTC. +// +// Set an offset to model a business day that ends elsewhere -- 0D09:00 rolls at 09:00 UTC, +// which is 17:00 in a UTC+8 timezone. +// +// NOTE an offset moves the business-day BOUNDARY, not just the event: with a 9h offset a UTC +// timestamp before 09:00 belongs to the previous business date. That is what the setting is +// for, but it does mean the current partition reads as the previous date until the roll +// fires -- and therefore that .z.D and the live partition disagree for offset hours a day. +// The reader takes the live partition from disk rather than .z.D for exactly this reason +// (docs/virtual-table-capture-pack.md 5.8). +// +// code/common/eodtime.q loads after this file and reads the value with @[value;...], so a +// setting here survives rather than being clobbered. +// --------------------------------------------------------------------------- +\d .eodtime +rolltimeoffset:0D00:00:00.000 +\d . diff --git a/appconfig/settings/feed.q b/appconfig/settings/feed.q new file mode 100644 index 0000000..f647660 --- /dev/null +++ b/appconfig/settings/feed.q @@ -0,0 +1,8 @@ +// Bespoke Feed config : Finance Starter Pack + +\d .servers +enabled:1b +CONNECTIONS:enlist `segmentedtickerplant // Feedhandler connects to the tickerplant +HOPENTIMEOUT:30000 + +\d . diff --git a/appconfig/settings/idb.q b/appconfig/settings/idb.q new file mode 100644 index 0000000..c1f8357 --- /dev/null +++ b/appconfig/settings/idb.q @@ -0,0 +1,38 @@ +// Virtual-table capture pack : IDB config +// see docs/virtual-table-capture-pack.md §5 + +\d .vtidb +roots:enlist hsym`$getenv`KDBDB // database roots to scan. a list rather than an atom + // so one reader can serve several capture stacks (§8.3) +tabs:` // ` = discover the table list from disk. the scan looks + // at the LIVE partition only (every date once, when + // the catalogue is empty), so it does not grow with + // retention. a table added to a date that has already + // rolled needs dropcache[] - see 6.1. + // set explicitly to restrict, e.g. `trade +historydays:0W // how many days back to attach. 0W = everything +sweep:0D00:00:30 // backstop rescan. the primary path is a notification + // from the wdb (§4.1); this only bounds the damage + // from a dropped message, so it is deliberately slack +symsweep:0D00:00:01 // how often to check whether the enumeration domain + // has grown. a new value in a data symbol column of + // an EXISTING partition creates no directory, so the + // writer never announces it, and the value reads as + // null until the domain is reloaded (§5.4). the check + // is one hcount per root, so this can be fast + +partitioncol:`sym // name the partition column is exposed under. + // the reader cannot derive it - the column is not + // stored on disk - so it must match the schema's + // name, or client queries will not port across +wdbtypes:`wdb +wdbcheckcycles:3 // wait this many cycles for the wdb, then start anyway. +wdbconnsleepintv:5 // the reader does not need the writer to function - + // without it the sweep keeps it current, just slower + +\d .servers +CONNECTIONS:`wdb`discovery // wdb: to register for new-partition notifications +STARTUP:1b + +\d .proc +loadprocesscode:0b // process code comes from -load, not $KDBAPPCODE/idb/ diff --git a/appconfig/settings/segmentedtickerplant.q b/appconfig/settings/segmentedtickerplant.q new file mode 100644 index 0000000..90a49c3 --- /dev/null +++ b/appconfig/settings/segmentedtickerplant.q @@ -0,0 +1,21 @@ +\d . + +createlogs:1b; // create a logs + +\d .stplg + +multilog:`tabperiod; // [tabperiod|singular|periodic|tabular|custom] +multilogperiod:0D01; +errmode:1b; +batchmode:`defaultbatch; // [memorybatch|defaultbatch|immediate] +customcsv:hsym first .proc.getconfigfile["stpcustom.csv"]; +replayperiod:`day // [period|day|prior] + +\d .proc + +loadprocesscode:1b; + +\d .eodtime + +datatimezone:`$"GMT"; +rolltimezone:`$"GMT"; diff --git a/appconfig/settings/wdb.q b/appconfig/settings/wdb.q new file mode 100644 index 0000000..2749978 --- /dev/null +++ b/appconfig/settings/wdb.q @@ -0,0 +1,55 @@ +// Virtual-table capture pack : WDB config +// see docs/virtual-table-capture-pack.md §3.3 + +\d .wdb +savedir:hdbdir:hsym`$getenv`KDBWDB // one directory; sym file lives at its root +writedownmode:`partbyattr // date + instrument directories. + // NB necessary but NOT sufficient - on its own it + // also writes the partition column into the files, + // which defeats the purpose. see code/wdb/vtwrite.q +mode:`saveandsort // the sort phase is overridden to a no-op (4.2) +immediate:1b // flush on every timer tick, ignore maxrows +settimer:0D00:00:01 // ...every second +gc:0b // at 1s cadence do not gc on every flush +rdbtypes:hdbtypes:gatewaytypes:() // none of these processes exist +sorttypes:sortworkertypes:() +idbtypes:`idb +permitreload:0b // nothing to reload +sortcsv:hsym`$getenv[`KDBAPPCONFIG],"/sort.csv" +// --------------------------------------------------------------------------- +// seed the partition from the BUSINESS date, not the calendar date. +// +// TorQ initialises .wdb.currentpartition from .proc.cd[] - the calendar date - and +// clearwdbdata[] then deletes THAT partition before the tickerplant log is replayed. With a +// roll offset the two disagree: at 07:06 UTC under a 09:00 roll the calendar says the 19th +// while the tickerplant is still logging the 18th. The delete then misses (nothing exists for +// the 19th yet), fixpartition corrects currentpartition afterwards from the tp log date, and +// the replay writes the whole day on top of data that was never removed - duplicating every +// row already on disk. Measured on this stack after one restart: 442 duplicate rows. +// +// .eodtime.getday is the same function the tickerplant uses to date its own logs, so seeding +// from it makes the writer agree with the tickerplant by construction at ANY roll offset - +// including none, where it reduces to the plain date and nothing changes. +// +// NOTE .eodtime is not loaded when this settings file runs (settings load ~11ms earlier), so +// the lookup sits inside the function body, not at the top level. writedown.q calls +// getpartition[] well after eodtime.q has loaded; the trap covers it never arriving at all. +// --------------------------------------------------------------------------- +startpartition:{[] + d:@[{[x] .eodtime.getday .z.p};(::);{[e] .proc.cd[]}]; + (`date^@[value;`.wdb.partitiontype;`date])$d + }; + +getpartition:{[] @[value;`.wdb.currentpartition;{[e] .wdb.startpartition[]}]}; + +symdomain:`sym // name of this stack's enumeration domain file. + // leave as `sym for a single stack. when several + // stacks are to be served by ONE reader, give each + // its own name (`syma, `symb...) - two roots both + // calling it `sym cannot be read together (8.3.1) + +\d .servers +CONNECTIONS:`segmentedtickerplant`idb`discovery + +\d .proc +loadprocesscode:1b // loads $KDBAPPCODE/wdb/vtwrite.q diff --git a/appconfig/sort.csv b/appconfig/sort.csv new file mode 100644 index 0000000..0ebdedb --- /dev/null +++ b/appconfig/sort.csv @@ -0,0 +1,4 @@ +tabname,att,column,sort +default,p,sym,1 +quote,p,sym,1 +trade,p,sym,1 diff --git a/code/loadtest.q b/code/loadtest.q new file mode 100644 index 0000000..86c66c6 --- /dev/null +++ b/code/loadtest.q @@ -0,0 +1,82 @@ +/ VT-12 load test, measured end to end from one process. +/ . +/ Publishes a burst of trades to the tickerplant, then polls the IDB until every row is +/ visible. That measures the whole chain - feed, tickerplant, writer, disk, reader - rather +/ than just how fast we can fill the tickerplant's queue. +/ . +/ Configured by environment: LOADROWS, LOADPAIRS, LOADBATCH. +/ . +/ NOTE a line containing only "/" opens a block comment in q, so every comment line here +/ carries text after the slash. + +getn:{[k;d] $[count v:getenv k; "J"$v; d] }; +rows:getn[`LOADROWS;200000]; +pairs:getn[`LOADPAIRS;50]; +batch:getn[`LOADBATCH;1000]; + +/ ports come from KDBBASEPORT, as selftest.q does. hardcoding them means a run against a +/ throwaway stack on another base silently drives the DEFAULT stack instead, and writes its +/ synthetic instruments into that database. +tpport:`$"::",getenv[`KDBBASEPORT],":feed:pass"; +idbport:`$"::",string[30+"J"$getenv`KDBBASEPORT],":idb:pass"; +tp:@[hopen;tpport;{[e] -1 "no tickerplant on ",string[tpport],": ",e; exit 1}]; +idb:@[hopen;idbport;{[e] -1 "no idb on ",string[idbport],": ",e; exit 1}]; + +insts:`$"P",/:string til pairs; +conds:" 89ABCEGJKLNOPRTWZ"; +exch:"NOL"; +mkbatch:{[n] + (n?insts; "f"$1+n?1000; "i"$1+n?1000; n?0b; n?conds; n?exch; n?`buy`sell) + }; + +/ NOTE on a completely empty database the reader defines no tables at all, so this query +/ is a value error rather than 0. Trap it - and see the finding in §12.3. +cnt:{[idb] @[idb;"count select from trade";{[e] 0}] }; +count0:cnt idb; +nb:rows div batch; + +-1 " publishing ",(string nb*batch)," rows in batches of ",string batch; +t0:.z.p; +{[tp;b;i] neg[tp](".u.upd";`trade;mkbatch b); if[0=i mod 50; tp"1+1"]; }[tp;batch] each til nb; +tp"1+1"; / everything is now in the tickerplant +tsent:.z.p; + +/ poll until the reader can see every row. rebuild first each time, so new instrument +/ directories are picked up without waiting for the sweep. +target:count0+nb*batch; +deadline:.z.p+0D00:05; +seen:{[idb;target;deadline] + while[(.z.pc:cnt idb; + idb".vtidb.rebuild[]"; + ]; + c + }[idb;target;deadline]; +t1:.z.p; + +secs:{`float$(x)%1000000000}; +-1 ""; +-1 " offered to tickerplant : ",(string `long$(nb*batch)%secs tsent-t0)," rows/sec (",(string tsent-t0),")"; +-1 " end to end to reader : ",(string `long$(nb*batch)%secs t1-t0)," rows/sec (",(string t1-t0),")"; +-1 " rows visible / target : ",(string seen)," / ",string target; +-1 " all rows captured : ",string seen>=target; +-1 ""; + +np:idb"count .vtidb.parts[`trade]"; +-1 " partition dirs : ",string np; +-1 " rows per partition : ",string `long$seen%np; + +pc:string idb".vtidb.partitioncol"; +one:first idb"1#exec ",pc," from .vtidb.parts`trade"; +q1:"select from trade where ",pc,"=`",string one; +idb q1; +u:{[idb;q] t:.z.p; r:idb q; (`long$(.z.p-t)%1000; count r) }[idb]; +r:u q1; +-1 " selective query : ",(string r 0)," us for ",(string r 1)," rows"; +r:u"select n:count i by ",pc," from trade"; +-1 " group by instrument : ",(string `long$(r 0)%1000)," ms"; +r:u"select total:sum price from trade"; +-1 " full aggregate : ",(string `long$(r 0)%1000)," ms"; +r:u".vtidb.rebuild[]"; +-1 " reader rebuild : ",(string `long$(r 0)%1000)," ms"; +hclose tp; hclose idb; +exit 0 diff --git a/code/processes/vtcompress-report.q b/code/processes/vtcompress-report.q new file mode 100644 index 0000000..c1d3e14 --- /dev/null +++ b/code/processes/vtcompress-report.q @@ -0,0 +1,48 @@ +// dry-run report for the weekend compression job (§7, VT-15). +// loaded by vtcompress.q when --dry-run is passed; reads .cmp.scope. +// +// the interesting number is not how many files are in scope but how big they are. kdb+ +// compresses a file in logical blocks of 2^cblocksize bytes, and the filesystem allocates +// space in blocks of its own, so a column file that already fits inside one filesystem +// block cannot get smaller on disk however well its bytes compress. + +\d .cmprep + +t:.cmp.scope; +fsblock:4096; // ext4 default: the floor on any file's disk usage +alloc:{[b;x] b*ceiling x%b}[fsblock]; // bytes actually allocated for a file of x bytes +s:asc t`currentsize; +pct:{[s;p] s `long$(count[s]-1)*p%100}[s]; +mb:{.Q.f[2;x%2 xexp 20]}; + +// hcount reports a compressed file's UNCOMPRESSED length, so currentsize alone cannot tell +// you whether the job has already run. ask each file's header instead +done:sum {00; + .cmp.toosmall:exec count i from .cmp.scope where currentsize<=.cmp.minfilesize; + .cmp.scope:select from .cmp.scope where currentsize>.cmp.minfilesize; + if[.cmp.toosmall; .lg.o[`compression;"size gate: skipping ",string[.cmp.toosmall], + " files of ",string[.cmp.minfilesize]," bytes or less - they cannot free a block"]]]; + +if[not count .cmp.scope; + .lg.o[`compression;"nothing in scope - either the tree is empty, or every partition is younger than minage"]; + exit 0]; + +.lg.o[`compression;"in scope: ",string[count .cmp.scope]," files across ", + string[count distinct .cmp.scope`partition]," partitions"]; + +if[.cmp.dryrun; + system"l ",getenv[`TORQAPPHOME],"/code/processes/vtcompress-report.q"; + exit 0]; + +.cmp.compressfromtable[.cmp.scope]; +.cmp.summarystats[]; +.lg.o[`compression;"finished compression"]; +exit 0 diff --git a/code/processes/vtidb.q b/code/processes/vtidb.q new file mode 100644 index 0000000..2f0c3fe --- /dev/null +++ b/code/processes/vtidb.q @@ -0,0 +1,369 @@ +/ Virtual-table capture pack : IDB reader +/ . +/ Implements section 5 of docs/virtual-table-capture-pack.md. +/ . +/ Reads the capture database in place. There is no load, no copy, no partitioned +/ database and no RDB: each table is a kx.pq.t virtual table whose partitions are the +/ date/instrument directories the WDB writes, opened as live views. History and the +/ current day are the same objects, so one process answers both. +/ . +/ NOTE a line containing only "/" opens a block comment in q, so every comment line here +/ carries text after the slash. + +/ bind mkP at the root and fully qualified: inside a \d block an undotted name is +/ resolved against that namespace, so this avoids depending on how it resolves +.vtidb.mkp:(use`kx.pq.t)`mkP; + +\d .vtidb + +/ --------------------------------------------------------------------------- +/ config - defaults here, overridden by appconfig/settings/idb.q +/ --------------------------------------------------------------------------- +roots:@[value;`roots;enlist hsym`$getenv`KDBDB]; +tabs:@[value;`tabs;`]; +historydays:@[value;`historydays;0W]; +sweep:@[value;`sweep;0D00:00:30]; +symsweep:@[value;`symsweep;0D00:00:01]; +/ the name the partition column is exposed under, and the name used in the catalogue. +/ the reader cannot derive this: the column is not stored on disk, which is the point of the +/ design (§2.2). set it to whatever the source schema calls the parted column, so that client +/ queries read the same as they would against a conventional database. +partitioncol:@[value;`partitioncol;`instrument]; +wdbtypes:@[value;`wdbtypes;`wdb]; +wdbcheckcycles:@[value;`wdbcheckcycles;3]; +wdbconnsleepintv:@[value;`wdbconnsleepintv;5]; + +/ --------------------------------------------------------------------------- +/ state +/ --------------------------------------------------------------------------- +current:0Nd; / the partition the writer is currently filling +symsize:0; / total size of the sym files at the last load +parts:(`$())!(); / table -> catalogue of (date;;path) +opened:(`$())!(); / table -> opened live views, one per parts row. + / NOTE not called "views" - that is a q keyword +lastgap:(`$())!(); / table -> dates missing at the last check (4.6) + +/ --------------------------------------------------------------------------- +/ the enumeration domain. +/ symbol columns in a partition are enumerations against $KDBDB/sym, so the domain +/ has to cover a directory's values BEFORE that directory is opened, or the symbols +/ resolve to the wrong values. same check stock TorQ's idb.q makes. +/ --------------------------------------------------------------------------- +/ every enumeration domain file in the tree. anything at a root that is a file rather than a +/ date directory is one: a stack written with .Q.ens[dir;t;`symb] leaves `symb` here, not `sym`. +/ discovering them rather than assuming `sym` is what lets two stacks keep separate domains +/ without colliding (§8.3.1). +symfiles:{[] + raze {[r] + k:key r; + k:k where not k like "*.*"; / drops date directories and par.txt alike: + / a domain file is named like an identifier + f:.Q.dd[r;] each k; + f where {x~key x} each f / a directory keys to its contents, a file to itself + } each roots + }; + +/ `load` binds a global named after the FILE, so two roots can keep independent domains as +/ long as they are named differently - each column file records which domain it belongs to and +/ resolves through that one. what is not safe is two roots using the same NAME for different +/ content: one load wins and the other root's symbols then resolve wrongly, and silently (§8.3.1). +loadsym:{[] + f:symfiles[]; + {@[load;x;{[p;e] .lg.e[`vtidb;"failed to load ",string[p],": ",e]}[x]]} each f; + / group returns INDICES into f, not the paths themselves + g:group {last ` vs x} each f; + {[f;g;n] + if[1c:symbytes[];[symsize::c;1b];0b] }; + +/ --------------------------------------------------------------------------- +/ 5.4 - the enumeration domain can grow with NO directory appearing. +/ . +/ a new value in a data symbol column - a new src, a new venue code - is appended to the +/ domain file by the writer and lands in an existing partition. no directory is created, so +/ the writer sends nothing (4.1 is edge-triggered on new directories), and the reader's +/ in-memory domain is then short by one entry. the rows are visible immediately, as any +/ append is, but that column reads as NULL until the domain is reloaded. silently. +/ . +/ this is why the domain gets its own timer rather than waiting for the rebuild sweep: +/ checking is one hcount per root, cheap enough to run every second, where a rebuild is +/ ~10 ms and has no reason to run that often. +/ --------------------------------------------------------------------------- +refreshsym:{[] + if[symchanged[]; + loadsym[]; + .lg.o[`vtidb;"enumeration domain grew - reloaded"]]; + }; + +/ --------------------------------------------------------------------------- +/ scanning the tree +/ --------------------------------------------------------------------------- + +/ date directories under one root, honouring historydays +datedirs:{[r] + d:key r; + if[not count d; :0#`]; + d:d where d like "[0-9][0-9][0-9][0-9].[0-9][0-9].[0-9][0-9]"; + $[historydays=0W; asc d; asc d where ("D"$string d) >= .z.D - historydays] + }; + +/ table names under the given date directories, across every root +scantabs:{[ds] + distinct raze {[ds;r] raze {[r;d] k:key .Q.dd[r;d]; $[()~k; 0#`; k]}[r] each ds}[ds] each roots + }; + +/ which tables to build. +/ . +/ discovering these from the tree rather than configuring them means a table added to +/ database.q needs no change here. Doing it by scanning EVERY date on every rebuild does not: +/ that is a readdir per date, so the cost grows with retention for ever, and it is paid on +/ every sweep to notice something that happens once in a deployment's life. Measured at 250 +/ dates it was 2.6 ms of a 4.2 ms rebuild - the last cost in this reader still proportional to +/ history, which is what 8.2 removed everywhere else. +/ . +/ A new table can only appear where the writer is writing, so once anything is known, only the +/ live partition needs looking at. The cold path still scans everything, because a table that +/ has STOPPED receiving data exists only in history and would never be found on the live date. +tablelist:{[ds] + if[not tabs~`; :(),tabs]; + known:key parts; + if[not count known; :scantabs ds]; + distinct known,scantabs $[null current; ds; enlist `$string current] + }; + +empty:flip (`date,partitioncol,`path)!(0#0Nd;0#`;0#`); + +/ the (date;;path) rows for one table on one date under one root. +/ WARNING the trailing ` on the path is what makes the view live - see §5.2. without it +/ each partition is a frozen snapshot and the reader never sees another row +scandate:{[t;r;d] + p:.Q.dd[.Q.dd[r;d];t]; + if[()~i:key p; :empty]; + flip (`date,partitioncol,`path)!(count[i]#"D"$string d; i; {.Q.dd[.Q.dd[x;y];`]}[p] each i) + }; + +/ one table on one date, across every root +scanone:{[t;d] raze scandate[t;;d] each roots }; + +/ every partition directory NAME currently on disk, across every root. these are symbols +/ (e.g. `2026.08.17), not dates - scandate needs the symbol to build the path +alldates:{[] d:raze datedirs each roots; $[count d; asc distinct d; 0#`] }; + +/ open one partition, tolerating a directory that is mid-creation +/ . +/ NOTE this does NOT catch a directory whose .d names columns that are not on disk. get is +/ lazy, so such a partition attaches cleanly and even counts, and then breaks every query that +/ touches a missing column - for the whole table, since all partitions must be opened. That is +/ a real state (a full disk leaves one behind; every new partition passes through it while +/ being written) and it is deliberately not guarded here. See 5.7 for the measurement and the +/ reasoning: the transient race is rare and self-healing, and in the permanent case a loud +/ failure is worth more than a reader that keeps answering while quietly omitting an +/ instrument. testfiles/vt-diskfull-test.q pins down what actually happens. +open:{[p] @[get;p;{[p;e] .lg.w[`vtidb;"cannot open ",string[p],": ",e]; ::}[p]]}; + +/ can this date still gain a directory? only the live partition can. once a date has +/ rolled its directory list never changes again, so its catalogue and its opened views +/ can be reused instead of rescanned - which is what keeps rebuild off the critical path. +/ a null current (before the writer has been found) forces a full scan. +/ vectorised deliberately: "mutable each" on an empty list yields an untyped () which +/ then breaks the boolean and in build. +mutable:{[d] $[null current; count[d]#1b; d>=current] }; + +/ which partition is the writer filling? +/ . +/ getting this wrong is not cosmetic. a date classified immutable is CACHED and never +/ rescanned, so believing the live date has already rolled freezes it: every instrument that +/ starts trading afterwards is on disk, absent from every query, and nothing is logged - +/ there is no error to log, the reader simply stopped looking. +/ . +/ .z.D is the wrong answer, and wrong in the ordinary case rather than an exotic one. Whenever +/ .eodtime.rolltimeoffset is non-zero - any business day that does not end at midnight in +/ rolltimezone - the writer goes on filling YESTERDAY for offset hours after .z.D has advanced. A reader that comes up in that window without +/ a writer to ask (the writer is down, or the reader started first) would freeze the live +/ partition for the rest of the day. +/ . +/ The latest date on disk is a lower bound the writer cannot contradict: it cannot be filling +/ a date older than the newest directory it has itself created. So take the writer's answer +/ when we have one, but never let it sit behind the disk. That needs no connection tracking +/ and degrades correctly when the writer dies after telling us once. +/ . +/ NOTE max ignores nulls, which is what makes this work before any writer has been found. +livepart:{[ds] $[count ds; max current,"D"$string last ds; current] }; + +/ discard the whole cache, so the next rebuild rescans every date. the manual recovery path +/ for a change this reader cannot see by itself - a directory added to a PAST date (§6.1). +/ not needed after compression: §7.1 measured that a rename-over is picked up immediately, +/ because a trailing-slash view holds no inode to go stale on. +dropcache:{[] parts::(`$())!(); opened::(`$())!(); }; + +/ forget specific dates, keeping the rest of the cache, so the next rebuild rescans just +/ those. this is what makes end of day O(instruments) rather than O(history): the date that +/ just closed needs one final scan, and nothing older does. +/ parts and opened are row-aligned and must be filtered together. +dropdates:{[ds] + if[not count ds; :()]; + {[ds;t] + k:where not parts[t][`date] in ds; + parts[t]:parts[t] k; + opened[t]:opened[t] k; + }[ds] each key parts; + }; + +/ --------------------------------------------------------------------------- +/ build one virtual table. +/ immutable dates already held are reused as-is; only the live date and any date not +/ yet seen are scanned and opened. rescanning all of history on every tick is what made +/ rebuild linear in days as well as instruments (§8.2). +/ the global has to land in the ROOT namespace so clients can write "select from +/ trade" - `t set ...` inside a \d block defines .vtidb.t instead +/ --------------------------------------------------------------------------- +build:{[t;ds] + if[not count ds; .lg.w[`vtidb;"no partitions found for ",string t]; :0]; + / ds holds directory NAMES as symbols; the catalogue holds real dates. keep both. + dsd:"D"$string ds; + old:$[t in key parts; parts t; empty]; + oldv:$[t in key opened; opened t; ()]; + / keep rows whose date is immutable and still on disk + keep:where (old[`date] in dsd) and not mutable old`date; + m:old keep; + v:oldv keep; + / everything else gets scanned: the live date, plus any date we do not already hold + torescan:ds where not dsd in distinct m`date; + if[count torescan; + nm:raze scanone[t] each torescan; + if[count nm; + nv:open each nm`path; + ok:where 98h=type each nv; + if[count[ok]count bytab; :()]; + full:asc distinct raze value bytab; + g:{[full;ds] full except ds}[full] each bytab; + gaps:(where 0 ",.Q.s1 after]]; + checkcoverage[]; / 4.6 - the silent failure needs a voice + after + }; + +/ called by the wdb at end of day (§4.2). no data moves; a new date is just new directories. +/ . +/ the ONLY date whose catalogue can still be wrong here is the one that just closed: the +/ writer's final flush of the day may have created directories the reader has not scanned +/ yet. so forget that one date and let rebuild pick it up, and keep every older date. +/ . +/ dropping the WHOLE cache here - which is what this used to do - made end of day a full +/ rescan of history, ~47 us per directory, and it was the last cost that grew with retention. +/ the justification for it (compression needs a genuine re-open) was measured false in §7.1. +/ . +/ NOTE the drop has to happen BEFORE current moves. once current is the new date, the date +/ that just closed reads as immutable and build would reuse its stale catalogue instead. +rollover:{[pt] + .lg.o[`vtidb;"rollover to ",string pt]; + dropdates enlist current; + current::pt; + loadsym[]; + rebuild[]; + }; + +/ --------------------------------------------------------------------------- +/ startup +/ --------------------------------------------------------------------------- + +/ find the writer. it is NOT required - without it the sweep keeps the reader current - +/ but registering with it turns new-partition latency from into <1s +/ NOTE the arguments go in @'s second slot, not into a projection: @[f[a;b];::;h] applies +/ f OUTSIDE the trap, so a failure there is never caught +findwdb:{[] + h:@[{[a] .servers.startupdepcycles . a; .servers.gethandlebytype[first a;`any]}; + (wdbtypes;wdbconnsleepintv;wdbcheckcycles); + {[e] .lg.w[`vtidb;"no wdb: ",e]; ()}]; + if[not count h; + .lg.w[`vtidb;"running without a wdb - the live partition is taken from disk, and new ", + "partitions will be picked up by the ",string[sweep]," sweep"]; + :()]; + w:first h; + / a failed read leaves current where livepart put it. it must NOT fall back to .z.D - see + / livepart for why that freezes the live partition for the rest of the day + pt:@[w;(value;`.wdb.currentpartition);{[e] .lg.w[`vtidb;"could not read the wdb partition: ",e]; 0Nd}]; + if[not null pt; current::pt]; + @[w;(`.servers.registerfromdiscovery;`idb;0b);{.lg.e[`vtidb;"registration with the wdb failed: ",x]}]; + .lg.o[`vtidb;"registered with the wdb, current partition ",string current]; + }; + +init:{[] + .lg.o[`vtidb;"scanning roots ",.Q.s1 roots]; + loadsym[]; + symsize::symbytes[]; + / current is left null: the first rebuild sets it from disk (livepart), and findwdb below + / replaces it with the writer's own answer if there is a writer to ask + n:rebuild[]; / cache is empty, so this one scans everything + .lg.o[`vtidb;"attached ",(.Q.s1 n)," across ",(.Q.s1 count distinct raze {exec date from x} each value parts)," date(s)"]; + findwdb[]; / replaces current with the writer's actual partition + if[.timer.enabled; + .timer.repeat[.proc.cp[];0Wp;sweep;(`.vtidb.rebuild;`);"virtual table sweep - backstop for a missed wdb notification"]; + .timer.repeat[.proc.cp[];0Wp;symsweep;(`.vtidb.refreshsym;`);"enumeration domain check - 5.4"]]; + .lg.o[`vtidb;"initialised"]; + }; + +\d . + +/ tables[] does not see virtual tables - they are type 112h, not 98h - so the attribute +/ has to come from our own catalogue +.proc.getattributes:{`partition`tables!(.vtidb.current;key .vtidb.parts)}; + +.vtidb.init[]; diff --git a/code/selftest.q b/code/selftest.q new file mode 100644 index 0000000..5daa36f --- /dev/null +++ b/code/selftest.q @@ -0,0 +1,109 @@ +/ End-to-end self test for the virtual-table capture pack. +/ . +/ Publishes a brand new symbol to the tickerplant and checks that it travels all the +/ way through to the IDB: writer creates a partition directory, notifies, reader rebuilds, +/ the rows come back through the virtual table, and the partition column is NOT in the files. +/ . +/ Run it with ./selftest.sh while the stack is up. +/ . +/ NOTE a line containing only "/" opens a block comment in q, so every comment line here +/ carries text after the slash. + +tpport:`$"::",getenv[`KDBBASEPORT],":feed:pass"; +idbport:`$"::",string[30+"J"$getenv`KDBBASEPORT],":idb:pass"; + +pass:0; fail:0; +/ WARNING the first parameter is NOT called "desc" - that is a q keyword, and using it as a +/ parameter name makes the function raise 'nyi when applied +check:{[msg;ok;detail] + if[ok; pass::pass+1; -1 " PASS ",msg; :()]; + fail::fail+1; + -1 " FAIL ",msg," -- ",detail; + }; + +-1 "connecting to the tickerplant and the idb..."; +tp:@[hopen;tpport;{'"no tickerplant on ",string[tpport],": ",x}]; +h:@[hopen;idbport;{'"no idb on ",string[idbport],": ",x}]; + +/ a fresh instrument each run, so this really does create a NEW partition directory. +/ WARNING seed from the clock first - every q process starts with the same rng seed, so +/ without this the "random" instrument is identical on every run +system "S ",string "i"$.z.t; / .z.t is ms since midnight - .z.n overflows an int +inst:`$"ZZ",5?.Q.A; +-1 "test symbol: ",string inst; + +/ wait until the reader actually has the trade table. on a brand new database it has no +/ tables at all until the writer's first flush, so "count select from trade" is a value +/ error rather than 0 - see §12.3. without this wait the test fails spuriously when run +/ immediately after a wipe. +/ the reader exposes the partition column under a configurable name (§5.3), so ask it rather +/ than hardcoding one - this test has to work whatever the schema calls it +pc:string h".vtidb.partitioncol"; +hastrade:{[h] @[{[x] x"count select from trade"; 1b};h;{[e] 0b}] }; +/ WARNING do not name the limit "maxs" - it is a q keyword (running maximum) and using it +/ as a parameter gives 'match when the function is applied +waitready:{[h;limit] + n:0; + while[(n=before+n;"before ",string[before]," after ",string after]; + +/ NOTE assert that OUR instrument is now in the catalogue, not that the count rose by exactly +/ one: the demo feed introduces symbols of its own, so an exact-count assertion fails +/ spuriously on a freshly started stack. +partsafter:h"count .vtidb.parts[`trade]"; +haspart:h"(`",(string inst),") in exec ",pc," from .vtidb.parts`trade"; +check["a new partition directory was picked up";haspart and partsafter>partsbefore; + "instrument present: ",(string haspart),", partitions ",string[partsbefore],"->",string partsafter]; + +got:h"select from trade where ",pc,"=`",string inst; +check["the new symbol is queryable";n=count got;"got ",string[count got]," rows"]; +check["the partition column comes back as a column";inst~first got[`$pc];.Q.s1 got]; + +/ the whole point of the design: the partition column must NOT be stored in the files +p:h"first exec path from .vtidb.parts[`trade] where ",pc,"=`",string inst; +ondisk:@[{cols get x};p;{`FAILED_TO_READ}]; +check["the partition column is NOT in the files";not `sym in ondisk;.Q.s1 ondisk]; +check["the data columns are";7=count ondisk;.Q.s1 ondisk]; + +/ appends to an EXISTING partition directory must become visible. +/ NOTE assert on this instrument's own rows, not on the global count or the global partition +/ count: the demo feed publishes concurrently and introduces new instruments of its own, so a +/ global assertion fails spuriously on a freshly started stack. The stronger claim - that no +/ rebuild is needed at all - is measured under controlled conditions in §5.2, not here. +mine:{[h;inst] h"count select from trade where ",pc,"=`",string inst}; +a:mine[h;inst]; +do[n; tp(".u.upd";`trade;(enlist inst;enlist 101f;enlist 20i;enlist 0b; + enlist " ";enlist "N";enlist`sell))]; +system"sleep 3"; +b:mine[h;inst]; +check["appends to an existing partition are visible";b>=a+n; + "rows for ",(string inst),": ",string[a],"->",string b]; + +hclose tp; hclose h; + +-1 ""; +-1 (40#"-"); +-1 " ",string[pass]," passed, ",string[fail]," failed"; +-1 (40#"-"); +exit $[fail>0;1;0]; diff --git a/code/tick/feed.q b/code/tick/feed.q new file mode 100644 index 0000000..0af0a54 --- /dev/null +++ b/code/tick/feed.q @@ -0,0 +1,101 @@ +/ Market-data feed, following the Finance Starter Pack generator. +/ . +/ Publishes trades and quotes for a small equity universe, with deliberately skewed volumes +/ so some symbols are far busier than others - which is what makes the date+instrument +/ layout worth looking at, since partition sizes then vary the way they do in real data. +/ . +/ Configured by environment: +/ REPLAYINTERVAL time between publishes (default 200ms) +/ . +/ NOTE a line containing only "/" opens a block comment in q, so every comment line here +/ carries text after the slash. + +REPLAYINTERVAL:@[value;`REPLAYINTERVAL;0D00:00:00.200]; + +syms:`AMD`AIG`AAPL`DELL`DOW`GOOG`HPQ`INTC`IBM`MSFT; +px:33 27 84 12 20 72 36 51 42 29f; / starting price per symbol +modes:" ABHILNORYZ"; / quote mode +conds:" 89ABCEGJKLNOPRTWZ"; / trade condition +exch:"NONNONONNN"; / exchange, one per symbol +srcs:`BARX`GETGO`SUN`DB; +sides:`buy`sell; + +cnt:count syms; +maxtrades:15; / max trades per tick +quotespertrade:5; + +/ WARNING every q process starts on the same rng seed, so without this the "random" feed is +/ byte-identical on every run - which quietly makes a load test measure the same data twice +system "S ",string "i"$.z.t; / .z.t is ms since midnight; .z.n overflows an int + +/ weights skew how often each symbol appears and how large its sizes are, so partitions do +/ not all come out the same size +weight:0.1*1+neg[cnt]?2*cnt; +volmap:syms!neg[cnt]?weight; +bidmap:syms!neg[cnt]?weight; +askmap:syms!neg[cnt]?weight; + +/ a weighted index list: symbols with a higher weight appear in it more often +skew:{[weights;items] raze weights#'neg[count items]?items}; +weighted:skew[`long$weight*10;til cnt]; + +pi:acos -1; +normalrand:{[n] (cos 2*pi*n?1f)*sqrt neg 2*log n?1f}; +rnd:{[x] 0.01*floor 0.5+x*100}; +vol:{[n] 10+`int$n?90}; + +/ a batch of correlated prices: each symbol random-walks from where it left off +qx:qb:qa:qp:(); +qn:0; +batch:{[n] + d:exp 0.001*normalrand n; + qx::n?weighted; + qb::rnd n?1.0; + qa::rnd n?1.0; + idx:where each qx=/:til cnt; + s:px*prds each d idx; + qp::n#0.0; + (qp raze idx):rnd raze s; + px::last each s; + qn::0; + }; + +len:10000; +batch len; + +/ column vectors in the order .u.upd expects. time is added by the tickerplant +mktrade:{[n] + if[not (qn+n) ", + (string `long$(nb*batch)%(`float$el)%1000000000)," rows/sec offered"]; + .lg.o[`load;"LOADDONE"]; + }; + +\d . + +.servers.startupdepcycles[`segmentedtickerplant;5;0W]; +.load.run[]; diff --git a/code/wdb/vtwrite.q b/code/wdb/vtwrite.q new file mode 100644 index 0000000..9085695 --- /dev/null +++ b/code/wdb/vtwrite.q @@ -0,0 +1,157 @@ +/ Virtual-table capture pack : WDB overlay +/ . +/ Implements sections 4.1, 4.2, 4.5 and 4.6 of docs/virtual-table-capture-pack.md. +/ . +/ Load order note: $KDBAPPCODE/wdb/ is loaded by .proc.reloadcode BEFORE the stock +/ code/processes/wdb.q, so anything defined here directly would be clobbered. Everything +/ is therefore defined under a private name and swapped in from .proc.initlist, which +/ runs last. Same pattern as the No-RDB pack's code/wdb/rollover.q. +/ . +/ NOTE a line containing only "/" opens a block comment in q, so every comment line here +/ carries text after the slash. + +\d .wdb + +/ partitions created during the current flush, as (partition;instrument) pairs +vtnew:(); + +/ this stack's enumeration domain (8.3.1). settings override it; default keeps stock behaviour +symdomain:@[value;`symdomain;`sym]; + +/ TorQ's directory-name sanitiser, factored out of upserttopartition so the fill logic +/ below builds identical names. non-alphanumerics become "_", nulls become TORQNULLSYMBOL. +/ WARNING lossy - EUR-USD and EUR_USD collapse into the same directory +vtdirname:{[expt] `$"_"^.Q.an .Q.an?"_" sv string `TORQNULLSYMBOL^ensuresymlist[expt]}; + +/ the schema a partition directory should have: the table minus its partition column(s), +/ because those are carried by the directory name (4.5) +vtschema:{[t;expttype] ![0#value t;();0b;expttype]}; + +/ --------------------------------------------------------------------------- +/ 4.5 - write the data WITHOUT the partition column. +/ a column stored inside the files can never be used to skip directories, so leaving +/ sym in the data would make every query on it scan the whole database. +/ also records directories that did not exist beforehand, for 4.6 below. +/ --------------------------------------------------------------------------- +vtupserttopartition:{[dir;tablename;tabdata;pt;expttype;expt;writedownmode] + base:` sv .Q.par[dir;pt;tablename],vtdirname[expt]; + if[()~key base; vtnew,:enlist (pt;expt)]; + keep:{x!x} cols[tabdata] except expttype; + r:?[tabdata;{(x;y;(),z)}[in;;]'[expttype;expt];0b;keep]; + .[upsert;(` sv base,`;r);{[e] .lg.e[`vtwrite;"failed to save partition: ",e];'e}]; + .merge.partsizes[base]+:(count r;-22!r); + }; + +/ --------------------------------------------------------------------------- +/ 4.6 - every table must have a directory in every partition. +/ a partition holding trade but not quote breaks a reader's load outright, and if it +/ sorts first it silently truncates the reader's table list. cheap to prevent here. +/ --------------------------------------------------------------------------- +vtfill:{[pt;expt] + {[pt;expt;t] + d:` sv .Q.par[savedir;pt;t],vtdirname[expt],`; + if[()~key first ` vs d; + .lg.o[`vtwrite;"creating empty ",(string t)," for new partition ",1_string d]; + d set .Q.en[hsym hdbdir; vtschema[t;.merge.getextrapartitiontype t]]]; + }[pt;expt] each tablelist[]; + }; + +/ --------------------------------------------------------------------------- +/ 4.1 - tell readers about NEW partitions only. +/ appends need no notification: readers hold live views (5.2) and see them already. +/ the only event a reader must react to is a directory appearing. +/ ORDER MATTERS - fill every table's directory before notifying, or a reader can +/ rebuild against a half-created partition and fail (4.6). +/ --------------------------------------------------------------------------- +/ . +/ NOTE the pending carry-over is what covers a tickerplant log REPLAY. TorQ's replay does not +/ come through here at all: replaymaxrowcheck calls savetables[savedir;getpartition[];0b;t] +/ directly, once per table, whenever a table exceeds replaymaxrows. So vtupserttopartition runs +/ and vtnew fills correctly - with every directory, since deletewdbdata wiped the partition +/ first - but vtfill and the notification never fire. Clearing vtnew unconditionally here then +/ THREW THAT LIST AWAY on the first flush after the replay, so an instrument that only ever had +/ rows in one table came back without its empty directory in the other. Harmless while every +/ table is busy; it is 4.6's silently-absent-date the moment a table receives nothing all day. +vtsavetodisk:{[] + pending:vtnew; / anything a replay's direct calls left + vtnew::(); / edge-triggered: only this flush counts + savetables[savedir;getpartition[];immediate;] each tablelist[]; + news:distinct pending,vtnew; + if[count news; + vtfill . ' news; + .lg.o[`vtwrite;"new partitions: ",.Q.s1 news]; + notifyidbs[`.vtidb.rebuild;enlist()]]; + }; + +/ --------------------------------------------------------------------------- +/ 4.2 - end of day does nothing but announce the new date. +/ stock endofdaysort would merge the instrument directories back into one table per +/ date, which is precisely the layout this design exists to avoid. +/ --------------------------------------------------------------------------- +vteodsort:{[dir;pt;tablist;writedownmode;mergelimits;hdbsettings;mergemethod] + .lg.o[`vtwrite;"no-merge eod - partition ",string[pt]," stays in place"]; + notifyidbs[`.vtidb.rollover;enlist pt+1]; + }; + +/ --------------------------------------------------------------------------- +/ 8.3.1 - name this stack's enumeration domain. +/ symbol columns are indices into a file at the database root, and a reader binds a global +/ named after that FILE. two stacks that both call it `sym` cannot be served by one reader: +/ one load wins and the other's symbols resolve to the wrong values, silently. giving each +/ stack its own name removes the coupling entirely. +/ . +/ .Q.en[d;t] is .Q.ens[d;t;`sym], so redirecting .Q.en covers every enumeration site in the +/ writer at once - savetables, the empty-partition fill, and the initial table creation - +/ without copying a forty-line TorQ function to change one symbol in it. +/ --------------------------------------------------------------------------- +applysymdomain:{[] + if[symdomain~`sym; :()]; + .lg.o[`vtwrite;"enumerating against `",string[symdomain]," instead of `sym (8.3.1)"]; + .Q.en:{[dom;d;t] .Q.ens[d;t;dom]}[symdomain]; + }; + +/ --------------------------------------------------------------------------- +/ 4.7 - the overrides must be installed BEFORE the tickerplant log is replayed. +/ . +/ .proc.addinitlist alone is not enough, and the gap is silent. Load order is: +/ . +/ code/wdb/origstartup.q defines .wdb.startup +/ $KDBAPPCODE/wdb/vtwrite.q this file +/ code/processes/wdb.q defines savetables/upserttopartition, then CALLS startup +/ .proc.init[] runs the init list +/ . +/ startup[] is what subscribes to the tickerplant and replays its log, and it runs at the +/ bottom of wdb.q - a full second before the init list. So on any restart with a populated +/ log, every partition rebuilt by the replay was written by the STOCK writer, which keeps +/ the partition column in the files. That is the one thing this design cannot tolerate +/ (4.5), and it puts old and new partitions into the mismatched-column state of 9.3. +/ . +/ It only shows up on the normal recovery path - restart a writer whose log has data - so a +/ test that wipes var/ first will never see it. testfiles/vt-replay-test.q covers it. +/ . +/ startup is defined in origstartup.q, which loads BEFORE this file, and wdb.q only calls it. +/ So wrapping it here survives, where redefining anything wdb.q owns would not. +/ --------------------------------------------------------------------------- +origstartup:startup; +startup:{[] + applyvtwrite[]; + origstartup[] + }; + +/ --------------------------------------------------------------------------- +/ swap everything in once the stock wdb.q has finished loading. +/ still registered: the wrapper above covers the replay path, this covers a writer that +/ never subscribes (saveenabled off, or no tickerplant). applyvtwrite is idempotent. +/ --------------------------------------------------------------------------- +applyvtwrite:{[] + .lg.o[`vtwrite;"installing virtual-table capture overrides (4.1, 4.2, 4.5, 4.6)"]; + upserttopartition::vtupserttopartition; + savetables::savetablesbypart[;;;;writedownmode]; / rebind: it closed over the old upsert + savetodisk::vtsavetodisk; + endofdaysort::vteodsort; + applysymdomain[]; + }; + +\d . + +.proc.addinitlist".wdb.applyvtwrite[]"; diff --git a/compress.sh b/compress.sh new file mode 100755 index 0000000..2c12e68 --- /dev/null +++ b/compress.sh @@ -0,0 +1,48 @@ +#!/bin/bash +# Run the weekend compression job over the capture tree, then exit. +# +# ./compress.sh compress everything older than minage (compressionconfig.csv) +# ./compress.sh --dry-run list what would be compressed, change nothing +# ./compress.sh --test compress underneath the running reader and verify it copes +# +# Two gates decide what gets touched, and both are deliberate (see §7.3 of the doc): +# age tier minage in appconfig/compressionconfig.csv - recent data stays uncompressed so +# interactive queries run at full speed. also keeps the job off the live partition +# size gate .cmp.minfilesize in appconfig/settings/compression.q - a column file that fits +# in one filesystem block frees nothing when compressed, so it is skipped +# +# Cron it for a quiet period, e.g. Saturday 02:00: +# 0 2 * * 6 /path/to/TorQ-VT-Capture-Pack/compress.sh + +set -e +. "$(cd "$(dirname "$0")" && pwd)/vt-env.sh" + +if [ ! -f "${TORQHOME}/torq.q" ]; then + echo "ERROR: no torq.q under TORQHOME=${TORQHOME}" >&2 + exit 1 +fi + +cd "$TORQHOME" +ACL="${KDBAPPCONFIG}/passwords/accesslist.txt" + +if [ "$1" = "--test" ]; then + # VT-15: compress with the reader live, and check it never notices. uses the test config, + # whose age tier is 1 day, so it has something to work on in a database a few days old + export VTCMP_CONFIG="${KDBAPPCONFIG}/compressionconfig-test.csv" + exec q "${TORQAPPHOME}/testfiles/vt-compress-test.q" &1 | tee "${KDBLOG}/cmp1.console.log" + +echo "" +echo "done. per-file detail was written to ${KDBLOG}/cmp1.console.log" diff --git a/database.q b/database.q new file mode 100644 index 0000000..ffc7f48 --- /dev/null +++ b/database.q @@ -0,0 +1,28 @@ +/ Schema, as published by the Finance Starter Pack feed. +/ . +/ sym is the partition column: the writer strips it from the files and carries it in the +/ directory name instead (§4.5). It is declared in appconfig/sort.csv and exposed back to +/ clients under the same name by the reader's `partitioncol` setting. + +quote:([] + time:`timestamp$(); + sym:`g#`symbol$(); + bid:`float$(); + ask:`float$(); + bsize:`long$(); + asize:`long$(); + mode:`char$(); + ex:`char$(); + src:`symbol$() + ) + +trade:([] + time:`timestamp$(); + sym:`g#`symbol$(); + price:`float$(); + size:`int$(); + stop:`boolean$(); + cond:`char$(); + ex:`char$(); + side:`symbol$() + ) diff --git a/docs/status-report.md b/docs/status-report.md new file mode 100644 index 0000000..8bbbf85 --- /dev/null +++ b/docs/status-report.md @@ -0,0 +1,657 @@ +# Virtual-Table Capture Pack — status report + +Point-in-time summary for ticket tracking. Written 2026-08-14, updated 2026-08-24 +(VT-18 to VT-21: the load-order and live-partition defects, end-of-day catalogue reuse, +the remaining testing gaps, and the move of all tests into `testfiles/`). +Technical detail: `docs/virtual-table-capture-pack.md`. Presentable summary: the published +design brief (link in the ticket). + +--- + +## Epic summary + +Prototype a kdb+ market-data capture stack that partitions by **date and instrument** rather +than date alone, so that a lookup for one instrument becomes a directory lookup instead of a +scan. Data is written once and never moved: there is no RDB, no HDB, no overnight +sort/merge, and no gateway. History and live data are the same files, served by one process +type. + +Built as a standalone application overlay onto a TorQ checkout. Runs on kdb-x, using its +virtual-table module to make the layout queryable. Captures the Finance Starter Pack's +`trade` and `quote` schema, partitioned on `sym`. + +**Status: prototype complete and verified end to end. Agrees with stock kdb+ on identical data, +and sustains 1.45M rows/sec with zero loss. No blocking gaps remain in the prototype; the +outstanding items are deployment concerns and one external dependency on KX.** + +Weekend compression is built, measured and shipped on a retention tier: it frees 83 % of the +bytes and 81 % of the disk at realistic partition sizes, for roughly double the query latency — +which is why it runs behind an age tier rather than over everything. Measuring it also settled +the last open scaling question: readers need no notification across a compression run, which +unblocked VT-17. **End of day is now flat — no operation in this design scales with how much +history is attached.** + +Multi-stack read is no longer blocked. One reader serves several capture stacks by naming each +stack's enumeration domain apart rather than sharing one file, which keeps the stacks +independent. One documented limitation remains, on cross-stack grouping by a symbol column. + +**Everything in the prototype is now built and verified.** VT-14 (legacy data migration) is +proven but deliberately unbuilt: greenfield deployment, and its shape depends on the open KX +item. That external dependency is the only thing outstanding. + +--- + +## Status at a glance + +| area | state | +|---|---| +| Architecture and design decisions | Complete, documented | +| Writer (date+instrument partitions) | Complete, verified | +| Reader (virtual tables over the tree) | Complete, verified | +| End-to-end capture → query | Verified, 2 ms latency for a new instrument | +| Scaling behaviour | Measured to 40,000 partitions; no operation now scales with retention | +| Client compatibility | Measured, 38 operations (`vt-compat-test.q`) | +| Correctness vs stock kdb+ | Verified — 16/19 queries identical, no silent disagreement | +| Load / volume testing | Verified — 1.45M rows/sec peak, zero loss | +| Silent-failure detection | Complete, verified | +| Weekend compression | Complete, measured, shipped on a retention tier with a size gate | +| Multi-stack read | Complete, verified, one documented limitation | +| Legacy migration | Designed and proven; build deferred pending KX (§9.6) | +| External blocker on KX | 1 open | + +--- + +# Completed + +## VT-1 · Architecture and design specification +1,201-line design document, 11 sections. Every decision recorded against the alternative that +was tried and rejected. + +- **VT-1.1** Capture requirements and target topology defined (segmented tickerplant, writer, + N replicable readers, minimal end-of-day, weekend housekeeping) +- **VT-1.2** Process inventory decided — what is kept, what is deleted, and why each deletion + is safe (RDB, HDB, sort, sort workers, gateway all removed) +- **VT-1.3** On-disk contract specified: exact directory layout produced, and the three + consequences that drive the rest of the design +- **VT-1.4** Gap analysis against TorQ 5.2.15 source — six changes identified, each traced to + a specific line, with the root cause established (the write mode is a *staging* format in + stock TorQ, so every "nobody reads this" exclusion becomes a defect once the merge is removed) +- **VT-1.5** Configuration design: process list, per-process settings, partition-column + declaration, environment contract +- **VT-1.6** End-of-day sequence designed and documented (no data movement, no restart) +- **VT-1.7** Decisions register written — 8 decisions, each with the rejected alternative and + the reason, so the design can be picked up cold +- **VT-1.8** Backward-compatibility analysis for existing date-partitioned data, including + evaluation of three candidate approaches proposed at kick-off +- **VT-1.9** Comparison against the No-RDB Starter Pack, to establish what is genuinely new + here versus already solved +- **VT-1.10** Third-party multipart-table module evaluated and its approach to partition-column + handling adopted + +## VT-2 · Establish query-engine behaviour +Determined empirically how the engine decides what to read. This drove the single most +important design decision. + +- **VT-2.1** Constraint routing established: which conditions are answered from directory + names versus by opening files +- **VT-2.2** Confirmed that a column stored inside the files can *never* be used to skip + directories — so the partition column must be stripped on write, or the layout delivers no + benefit at all and is slower than what it replaces +- **VT-2.3** Range-based partition-skipping mechanism investigated (an undocumented feature), + including which operators it supports; concluded unusable for this schema +- **VT-2.4** Findings made reproducible as `testfiles/vt-probe.q` + +## VT-3 · Writer implementation +Four overrides to the TorQ write path, 95 lines, applied as an overlay with no fork of TorQ. + +- **VT-3.1** Strip the partition column from stored data (the load-bearing change) +- **VT-3.2** Detect newly created partition directories and notify readers — only on creation, + not on every write +- **VT-3.3** Guarantee every table has a directory in every partition, before notifying +- **VT-3.4** Replace the overnight merge with a no-op plus a rollover notification +- **VT-3.5** Overlay install mechanism, working around TorQ's load order so the stock process + code does not clobber the overrides +- **VT-3.6** Compression path corrected for the deeper directory tree +- **VT-3.7** Writer configuration and partition-column declaration + +## VT-4 · Reader implementation +New process, 203 lines, serving history and the live day as one queryable object. + +- **VT-4.1** Tree scan and partition catalogue construction +- **VT-4.2** Live-view file opening, so appended rows are visible with no reload +- **VT-4.3** Virtual-table construction, one per table, in the namespace clients query +- **VT-4.4** Table list discovered from disk rather than configured, so a schema addition needs + no reader change +- **VT-4.5** Symbol-domain handling, ordered so directories are never opened against a stale + domain +- **VT-4.6** Registration with the writer, plus handlers for the new-partition and end-of-day + notifications +- **VT-4.7** Backstop rescan timer, so a lost notification degrades to staleness rather than + blindness +- **VT-4.8** Operates correctly with no writer present — starts in any order, survives a writer + restart, can serve an archived tree +- **VT-4.9** Reader configuration, including the history-window control +- **VT-4.10** Process attributes published for service discovery + +## VT-5 · Packaging and operability +- **VT-5.1** Extracted into a standalone package, independent of the Finance Starter Pack it + was derived from +- **VT-5.2** Single environment file; verified portable across paths and TorQ versions +- **VT-5.3** Start and stop scripts covering the full stack +- **VT-5.4** 7-assertion end-to-end self test, exits non-zero on failure, suitable for a smoke + test +- **VT-5.5** README covering setup, running, querying and layout +- **VT-5.6** Source repository restored to its prior state, with the parallel FX demo intact + +## VT-6 · End-to-end verification +- **VT-6.1** Capture layer verified: directory shape correct, partition column confirmed absent + from stored files +- **VT-6.2** Directory skipping proven by deliberately corrupting one instrument — queries for + other instruments continued to work, proving the corrupt directory was never opened +- **VT-6.3** Appended rows confirmed visible with **no reload and no notification** +- **VT-6.4** New instrument measured end to end at **2 ms** from directory creation to + queryable +- **VT-6.5** Cold start verified with all state wiped and launched from an unrelated directory — + reader bootstraps from an empty database and grows as the writer works +- **VT-6.6** End-of-day rollover path exercised +- **VT-6.7** Zero errors logged across all runs + +## VT-7 · Failure-mode verification +- **VT-7.1** Writer's ordering guarantee deliberately broken to test the failure it prevents +- **VT-7.2** Tested with the gap in the first partition, the last partition, and for both an + empty and a populated table +- **VT-7.3** Established the true symptom — quietly incomplete results, not the documented hard + failure — and quantified it (half the data returned, no error) +- **VT-7.4** Confirmed the damage is bounded by the rescan interval rather than permanent +- **VT-7.5** Design document corrected; reproduction shipped as `testfiles/vt-gap-test.q` + +## VT-8 · Scaling measurement +- **VT-8.1** Synthetic tree generator built, to test sizes beyond what the demo feed produces +- **VT-8.2** Kernel resource usage measured (memory mappings, file descriptors, resident + memory) from 400 to 40,000 partitions +- **VT-8.3** System limits confirmed on the host +- **VT-8.4** Query latency measured against partition count — **targeted queries confirmed flat + at 300–400 µs**, which verifies the design's central claim at scale +- **VT-8.5** Established that the documented memory-mapping ceiling **does not apply**, and + identified why the original measurement was misleading +- **VT-8.6** Identified the real scaling limit (rescan time) and quantified it +- **VT-8.7** Design document corrected; reproduction shipped as `testfiles/vt-scale-test.q` + +## VT-9 · Rescan optimisation +- **VT-9.1** Designed around the observation that historical partitions are immutable, so only + the live partition can change +- **VT-9.2** Implemented catalogue and open-view reuse for immutable dates +- **VT-9.3** Cache invalidation handled for the cases that need it — end of day, and after + compression rewrites historical files +- **VT-9.4** Verified the optimised path returns results identical to a full rescan +- **VT-9.5** Confirmed it never blocks discovery of a late-appearing historical partition +- **VT-9.6** Measured: **154x faster at 40,000 partitions, and now flat with respect to history + depth** — the scaling limit from VT-8 is removed + +## VT-10 · Client compatibility assessment +- **VT-10.1** 38 common query operations probed, now by a runnable test rather than by hand +- **VT-10.2** Established that 22 work directly and 9 fail only when applied to the table + object — all 9 work when wrapped in a `select`, making migration mechanical +- **VT-10.3** Identified the one genuine gap: tooling that *discovers* table names sees nothing +- **VT-10.4** Corrected two incorrect claims in the design document, one of which was a faulty + test rather than a real limitation + +--- + +## Key findings + +**The core premise is verified, not just argued.** A targeted query — one date, one instrument +— runs in 300–400 µs and **does not slow down as history grows**, measured from 400 to 40,000 +partitions. This is the benefit the whole design exists to deliver. + +**Two documented assumptions were proved wrong by measurement.** Both had been stated +confidently in the design and were corrected once tested: + +- A predicted memory-mapping ceiling, which would have limited the design to roughly nine days + of history, **does not exist**. The original measurement was valid but measured a file-open + method the reader does not use. Actual cost is 891 bytes of memory per partition. +- A predicted hard failure when a partition is incomplete **does not occur**. The reader + instead returns quietly incomplete results — a worse failure mode than the one expected, + because nothing surfaces it. + +Both corrections are recorded in the design document alongside the original claim, so the +reasoning is auditable rather than silently rewritten. + +**Client impact is smaller than first thought.** 27 of 38 operations work directly. Eleven fail +on the table object but work when wrapped in a `select` — a mechanical edit to existing +scripts, not a redesign. One genuine gap remains: tooling that discovers table names rather +than being told them sees nothing. + +**It agrees with kdb+.** Given identical data, 16 of 19 queries return identical results. The +three that differ all *error* rather than returning a wrong number, and all three work when the +query is wrapped in a `select`. **No silent disagreement was found** — the important property, +since an error is recoverable and a quietly wrong number is not. + +**One naming decision needed.** The reader exposes the partition column as `instrument` +regardless of what the source schema calls it (`sym` here). Existing queries naming +the schema column would need changing. Cheap to fix now, expensive once clients are written. + +**One external blocker.** A single virtual table cannot span partitions with different column +layouts, which prevents presenting old-format and new-format data under one table name. +Requires a change from KX. Workaround exists (separate table names); it pushes complexity onto +clients. + +## VT-11 · Validation against stock kdb+ +Two databases built from the **same captured bytes** — the new date+instrument format, and a +conventional date-partitioned kdb+ database with the partition column stored as a real column. +The conventional one stood up as a plain q process with no TorQ involved. 19 queries run against +both and compared. + +- **VT-11.1** Conventional control database built from identical data +- **VT-11.2** Query battery defined across filters, aggregations, grouping, time ranges, + weighted averages, empty results and the partition column +- **VT-11.3** Results compared after normalising row order, column order and symbol + representation — **16 of 19 identical** +- **VT-11.4** The 3 differences characterised: in every case the virtual table **raises an error + rather than returning a wrong answer**, and all 3 work when wrapped in a `select` +- **VT-11.5** Two representation differences identified and assessed as non-defects; the + partition column's exposed name has since been made configurable and set to match the schema, + so client queries port across unchanged +- **VT-11.6** Reproducible as `testfiles/vt-compare-kdb.sh` + +## VT-12 · Load and volume testing +Rate-controlled generator plus a harness that bursts volume through a clean stack and polls the +reader until every row is visible, so the figure measures the whole chain. + +- **VT-12.1** Load generator built (`code/tick/loadfeed.q`, `code/loadtest.q`) with configurable + volume, batch size and instrument universe; harness `./loadtest.sh` +- **VT-12.2** Throughput established across five configurations up to 4M rows — + **peak 1,447,000 rows/sec end to end, with zero data loss and no errors in any run** +- **VT-12.3** Instrument-count cost quantified: same volume across 50, 500 and 2,000 instruments + reduces write throughput (637k → 216k → 97k rows/sec) and scales whole-database operations + with directory count +- **VT-12.4** Query latency re-measured at realistic partition sizes: selective queries track + **rows read, not database size** (0.2–0.3 µs/row), and a wider instrument universe makes a + single-instrument query *faster* +- **VT-12.5** Storage footprint quantified: **4.9x amplification** from 35 bytes/row at 50 + instruments to 172 bytes/row at 2,000 — the small-files cost, now measured rather than + asserted. The narrower the row, the worse the ratio: per-file overhead is fixed +- **VT-12.6** Burst-absorption behaviour identified: the writer buffers in memory when a burst + outpaces the flush (2,098,000 rows observed in one cycle), so a large burst is bounded by + writer RAM — needs a memory limit and an alert in a real deployment + +## VT-13 · Detection for the silent failure mode +The incomplete-partition failure from VT-7 produces wrong answers with no signal. The reader now +compares partition coverage across tables on every rebuild and says so. + +- **VT-13.1** Coverage-gap warning implemented: a table whose date coverage is a strict subset of + another's is reported by name, with the missing dates +- **VT-13.2** `coverage[]` exposed for monitoring, so a health check can poll it rather than + scrape logs +- **VT-13.3** Verified three ways: silent on a healthy database, fires on a real gap, and does + **not** repeat on subsequent rebuilds — it logs only when the gap set changes, and logs a + recovery message when gaps clear + +## VT-15 · Weekend compression +Scheduled, run against a live stack, and measured. Two of the three sub-points changed a +documented design claim. + +- **VT-15.1** Compression job built and scheduled: `code/processes/vtcompress.q` replaces the + stock process and applies the directory-classifier fix; `./compress.sh` runs it, with + `--dry-run` to report scope and `--test` to exercise it against the running stack. A cron + line for a Saturday window is documented +- **VT-15.1a** *Defect found in the documented fix.* The classifier override was specified to + live in the process's settings file. It cannot: settings load ~13 ms **before** + `code/common/compress.q`, which then reinstates the stock definition. The job runs, reports + success, and compresses nothing — the exact failure the fix was written to prevent. The + override now lives in process code, which loads after common code +- **VT-15.2** *Documented caveat disproved.* The design said readers must be sent a rollover + after compression or they would serve pre-compression data indefinitely. Measured: a reader + returns byte-identical results through the **same handle**, with no rollover, no re-map and + no restart, after every eligible column file was compressed and renamed over beneath it. The + reason is the one already established for memory mapping — a trailing-slash open is a live + view, so there is no stale inode to hold. **This unblocks VT-17** +- **VT-15.3** Compression ratio measured on real partition sizes: **83 % of the bytes and + 81 % of the disk** at 20,000 rows per instrument per day. The two numbers converge once + column files clear a filesystem block — and diverge sharply below it, where the same job + frees 83 % of the bytes and under 10 % of the disk. The ratio is a function of partition + size, not of the data, which is the small-files cost of instrument-splitting quantified +- **VT-15.4** Read cost measured properly — 1,000 samples per state, min and median: the same + single-instrument select goes from 382 µs to ~800 µs, **about +100 %** at these partition + sizes. So the trade is 75–80 % of disk for roughly double the query latency, not a free win. + The age tier is what makes that tolerable: recent data stays uncompressed and fast +- **VT-15.5** Threshold established for the enable/disable decision: compressed size lands on + the one-block-per-file floor immediately and stays flat, so the saving is decided by rows per + instrument per day. Below ~1,000 it recovers under 40 %; above ~10,000 it recovers over 80 % +- **VT-15.6** Recommendation implemented and shipped: an **age tier** (`minage 7`, so recent + data stays uncompressed and interactive queries run at full speed) and a **size gate** + (`.cmp.minfilesize 4096`, skipping files that cannot free a block). A/B measured across + uncompressed / gated / ungated: the gate compresses 350 files instead of 750 for an + **identical** result — same 8,220 kB on disk, same latency within noise. It buys less job + time and 53 % fewer files rewritten per weekend, not faster queries; the expectation that it + would cut read cost was tested and disproved + +## VT-16 · Multi-stack read +One reader now serves several capture stacks safely. The blocker was resolved by a third option +neither of the two originally proposed: name the domains apart rather than share one. + +- **VT-16.1** *Blocker, as previously reported:* two capture trees could not have independent + `sym` files. The reader loaded each root's domain in turn and the last won, so every earlier + root's symbol columns silently resolved to the wrong values +- **VT-16.2** *Decision.* The ticket offered a shared enumeration domain or one reader per + stack. One reader per stack was rejected outright — it pushes the join to the client, which + this design exists to avoid. `load` binds a global named after the **file**, and each column + records which domain it belongs to, so `sym` and `symb` coexist in one process with each + column resolving through its own. Verified at the kdb+ level before committing to it +- **VT-16.2a** *An objection of ours turned out to be wrong.* The shared domain was also + rejected on the grounds that two writers appending to one `sym` file risk corrupting it. + Measured (`testfiles/vt-sym-concurrency.q`): six concurrent writers, 1,800 enumeration calls on an + overlapping vocabulary, **zero duplicates and zero indices invalidated** — the primitive + locks. The real constraint is storage, not concurrency: sharing means both stacks writing one + inode, symlinked into each root, so it needs a shared filesystem. Two *copies* is the unsafe + configuration, because they diverge on the first new symbol. Both options are now supported + and documented, with the choice driven by storage topology +- **VT-16.3** Implemented both sides as configuration: `symdomain` in the writer's settings + (redirecting `.Q.en` to `.Q.ens`, one line covering every enumeration site), and domain + *discovery* in the reader instead of assuming `sym`. The collision check now reports only the + genuinely unsafe case — two roots using the same name for different contents +- **VT-16.4** **Both modes work, and the reader needs no configuration to tell them apart** — + it discovers whatever domain files exist per root. The choice is made per stack on the writer + side: leave `symdomain` at `` `sym `` and symlink one file into each root, or set a different + `symdomain` per stack. Verified across all three configurations a deployment can reach — + separate domains, one shared file symlinked into both roots, and the broken middle case of two + copies under one name — by `testfiles/vt-multistack-test.q`, 17 assertions. The shared case checks + that a write through the link *extends the shared file and leaves it a link*, which is what + would otherwise silently turn mode two into mode three +- **VT-16.5** *Limitation found, measured and documented.* Grouping on a symbol column held + inside the files splits per domain (`` `sym$`book1 `` and `` `symb$`book1 `` are distinct), so + a cross-stack `by side` returns one group per domain rather than one per value. Grouping by + the **partition column** — the query this layout exists to serve — is correct, as is + filtering, and `value` on the column fixes the rest. Asserted in the test so an engine change surfaces it +- **VT-16.6** Cost measured: same data across one tree versus two costs 73 → 119 µs on rebuild + (one extra directory read per date per table) and 254 → 278 µs on a selective query + +## VT-17 · Make end of day flat +Rollover dropped the reader's whole cache, so end of day rescanned all history — the last cost +that still grew with retention. It now forgets one date. **No operation in this design scales +with how much history is attached.** + +- **VT-17.1** Cache drop removed from `rollover`, replaced with `dropdates` — targeted + invalidation of a single date, keeping the rest of the catalogue and its opened views +- **VT-17.2** *Silent data-loss trap found and avoided.* Simply deleting the `dropcache[]` call + is wrong: `mutable` is `d>=current`, so once `current` moves to the new date the date that + just **closed** reads as immutable, and the reader reuses its stale catalogue. Any directory + the writer created in its final flush of the day would be on disk and permanently invisible, + with no error. The drop must therefore happen **before** `current` moves +- **VT-17.3** Measured: end of day now tracks the live rebuild instead of the cold rescan — + 2/3/7/8/11 ms at 400 to 40,000 directories, against 18/90/388/731/1,540 ms before. + **Flat with retention, 140x cheaper at 40,000 directories.** 2.4 ms on the running stack +- **VT-17.4** `testfiles/vt-rollover-test.q` added, covering the trap in VT-17.2 specifically: it + creates a directory the reader has not scanned, rolls over, and checks the rows survive. + Verified to fail against the naive version of the change and pass against the shipped one + +## VT-18 · Writer restart preserved the wrong layout — FOUND AND FIXED +Found by accident on 2026-08-19, while restarting the stack for an unrelated config change. +Not a TorQ gap: a gap in this pack's own overlay, and the one class of defect the existing +tests could not reach. + +- **VT-18.1** *Defect.* Restarting the writer is the normal recovery path: TorQ deletes the + current partition and rebuilds it from the tickerplant log. That replay runs inside + `.wdb.startup[]`, roughly a second **before** `.proc.init[]` runs the init list — which is + where the pack installed its overrides. So every partition rebuilt by a replay was written by + the **stock** writer, keeping the partition column in the files. All 22 partitions came back + 8 columns wide instead of 7, putting the database into the mismatched-column state of §9.3: + silently wrong answers, no error +- **VT-18.2** *Why it was never caught.* Every test in the pack wipes `var/` first, so the + tickerplant log is empty and the replay is a no-op. The defect is only reachable by + restarting a writer whose log has data — which is exactly what a real deployment does +- **VT-18.3** *Fix.* `startup` is defined in `code/wdb/origstartup.q`, which loads before the + overlay, and `wdb.q` only calls it — so the overlay wraps it and installs the overrides + first. Verified from the writer's log: overrides at line 171, replay at 184 (previously 3573 + and 185). The init-list registration stays as a fallback for a writer that never subscribes +- **VT-18.4** `testfiles/vt-replay-test.q` added, asserting the ordering out of the writer's own log + plus the consequence in the tree — no table may hold two column widths, and the partition + column may not appear in any file + + +## VT-19 · A new symbol value read as null for up to 30 seconds — FOUND AND FIXED +Found on 2026-08-19 while answering a question about what the live view guarantees. Second +defect in the pack's own code, and the same shape as VT-18: an invariant that was true for the +case the tests covered and false for one they did not. + +- **VT-19.1** *Defect.* §5.2's result — appends need no rebuild — holds for rows but not for + symbol *values*. Symbol columns are indices into an enumeration domain the reader holds in + memory. A value never seen before is appended to the domain file and written into an + **existing** partition, so no directory is created and nothing is announced (§4.1 is + edge-triggered on directories). The rows appear immediately; that column reads as **null** + until the domain is reloaded. Silently +- **VT-19.2** *Why it was never caught.* The only caller of `loadsym` was `rebuild`, so the + window was the 30-second backstop sweep. Every test either wipes the database first or works + with a fixed symbol universe, so no test ever introduced a new value into a live partition +- **VT-19.3** *Fix.* The domain now has its own timer, `symsweep`, defaulting to one second. + `symchanged` is a single `hcount` per root — far cheaper than the ~10 ms rebuild it used to + ride on, so it can run often without making the reader rescan directories +- **VT-19.4** Measured: **1.1 s, 1.9 s, 2.6 s** over three runs, against up to 30 s before. The + floor is the writer's own flush interval, since the value is not in the domain file until the + writer flushes +- **VT-19.5** `testfiles/vt-symdomain-test.q` added, asserting both halves: that the rows arrive + immediately, and that the value resolves inside a five-second budget + +## VT-20 · The tests all started from a clean, static world +Both defects found on 2026-08-19 (VT-18, VT-19) were invisible for the same structural reason, +not by coincidence: every test either builds a scratch tree from nothing or assumes a fixed +symbol universe. Neither *restart with prior state* nor *novelty at runtime* was reachable by +any of them. Four more cases from that class were then worked through deliberately. + +- **VT-20.1** *Restart under load — clean.* A writer restart deletes and replays the live + partition; done underneath a running reader, the reader and disk agreed throughout and counts + kept rising. TorQ's source only warns about the Windows case (delete fails); on Linux it + succeeds and the trailing-slash view simply re-resolves +- **VT-20.2** *Name collision — confirmed, and worse than documented.* Two instruments whose + names differ only in punctuation share one directory. Measured: the hyphenated one returns + **0 rows** and the other returns **all of them**. Both answers wrong, neither errors. The + design had called this "rows interleave". No detection exists and none is cheap — by the time + the writer holds a directory name the distinguishing character is gone. + `testfiles/vt-collision-test.q` +- **VT-20.3** *On-disk damage — three kinds, one dangerous.* A truncated column attaches and + **silently returns the shortest column's row count**; a missing `.d` is detected, excluded and + logged; a corrupt column errors loudly. Blast radius is contained by partition elimination: + selective queries on healthy instruments are unaffected, whole-table queries fail — including + ones that never name the damaged column. `testfiles/vt-damage-test.q` +- **VT-20.4** *A table added mid-life works, and trips the gap check.* It is discovered from the + tree with no config change and is immediately queryable. It also has fewer dates than its + peers, which is indistinguishable on disk from §4.6's failure, so it is reported as a coverage + gap on every rebuild for as long as the older dates are attached. Correct by its own rules; + worth knowing before adding a table to a live database. `testfiles/vt-newtable-test.q` +- **VT-20.5** *Tickerplant restart — capture stalls permanently.* See §4.8. The feed half is + fixed; the writer half is an operational procedure (restart it — the replay loses nothing) and + a design question deferred rather than patched. `testfiles/vt-tprestart-test.q` + +## VT-21 · The three moments a test never covered — TWO DEFECTS FOUND AND FIXED +VT-20 closed the *stale state* and *novelty* gaps. Three remained, all of the same shape: a +query or a process arriving in the middle of something rather than after it. Working through +them found two defects, one of which had been running in production configuration all along. + +- **VT-21.1** *A query arriving mid-write — safe, and for a reason worth knowing.* A splayed + write extends columns one at a time, so a partition genuinely has ragged column lengths on + disk while it is written; **200 of 5,000 concurrent reads landed in that window**. Not one + returned a torn row or an error. A splayed table is cut to its shortest column, so a short + read is a consistent *prefix* — the same rule that makes truncation silent (VT-20.3) is what + makes this safe. Mid-rebuild is safe for a different reason: q's main loop serialises, so the + cost of a rebuild is paid as **queuing**, never as inconsistency. `testfiles/vt-inflight-test.q` +- **VT-21.2** *A reader with no writer froze the live partition — FOUND AND FIXED.* `.vtidb.current` + decides which dates are cached forever. When the reader could not ask the writer, it fell back + to `.z.D` — and with a non-zero roll offset the writer is still filling *yesterday* for the hours + after midnight GMT. In that window the live date was marked immutable and cached: new + instruments were on disk, absent from every query, with nothing logged. The live partition is + now taken from the newest directory on disk, which the writer cannot contradict, and the + writer's own answer is still preferred when it is ahead. §5.8, `testfiles/vt-restart-test.q` +- **VT-21.3** *A fourth kind of damage — FOUND, MEASURED, DELIBERATELY NOT GUARDED.* + `.d` is written before the columns, so every new partition passes through a state where it + names files that do not exist. A lazy `get` accepts it and even counts it correctly, and then + **every whole-database query fails** — one directory is enough, because all of them must be + opened. A guard was built and reverted. The transient race is the 30-second sweep landing in + the daily creation burst: **once per ~9 years at 10 instruments, once per ~2 months at 500, + once per ~6 days at 5,000**, and it heals on the next sweep. The only permanent source is a + full disk — where capture has already stopped, and a reader that keeps answering while + silently omitting an instrument is worse than one that fails loudly. Cost of the guard was + ~45% of a full rescan. §5.7, `testfiles/vt-diskfull-test.q` asserts the unguarded behaviour +- **VT-21.4** *Disk full — nothing is lost, but a retry duplicates.* ENOSPC arrives as a normal + q error naming the file; the process stays up and keeps serving. TorQ empties the in-memory + table only after the upsert loop and the pack's override rethrows, so the rows survive for the + next flush. But the partitions written *before* the failure are already on disk, and the retry + re-upserts the whole buffer with nothing to dedupe it. After any ENOSPC, check the partitions + that succeeded, not only the one that reported. §8.5, `testfiles/vt-diskfull-test.q` + +- **VT-21.5** *Table discovery still scanned all of history — REMOVED.* Asked to justify each + gap's code by probability rather than by possibility, one item did not survive: `tablelist` + did a `readdir` per **date** on every rebuild so that a table appearing mid-life would be + found. At 250 dates that was 2.6 ms of a 4.2 ms rebuild, growing with retention for ever, to + notice something that happens once in a deployment's life. It now scans the live partition + only — where a new table can actually appear — plus every date once on a cold catalogue. + Live rebuild at 250 dates: **4,194 µs → 1,428 µs**, the same as hard-configuring `tabs`. + At 40,000 partitions the sweep's rebuild went 10 ms → 6 ms and is now flat. The cost is that + a table added to an already-rolled date needs `dropcache[]`, which is the rule §6.1 already + states for a directory added to a past date. §5.3, `testfiles/vt-newtable-test.q` + +- **VT-21.6** *A writer restart duplicated the day it was meant to rebuild — FOUND AND FIXED.* + Recovery deletes the live partition and replays the tickerplant log, but `clearwdbdata` runs + against `getpartition[]`, which TorQ seeds from `.proc.cd[]` — the **calendar** date. Under a + roll offset the tickerplant is on a different date, so the delete missed, the real partition + survived untouched, and the replay wrote the whole day on top of it. `fixpartition` corrects + the date afterwards, too late, and its corrective branch only fires when the wrong directory + exists. **Measured on the live stack: 442 duplicate rows on 2026.08.18 from a single restart**, + all inside the replayed window. The pack now seeds from `.eodtime.getday`, the same function + the tickerplant uses to date its own logs, so the two agree by construction at any offset — + including none, where it reduces to the plain date. `appconfig/settings/wdb.q`, + `testfiles/vt-partition-test.q` (12 assertions, fails against the stock seeding) +- **VT-21.7** *Partitions created during log replay are never announced.* Found while verifying + the above. `replaymaxrowcheck` calls `savetables` directly rather than through `savetodisk`, + so the `vtnew` edge signal accumulates during the replay and is then cleared by the next + timed flush before anything reads it. Consequences are small and measured: readers pick the + partitions up on the 30-second sweep instead of immediately, and `vtfill` does not run for + them, so a table with no data in that partition keeps no empty directory. **Not fixed** — + logged here because the impact is bounded and the fix belongs with the §4.8 re-subscribe + question rather than on its own. + +- **VT-21.8** *A writer restart under-reports to readers for the length of the replay.* The + restart deletes the live date and rebuilds it, and the reader is not told. Three states, + measured: while the catalogue still points at deleted directories every query **fails loudly** + naming the missing file; once the sweep rescans, the reader serves history only and today is + **silently absent**; through the replay, whole-database counts climb monotonically + (650 → 1200) and are **short but not wrong**. Selective queries on instruments already rebuilt + are exact throughout and history is never at risk. §4.6's coverage check cannot see any of it, + because the date leaves every table at once and the check only compares tables against each + other. Not a reader defect — the rows genuinely are off disk — but worth knowing before + restarting a writer underneath anything that reports numbers to people. + `testfiles/vt-wdbrestart-test.q` (16 assertions) + +- **VT-21.6** *`vtfill` never ran during a log replay — FOUND AND FIXED.* The load-order fix of + VT-18 makes the replay write the right layout; it does not make it run the rest of a flush. + TorQ's `replaymaxrowcheck` calls `savetables` directly, so `vtnew` filled correctly and was + then thrown away by the `vtnew::()` opening the next flush — meaning §4.6's empty-directory + fill was absent on the routine recovery path. Found on a real restart after an overnight + shutdown: nine instruments had a `trade` directory and no `quote` one. Fixed by carrying the + pending list across the boundary, which costs nothing in steady state. §4.7, + `testfiles/vt-replay-test.q` + +VT-21.2 shares the signature the whole project keeps returning to: correct bytes on disk, no +error anywhere, and an answer that is quietly incomplete. VT-21.3 is the counter-case, and the +reason the guard was reverted — there, *failing* is the honest behaviour, and suppressing it +would have manufactured exactly that signature. + +--- + +# Remaining work + +## VT-14 · Legacy data migration — DESIGNED, PROVEN, DELIBERATELY NOT BUILT +Held open on purpose. This is a greenfield deployment with no history to migrate, and the +approach depends on the open KX item below: if one table can span both formats, the right +design is a single table name and the two-name workaround becomes dead code. Building it now +would ship a migration path nobody has asked to use. + +Prototyped far enough to retire the risk, then reverted (§9.6 records the findings and what it +takes to rebuild — roughly 40 lines in the reader). + +- **VT-14.1** ✓ *Proven.* A conventional date-partitioned database built from real captured + data attaches beside the capture tree, keyed on date alone, under its own table name +- **VT-14.2** ✓ *Proven.* Row counts, symbols and numerics identical to a plain read of the + same files; and **280 µs conventional load versus 320 µs through the reader**, so the + design's "correct, and no faster" claim is now measured — ~14 % overhead, not a regression. + (Measured before the schema was switched to the Starter Pack's `trade`/`quote`; the design + is schema-independent and the legacy path is no longer in the code) +- **VT-14.3** ✓ *Proven.* count, aggregate, time filter, distinct, sort, `meta`, and a union + across both table names all behave +- **VT-14.4** Cutover procedure — **open**, and correctly so: it depends on whether clients end + up with one table name or two, which is what the KX item decides + +**Trigger to build it: KX answers on differing column lists, or a deployment with real history +appears.** + +--- + +## Risks + +| risk | impact | mitigation | +|---|---|---| +| ~~Results may not match stock kdb+~~ | Retired — VT-11 verified agreement | closed | +| ~~Untested at realistic volume~~ | Retired — VT-12 measured to 4M rows, no loss | closed | +| Storage amplifies 4.9x at wide instrument universes | Capacity planning | Measured; size the estate on 172 B/row, not row data | +| Large bursts bounded by writer RAM | Writer could exhaust memory | Set a `-w` limit and alert | +| ~~Partition column exposed under a different name than the schema~~ | Retired — now configurable via `partitioncol`, set to the schema's name | closed | +| ~~Incomplete partitions fail silently~~ | Retired — prevented on the writer, detected on the reader (VT-13) | closed | +| ~~Several capture roots cannot have independent sym files~~ | Retired — VT-16 named the domains apart | closed | +| Cross-stack `by` on a symbol column splits per domain | Wrong group count in multi-stack reports | Documented; use `value`, or share one domain (§8.3.1) | +| ~~End of day rescans all history~~ | Retired — VT-17 made it O(instruments) | closed | +| One table cannot span both data formats | Clients must know two table names | Raised with KX; workaround in place | +| Small-files count | Constrains filesystem choice and backup tooling | Known and quantified; inherent to the design | +| **A tickerplant restart stalls capture until the writer is restarted** | Silent — every process stays up and looks healthy | Feed fixed; writer needs a manual restart, which replays and loses nothing. Detect with `vt-tprestart-test.q` (§4.8) | +| **A truncated column file returns fewer rows, silently** | Wrong answers, no warning | Demonstrated (`vt-damage-test.q`). Specific to **uncompressed** columns: a compressed one carries a metadata header kdb+ validates, so the same damage raises instead. No detection exists for the uncompressed case | +| **Instrument names differing only in punctuation** | **One becomes unqueryable, the other absorbs its rows — silently** | Demonstrated (`vt-collision-test.q`). No detection exists. Hash or escape identifiers containing `.` `-` `/` before they reach the parted column | +| Client scripts need edits | Migration effort for existing dashboards | Quantified: 11 of 38 operations need a `select` wrapper (`vt-compat-test.q`) | +| ~~A new symbol value reads as null until the next rebuild~~ | Retired — VT-19 gave the domain its own timer | closed | +| ~~Writer restart rebuilt partitions in the wrong layout~~ | Retired — VT-18 installs the overrides before replay | closed | +| A writer restart deletes and rebuilds the live partition | Anything not in the current tp log is not restored | Stock TorQ recovery; know it before restarting a writer | +| **A partition whose `.d` names columns that are not on disk** | **Every whole-database query fails, not just that partition** | Not guarded, by decision (§5.7): transient case heals within one sweep, permanent case is a full disk where failing loudly is correct. Pinned down by `vt-diskfull-test.q` | +| **A disk-full retry re-writes partitions that already succeeded** | Duplicate rows, silently, in the partitions written *before* the failure | Demonstrated (`vt-diskfull-test.q`, §8.5). No dedupe exists — check those partitions after any ENOSPC | +| **The same `(date;instrument)` under two roots** | Rows served twice, no error, one key | Demonstrated (`vt-inflight-test.q`, §8.3.1). Keep stack instrument universes disjoint | +| ~~Reader with no writer freezes the live partition~~ | Retired — the live date is taken from disk, not `.z.D` | closed (§5.8) | +| ~~A writer restart duplicates the day under a roll offset~~ | Retired — the partition is seeded from the business date | closed (VT-21.6). Rows duplicated by a restart *before* the fix stay on disk — check any partition written across a pre-fix restart | +| Partitions created during tp log replay are not announced | Up to 30s of staleness after a writer restart; `vtfill` skipped for them | Known, not fixed (VT-21.7) | + +--- + +## Notes for reviewers + +Every claim in the design document has a runnable script that reproduces it, all under +`testfiles/`: + +``` +testfiles/vt-probe.q how the query engine routes conditions +testfiles/vt-limitations.q what works and what does not +testfiles/vt-sample-legacy.q attaching an existing date-partitioned database +testfiles/vt-gap-test.q what an incomplete partition actually does +testfiles/vt-scale-test.q resource use and latency versus partition count +testfiles/vt-compare-kdb.sh agreement with a conventional kdb+ database, same data +./compress.sh --dry-run what the weekend job would touch, and the ceiling on what it can free +./compress.sh --test compression underneath a live reader; ratio and read cost +testfiles/vt-compress-ratio.q where the saving actually lands, bucketed by original file size +testfiles/vt-compress-sizes.q how the saving scales with rows per instrument per day +testfiles/vt-compress-ab.q uncompressed vs gated vs ungated: disk and latency, 1000 samples +testfiles/vt-rollover-test.q end of day keeps the date it just closed, and does not rescan history +testfiles/vt-multistack-test.q one reader over two capture stacks: all three domain configurations +testfiles/vt-sym-concurrency.q concurrent writers against one shared enumeration domain +testfiles/vt-replay-test.q overrides installed before the tp log replay, and 4.6's fill survives it +testfiles/vt-symdomain-test.q rows arrive live; a brand new symbol value resolves within seconds +testfiles/vt-collision-test.q what two instruments sharing a sanitised directory name actually do +testfiles/vt-damage-test.q truncated / .d-less / corrupt partitions, and the blast radius +testfiles/vt-newtable-test.q a table appearing mid-life is discovered, and trips the coverage check +testfiles/vt-tprestart-test.q liveness: writer subscribed, database growing, feed not holding a handle +testfiles/vt-compat-test.q 38 client operations: what works, what needs a select wrapper +testfiles/vt-inflight-test.q a query arriving mid-append and mid-rebuild: short reads, never wrong ones +testfiles/vt-restart-test.q a reader started in the middle of a flush, and whether it heals +testfiles/vt-diskfull-test.q a genuinely full filesystem: what breaks, what survives, what duplicates +testfiles/vt-partition-test.q which partition a restart deletes, at any roll offset or none +testfiles/vt-wdbrestart-test.q what a reader serves while the writer deletes and replays the day +./loadtest.sh throughput, storage and latency under real volume +./selftest.sh end-to-end check against a running stack +./regress.sh runs the sixteen assertion tests above and summarises them +``` + +This was deliberate: two of the design's stated assumptions turned out to be wrong when +measured, and both were caught because the claims were made reproducible rather than asserted. diff --git a/docs/virtual-table-capture-pack.md b/docs/virtual-table-capture-pack.md new file mode 100644 index 0000000..1b3765d --- /dev/null +++ b/docs/virtual-table-capture-pack.md @@ -0,0 +1,2221 @@ +# TorQ Virtual-Table Capture Pack — Architecture + +A minimal TorQ capture stack that writes date+instrument partitioned data once and never +moves it again. History and live data are the same files; virtual tables are what make +those files queryable. + +This pack is standalone. It is an application overlay onto a TorQ checkout, in the same way +the FX positions POC is — and it carries its own copy of the Finance Starter Pack's `trade` +and `quote` schema and feed, so nothing outside it has to exist. + +Reference points, in the versions this was built and checked against: TorQ 5.2.12 and 5.2.15 +(`$TORQHOME`), the kdb-x `kx.pq.t` module (`$QPATH/kx/pq/t.k`), Jonathon McMurray's `mp` +multipart module, and the +[No-RDB Starter Pack](https://github.com/DataIntellectTech/TorQ-No-RDB-Starter-Pack) — see §11. + +Two companion scripts reproduce the evidence behind the decisions below: +`testfiles/vt-probe.q` (query-engine behaviour) and `testfiles/vt-limitations.q` (a presentable +summary of what does and does not work). + +--- + +## 0. Design decisions and why + +Every choice here was made against a specific alternative that was tried and rejected. If +you are picking this up cold, read this table first — most of the document is the evidence +behind one of these rows. + +| decision | the alternative | why we chose this | +|---|---|---| +| **Strip the partition column from the files** | leave `sym` in the data, as TorQ's `partbyattr` does | a column stored inside the files can never be used to skip directories — the engine opens the files to check it instead. Leaving `sym` in means `where sym=X` scans the entire database, and you have paid the small-files cost for nothing. §2.2, §4.5 | +| **Partition by date + instrument** | date only, as a normal HDB does | instrument becomes a directory, so a selective lookup is a directory lookup rather than a scan. This is the whole point of the design, and it only pays off given the row above. §4.2 | +| **No EOD sort** | sort and apply `p#` nightly, as the No-RDB pack does | `p#` exists to make rows for one instrument contiguous. Directory structure already achieves that, cannot go stale under append, and needs no maintenance. §4.2 | +| **Open files with a trailing slash** | plain `get`, then re-read on every flush | a trailing slash gives a *live* view that tracks appends, verified across processes. Without it every reader must re-open every file it holds, every second. This removes most of the refresh machinery. §5.2 | +| **Rebuild only when a directory appears** | rebuild on every write | appends are already visible via the row above, so the only event a reader must react to is a *new* directory — a few times a day, not once a second. §5.3 | +| **Writer creates every table's directory** | let readers cope with gaps | measured: a reader given a partition missing one table does not error — it serves that table with the whole date absent, silently, until its next rebuild. Cheap to prevent on write, invisible on read. §4.6 | +| **Legacy data partitioned by date only** | replicated links, or a nested list of instruments | both were tried. Replicated links duplicate every row once per instrument; nested lists and nulls are simply ignored. Neither can reduce what is read, because all the entries point at the same file. §9.2 | +| **Separate table names for old and new formats** | one table spanning both | a virtual table reads its column list from the first file only and assumes the rest match. Mixing formats silently drops rows or returns the wrong instrument. §9.3 — this is the one genuine blocker | + +--- + +## 1. Process inventory + +| proctype | procname | port | role | status | +|---|---|---|---|---| +| `discovery` | `discovery1` | 6001 | service discovery | unchanged | +| `segmentedtickerplant` | `stp1` | 6000 | capture, log roll | unchanged | +| `wdb` | `wdb1` | 6005 | date+instrument partitions, 1s flush | **config + 5 overrides** (§4.1, 4.2, 4.5, 4.6, 4.7) | +| `idb` | `idb1..n` | 6030+ | virtual table over the whole tree | **new implementation** | +| `feed` | `feed1` | 6014 | dummy feed | unchanged | +| `compression` | `cmp1` | 6040 | weekend column compression; not started with the stack (`startwithall=0`), binds its port only while running | **replaced** (`vtcompress.q`) | + +Deleted relative to the Finance Starter Pack: `rdb`, `hdb`, `sort`, `gateway`, sort workers. + +There is no RDB (the WDB's 1s flush makes on-disk data fresh enough), no HDB (the same tree +serves both), no gateway (one process already spans all dates), and no sort/merge +(the write layout *is* the final layout). + +--- + +## 2. The on-disk contract + +Everything hinges on what `partbyattr` actually writes. From +`code/processes/wdb.q:135-154` (`upserttopartition`), the directory is built as: + +```q +directory:` sv .Q.par[dir;pt;tablename],(`$"_"^.Q.an .Q.an?"_" sv string `TORQNULLSYMBOL^ensuresymlist[expt]),` +``` + +which produces: + +``` +$KDBWDB/ + sym <- enumeration domain (.Q.en target) + 2026.08.03/ + trade/ + AMD/ time price size stop cond ex side .d + AAPL/ time price size stop cond ex side .d + quote/ + AMD/ + 2026.08.04/ + ... +``` + +Three consequences, all of which drive the rest of the design: + +**2.1 This is not a q partitioned database.** `$KDBWDB/2026.08.03/trade/` contains +directories, not column files, and has no `.d`. `\l $KDBWDB` will not load it. In stock TorQ +this is fine because `partbyattr` is a *staging* format that only becomes queryable after the +EOD merge (`wdb.q:451`, `endofdaymerge`). We are deleting the merge, so something else has to +make the tree queryable. That something is the virtual table. This is the single reason the +whole design needs `kx.pq.t` rather than being a config change. + +**2.2 The partition column is written into the files, and it must not be.** This is the +single most important correction to make to stock TorQ, so it is worth being precise about. + +`upserttopartition` writes `r:?[tabdata;;0b;()]` — the full row, `sym` +column included. So every file directory carries a `sym` column whose value is constant and +equal to the directory name it sits in. + +That looks like harmless redundancy. It is not. When you query, the engine decides for each +condition in the `where` clause whether it can answer it from the **directory names** (in +which case it skips directories it doesn't need) or whether it must **open the files** and +check. The rule is simply: *if the column is stored inside the files, open the files*. + +So with `sym` written into the data, `` where sym=`AMD `` opens every instrument +directory in the database and filters each one down to all-or-nothing. The answer is +correct, but the instrument partitioning has bought you exactly nothing — while still +costing you the file count of §8.1 and the rebuild cost of §8.2. That is strictly worse +than a plain date-partitioned database. + +Remove the column and the same query becomes a directory lookup. This is what Jonathon +McMurray's `mp` module does on write: + +```q +/ exclude partkey cols from saved data (these will be virtual cols) +selcols:{x!x} cols[data] except key partkey; +``` + +and it is why his benchmark reports 42ms against 778ms for a standard splay with `g#`. + +The fix for our WDB is in §4.5. Note that the column is not lost — it is still queryable, +because the directory name supplies it. + +**2.3 Directory names are lossy, and the consequence is worse than "interleaved".** `.Q.an?` +maps non-alphanumeric characters to `"_"`, so `BRK-B` and `BRK_B` produce the same directory +name. Earlier drafts of this document described the result as the two instruments' rows +interleaving. Measured (`testfiles/vt-collision-test.q`), it is sharper than that: + +``` +publishing 4 rows for `ZZ-PQCA and 6 for `ZZ_PQCA + +directories created : ,`ZZ_PQCA <- one, not two +rows for `ZZ-PQCA : 0 <- 4 published, none returned +rows for `ZZ_PQCA : 10 <- 6 published, 10 returned +``` + +**One instrument becomes completely unqueryable** — its rows are on disk, under the other +instrument's name — and **the other silently absorbs them**, returning more rows than were ever +published for it. Both answers are wrong and neither errors. + +This is the one failure mode the writer cannot detect for itself: by the time it holds a +directory name, the character that distinguished the two instruments is already gone. Detecting +it would mean keeping a sanitised-to-original map and rejecting a second original that lands on +an existing name. + +For plain uppercase tickers it never fires. For identifiers containing `.`, `-` or `/` it does, +on the first collision. If the universe can contain those, hash or escape the value before it +reaches the parted column. + +--- + +## 3. Configuration + +### 3.1 `appconfig/process.csv` + +``` +host,port,proctype,procname,U,localtime,g,T,w,load,startwithall,extras,qcmd +localhost,{KDBBASEPORT}+1,discovery,discovery1,${TORQAPPHOME}/appconfig/passwords/accesslist.txt,1,0,,,${KDBCODE}/processes/discovery.q,1,,q +localhost,{KDBBASEPORT},segmentedtickerplant,stp1,${TORQAPPHOME}/appconfig/passwords/accesslist.txt,1,0,,,${KDBCODE}/processes/segmentedtickerplant.q,1,-schemafile ${TORQAPPHOME}/database.q -tplogdir ${KDBTPLOG},q +localhost,{KDBBASEPORT}+5,wdb,wdb1,${TORQAPPHOME}/appconfig/passwords/accesslist.txt,1,1,,,${KDBCODE}/processes/wdb.q,1,,q +localhost,{KDBBASEPORT}+30,idb,idb1,${TORQAPPHOME}/appconfig/passwords/accesslist.txt,1,1,60,4000,${KDBAPPCODE}/processes/vtidb.q,1,-s 8,q +localhost,{KDBBASEPORT}+31,idb,idb2,${TORQAPPHOME}/appconfig/passwords/accesslist.txt,1,1,60,4000,${KDBAPPCODE}/processes/vtidb.q,1,-s 8,q +localhost,{KDBBASEPORT}+14,feed,feed1,,1,0,,,${KDBAPPCODE}/tick/feed.q,1,,q +``` + +IDBs are stateless and identical — replicate by adding rows. `-s 8` per the "heavy use of -s" +goal; each IDB is a read-only access point over the same files, so there is no coordination +cost to adding more. Note the `wdb1` load path is stock `${KDBCODE}/processes/wdb.q` — the +WDB changes in §4 are an app-code overlay, not a replacement. + +On the kdb-x community edition, concurrent connection and memory caps apply. Start with one +IDB (as the No-RDB pack deliberately does) and add more as the licence allows. + +### 3.2 Environment + +Follow the No-RDB pack's convention of a single named data root with the stock variables as +aliases, so TorQ core and any stock settings that read them by name resolve to the same +place: + +```sh +export KDBDB=${TORQDATAHOME}/db +export KDBHDB=${KDBDB} +export KDBWDB=${KDBDB} +``` + +A kdb-x install also needs three variables that are typically not set in the +shell profile, without which `q` fails with `license error: no license loaded` and `use` +cannot resolve modules: + +```sh +export QHOME=~/.kx/q # empty directory, but q requires it to be set +export QLIC=~/.kx # where kc.lic lives +export QPATH=~/.kx/mod # module search path for `use` +``` + +### 3.3 `appconfig/settings/wdb.q` + +```q +// Virtual-table capture pack : WDB config + +\d .wdb +savedir:hdbdir:hsym`$getenv`KDBDB // one directory; sym file lives at $KDBDB/sym +writedownmode:`partbyattr // split by date + instrument (necessary, not sufficient - see 4.5) +mode:`saveandsort // sort phase is overridden to a no-op (see 4.2) +immediate:1b // flush on every timer tick, ignore maxrows +settimer:0D00:00:01 // ...every second +gc:0b // 1s cadence: do not gc on every flush +rdbtypes:hdbtypes:gatewaytypes:() // none of these exist +sorttypes:sortworkertypes:() +idbtypes:`idb +permitreload:0b // nothing to reload + +\d .servers +CONNECTIONS:`segmentedtickerplant`idb`discovery +``` + +`writedownmode:`partbyattr` gets you the directory *shape* but not the behaviour — on its +own it also writes the partition column into the files, which defeats the purpose (§2.2). +It must be paired with the override in §4.5. + +`savedir:hdbdir` matters: `savetablesbypart` enumerates with +`.Q.en[hdbsettings[`hdbdir];...]` (`wdb.q:173`), so this is what decides where `sym` lives. +The IDB must load that same file before opening any partition directory, and reload it whenever it +grows (§5.3). + +`gc:0b` is deliberate. `savetablesbypart` calls `.gc.run[]` after every table save +(`wdb.q:181`); at a 1s cadence with `immediate:1b` that is a garbage collection every second. +Let the timer-based `.gc` config handle it instead. + +### 3.4 `appconfig/sort.csv` + +Still required — `getsortparams` (`wdb.q:619-639`) exits if a `partbyattr` process has no +`p` attribute defined, and `getextrapartitiontype` reads this file to decide which column +becomes the directory level. + +``` +tabname,att,column,sort +default,p,sym,1 +default,,time,1 +``` + +Note the practical restriction: the parted column must be named consistently across tables, or +given a per-table row here. Both tables in this pack key on `sym` (`database.q`), so a single +`default` row would do; the per-table rows are written out anyway, so that adding a table with +a differently named identifier is an edit rather than a debugging session: + +``` +tabname,att,column,sort +default,p,sym,1 +trade,p,sym,1 +quote,p,sym,1 +``` + +### 3.5 `appconfig/settings/idb.q` + +```q +// Virtual-table IDB config +\d .vtidb +roots:enlist hsym`$getenv`KDBDB // list — one entry per capture stack (see 8.3) +tabs:`trade`quote +refreshmode:`notify // `notify (WDB-driven) or `timer +historydays:0W // how far back to map; 0W = everything + +\d .servers +CONNECTIONS:`wdb`discovery +STARTUP:1b + +\d .proc +loadprocesscode:0b +``` + +`database.q`, `code/tick/feed.q` and the STP settings are unchanged from this repo. + +--- + +## 4. What has to change in TorQ + +Six gaps, all verified against 5.2.15 source. None are large; all are load-bearing. + +They share one root cause. In stock TorQ, `partbyattr` is a **staging** format: data written +that way is not meant to be read, it exists only between a flush and the nightly merge that +turns it into a normal database. So every code path concerned with "data somebody might +query" deliberately excludes it. Delete the merge, as this design does, and each of those +exclusions becomes a bug. + +| gap | what TorQ assumes | fix | +|---|---|---| +| 4.1 | nobody reads staging data | overlay, ~10 lines | +| 4.2 | staging gets merged nightly | overlay, ~4 lines | +| 4.3 | readers load a normal database | rewrite (§5) | +| 4.4 | column files are three levels deep | settings override | +| 4.5 | the partition column belongs in the data | overlay, one expression | +| 4.6 | a later merge will even out the partitions | overlay, small | + +### 4.1 The WDB never notifies IDBs in `partbyattr` mode + +`wdb.q:190-193`: + +```q +savetodisk:{[] + changes:savetables[savedir;getpartition[];immediate;] each tablelist[]; + if[any[changes] and writedownmode in `partbyenum`partbyfirstchar`default;filldb getpartition[];notifyidbs[`.idb.intradayreload;enlist()]]}; +``` + +and `wdb.q:643-652`: + +```q +idbreload:{[pt] + if[writedownmode in `partbyenum`default`partbyfirstchar; + ... + notifyidbs[`.idb.rollover;pt] + ]; +``` + +`partbyattr` is excluded from both. This is consistent with stock TorQ (the data is not +queryable until merged) but means an IDB would never hear about a flush or a rollover. + +Fix — as an app-code overlay in `$KDBAPPCODE/wdb/vtwrite.q`, following the pattern the +No-RDB pack uses in `code/wdb/rollover.q`: + +```q +\d .wdb + +/ notify idbs on every flush; no filldb - .Q.chk is meaningless for this layout +vtsavetodisk:{[] + changes:savetables[savedir;getpartition[];immediate;] each tablelist[]; + if[any changes; notifyidbs[`.vtidb.refresh;enlist()]]; + }; + +applyvtwrite:{[] + .lg.o[`vt;"installing partbyattr flush notification"]; + savetodisk::vtsavetodisk; + endofdaysort::vteodsort; // see 4.2 + }; + +\d . +.proc.addinitlist".wdb.applyvtwrite[]"; +``` + +The deferral matters. `.proc.reloadcode` loads `$KDBAPPCODE//` at torq.q:643-644, +*before* the `-load` file, so a direct redefinition here would be clobbered when +`code/processes/wdb.q` loads afterwards. `.proc.addinitlist` (torq.q:12) queues the swap onto +`.proc.initlist`, which `.proc.init[]` runs last (torq.q:697). This is why `process.csv` can +keep pointing at the stock `wdb.q`. + +Overriding `savetodisk` this late is safe: the timer holds `(`.wdb.savetodisk;`)` as a symbol +(`wdb.q:532`) and resolves it at fire time. + +### 4.2 EOD must not merge + +With `mode:`saveandsort`, `endofday` calls `endofdaysort` directly (`wdb.q:221`), which for +`partbyattr` runs `endofdaymerge` (`wdb.q:458-461`) — exactly the operation we are removing. + +Fix — replace it with a rollover notification, installed by the same `applyvtwrite` above: + +```q +\d .wdb +vteodsort:{[dir;pt;tablist;writedownmode;mergelimits;hdbsettings;mergemethod] + .lg.o[`eod;"no-merge eod - partition ",string[pt]," stays in place"]; + notifyidbs[`.vtidb.rollover;enlist pt+1]; + }; +\d . +``` + +That is the entire end-of-day operation. The STP rolls its log on its own schedule +(`.stplg.multilog:`tabperiod`), `endofday` bumps `.wdb.currentpartition` (`wdb.q:228`), and +the next flush creates the new date directory. Nothing is copied, sorted or reloaded. + +Choosing `mode:`saveandsort` over `mode:`save` avoids `informsortandreload` logging a +spurious "no sortandreload process detected" error before falling through to the same +function (`wdb.q:515-524`). + +Why nothing has to be sorted at all is worth stating explicitly, because it is the substantive +difference from the No-RDB pack (§11). The only reason that pack sorts at EOD is to apply a +`p#` to `sym`, and the only reason it needs `p#` is that its data is date-partitioned, so +rows for one instrument are scattered through the partition. **`partbyattr` is a physical +`p#`**: the grouping the attribute describes is already expressed as directory structure, and +directory structure needs no maintenance, cannot go stale under append, and does not have to +be rebuilt at a day boundary. That is what makes "minimal EOD" achievable rather than merely +desirable. + +### 4.3 The stock IDB cannot load this layout + +`code/processes/idb.q:20-24` is `system "l ",1_string idbdir` — an ordinary q database load, +which fails on the four-level tree for the reason in §2.1. The stock IDB also derives its +paths from `.wdb.writedownmode` and only handles `` ` `` vs `currentpartition` (`idb.q:37`). + +This is a rewrite, not a patch — see §5. + +### 4.4 The compression process sees nothing to compress + +`code/common/compress.q`, `hdbstructure` classifies paths purely by depth: + +```q +t:update partition:split[;base],table:`$split[;base+1],column:`$split[;base+2] from t where splitcount=base+3; / partitioned +t:update table:`$split[;base],column:`$split[;base+1] from t where splitcount=base+2; / splayed +``` + +A `partbyattr` column file is at `base+4` (`root/date/table/instrument/column`), so it +matches neither branch, `table` stays null, and `showcomp` then does +`pathstab:delete from pathstab where table in `` ` `` — dropping every row. The compression +process runs successfully and compresses nothing. + +Fix — add the missing depth: + +```q +hdbstructure:{ + t:([]fullpath:(raze/)traverse x); + base:count "/" vs string x; + t:update splitcount:count each split from update split:"/" vs' string fullpath,column:`,table:`,partition:(count t)#enlist"" from t; + / date partitioned : partition/table/column + t:update partition:split[;base],table:`$split[;base+1],column:`$split[;base+2] from t where splitcount=base+3; + / partbyattr : partition/table/instrument/column + t:update partition:split[;base],table:`$split[;base+1],column:`$split[;base+3] from t where splitcount=base+4; + / splayed : table/column + t:update table:`$split[;base],column:`$split[;base+1] from t where splitcount=base+2; + t:update partition:{$[not all null r:"D"$'x;r;not all null r:"M"$'x;r;"I"$'x]}[partition] from t; + $[14h=type t`partition; t:update age:.z.D - partition from t; + 13h=type t`partition; t:update age:(`month$.z.D) - partition from t; + t:update age:{$[all x within 1000 3000; x - `year$.z.D;(count x)#0Ni]}[partition] from t]; + delete splitcount,split from t} +``` + +The instrument level is folded away, so per-column rules in `compressionconfig.csv` keep +working unchanged. + +**Where the override goes matters, and the obvious place does not work.** This is +configuration in spirit, so it belongs in `appconfig/settings/compression.q` — but settings +files are loaded before `code/common/compress.q`, which then redefines `hdbstructure` with the +stock version. Measured on this stack, the two loads are 13 ms apart: + +``` +15:27:51.407262 loading /…/appconfig/settings/compression.q +15:27:51.420287 loading /…/code/common/compress.q <- clobbers it +``` + +The failure is silent and looks exactly like having made no change at all: the job runs, logs +success, and reports nothing in scope. The override therefore lives in +`code/processes/vtcompress.q`, which replaces the stock compression process and is loaded via +`-load`, after common code. `appconfig/settings/compression.q` keeps only `hdbpath` and +`maxage`, with a comment saying why the rest is not there. + +Compression is safe to run underneath live readers — see §7, where this is measured rather +than assumed. Keep `minage` at 1 or more so the live partition is never touched. + +### 4.5 The partition column is written into the files + +The most important of the six, for the reason set out in §2.2: a column stored inside the +files can never be used to skip directories, so leaving `sym` in the data means every query +on it scans the whole database and the instrument partitioning achieves nothing. + +`upserttopartition` (`wdb.q:147-148`) selects the rows for each instrument but keeps all +columns: + +```q +r:?[tabdata; ; 0b; ()]; +``` + +The last argument is the column selection, and `()` means "all of them". Fix — select +everything *except* the parted column, the same way `mp` does: + +```q +\d .wdb + +/ drop the parted column(s) from the data: the directory name supplies them +vtupserttopartition:{[dir;tablename;tabdata;pt;expttype;expt;writedownmode] + directory:` sv .Q.par[dir;pt;tablename], + (`$"_"^.Q.an .Q.an?"_" sv string `TORQNULLSYMBOL^ensuresymlist[expt]),`; + keep:{x!x} cols[tabdata] except expttype; // <- the change + r:?[tabdata;{(x;y;(),z)}[in;;]'[expttype;expt];0b;keep]; + .[upsert;(directory;r);{[e] .lg.e[`vtwrite;"failed to save: ",e];'e}]; + .merge.partsizes[first ` vs directory]+:(count r;-22!r); + }; +``` + +installed alongside the others in `applyvtwrite` (§4.1). + +Two things to know about this. The column is not lost — queries still return `sym`, because +the directory name supplies it. And it is not optional: without it the entire design is +slower than the date-partitioned database it replaces. + +### 4.6 A partition missing one table silently loses that date + +Stock TorQ never hits this because the nightly merge produces a uniform database. Here the +partitions are permanent, so an instrument that quotes but has not yet traded leaves a +`quotes` directory with no `trades` sibling — an entirely ordinary state for a live feed. + +**Measured, not assumed** — step 5 of §10. Earlier drafts of this section predicted the +failure modes of a `\l`-style load: an error for a gap in a later partition, and a silently +dropped table for a gap in the first. **Neither happens with this reader.** It never loads a +partitioned database; it scans each table on each date independently, and `scandate` returns +an empty catalogue when a table directory is absent (`vtidb.q`, the `()~i:key p` line). That +tolerance is deliberate — and it is exactly what makes the failure quiet. + +What actually happens, with `trade` removed from one of two dates: + +``` +reader load : LOADED / no error +tables attached : `quote`trade / nothing dropped +dates for trade : 2026.08.16 / 2026.08.17 simply absent +count select from trade : 4086 / was 8172 - half the data, no warning +``` + +Gap position is irrelevant: first partition and last behaved identically. The symptom is +uniform, and it is the bad kind — **the table answers queries, and the answers are quietly +incomplete.** + +The damage is bounded by the reader's sweep rather than permanent: + +``` +gap present : 4086 +directory restored, no rebuild : 4086 / a new directory needs a rebuild (§5.2) +after one rebuild : 8172 +``` + +So breaking the fill-then-notify ordering costs up to `sweep` seconds of wrong answers, not a +broken process. That is milder than the original prediction in one sense and worse in the +sense that matters: nothing surfaces it. + +This is a writer-side fix, not a reader-side one. By the time a reader sees it the database +on disk is already malformed, and repairing it on every load is fixing someone else's mess. +Stock TorQ takes the same view — its WDB calls `filldb` (which is `.Q.chk`) after writing, +precisely so readers never see an incomplete partition (`wdb.q:576-579`). + +So: **when the writer creates a partition, it creates a directory for every table, not just +the ones with data.** An empty table with the right schema is enough. + +The ordering matters, and getting it wrong reintroduces the same bug as a race: + +``` +WRONG RIGHT +───── ───── +create USDJPY/quotes create USDJPY/quotes +notify readers create USDJPY/trades (empty) +reader rebuilds -> FAILS notify readers +create USDJPY/trades reader rebuilds -> ok +``` + +Fill first, then notify. Never the other way round. + +For repairing a database that is already malformed — or validating one at startup — a +`.Q.chk` equivalent is worth keeping to hand. It must take the union of tables across *all* +partitions, not just the first, or it will reproduce the silent failure: + +```q +/ helpers must be globals: nested lambdas in q cannot see enclosing locals +.mpchk.sd :{[p] r where 11h=type each key each r:` sv/:p,/:key[p]}; +.mpchk.has :{[p;t] not ()~key ` sv (p;t;`)}; +.mpchk.tmpl:{[parts;t] 0#get ` sv ((first parts where .mpchk.has[;t] each parts);t;`)}; +.mpchk.fix :{[tmpl;p;t] d:` sv (p;t;`); $[()~key d; [d set tmpl t; enlist d]; ()]}; + +mpchk:{[path;nk] // nk = number of partition levels + parts:{raze .mpchk.sd each x}/[nk;path]; + tabs:distinct raze {last each ` vs' .mpchk.sd x} each parts; + tmpl:tabs!.mpchk.tmpl[parts] each tabs; + made:(raze/) {[tmpl;p] .mpchk.fix[tmpl;p] each key tmpl}[tmpl] each parts; + count made + }; +``` + +--- + +### 4.7 The overrides are not in force during tickerplant log replay + +The other six gaps are things stock TorQ does that this design cannot tolerate. This one is a +gap in **the overlay itself**, and it stayed hidden for the whole build because every test in +the pack wipes `var/` before starting. + +Restarting the writer is the ordinary recovery path, and TorQ handles it by deleting the +current partition and rebuilding it from the tickerplant log — `clearwdbdata`, then a replay, +the log being the source of truth for the day in flight. The replay happens inside +`.wdb.startup[]`, which `wdb.q` calls at the bottom of its own load: + +```q +upd:.wdb.replayupd; +.wdb.clearwdbdata[]; +.wdb.startup[]; / subscribes, and replays the log +``` + +`.proc.init[]` — which runs the init list, and therefore `applyvtwrite` — runs *after* the +`-load` file has finished. Measured on a real restart, about a second later: + +``` +185 subscribe|replaying the log file(s) +3560 subscribe|finished log file replay +3573 vtwrite|installing virtual-table capture overrides +``` + +So every partition rebuilt by the replay was written by the **stock** writer, which keeps the +partition column in the files (§4.5). The tree then holds a mixture of stripped and unstripped +directories, which is precisely the mismatched-column state of §9.3 — silently wrong answers, +no error. On the run that found this, all 22 partitions came back carrying a `sym` column file +and the reader returned a schema with `sym` in it twice. + +Fix — wrap `startup` rather than relying on the init list. `startup` is defined in +`code/wdb/origstartup.q`, which loads *before* `$KDBAPPCODE/wdb/`, and `wdb.q` only ever calls +it, so a wrapper here survives where redefining anything `wdb.q` owns would not: + +```q +origstartup:startup; +startup:{[] + applyvtwrite[]; + origstartup[] + }; +``` + +The `.proc.addinitlist` registration stays as well — it covers a writer that never subscribes +(`saveenabled` off, or no tickerplant), and `applyvtwrite` is idempotent. + +**This is the one defect in the pack that a wiped-database test can never find**, which is why +`testfiles/vt-replay-test.q` asserts the ordering out of the writer's own log rather than by +inspecting behaviour after a clean start. + +#### The replay does not go through `savetodisk` either + +The load-order fix above puts the overrides in force, so the replay writes the right *layout*. +It does not make the replay run the rest of a normal flush. TorQ's replay flushes through its +own door: + +```q +replaymaxrowcheck:{[t;lmt] + if[(rpc:count[value t]) > lmt; + savetables[savedir;getpartition[];0b;t]]; / direct, once per table + }; +``` + +`savetables` is called directly, so `vtupserttopartition` runs and `vtnew` fills correctly — +with *every* directory, because `deletewdbdata` wiped the partition first — but `vtfill` and +`notifyidbs` never fire, because those live in `vtsavetodisk`. The accumulated list was then +discarded by the `vtnew::()` at the top of the first flush after the replay. + +The consequence is §4.6's failure arriving by the recovery path: an instrument that has rows in +one table but not another comes back without its empty directory in the other. Observed on a +real restart — nine instruments with a `trade` directory and no `quote` one. Harmless while +every table is busy, since a query returns no rows either way. Not harmless the moment a table +receives nothing for a whole day: that date then has no directory for it, which is served as +silently absent. + +The fix carries the list across the boundary rather than clearing it: + +```q +vtsavetodisk:{[] + pending:vtnew; / anything a replay's direct calls left + vtnew::(); / edge-triggered: only this flush counts + savetables[savedir;getpartition[];immediate;] each tablelist[]; + news:distinct pending,vtnew; + if[count news; vtfill . ' news; notifyidbs[`.vtidb.rebuild;enlist()]]; + }; +``` + +In steady state `pending` is empty, so this costs nothing; it does work only on the one flush +that follows a replay. Verified on a live restart: nine `creating empty quote` lines one second +after the replay finished, and the instrument directories matched again. +`testfiles/vt-replay-test.q` asserts it from the tree, so it holds however the current state arose. + +### 4.8 A tickerplant restart stalls capture, permanently and silently + +Restart the tickerplant and the stack does not recover. Every process stays up, the writer keeps +logging `enumerated trade table` once a second, the reader answers queries — and the row count +does not move. Measured: still stalled ten minutes later, well past the five-minute +`.servers` `RETRY`. + +Two independent causes. + +**The feed cached its tickerplant handle.** `code/tick/feed.q` did what the Starter Pack's own +feed does — `h:.servers.gethandlebytype[…]` once at load, then `h(".u.upd";…)` on a timer. When +the tickerplant dies that handle is dead, and `.servers` reconnecting afterwards updates its own +table, not a copy somebody took at startup. The only trace is in a log nobody watches: + +``` +ERR|timer|timer ID 9 failed with error Cannot write to handle 7. + OS reports: Bad file descriptor. +``` + +Fixed — the feed now resolves the handle on every publish, and skips the tick when there is no +tickerplant to publish to: + +```q +tphandle:{[] .servers.gethandlebytype[`segmentedtickerplant;`any] }; + +send:{[] + tp:tphandle[]; + if[not count tp; :()]; / tickerplant down - the next tick tries again + … + }; +``` + +**The writer does not re-subscribe.** `subscribe[]` is called only from `.wdb.startup[]`. TorQ +defines `.wdb.notpconnected[]` for precisely this condition — and never calls it, in `wdb.q`, +`rdb.q` or `chainedtp.q`. The predicate exists; nothing invokes it. + +This one is **not** fixed here, deliberately. Re-subscribing also re-runs `fixpartition`, which +deletes the current partition and replays — and the tickerplant has just rolled its log, so a +naive timer risks rebuilding the day from the wrong subset. That needs designing, not patching. + +The operational answer is to **restart the writer**, which replays the tickerplant logs and +loses nothing. Verified: 435 rows before, 2,005 immediately after, and rising. Data written to +the tickerplant while the writer was disconnected is all recovered — the stall costs freshness, +not data. + +`testfiles/vt-tprestart-test.q` turns the silent stall into a check: it asserts an active +subscription, asserts the database is actually growing, and asserts the feed is not holding a +cached handle. + +> **Note for anyone reading `.wdb.notpconnected[]` as a fix.** It reads `tickerplanttypes` +> unqualified, so it only resolves when the calling context is already `.wdb`. Called over IPC +> it raises a value error rather than answering. Query `.sub.SUBSCRIPTIONS` instead, which is a +> root-namespace table. + + +--- + +## 5. The IDB + +### 5.1 What a virtual table actually is + +From `~/.kx/mod/kx/pq/t.k`, the module exports `([mkT;mkP;tt;mt;fv])`. + +The module's own term for "the table a partition points at" is a **leaf**; elsewhere this +document says *partition directory*, which is the same thing. + +- `mkP` takes a dictionary `t!v` where `t` is a simple table of partition columns and `v` is + the matching list of leaf tables. It stores them internally as `t:t!([]t:v)` — a keyed + table from partition-key row to leaf. +- `tt` wraps a plain q table as a leaf. Its select handler is a straight functional select + (`. (?;t;c;b;a),v`), so a memory-mapped splayed directory works as a leaf. Leaves do not + have to be Parquet. +- Leaves in one `mkP` need not share a root, a format, or a granularity. + +Query execution, from `mkP`'s handler (`t.k:11-12`): + +1. `fe[tc]'(c;b;a)` splits the where clause, by clause and aggregations into the parts that + reference *only* virtual columns and the parts that reference any leaf column. +2. `ct[c;t]` prunes the partition table using min/max statistics columns (§9.3). +3. `ex[...;*u;0b;()]` filters the partition table by the virtual-only constraints. +4. Each surviving leaf is queried with the leaf-referencing constraints. +5. Results are re-aggregated, with the partition key columns prepended. + +### 5.2 The key fact: a trailing slash gives a live view + +Before the code, the property everything else rests on. + +Opening a splayed directory **with a trailing slash** returns a view that keeps tracking the +file as it grows. Without the slash you get a frozen snapshot: + +```q +a:get `:/db/leaf / no trailing slash +b:get `:/db/leaf/ / WITH trailing slash + +/ another process appends 3 rows, then 2 more +count a -> 5 5 5 / never moves +count b -> 5 8 10 / tracks +``` + +Both are type 98h; nothing distinguishes them at the type level. Verified across processes: +a separate q process appended three times and the reader saw every append with no reload. + +This is why `mp` needs no reload for appends — its `vtable` builds paths as +`` ` sv (path;tablename;`) ``, and that final backtick is what produces the slash. + +**What it removes from this design.** An earlier draft of this document had the IDB evict +every live directory from a cache each second, re-open it, and rebuild — plus a file-size +guard to avoid doing that for instruments that hadn't traded. None of it is needed. Writers +and readers stay in step with no coordination at all, and the reader only has to react when +a *new directory appears*. + +**What it does not cover.** A new directory — a new instrument, or the next day — is not +picked up, because the virtual table holds a fixed list of directories. That is the only +event requiring a rebuild, and it happens a few times a day rather than once a second. + +### 5.3 Implementation — `code/processes/vtidb.q` + +Built and verified. The process is ~150 lines; the shape is: + +```q +/ bind mkP at the root and fully qualified - see "what the sketch got wrong" below +.vtidb.mkp:(use`kx.pq.t)`mkP; + +\d .vtidb + +/ the (date;instrument;path) rows for one table on one date under one root. +/ NB the trailing ` is what makes the view live (§5.2). without it each partition +/ is a frozen snapshot and the reader never sees another row. +scandate:{[t;r;d] + p:.Q.dd[.Q.dd[r;d];t]; + if[()~i:key p; :empty]; + ([]date:count[i]#"D"$string d; instrument:i; path:{.Q.dd[.Q.dd[x;y];`]}[p] each i) + }; + +build:{[t] + m:scanall t; + v:open each m`path; / open tolerates a directory mid-creation + ok:where 98h=type each v; + m:m ok; + parts[t]:m; + @[`.;t;:;mkp ([]date:m`date; instrument:m`instrument)!v ok]; + count m + }; + +/ §5.2 - appends need no work. the ONLY event a reader reacts to is a directory +/ appearing. called by the wdb when it creates one (§4.1), and by the sweep timer. +rebuild:{[] + if[symchanged[]; loadsym[]]; / must precede any open: the enum domain grew + before:count each parts; + build each tablelist ds; + if[not before~count each parts; .lg.o[`vtidb;"partitions ..."]]; + }; + +/ NOTE the drop has to precede the move: once current is the new date, the date that +/ just closed reads as immutable and build would reuse its stale catalogue +rollover:{[pt] dropdates enlist current; current::pt; loadsym[]; rebuild[]; }; +``` + +Config lives in `appconfig/settings/idb.q`: `roots` (a list, so one reader can serve several +capture stacks), `tabs`, `historydays`, `sweep`, and how long to wait for the writer. + +**Measured end to end.** Writer creates the directory at `12:36:55.799`, fills every table's +directory and notifies at `.800`, reader has rebuilt at `.802`. Two milliseconds, and the +only work in that window is one `key` per date per table. + +#### What the sketch got wrong + +Three things the earlier draft of this section would have got wrong, all found by running it: + +**`t set value` inside a `\d` block does not create a global.** It creates `.vtidb.t`. The +virtual tables have to land in the root namespace or a client's `select from trade` does not +resolve, so the assignment is `@[`.;t;:;v]`, which names the root namespace explicitly. + +**`tables[]` does not see virtual tables.** They are type `112h`, not `98h`, so every +TorQ-side facility that enumerates tables — including `.proc.getattributes`, which the +discovery service publishes — comes back empty. The process reports its own catalogue +instead. This is worth knowing before pointing anything at the IDB that discovers tables +rather than being told them. + +**`@[f[a;b;c];::;handler]` does not trap `f`.** Supplying every argument applies the function +where it is written, outside the trap, so the failure it exists to catch propagates anyway. +The arguments belong in `@`'s second slot: `@[f;(a;b;c);handler]`. + +#### Two departures from the sketch, on purpose + +**The table list is discovered from disk, not configured.** `tabs` defaults to `` ` ``, which +means "whatever table directories exist". A table added to `database.q` then needs no change +here. Set `tabs` explicitly only to *restrict* what a reader attaches. + +The scan looks at the **live partition only**, plus every date once when the catalogue is empty. +Scanning all of history on every rebuild — which is what this did originally — is a `readdir` +per date, so the cost grew with retention for ever, and it was paid every sweep to notice +something that happens once in a deployment's life. Measured at 250 dates and 10,000 +partitions: + +| | rebuild | +|---|---| +| scanning every date | 4,194 µs | +| scanning the live partition | **1,428 µs** | +| `tabs` hard-configured, no scan at all | 1,445 µs | + +Discovery is now free: it costs the same as not discovering. The scan itself went from 2,644 µs +to 113 µs. + +What this gives up is stated in `testfiles/vt-newtable-test.q` and asserted there: a table appearing +on a date that has **already rolled** is not found by a rebuild, because that date is not +scanned. `dropcache[]` finds it. That is the same recovery path §6.1 already prescribes for a +*directory* added to a past date, so it adds no new rule — a new table arrives where the writer +is writing, which is the partition that does get scanned. + +**The writer is optional.** The sketch blocked forever waiting for the WDB. The reader does +not need the writer — it reads a directory tree, and the sweep keeps it current on its own. +It waits a few cycles, logs a warning, and starts anyway. Registering with the writer is +purely a latency optimisation: it turns new-partition visibility from `sweep` (30s) into the +2ms measured above. + +### 5.4 Rows travel free; new symbol *values* do not + +§5.2's result is easy to over-read. Two different things travel at two different speeds, and +the difference is invisible until it bites. + +**Rows appended to a directory the reader already holds are visible immediately** — no rebuild, +no notification, nothing told to the reader. That is the trailing slash doing its job. + +**A symbol value that has never been seen before is not.** Symbol columns are indices into the +enumeration domain, and the reader holds that domain *in memory*. When the writer meets a new +value — a new `src`, a new venue code — it appends an entry to the domain file and writes the +rows into an **existing** directory. No directory is created, so nothing is announced: §4.1 is +edge-triggered on directories appearing. The reader's copy of the domain is then one entry +short, and that column reads as **null**: + +``` +rows for symbol : 859 -> 869 <- the rows arrived, live view, no rebuild +distinct sides : `buy`sell` <- the new value resolved to NULL, silently +``` + +Nothing on disk is wrong. The reader is misreading correct bytes, and it corrects itself the +moment the domain is reloaded. + +Before this was found the window was the 30-second rebuild sweep, because a rebuild is the only +thing that called `loadsym`. The domain now has its own timer: + +```q +refreshsym:{[] + if[symchanged[]; + loadsym[]; + .lg.o[`vtidb;"enumeration domain grew - reloaded"]]; + }; +``` + +`symchanged` is one `hcount` per root, so it is cheap enough to run every second where a +rebuild has no reason to. `symsweep` in `appconfig/settings/idb.q` sets the interval. + +Measured, three runs: **1.1 s, 1.9 s, 2.6 s** from publish to the value resolving. The floor is +the writer's own flush interval — the value is not in the domain file until the writer flushes — +so with `settimer` at one second, roughly two seconds is as good as this can get without the +writer announcing domain growth the way it announces directories. `testfiles/vt-symdomain-test.q` +holds it to a five-second budget and separately asserts that the *rows* arrive immediately, +which is the distinction worth protecting. + +The partition column is never affected: it comes from the directory name and never goes near +the domain. + +### 5.5 Why it is shaped this way + +**Appends need no code at all.** Section 5.2. This is the single biggest simplification, and +it is worth stating plainly because the obvious design — re-read everything on every flush — +costs about 0.25 ms per directory. At 500 instruments across two tables that would be ~250 ms +of every second spent re-opening files whose contents you already had. + +**Rebuild is driven by directory creation, not by writes.** The writer notifies when it +creates a partition (§4.6), not when it appends. On a static instrument universe that is +once a day; on a growing one, once per new instrument. Either way it is rare enough that +`rebuild` can afford to rescan everything rather than tracking deltas. + +**A timer exists, but only as a backstop.** `sweep` runs every 30 seconds and does the same +work as a notification. It is there so a dropped message degrades into 30 seconds of +staleness rather than a reader that is permanently blind to a new instrument. It is not the +primary path, and the interval is deliberately slack. + +**Sym is reloaded before any directory is opened.** Leaf `sym` columns are enumerations +against `$KDBDB/sym`; open a directory before the domain covers its values and it resolves +incorrectly. Stock TorQ uses the same `hcount` check (`idb.q:52-54`). One caveat inherited +from the No-RDB pack: at a fast cadence the reload happens every time the file grows, so a +high-cardinality symbol universe becomes a recurring cost. Since instrument is also the +partition column here, that is worth thinking about early. + +**`.Q.MAP` does not apply.** The No-RDB pack maps the whole database once and refreshes one +partition slot per flush. That is a `.Q` facility for `.Q` partitioned databases, and this is +not one (§2.1). The trailing-slash view is the equivalent, and it needs no framework support. + +**`mkP` reconstruction is free.** Because it captures directory handles rather than paths, a +new directory means rebuilding the whole virtual table. Measured, that costs 5–18 µs even at +500,000 partitions — it assembles a keyed table from vectors that already exist, with no +per-partition work. There is no reason to optimise it. + +### 5.6 What a query sees when it lands mid-write + +There is no lock anywhere on the read path, and the writer appends to the very directories a +client is querying. So the honest question is what a query returns when it arrives at the worst +possible moment. There are two such moments and they are governed by different mechanisms. + +**Mid-append.** A splayed write extends the column files **one at a time**. Between `price` +being extended and `size` being extended, the partition genuinely has columns of different +lengths on disk. This is not rare — measured with one process appending and another reading as +fast as it could, **200 of 5,000 reads** landed in that window. + +What comes back is a short read, never a wrong one: + +``` +reads 5000 +landed mid-append 200 +rows seen 192100 .. 800000 +invariant b = 2*a broken 0 <- not once +queries that errored 0 <- not once +``` + +The mechanism is the same rule that makes on-disk truncation silent in §4.6: **a splayed table +is cut to its shortest column.** Here that rule is what saves it. Truncation takes a *prefix* +of every column, so the rows that come back are internally consistent — a query simply misses +the last few rows, and the next query sees them. Nothing needs to be locked, retried or +coordinated. + +**Mid-rebuild.** `rebuild` replaces the global the client is querying. A client that caught it +half-done would see a table whose catalogue and contents disagree. It cannot: q's main loop is +the lock. A rebuild is one message and a query is another, and they do not interleave. Measured +with a client asking — in a single message, so the answer is one instant — for three numbers +that must agree, while directories appeared underneath a reader rebuilding every 3 ms: + +``` +queries 1500 +rebuilds meanwhile 1998 +partitions 20 -> 60 +catalogue disagreed with the served table : 0 times +row counts that went backwards : 0 times +``` + +The cost is therefore paid as **queuing**, not as inconsistency. A query that arrives during a +rebuild waits for it. That is the reason §8.2's work — making rebuild proportional to new +directories rather than to history — matters to readers and not only to the writer. At the +pathological 3 ms cadence above, median query latency was ~12 ms; at the configured 30-second +sweep it is invisible. + +`testfiles/vt-inflight-test.q` covers both, and asserts the invariants rather than the timings. + +### 5.7 A partition can promise columns it does not have + +§4.6 and `testfiles/vt-damage-test.q` cover three kinds of damage. There is a fourth, and it is not +guarded against — deliberately, after building the guard and measuring what it was worth. + +A splayed write lays down `.d` **first** and then the columns in order. So between those steps +the directory names columns that do not exist. `get` is lazy, so such a directory attaches +perfectly cleanly — it even *counts* correctly, because a count reads only the first column: + +``` +.d names `time`price`side +on disk `.d`time +get dir/ OK, type 98 +count 20000 <- correct, and completely misleading +select from 'No such file or directory +``` + +The blast radius is the usual one: every partition must be opened to answer a query that does +not name an instrument, so one such directory makes **every whole-database query fail** — +including queries that never mention the missing column. Selective queries on healthy +instruments are unaffected. + +**How often does this actually happen?** Only at partition *creation*. Appends never produce +it: the columns all exist already, and a mid-append read is §5.6's harmless short prefix. So +the exposure is one burst per instrument per day, and the main rebuild trigger cannot see it — +`vtsavetodisk` notifies readers only after `savetables` and `vtfill` have both returned. That +leaves the 30-second sweep as the only way in. Measured: writing one partition takes 0.4 ms +(500 rows) to 6.5 ms (100,000 rows), so the daily window is the creation burst: + +| instruments × 2 tables | creation burst | chance a sweep lands in it | expected | +|---|---|---|---| +| 10 | ~10 ms | 0.03% / day | once per ~9 years | +| 500 | ~0.5 s | 1.7% / day | once per ~2 months | +| 5,000 | ~5 s | 17% / day | once per ~6 days | + +And when a sweep does land there, the consequence is bounded: whole-database queries fail until +the next sweep re-opens the finished partition, at most 30 seconds, with no intervention. + +**Why it is not guarded.** The reader can check each partition against its own `.d` before +accepting it — `all (cols v) in key first ` vs p` — and that was implemented, tested and then +reverted. Two reasons. + +It costs about **45% of a full rescan** (measured: 60 ms unguarded, 83 ms guarded, on 2,000 +partitions; a further 31 ms if the check is placed before the `get`, which reads `.d` twice +because `get` parses it anyway). That is affordable, but it buys little against a race that +heals itself. + +The second reason is the one that decided it. The only *permanent* source of this state is a +full disk — and a full disk means capture has already stopped. You are in an outage. A reader +that fails loudly sends someone to look; a reader that skips the directory keeps answering +queries while silently omitting an instrument, with nothing but a log line to say so. That is +precisely the failure signature this design spends §4.6, §5.4 and §8.3.1 trying to eliminate. +Masking it would have been the wrong trade. + +`testfiles/vt-diskfull-test.q` asserts the unguarded behaviour, so it is pinned down rather than +merely known. + +### 5.8 Which partition is live — and why `.z.D` is the wrong answer + +`.vtidb.current` is the partition the writer is filling. It decides which dates are rescanned +and which are cached forever, so it is not a cosmetic variable: **a date wrongly believed to +have rolled is cached and never looked at again.** Every instrument that starts trading +afterwards is on disk, absent from every query, and nothing is logged — there is no error to +log, the reader has simply stopped looking. + +The reader asks the writer. When it cannot — the writer is down, the reader started first, or +the read of `.wdb.currentpartition` failed — it has to decide for itself, and the obvious +answer is wrong. Set any non-zero `.eodtime.rolltimeoffset` — a business day that ends somewhere +other than midnight in `rolltimezone` — and from midnight until the roll the writer is still +filling *yesterday* while `.z.D` already says today. The measurement below used `0D09:00`, +a 17:00 roll in a UTC+8 timezone: + +``` +idb .z.D 2026.08.19 +wdb currentpartition 2026.08.18 <- 9 hours a day, minimum +``` + +A reader that guessed `.z.D` in that window froze the live partition for the rest of the day. +Reproduced against the unfixed reader: a new instrument directory on disk, three rebuilds, and +the catalogue never moved off two partitions. + +The date on disk is knowable without asking anyone, and it is a bound the writer cannot +contradict — it cannot be filling a date older than the newest directory it has itself created: + +```q +livepart:{[ds] $[count ds; max current,"D"$string last ds; current] }; +``` + +Take the writer's answer when there is one, but never let it sit behind the disk. This needs no +connection tracking and it degrades correctly in both directions: a writer that has rolled but +not yet flushed stays *ahead* of the disk and is respected, and a writer that dies after +speaking once no longer freezes the reader behind it. `testfiles/vt-restart-test.q` drives the real +`init` path and fails on the old behaviour. + +## 6. End of day + +In full, in order: + +1. STP rolls its log file on its configured schedule and calls `.u.end`. +2. `.wdb.endofday` flushes remaining rows, calls the overridden `endofdaysort` (a no-op plus + notification), sets `currentpartition::pt+1` (`wdb.q:228`). +3. IDBs receive `.vtidb.rollover`, reload `sym`, rescan, and pick up the new empty date. +4. The next 1s flush creates `$KDBWDB////`. + +No data is copied. No sort. No merge. No process restart. The cost of EOD is one directory +scan per IDB. + +The one thing that *does* have to happen at EOD is the rebuild — a new date means new +directories, and new directories are the only event a reader has to react to (§5.2). Note +step 3 depends on the writer having created every table's directory for the new date before +notifying, per §4.6. + +--- + +### 6.1 Editing historical data + +`mutable` treats any date before the live partition as immutable, and the reader reuses its +catalogue and opened views rather than rescanning. That raises a fair question: can historical +data still be corrected or backfilled? + +Yes — but what "immutable" means here is narrower than it sounds. **It is an assumption about +the set of directories, not about their contents.** Measured on a copy of a real database: + +| change to a historical date | reader sees it | +|---|---| +| **append rows** to an existing instrument directory | **immediately, with no rebuild** | +| **replace a column file** (write new, rename over) | **immediately, with no rebuild** | +| **add a new instrument directory** | **not until `dropcache[]` then `rebuild[]`** | + +The first two work because the opened views are live regardless of which date they belong to — +liveness is a property of how the file was opened (§5.2), not of the date. Only the third is +blocked, because finding a new directory requires a rescan and immutable dates are not +rescanned. + +So the runbook for correcting history is: + +- **Backfilling rows into instruments that already exist** — just write them. Nothing else to do. +- **Backfilling a new instrument into a past date** — write it, then send the reader + `.vtidb.dropcache[]` followed by `.vtidb.rebuild[]`, or `.vtidb.rollover[.wdb.currentpartition]`, + which does both. Without that the rows are on disk and invisible. +- **Deleting a partition** — a removed directory *is* noticed, because `build` drops catalogue + rows whose date is no longer on disk. It is the addition that needs the cache drop. + +One caveat carried forward: the third row above also suggests §7's compression caveat may be +wrong. That caveat assumes a running reader holds a mapping on the old inode and keeps serving +pre-compression data. This reader does not memory-map at all (§8.2), and a rename-over was +picked up immediately here. That was tested with a plain column file, not a genuinely compressed +one, so **§7 stands until VT-15 tests compression itself** — but expect it to need the same +correction §8.2 did. + +--- + +## 7. Compression + +`code/processes/vtcompress.q` replaces the stock compression process. It applies the §4.4 +classifier, applies the two gates below, calls `.cmp.compressfromtable` over what is left, and +exits. The pack drives it through one script: + +``` +./compress.sh compress +./compress.sh --dry-run report what would be compressed, change nothing +./compress.sh --test compress underneath the running reader and verify it copes +``` + +Two gates decide what gets touched. **The age tier**, `minage` in +`appconfig/compressionconfig.csv`: + +``` +table,minage,column,calgo,cblocksize,clevel +default,7,default,2,16,9 +``` + +Recent partitions stay uncompressed so interactive queries on them run at full speed. It must +stay above 0 whatever tier is chosen: that is also what keeps the job off the live partition, +and never compressing a directory the WDB may still append to is the important safety property. +`appconfig/compressionconfig-test.csv` holds a 1-day copy, which is how `--test` exercises the +job against a database only a couple of days old. + +**The size gate**, `.cmp.minfilesize` in `appconfig/settings/compression.q`, defaults to 4096 +and is the subject of §7.3. Set it to 0 to compress everything, which is stock behaviour. + +Weekend scheduling: the process exits on completion, so drive it externally rather than from +the TorQ timer: + +``` +# Saturday 02:00 +0 2 * * 6 /path/to/TorQ-VT-Capture-Pack/compress.sh +``` + +### 7.1 Readers do not need to be told + +An earlier draft of this section carried a caveat: *"IDBs must re-map afterwards. +`cleancompressed` renames the compressed file over the original, so a running IDB keeps its +mapping on the old inode and serves pre-compression data indefinitely. Send +`.vtidb.rollover[…]` to each IDB when the job finishes, or restart them."* + +**That is wrong, and it mattered** — it was the sole reason `rollover` dropped the reader's +cache, which is what made end of day cost a full rescan of history (§10, VT-17). + +`./compress.sh --test` compresses the oldest complete partition while the stack is running, +and re-queries through the *same handle*, with no rollover sent and no cache dropped: + +``` + before 1000000 rows in 750 column files + 30474 kB logical, 33372 kB allocated, 0 already compressed + + compressing - reader stays up, no rollover sent + + PASS reader returns identical results through the same handle, with no rollover + PASS files are genuinely compressed (0 -> 350 of 750) + PASS no temporary files left behind + PASS the partition still holds rows, so the comparison above is not vacuous + PASS a cold process can open a compressed partition directly +``` + +The reason is the one already established in §8.2: a trailing-slash open is a *live view*, not +a memory map. There is no inode to go stale on, so the rename-over is picked up on the next +read. The reader needs no notification, no rollover and no restart, and end of day is free to +keep its cache. + +### 7.2 What it actually saves + +Measured by `./compress.sh --test` against a partition holding 20,000 rows per instrument — +a realistic day for a busy symbol: + +| | before | after | saved | +|---|---|---|---| +| logical (bytes in the files) | 29.8 MB | 5.0 MB | **83.1 %** | +| allocated (what `df` reports) | 30.2 MB | 5.7 MB | **81.2 %** | + +Both rows are large here, and that is the point: once a column file is comfortably bigger than +a filesystem block, the logical ratio and the disk ratio converge. `testfiles/vt-compress-ratio.q` +reads each compressed file's own header and buckets by the size it had before: + +``` +sizeband files logicalsaved disksaved blockswas blocksnow +--------------------------------------------------------- +16-64 kB 150 66 60 750 300 +>64 kB 200 84.9 83.5 6990 1152 +``` + +Nothing appears below 16 kB because the size gate of §7.3 skipped it: 400 of the partition's +750 column files were too small to free a block, and were left alone. + +**This is very sensitive to how much data lands in each directory**, which is the whole +small-files story of §8.1. On a thin partition — a few hundred rows per instrument — the same +job frees 83 % of the *bytes* and under 10 % of the *disk*, because almost every file already +fits in one block. Do not carry a compression ratio across from a different partition size. + +### 7.3 So does it pay? + +It costs read latency, and at these partition sizes it costs a lot. Same single-instrument +select, 1,000 samples (`testfiles/vt-compress-ab.q`): + +``` +state | uncompressed gated ungated +compressed| 0 350 750 +allocKB | 33372 8220 8220 +minus | 382 805 775 +medus | 526 982 985 +``` + +So roughly **+100 %** on the query the whole layout exists to make fast, in exchange for 75 % +of the disk. That is a real decision, not a free win — and note that the cost scales the same +way the benefit does: the bigger the partitions, the more there is to inflate on every read. + +`testfiles/vt-compress-sizes.q` sweeps the partition size directly. Compressed size lands on the +floor — one block per column file — almost immediately and then stays flat, so the *saving* is +decided entirely by how large the files were to begin with, which is rows per instrument per +day: + +``` +rows | 100 300 1000 3000 10000 30000 100000 +kbperfile | 1 1.9 4.9 13.6 43.8 130.3 433.1 +cmpallocKB | 28 28 28 36 64 152 468 <- flat: the one-block-per-file floor +disksaved | 12.5 12.5 36.4 64 80 83.6 84.6 +``` + +(Entropy is held constant across the sweep, so the absolute ratios are optimistic; the shape, +which is what sets the threshold, is not affected.) + +#### The size gate, and what it does not buy + +Since a file inside one filesystem block frees nothing, compressing it is pure cost. The job +therefore skips them — `.cmp.minfilesize`, default 4096. On the partition above that excludes +400 of 750 files. The expectation was that skipping them would also cut the read cost, because +a selective query opens every column of one instrument directory. + +**It does not.** Gated and ungated are indistinguishable on latency in the table above (805 vs +775 µs minimum, 982 vs 985 median), and identical on disk — 8,220 kB either way, which is the +block floor again, proving the 400 extra files freed nothing at all. + +The reason is structural rather than incidental. The gate only ever skips files in directories +that hold little data, and queries against those directories are already cheap. It cannot speed +up an expensive query, because an expensive query is by definition reading a directory with +enough data in it that none of its columns qualify. + +The gate is still worth keeping, for operational reasons rather than query ones: it compresses +350 files instead of 750 for the same result on disk, which is less job time and 53 % fewer +files whose inode changes each weekend — which matters to `rsync`-style backup over a tree that +already has a small-files problem (§8.1). + +#### Recommendation + +Enable it, on a retention tier, with the size gate on — which is how the pack now ships +(`minage 7`, `minfilesize 4096`). The age tier is what makes the read cost tolerable: recent +data, which is what gets queried interactively, stays uncompressed and fast, while older data +gives back most of its disk. Below roughly 1,000 rows per instrument per day the layout has +already made the files too small for compression to recover much and the read cost is paid +anyway; above ~10,000 it recovers more than 80 % of the disk and the case is clear. + +--- + +## 8. Scale and limits + +### 8.1 File count + +Files = `days × instruments × tables × columns`. For 500 pairs, 2 tables, 10 columns: +10,000 files/day, 2.5M files/year. Every one carries a `.d` sibling and a filesystem inode. +This is the drawback already identified, and it is real: it constrains the filesystem +choice, `rsync`-style backup, and any tooling that stats the tree. + +### 8.2 Memory mappings — measured, and not the limit after all + +**This section previously named memory mappings as the sharpest constraint in the design. Step 6 +of §10 measured it properly and that was wrong.** The correction matters enough to show the +reasoning. + +The original measurement was real and reproducible: opening a splayed directory with plain +`get` costs exactly one kernel mapping per column, so 500 directories of 7 columns added 3,500 +mappings. Extrapolating that gave a ceiling of roughly nine days of history at 500 instruments +on a stock Linux box. + +**But the reader does not open directories that way.** It appends a trailing slash to get a +live view (§5.2), and a trailing-slash open does *not* memory-map — it reads on demand. The +two are easy to confuse and cost completely different resources: + +``` +get `:.../AMD -> +8 mappings (8-column splay, 1 per column) +get `:.../AMD/ -> +0 mappings (live view, what the reader uses) +``` + +Measured against the real `vtidb.q` on synthetic trees, reading every column of every +partition to be sure the data was genuinely touched: + +| partition dirs | mappings added | file descriptors | RSS added | bytes/dir | +|---|---|---|---|---| +| 400 | 0 | 0 | 384 kB | 983 | +| 2,500 | 1 | 0 | 2.3 MB | 922 | +| 10,000 | 1 | 0 | 8.7 MB | 891 | +| 20,000 | 1 | 0 | 17.4 MB | 891 | +| 40,000 | 1 | 0 | 34.8 MB | 891 | + +Mappings do not scale with partition count at all. Neither do file descriptors. The cost is +ordinary heap, and it is small and linear: **891 bytes per partition directory**, so a million +partitions is about 850 MB of RSS. `vm.max_map_count` is not reachable by this design. + +The `historydays` knob therefore is not the memory-mapping defence this document previously +said it was. It is still useful, for the reason below. + +#### The real limit is rebuild time + +Rebuild rescans every date and re-opens every directory. That is linear in partition count, +and it runs on the sweep timer as well as on every new directory: + +| partition dirs | full rescan | live rebuild | end of day | selective query | +|---|---|---|---|---| +| 400 | 14 ms | 1 ms | 1 ms | 237 µs | +| 2,500 | 100 ms | 3 ms | 3 ms | 245 µs | +| 10,000 | 381 ms | 6 ms | 7 ms | 269 µs | +| 20,000 | 758 ms | 6 ms | 7 ms | 249 µs | +| 40,000 | 3,691 ms | 6 ms | 8 ms | 279 µs | + +That is the current `testfiles/vt-scale-test.q` output. The **live rebuild** column is the one that +runs on the sweep, and it is now flat at 6 ms from 10,000 directories to 40,000 — the table +scan no longer contributes to it. The full-rescan column is unchanged by that work (a cold +catalogue still scans every date) and it degrades faster than linearly at the top end as the +dentry cache stops holding the whole tree; it swings between roughly 2.7 s and 3.7 s at 40,000 +across runs, and is left as measured. The *full rescan* column is what rebuild +used to cost on every sweep, and is what the two subsections below remove; it is kept because +it is the measurement that located the problem. + +Two things to read off this. + +**The design's core promise holds.** A selective query — one date, one instrument — is flat at +roughly 250–290 µs from 400 partitions to 40,000: a hundredfold change in database size moves +it by less than the run-to-run noise. It does not degrade as history grows, because the engine +never looks at the directories it does not need. That is the whole point of the design, now +verified at scale rather than argued. + +**Rebuild was the constraint.** About 38 µs per partition directory when this was measured — +now ~69 µs — single-threaded, and it blocked the process while it ran. Against the default 30-second sweep: + +| partition dirs | rebuild | share of a 30s sweep | +|---|---|---| +| 40,000 | 1.4 s | 5% | +| 100,000 | 3.4 s | 11% | +| 250,000 | 8.5 s | 28% | +| 1,000,000 | 34 s | **cannot keep up** | + +That put the practical ceiling at roughly 100,000 partition directories — 500 instruments +across 2 tables for about 100 days — as a *latency* limit rather than a hard failure: past it, +queries stall for seconds at a time on every sweep. + +**This table describes the original full-rescan behaviour.** It is kept because it is the +measurement that located the problem; the next subsection removes it. + +#### The fix was in our own code, not KX's — and it is now implemented + +**History is immutable.** Once a date has rolled, its set of directories never changes again; +only the live partition can gain one. So `build` keeps the catalogue *and the opened views* +for every immutable date it already holds, and rescans only the live date plus any date it has +not seen before. Rebuild becomes O(instruments) instead of O(instruments × days). + +Measured, same harness (`testfiles/vt-scale-test.q`): + +| partition dirs | rebuild, full rescan | rebuild, live date only | speed-up | +|---|---|---|---| +| 400 | 14 ms | 1 ms | 14x | +| 2,500 | 100 ms | 3 ms | 33x | +| 10,000 | 381 ms | 6 ms | 64x | +| 20,000 | 758 ms | 6 ms | 126x | +| 40,000 | 3,691 ms | 6 ms | **615x** | + +The live-date rebuild is **flat** — 6 ms whether the database holds 10,000 directories or +40,000 — because the work no longer depends on how much history is attached. The sweep now +costs 8 ms every 30 seconds instead of seconds, at any depth. The table above showing rebuild +as the ceiling describes the pre-optimisation behaviour; that ceiling is gone. + +Three details worth knowing about the cache: + +- **A full rescan is still available**, via `dropcache[]`. It is the manual recovery path for + the one change this reader cannot see by itself — a directory added to a *past* date (§6.1). + It is no longer called at end of day; see §8.2.1. +- **The cache never blocks discovery.** It reuses only dates it already holds; a date that + appears late — backfill, or a reader whose `historydays` widens — is not in the catalogue, + so it lands in the rescan set. Verified by `testfiles/vt-gap-test.q`, which recovers a restored + historical directory through the cache. +- **A null `current` forces a full scan**, which is what makes the first build after startup + complete rather than partial. + +If ever needed, two lesser levers remain: lengthen `sweep` (the notification path already +covers the latency-sensitive case), or bound `historydays` and route by date across readers. + +#### 8.2.1 End of day is flat too + +Making the sweep flat left one operation that still scanned everything: `rollover` dropped the +whole cache, so end of day cost a full rescan — ~47 µs per directory, which is 12 s at 250,000 +partitions and 47 s at 1,000,000, as a blocking stall once a night. The justification was that +compression needed a genuine re-open, and §7.1 measured that it does not. + +It could not simply be deleted, though. `mutable` is `d>=current`, so the moment `current` +moves to the new date the date that just *closed* reads as immutable — and `build` would then +reuse its catalogue as-is. Any directory the writer created in its final flush of the day, +before the reader last rebuilt, would be on disk and permanently invisible. Silently. + +So `rollover` forgets exactly one date, the one closing, and does it **before** `current` +moves: + +```q +dropdates:{[ds] + if[not count ds; :()]; + {[ds;t] + k:where not parts[t][`date] in ds; + parts[t]:parts[t] k; + opened[t]:opened[t] k; + }[ds] each key parts; + }; + +rollover:{[pt] dropdates enlist current; current::pt; loadsym[]; rebuild[]; }; +``` + +End of day now costs one date's scan, and tracks the live rebuild rather than the cold one: + +| partition dirs | cold rescan | live rebuild | **end of day** | +|---|---|---|---| +| 400 | 18 ms | 1 ms | **2 ms** | +| 2,500 | 90 ms | 3 ms | **3 ms** | +| 10,000 | 388 ms | 7 ms | **7 ms** | +| 20,000 | 731 ms | 8 ms | **8 ms** | +| 40,000 | 1,540 ms | 10 ms | **11 ms** | + +Flat, and 140x cheaper at 40,000 directories. On the running stack a rollover takes 2.4 ms and +keeps every partition attached. `testfiles/vt-rollover-test.q` covers the failure mode above — it +creates a directory the reader has not scanned, rolls over, and checks the rows survive. It is +written to fail against the naive version of this change, and does. + +**No operation left in this design scales with how much history is attached.** + +#### What this means for the rejected mitigation + +The granularity-mixing plan in earlier drafts existed to keep the mapping count flat as data +aged. Since the mapping count was never growing, that motivation is gone. Mixing granularities +is still blocked by §9.3, and that blocker is still real — but it is no longer entangled with +scaling, which makes the ask of KX narrower than this document previously claimed. +### 8.3 Adding tickerplants + +`.vtidb.roots` is a list. A second capture stack writing to a different `savedir` is picked +up by adding its root — the scan produces more partitions, `mkP` takes them all, and nothing +downstream changes. The claim that TPs can be added without modifying anything downstream +holds, with one condition on the enumeration domain that §8.3.1 settles. + +Measured cost of the extra root, same data split one tree versus two +(`testfiles/vt-multistack-test.q`): + +| roots | partitions | rebuild | selective query | +|---|---|---|---| +| 1 | 22 | 73 µs | 254 µs | +| 2 | 22 | 119 µs | 278 µs | + +A root costs one extra directory read per date per table on rebuild, and nothing at query time +beyond the wider partition table. + +--- + +### 8.3.1 Each stack needs its own enumeration domain + +Symbol columns are not stored as text — they are integer indices into a domain file at the +database root. The reader loads that file with `load`, and **`load` binds a global named after +the file**. Two stacks that both call theirs `sym` therefore collide: one load wins, and every +other root's symbol columns resolve against the wrong domain. + +``` +root a sym : `AAA`BBB`CCC +root b sym : `XXX`YYY`ZZZ + +after loading a : sym = `AAA`BBB`CCC +after loading b : sym = `XXX`YYY`ZZZ + +index 0 from root a's data should be AAA; it resolves to `XXX +``` + +No error, just wrong symbols — and two stacks that grew their domains independently will have +assigned different indices to the same symbols, so this is the default outcome rather than an +edge case. + +**The fix is to name the domains apart, not to share one file.** `load` binds by filename, and +each column file records which domain it belongs to, so `sym` and `symb` coexist in one process +with each column resolving through its own. Both sides are configuration: + +- writer — `symdomain` in `appconfig/settings/wdb.q`. `.Q.en[d;t]` is `.Q.ens[d;t;`sym]`, so + the override redirects `.Q.en` once and covers every enumeration site in the writer rather + than copying a forty-line TorQ function to change one symbol in it +- reader — `symfiles` discovers whatever domain files exist at each root instead of assuming + `sym`, and `loadsym` now reports only the configuration that is actually unsafe: two roots + using the **same name** for different contents + +Three configurations, all verified by `testfiles/vt-multistack-test.q`: + +| domains | outcome | +|---|---| +| `` `sym `` and `` `symb `` | **works** — stacks stay independent, one reader serves both | +| both `` `sym ``, different contents | **refused loudly** — the error names the domain | +| both `` `sym ``, identical contents | works, and symbol columns unify (see below) | + +#### What separate domains cost + +One thing does not unify across domains, and it is worth knowing before choosing: + +| operation | separate domains | shared domain | +|---|---|---| +| group by the **partition column** | correct | correct | +| filter on a symbol column, `` where side=`buy `` | correct | correct | +| `` select … by value side `` | correct | correct | +| `` select … by side `` | **splits per domain** | correct | + +The partition column comes from directory names rather than a domain, so the query this layout +exists to serve is unaffected. A filter is correct because comparing an enum to a symbol +resolves per element. But a *grouping* on a symbol column held inside the files gets one group +per domain — `` `sym$`book1 `` and `` `symb$`book1 `` are distinct values — so a cross-stack +aggregate by `side` returns one group per domain rather than one per value. `value` on the column fixes it. + +That is visible rather than silent (the group labels carry their domain), and the test asserts +it so a change in the engine's behaviour shows up. + +#### Sharing one domain is safe — the constraint is storage, not concurrency + +An earlier draft of this section rejected the shared domain partly on the grounds that two +writers appending to one `sym` file risk corrupting it. **That is wrong.** The enumeration +primitive `.Q.en` calls is `path?syms` on a file handle, and it locks. Measured by +`testfiles/vt-sym-concurrency.q` — six concurrent processes, 1,800 enumeration calls, an overlapping +vocabulary: + +``` + final domain size 507 + duplicate entries 0 + indices handed out 3012 + now resolving WRONG 0 +``` + +No duplicates, and every index handed to a writer still resolves to the symbol it was issued +for — which is the property that matters, because column files already written carry those +indices. + +The real constraint is physical. A domain file lives at a database root and the reader loads +`/`, so "sharing" means both stacks writing **one inode**, symlinked into each +root: + +``` +/data/stackA/sym the file +/data/stackB/sym -> /data/stackA/sym +``` + +That works — verified, both roots resolve against one domain and each stack's writes extend it. +It requires the stacks to share a filesystem. + +What is *not* safe is two **copies**. They are identical the day you set them up and diverge the +moment either stack sees a symbol the other has not, at which point the configuration silently +becomes the colliding row in the table above. + +#### Both modes are supported — how to configure each + +The reader does not have to be told which mode is in use. `symfiles` discovers whatever domain +files exist at each root, and `loadsym` objects only to the one configuration that is actually +broken. So the choice is made entirely on the writer side, per stack. + +**Shared domain** — one file, symlinked into each root. **The link has to exist before the +second stack writes anything.** Symlinking a domain over a tree whose columns were already +enumerated against a different one shares nothing: it reinterprets existing indices against the +wrong list, which is the collision case again. Leave `symdomain` at its default: + +```q +/ appconfig/settings/wdb.q, on every stack +symdomain:`sym +``` + +```sh +# one physical file; every other stack links to it +ln -s /data/stackA/sym /data/stackB/sym +``` + +Verified: the link survives repeated writes — `.Q.en` appends to the file rather than replacing +it — both roots stay on one inode, and `hcount` follows the link, so the reader's +`symchanged` check still notices when the *other* stack extends the domain. + +**Separate domains** — one file per stack, named apart. Set `symdomain` per stack: + +```q +/ appconfig/settings/wdb.q, stack A / stack B +symdomain:`syma symdomain:`symb +``` + +Nothing else changes; nothing needs to be linked. + +| | separate domains | shared domain | +|---|---|---| +| stacks on separate storage | **yes** | not possible | +| stacks on shared storage | yes | **yes** | +| cross-stack `by` on a symbol column | needs `value` | correct | +| coupling between stacks | none | one shared file | +| concurrent writers | n/a | safe — the primitive locks | + +Use the shared domain when the stacks already share storage and cross-stack aggregation by a +symbol column matters. Use separate domains otherwise — it is the only option across +filesystems, and it keeps the stacks independent, which is usually why a second one exists. + +What must **not** happen in either mode is two *copies* of a domain under the same name. That +is the middle row of the table above: correct on the day it is set up, silently wrong from the +first symbol one stack sees and the other does not. The reader detects it and logs an error +naming the domain. + +One further limit applies whichever mode is chosen: **the same `(date;instrument)` under two +roots is served twice.** `mkP` does not reject a duplicate key — it builds cleanly, the keyed +table shows one key, and a query returns the rows of *both* directories. Two stacks capturing +the same instrument therefore double-count, silently. Keep the instrument universes of two +stacks disjoint, or expect to deduplicate downstream. Asserted in `testfiles/vt-inflight-test.q`. + +### 8.4 Under load — measured + +Everything in §8.2 was measured on synthetic trees of 10-row partitions, driven by the demo +feed's handful of rows every 200 ms. This section replaces that with real volume. +`./loadtest.sh` starts a clean stack with no demo feed, bursts a configurable number of rows +through the tickerplant, and polls the reader until every row is visible — so the figure is the +whole chain, not the rate at which the tickerplant's queue can be filled. + +| rows | instruments | offered | end to end | rows/dir | selective query | group by all | rebuild | files | on disk | +|---|---|---|---|---|---|---|---|---|---| +| 200k | 50 | 4.99M/s | **137k/s** | 4,000 | 1,048 µs | 3 ms | 4 ms | 851 | 8.8 MB | +| 1M | 50 | 4.77M/s | **637k/s** | 20,000 | 1,993 µs | 3 ms | 4 ms | 851 | 33 MB | +| 4M | 50 | 4.57M/s | **1,447k/s** | 80,000 | 6,087 µs | 5 ms | 4 ms | 851 | 122 MB | +| 1M | 500 | 4.59M/s | **216k/s** | 2,000 | 886 µs | 27 ms | 35 ms | 8,501 | 60 MB | +| 1M | 2,000 | 4.33M/s | **97k/s** | 500 | 807 µs | 71 ms | 142 ms | 34,001 | 164 MB | + +**No data was lost in any run, and no errors were logged.** Every row published was visible to +the reader. + +**Throughput rises with burst size**, because the ~1 second flush interval is a fixed cost that +amortises: 137k/s for 200k rows becomes **1.45M/s** for 4M. The writer is not the constraint at +these volumes — the flush cadence is. If low latency matters more than throughput, shorten +`settimer`; if throughput matters more, lengthen it. + +**Selective queries track rows read, not database size.** 488 rows in 807 µs, 1,897 in 886 µs, +20,013 in 2.0 ms, 80,191 in 6.1 ms — roughly 0.1 µs per row at the larger sizes. This is the +design paying off in the way intended, and note the direction: *more* instruments makes a +single-instrument query **faster**, because each directory holds fewer rows. + +**Instrument count is what costs.** Same 1M rows, three universes: + +- write throughput falls (637k/s → 216k/s → 97k/s), because each flush touches more directories +- whole-database operations scale with directory count (group by all: 3 ms → 27 ms → 71 ms; + rebuild: 4 ms → 35 ms → 142 ms), consistent with the per-directory cost in §8.2 +- **storage amplifies 4.9x**: 35 bytes/row at 50 instruments, 63 at 500, **172 at 2,000** — the + small-files cost of §8.1, now quantified rather than asserted + +That last figure is the one to carry into capacity planning. Fragmentation, not row data, +dominates the footprint once the instrument universe is wide and each partition holds only a +few hundred rows — and the narrower the row, the worse the ratio, because the per-file overhead +is fixed. This schema's 7-column `trade` row is about half the width of the FX schema measured +in an earlier draft, and its amplification is correspondingly worse: 4.9x against 2.5x. + +**The writer's memory is the burst buffer.** During the 4M-row run the writer was observed +holding 2,098,000 rows in memory in a single flush cycle, because the burst arrived faster than +the flush could drain it. That is correct behaviour and it absorbed the burst without loss, but +it means a sufficiently large burst is bounded by writer RAM rather than by disk. Worth a `-w` +limit and an alert in a real deployment. + +### 8.5 When the disk fills up + +The writer writes straight into the directories readers are reading; there is no staging area +to fail into. So a full disk is worth knowing in detail rather than guessing at. +`testfiles/vt-diskfull-test.q` runs the whole thing inside a private mount namespace with a small +tmpfs in it, so it fills a real filesystem without root and without touching anything outside. + +**ENOSPC is an ordinary q error, not a death.** It names the file: + +``` +/db/2026.01.01/trade/I12/price. OS reports: No space left on device +``` + +The process stays up and keeps serving. (This is worth stating because the obvious way to +simulate a full disk — `ulimit -f` — is *not* equivalent: that raises `SIGXFSZ` and kills q +outright, which is a different failure and would teach the wrong lesson.) + +**The data is not lost.** `savetablesbypart` upserts every partition and only *then* empties +the in-memory table, and the pack's `upserttopartition` override rethrows rather than +swallowing. The error therefore escapes before the table is cleared, and the rows are still +there for the next flush to retry. + +**But a retry duplicates.** The failure happens part way through the partition loop, so +partitions written before it are already on disk — complete and readable. A retry re-upserts +the *whole* buffer, and nothing dedupes it: + +``` +save failed part way through yes +rows still in memory 80000 <- nothing lost +partitions already written 100 101 102 <- these get the same rows twice +``` + +So after a disk-full it is the partitions written **before** the failure that need checking, +not just the one that reported it. + +**What is left on disk** is the incomplete partition of §5.7 — `.d` naming columns that were +never written. The reader attaches it without complaint, and every whole-database query then +fails naming the missing file; selective queries on healthy instruments keep working. That is +the intended signal rather than a defect (§5.7). Recovery is to free space and write the +partition again; the reader attaches the repaired copy on the next rebuild, with no restart. + +## 9. Backward compatibility + +Everything below is **verified by execution** — `testfiles/vt-probe.q` and +`testfiles/vt-sample-legacy.q` reproduce it: + +```sh +QHOME=~/.kx/q QLIC=~/.kx QPATH=~/.kx/mod ~/.kx/bin/q testfiles/vt-probe.q +``` + +The probe makes one partition a poison value (an int, not a table) so any query against it +throws. A clean result therefore proves the partition was skipped; `THREW` proves it was +opened. + +**Headline: attaching an existing HDB works today, and is simpler than expected.** What does +not work is putting old and new formats under one table name. Earlier drafts of this document +had that backwards. + +### 9.1 Why the greenfield case is not affected + +The mechanism is the one from §2.2: a condition on a column stored inside the files is +answered by opening the files, never by skipping directories. + +| probe | virtual column | in the files? | result | +|---|---|---|---| +| 1a | `date` | no | **skips** | +| 1b | `instrument` | no | **skips** | +| 2a | `sym` | yes | **opens everything** | + +1b and 2a are the same query on the same data; only the column name differs. + +For new data this is settled by §4.5 — strip the column on write and the collision never +arises. For an existing HDB you cannot: `sym` is written into every partition, and removing +it means rewriting the database. + +### 9.2 Attaching an existing HDB: partition on date only + +The options in the original brief all added `sym` to the partition key in some form. None of +them help, and one is actively harmful. With a legacy table + +```q +trade:([]sym:`AMD`AAPL`MSFT`AAPL; px:10 20 30 40f); +``` + +the correct answer for `` where sym=`AAPL `` is px 20 and 40. + +| approach | result | why | +|---|---|---| +| `([]date:…)` — **date only** | **correct, 2 rows** | `sym` is filtered by the table itself, exactly as in a normal HDB | +| replicated links, key named `sym` | **6 rows — each duplicated 3×** | nothing is skipped, so all three entries query the same table and the results stack up | +| replicated links, key named `instrument` | **wrong instruments returned** | skips to the right entry, but the table receives no condition and returns the whole day mislabelled | +| null / wildcard key | correct, but the key is **ignored** | one entry, so nothing stacks; the null is never consulted | +| nested list of instruments | correct, but the key is **ignored** | likewise | + +The last two are provably inert — put deliberate nonsense in the key and the answer does not +change: + +```q +bad:vt.mkP ([]date:enlist 2020.01.01; sym:enlist `AAA`BBB`CCC)!enlist trade; +select from bad where sym=`AAPL / still returns px 20 and 40 +``` + +**And none of them could help even if they worked.** Every entry for a given date points at +the same table: + +``` +(2020.01.01, AMD) ─┐ +(2020.01.01, AAPL) ─┼─> trade one set of files +(2020.01.01, MSFT) ─┘ +``` + +Filtering on `sym` might drop an entry from the list, but it never avoids opening a file — +they all lead to the same place. `date` is different, because each date genuinely is its own +set of files. + +> **Instrument-level partitioning over an existing HDB cannot be achieved by configuration.** +> It requires the files physically rewritten into the new layout. Anything else is bookkeeping +> that costs correctness and buys no I/O. + +So: attach legacy partitions keyed on date alone. Queries are correct and perform exactly as +they do today — not a regression, just not an improvement. + +### 9.3 The blocker: one table cannot span both formats + +New-format partitions have the instrument column stripped; legacy ones still contain it. A +virtual table reads its column list from the **first** partition only and assumes the rest +match. + +```q +neweur:([]px:100 200f); / new format: sym is in the directory name +old :([]sym:`AMD`AAPL; px:10 20f); / legacy: sym is in the data +``` + +Correct answer for `` sym=`AMD `` is px 100, 200 and 10. + +```q +q) select from c1 where sym=`AMD / new-format partition listed first +2026.08.03 AMD 100 +2026.08.03 AMD 200 <- legacy row SILENTLY DROPPED + +q) select from c2 where sym=`AMD / legacy partition listed first +2020.01.01 AMD AMD 10 +2026.08.03 AMD AMD 100 +2026.08.03 AAPL AAPL 200 <- AAPL returned for a sym=`AMD query +``` + +Same data, same query. Which failure you get depends on listing order — an implementation +detail no one should have to reason about. + +The workaround is two table names: + +```q +tradenew :vt.mkP ([]date:…; sym:…)!(new partitions); / correct +tradehist:vt.mkP ([]date:…)!(legacy partitions); / correct +``` + +Each is right in isolation. But a client wanting full history must know both names exist, +know the cutover date, query both and stitch — which is the RDB/HDB split this design set out +to remove. + +It also blocks granularity mixing, though that matters less than earlier drafts claimed: the +mixing plan existed to hold the memory-mapping count flat, and §8.2 measured that the count +never grows. So this blocker is now purely about presenting one table name to clients, not +about scaling. + +### 9.4 Partition-skipping hints: present, undocumented, and unusable here + +There is a mechanism for skipping partitions based on a column's *range* rather than its +value. Virtual columns named `99min` and `99max` record per-partition minimum and +maximum, and are hidden from results. It works: + +```q +q) select from v where px<2.5 / partitions whose min px is 10 and 100 are skipped +``` + +Two limits make it useless for this design. + +**Only `< > <= >=` are recognised.** `=`, `in` and `within` return correct results but ignore +the hint and open everything, even though the hint plainly contains enough information: + +``` +where px<2.5 pruned +where px<=2 pruned +where px>=1, px<=2 pruned +where px=1.0 NOT pruned +where px in 1 2f NOT pruned +where px within 1 2 NOT pruned +``` + +**Symbol columns cannot use it at all.** The engine works out which side of the condition is +the column name by testing which one is a symbol — for `` sym>=`AMD `` both sides are, so +it gives up. Since the partition key here *is* an instrument symbol, the entire mechanism is +unavailable exactly where it would be most useful. That closes off the last workaround for +§9.2. + +### 9.5 Asks for KX, in priority order + +1. **Support partitions with differing column lists** — or at minimum detect and reject them + rather than silently dropping rows. Unblocks §9.3 *and* §8.2; by far the highest value. +2. **Guard against silent row multiplication.** Several partition entries sharing one table + multiply result rows with no warning, even with no `where` clause at all. +3. **Apply a condition in both places when the column exists in both** the partition key and + the data. Would make the legacy options in §9.2 correct — though see the caveat there + about whether they are worth having. +4. **Identify the constrained column by position, not by type**, so symbol columns can use + the hints in §9.4. +5. **`=`, `in`, `within` in hint matching.** +6. **Reject nested partition values** until they are properly supported — currently they are + accepted and then conformed element-wise, which mislabels rows. +7. **Document the `99min`/`99max` convention.** It is a real, working feature with + no public description and a name order that is easy to get backwards. + +Items 1 and 2 are correctness. Nothing here requires a new storage format. + +--- + +### 9.6 Status: designed and proven, deliberately not built + +**The reader has no legacy path, and that is a decision, not an omission.** This is a +greenfield deployment with no history to migrate, and item 1 above — one table spanning both +formats — is with KX. Building the attachment now would mean shipping the two-table-name +workaround into a codebase that may not need it: if KX supports differing column lists, the +right design is one table name and the workaround becomes dead code carrying a migration story +nobody used. + +It was prototyped far enough to retire the risk before being set aside. A conventional +date-partitioned database was built from real captured data, attached beside the capture tree, +and checked against a plain `\l` of the same files. (Measured on the FX schema this pack +carried at the time; the design is schema-independent, and the reader now has no legacy path +at all — see below.) + +| claim | result | +|---|---| +| attaches keyed on **date alone** (§9.2) | one partition per date, not per instrument | +| exposed under its **own table name** (§9.3) | `trade` and `tradehist` side by side, both correct | +| row counts, symbols and numerics vs a direct read | identical | +| filter and group by the partition column | correct — it is in the data, as the legacy DB stored it | +| client query shapes | count, aggregate, time filter, distinct, sort, `meta`, and a union of both tables — all fine | +| **performance vs a conventional load** | **280 µs conventional, 320 µs through the reader** | + +So the design in §9.2 and §9.3 holds, and the performance claim — "correct, and no faster" — is +now measured rather than asserted: about 14 % overhead, not a regression. + +**What it takes when it is needed**, from the prototype: + +- `legacyroots` and `legacysuffix` in the reader's settings +- a scan one level shallower (`root/date/table` *is* the splayed table, so `key` on it returns + column names — its only use is existence) +- a build keyed on `date` alone, into `
`, cached like any other immutable history +- the legacy catalogue kept **apart** from `parts`, or §4.6's coverage check reads a legacy + table's older dates as a gap in every other table +- legacy roots included in `symfiles`, under the same domain rules as §8.3.1 + +Roughly 40 lines in `vtidb.q`. Revisit when KX answers item 1, or when a deployment with real +history appears — whichever comes first. +## 10. Build order + +Steps 0-4 have been done; the rest is the build. + +0. ~~Establish how the query engine actually behaves.~~ **Done** — `testfiles/vt-probe.q`. + Outcome: a column stored in the files can never be used to skip directories, which is what + drives §4.5. +1. ~~Determine whether appends need a reload.~~ **Done** — they do not, given a trailing + slash (§5.2). This removed most of the planned refresh machinery. +2. ~~Determine whether legacy data can be attached.~~ **Done** — yes, keyed on date (§9.2). + Mixing formats under one table name cannot (§9.3). + +3. ~~**WDB writing the right shape.**~~ **Done** — `code/wdb/vtwrite.q`. The tree is + `date/table/instrument/`, the partition column is absent from the files, and a + poison-partition test confirms a query on one instrument never opens the others. +4. ~~**IDB over a single day.**~~ **Done** — `code/processes/vtidb.q` (§5.3). The §5.2 + property is verified end to end through the real process: two IPC queries seconds apart + returned 389 then 395 rows with the partition count unchanged, i.e. the writer's appends + were visible with no reload and no notification. + ~~*Still outstanding:* comparing results against stock kdb+.~~ **Done** — see §12. +5. ~~**New-partition handling.**~~ **Done.** Publishing a new instrument mid-session was + verified end to end: the writer created every table's directory (§4.6), notified, and the + reader rebuilt 19 → 20 partitions, 2 ms from directory creation to reader visibility. + The fill-then-notify ordering was then deliberately broken, and **the failure is real but + not the one this document predicted** — see the measured results in §4.6. The reader does + not error and does not drop the table; it serves that table with the affected date missing, + silently, until the next rebuild. The guard is still justified — arguably more so, since a + silent wrong answer is worse than a crash — but the stated rationale was wrong and has + been corrected. +6. ~~**Multi-day, and measure mappings** against `vm.max_map_count`.~~ **Done, and it + overturned §8.2.** Measured on synthetic trees from 400 to 40,000 partition directories: + the reader adds **0–1 mappings and 0 file descriptors regardless of scale**, because a + trailing-slash open does not memory-map. `vm.max_map_count` is unreachable by this design. + Cost is 891 bytes of heap per partition directory, linear. A selective query stays flat at + 300–400 µs across the whole range, which verifies the design's central claim at scale. + The real limit turned out to be **rebuild time** — ~34 µs per directory, single-threaded, + on the sweep timer — giving a practical ceiling around 100,000 directories. **That ceiling + has since been removed**: `build` now reuses the catalogue and opened views for immutable + dates and rescans only the live partition, which is 154x faster at 40,000 directories and + flat with respect to history depth. See §8.2. +7. **Compression.** Done — `./compress.sh`, driven by `code/processes/vtcompress.q`. Two + findings. The classifier override cannot live in the settings file, because settings load + before `code/common/compress.q` overwrites it (§4.4); and readers need no rollover, no + re-map and no restart across a compression run, which was the documented caveat and is + false (§7.1) — which unblocked the end-of-day work in §8.2.1. What it saves + is 93% of the bytes but only 37% of the disk, for ~30% more query latency, because a third + of the column files already fit inside one filesystem block (§7.2). It now ships with an age + tier (`minage 7`) and a size gate (`minfilesize 4096`) — the gate compresses 137 files + instead of 308 for an identical result on disk and on latency (§7.3). +8. **Attach an existing HDB** as date-keyed partitions under a *separate* table name (§9.3). + Prototyped and proven — correct, and 280 µs versus 320 µs against a conventional load — then + deliberately reverted. Greenfield deployment, and the shape depends on the open KX item. + §9.6 has the findings and what it takes to rebuild. +9. **Second capture stack** into a second root, to test the §8.3 claim. +10. **Make end of day flat.** Done — `rollover` forgets only the date it just closed rather + than the whole cache, so end of day tracks the live rebuild (8 ms at 20,000 directories, + against 857 ms) and no longer grows with retention. The naive version of this change loses + the writer's final flush silently; §8.2.1 has the reason and `testfiles/vt-rollover-test.q` + guards it. + +Step 6 is the one that can invalidate the design at scale, so do not leave it until last. For +steps 4 and 6, reuse `bench/` from the No-RDB pack rather than writing a new harness — +`bench/run.sh` already seeds ~50M rows, stands up an RDB control, and runs a latency matrix, +which gives a directly comparable three-way number (RDB / date-partitioned / this design) on +the selective live lookup that the whole design is meant to win. + +**Client compatibility is smaller than earlier drafts of this document claimed.** +`testfiles/vt-compat-test.q` probes 38 common operations against a virtual table: **27 work +directly, 11 fail, and all 11 work when applied to the result of a `select`**. Nothing is +unreachable. + +An earlier figure of "33 operations, 22 direct, 9 failing" appeared here and in the status +report. It came from an ad-hoc session rather than a script, it did not add up (22 + 9 = 31), +and it could not be re-derived. The test above replaces it, and runs against a scratch tree of +its own so it needs no live stack. + +``` +meta trade 'length meta select from trade ok +`time xasc trade 'type `time xasc select from trade ok +update flag:1b from trade 'type update flag:1b from select from trade ok +trade[`side] 'rank (select from trade)[`side] ok +``` + +The failures are all the same thing: applying an operation to the *table object* rather than +to data. The practical rule is "put a `select` in front of it" — a mechanical edit to existing +scripts, not a redesign. + +Two corrections to earlier drafts, both found by probing rather than assuming: + +- **Dot notation works.** `select time.minute from trade` returns correct values. +- **`ungroup` and `uj` are not virtual-table limitations.** The expressions used to test them + fail identically on an ordinary in-memory table — they were bad tests. + +The one genuine gap is `tables[]`, which does not fail — it succeeds and omits the virtual +tables, because they are type `112h` (§5.3). Anything that *discovers* table names rather than +being told them sees nothing. That affects tooling, not analysts. Checking dashboards for +`tables[]` and bare table references is still worth doing, but it is no longer the thing most +likely to change the design — §8.2 is. + +--- + +## 11. Relationship to the No-RDB Starter Pack + +Checked against [DataIntellectTech/TorQ-No-RDB-Starter-Pack](https://github.com/DataIntellectTech/TorQ-No-RDB-Starter-Pack) +at commit `73956f9`. + +The two designs agree on the whole skeleton. It is worth being explicit that this is a change +of *one variable* rather than a different architecture: + +| | No-RDB pack | this design | +|---|---|---| +| one directory, `savedir==hdbdir` | yes (`KDBDB`) | yes, adopted | +| continuous 1s flush, `immediate:1b` | yes | yes | +| N identical readers, no gateway | yes | yes | +| RDB / HDB / sort workers | none | none | +| **partition scheme** | **date only** | **date + instrument** | +| on-disk shape | real q partitioned DB | 4-level, not q-loadable | +| reader | stock `idb.q` + `.Q.MAP` overlay | replacement, `kx.pq.t` virtual table | +| EOD | staged copy, sort, atomic swap | nothing but a notification | +| sort process | yes | not needed | + +Everything downstream follows from the partition scheme. Because that pack stays +date-partitioned its database is a normal q partitioned DB, so `\l` works, `.Q.MAP` works, +and the stock IDB needs only a small overlay. Because this design partitions by instrument as +well, none of that holds (§2.1) and the virtual table does the job `.Q` does there. + +**Adopted from it:** + +- **The `.proc.addinitlist` overlay pattern** (§4.1). Its `code/wdb/rollover.q` and + `code/idb/mapping.q` both define under a private name and swap in from `.proc.initlist`, + because `$KDBAPPCODE//` loads *before* the stock process code and a direct + redefinition gets clobbered. This is why `process.csv` can keep pointing at stock `wdb.q`. +- **Touch only what can have changed** (§5.4). Its `refreshliveslot` refreshes one partition + slot and leaves the rest alone; an early draft here rescanned every date on every flush. + The trailing-slash view (§5.2) later made even that unnecessary, but the principle drove + the design. +- **Reload sym on growth, and treat it as a recurring cost** (§5.4), which an early draft + omitted and which would have broken on the first new instrument. + +**Deliberately not adopted:** + +- **The EOD staged sort.** That pack copies the day's partition, sorts the copy and swaps it + in with two atomic renames — transiently 2× the day's disk — solely to apply `p#` to `sym`. + Partitioning by instrument makes that unnecessary (§4.2). This is the design's clearest + win: it removes the last EOD operation *and* the transient disk requirement. +- **`.Q.MAP` and the two read modes.** Not applicable to a non-`.Q` database. The + trailing-slash view is the equivalent and needs no framework support. + +**Honest comparison.** On EOD cost this design is unambiguously better, and that needs +nothing from KX. On live selective lookups — the thing it exists to win — it should be +dramatically better: that pack's own benchmark puts an indexed RDB lookup at 0.7 ms against +370 ms for a mapped on-disk scan, and its README is explicit that attributes cannot be +maintained under continuous append, so the live day is always un-indexed. Turning that scan +into a directory lookup is the entire point, and §4.5 is what makes it happen. Verify it at +step 4 rather than assuming it. + +The area where that pack remains ahead is ordinary q compatibility: its database is a real +partitioned database, so everything works on it. See the note at the end of §10. + +## 12. Agreement with stock kdb+ + +Everything up to here shows the design behaves as intended. This section asks a different +question: **given the same data, does it return the same answers as ordinary kdb+?** + +Method — `testfiles/vt-compare-kdb.sh`. Two databases are built from the *same captured bytes*: the +new date+instrument tree, and a conventional date-partitioned splay with `sym` stored +as a real column. The conventional one is stood up as a plain q process, with no TorQ involved. +19 queries are then run against both and the results compared after normalising for row order, +column order and symbol representation. + +**Result: 16 of 19 identical. 3 differ, and in all 3 the virtual table raises an error rather +than returning a wrong answer.** + +``` +PASS total row count PASS min / max / avg +PASS count by date PASS distinct on the partition column +PASS filter on the partition column PASS select specific columns +PASS filter on partition column + date PASS time-range filter +PASS filter on a data column FAIL dot notation on a temporal column +PASS combined partition and data filter PASS empty result - instrument that does not exist +PASS in on a list of instruments PASS weighted average +PASS sum aggregate FAIL fby +PASS group by the partition column FAIL count distinct +PASS group by date and a data column +``` + +**No silent disagreement was found.** That is the material result: an error is recoverable, a +quietly different number is not. + +The three failures follow the pattern established in §10 — they fail on the table object and +work when wrapped in a `select`: + +| query | direct | wrapped in a select | +|---|---|---| +| `select n:count i by time.hh from t` | `'time.hh` | works | +| `where price=(max;price) fby sym` | `'length` | works | +| `select nd:count distinct sym from t` | `'type` | works | + +Note that dot notation works in a *select list* (`select time.minute from t`) and fails only in +a *by* clause; and `fby` works on a data column and fails only on a partition column. The +failures are narrower than the operation names suggest. + +`count distinct` deserves a caveat: it is not a true aggregation, so kdb+ itself returns +per-partition rows for it against a partitioned database. That query needs rethinking whichever +backend it runs on. + +### 12.1 Two representation differences, neither a defect + +**Symbol columns come back as enumerations in-process.** A local query against the virtual +table returns symbol columns as unresolved enumerations (type `20h`) where a conventional +partitioned select resolves them to symbols (`11h`). Values compare equal, but `~` does not +match. **Over IPC both send plain symbols**, so remote clients — dashboards, other processes — +see no difference at all. This only matters to code running inside the reader. + +**The partition column name is configurable, and now matches the schema.** The reader cannot +derive this name — the column is not stored on disk, which is the whole point of the design +(§2.2) — so it comes from `partitioncol` in `appconfig/settings/idb.q`, defaulting to +`instrument` and set to `sym` here. That means a client query reads exactly as it +would against a conventional database, and the comparison in this section needed no renaming on +either side. + +Set it to whatever the source schema calls the parted column. Getting it wrong is not subtle — +queries fail with a value error on the column name rather than returning anything misleading. + +### 12.3 On a brand new database, the tables do not exist yet + +Found while building the load test. A reader started against an *empty* database defines no +tables at all — `build` finds no partitions, warns, and returns without creating the global. +So `select from trade` is a **value error**, not an empty result: + +``` +q)count select from trade +'trade +``` + +A conventional kdb+ stack would have the schema in memory from `database.q` and return an empty +table. Here the tables only come into existence once the writer has flushed something. + +This is narrow — it lasts from process start until the first flush, so about a second in a live +stack — but it is real at the beginning of a deployment, and a client that starts up and +immediately queries can hit it. Two options if it matters: have the reader define empty schema +tables when a partition is missing, or have clients treat a value error on first query as "not +ready yet". Not currently handled either way. + +### 12.2 One case where the virtual table is better + +`exec side from trade` against the conventional partitioned database **fails with `'nyi` over +IPC**, whether or not the client has the sym file. The same query against the virtual table +returns correctly. Not a reason to choose the design, but worth knowing that the compatibility +gap is not entirely one-directional. diff --git a/loadtest.sh b/loadtest.sh new file mode 100755 index 0000000..775581e --- /dev/null +++ b/loadtest.sh @@ -0,0 +1,62 @@ +#!/bin/bash +# VT-12 load test. Starts a clean stack WITHOUT the demo feed, then hands off to +# code/loadtest.q which drives the load and measures the whole chain from one process. +# +# ./loadtest.sh 200k rows, 50 instruments +# ROWS=2000000 PAIRS=500 ./loadtest.sh heavier +# +# Wipes var/ so every run starts from a known state. + +cd "$(dirname "$0")" +. ./vt-env.sh >/dev/null 2>&1 + +if [ ! -f "${TORQHOME}/torq.q" ]; then + echo "ERROR: no torq.q under TORQHOME=${TORQHOME}" >&2 + echo " set TORQHOME to your TorQ checkout, or edit vt-env.sh" >&2 + exit 1 +fi + +ROWS=${ROWS:-200000} +PAIRS=${PAIRS:-50} +BATCH=${BATCH:-1000} + +./stop.sh >/dev/null 2>&1 +rm -rf var +mkdir -p "$KDBDB" "$KDBLOG" "$KDBTPLOG" + +echo "load test: $ROWS rows, $PAIRS instruments, batches of $BATCH" + +APPHOME="$TORQAPPHOME" +ACL="${KDBAPPCONFIG}/passwords/accesslist.txt" +cd "$TORQHOME" +launch () { + q torq.q -load "$4" $KDBSTACKID -proctype "$2" -procname "$1" -localtime $3 \ + "${KDBLOG}/$1.console.log" 2>&1 & + disown +} +launch discovery1 discovery "-U $ACL" "${KDBCODE}/processes/discovery.q" +sleep 2 +q torq.q -load "${KDBCODE}/processes/segmentedtickerplant.q" \ + -schemafile "${APPHOME}/database.q" -tplogdir "$KDBTPLOG" $KDBSTACKID \ + -proctype segmentedtickerplant -procname stp1 -U "$ACL" -localtime \ + "${KDBLOG}/stp1.console.log" 2>&1 & +disown +sleep 2 +launch wdb1 wdb "-U $ACL -g 1" "${KDBCODE}/processes/wdb.q" +sleep 3 +launch idb1 idb "-U $ACL -s 4" "${KDBAPPCODE}/processes/vtidb.q" +sleep 4 +# count only THIS pack's processes - the procnames are TorQ defaults, so an unscoped +# pgrep counts every stack on the machine (it reported 8/4 with two stacks up) +up=$(pgrep -f 'procname (discovery1|stp1|wdb1|idb1)' 2>/dev/null | while read -r pid; do + ps -p "$pid" -o args= 2>/dev/null | grep -qF -- "$TORQAPPHOME" && echo x +done | wc -l) +echo "stack up: ${up}/4" +echo "" + +LOADROWS=$ROWS LOADPAIRS=$PAIRS LOADBATCH=$BATCH q "${APPHOME}/code/loadtest.q" /dev/null | wc -c)" diff --git a/regress.sh b/regress.sh new file mode 100755 index 0000000..92ab483 --- /dev/null +++ b/regress.sh @@ -0,0 +1,138 @@ +#!/bin/bash +# Regression runner for testfiles/. +# +# ./regress.sh every test the environment allows +# ./regress.sh --quick self-contained tests only, never touches a running stack +# ./regress.sh -v stream each test's output instead of summarising +# ./regress.sh --no-mutate skip the one test that rewrites files in var/db +# +# Tests come in two kinds. SELF-CONTAINED ones build their own database in a scratch +# directory and clean it up; they never touch var/ and are safe to run any time. The rest +# need the stack already up (./start.sh) because they publish through the live tickerplant. +# +# ONE TEST MUTATES var/db: vt-compress-test compresses every partition older than a day and +# leaves it compressed. That is what it is for - it checks a live reader copes with files +# being rewritten underneath it - and compression is transparent and idempotent, so nothing +# is lost. But it does change query latency on those partitions. Pass --no-mutate to skip it. +# +# Every test prints " N passed, M failed" and exits non-zero if anything failed, so this +# script reports both the exit status and the counts. + +cd "$(dirname "$0")" +. ./vt-env.sh >/dev/null 2>&1 + +# vt-partition-test loads the real timezone.q and eodtime.q from TorQ core rather than a +# copy of their formula, so the suite needs TORQHOME even though the databases are scratch. +if [ ! -f "${TORQHOME}/torq.q" ]; then + echo "ERROR: no torq.q under TORQHOME=${TORQHOME}" >&2 + echo " set TORQHOME to your TorQ checkout, or edit vt-env.sh" >&2 + exit 1 +fi + +TIMEOUT=${TIMEOUT:-300} +IDBPORT=$((${KDBBASEPORT:-6000}+30)) # the idb, per appconfig/process.csv +QUICK=0; VERBOSE=0; NOMUTATE=0 +for a in "$@"; do + case "$a" in + --quick) QUICK=1 ;; + --no-mutate) NOMUTATE=1 ;; + -v|--verbose) VERBOSE=1 ;; + -h|--help) sed -n '2,20p' "$0"; exit 0 ;; + *) echo "unknown option: $a"; exit 2 ;; + esac +done + +SELFCONTAINED="vt-partition-test vt-rollover-test vt-newtable-test vt-restart-test + vt-inflight-test vt-damage-test vt-multistack-test vt-wdbrestart-test + vt-compat-test vt-diskfull-test" +NEEDSTACK="vt-replay-test vt-symdomain-test vt-collision-test vt-tprestart-test" + +LOGDIR=$(mktemp -d); PASS=0; FAIL=0; SKIP=0; ROWS="" + +run () { # run + local name=$1; shift + printf " %-24s " "$name" + local log="$LOGDIR/$name.log" rc counts + if [ "$VERBOSE" = 1 ]; then + echo ""; timeout -k 10 "$TIMEOUT" "$@" &1 | tee "$log"; rc=${PIPESTATUS[0]} + printf " %-24s " "$name" + else + timeout -k 10 "$TIMEOUT" "$@" "$log" 2>&1; rc=$? + fi + counts=$(grep -oE '[0-9]+ passed, [0-9]+ failed' "$log" | tail -1) + [ -z "$counts" ] && counts="no assertions reported" + if [ "$rc" = 0 ]; then + echo "PASS $counts"; PASS=$((PASS+1)) + elif [ "$rc" = 77 ]; then + # 77 is the pack's convention for "precondition not met" - the database has no + # partitions yet, too few instruments, or only one date. That is not a failure: + # it is a test that cannot run yet, and it must not read as one on a fresh clone. + echo "SKIP $(tail -1 "$log" | sed 's/^ *//')"; SKIP=$((SKIP+1)) + elif [ "$rc" = 124 ]; then + echo "TIMEOUT after ${TIMEOUT}s"; FAIL=$((FAIL+1)) + else + echo "FAIL rc=$rc $counts"; FAIL=$((FAIL+1)) + fi + ROWS="$ROWS$name|$rc|$counts\n" +} + +# For tests whose non-zero exit is expected: pass if the result still matches the recorded +# baseline, fail if it moves. vt-compare-kdb exits 1 because differences exist at all, but +# the three that differ are known and documented - what matters is that it is still three. +run_expect () { # run_expect + local name=$1 want=$2; shift 2 + printf " %-24s " "$name" + local log="$LOGDIR/$name.log" got + timeout -k 10 "$TIMEOUT" "$@" "$log" 2>&1 + got=$(grep -oE "$want" "$log" | tail -1) + if [ -n "$got" ]; then + echo "PASS $got (baseline)"; PASS=$((PASS+1)); ROWS="$ROWS$name|0|$got\n" + else + echo "FAIL baseline moved - expected /$want/"; FAIL=$((FAIL+1)); ROWS="$ROWS$name|1|baseline moved\n" + fi +} + +echo "" +echo "regression run - $(date '+%Y-%m-%d %H:%M:%S')" +echo " timeout per test : ${TIMEOUT}s" +echo " logs : $LOGDIR" +echo "" +echo "self-contained (build their own database, safe any time)" +for t in $SELFCONTAINED; do run "$t" q "testfiles/$t.q"; done + +echo "" +if [ "$QUICK" = 1 ]; then + echo "stack tests skipped (--quick)" + SKIP=$(echo $NEEDSTACK | wc -w); SKIP=$((SKIP+2)) # +vt-compress-test +vt-compare-kdb +elif (ss -ltn 2>/dev/null || netstat -ltn 2>/dev/null) | grep -qE ":${IDBPORT}\b"; then + echo "against the running stack" + for t in $NEEDSTACK; do run "$t" q "testfiles/$t.q"; done + # must go through its wrapper: --test swaps in the 1-day age tier, without which + # nothing in a few-days-old database is in scope and the test has nothing to compress + if [ "$NOMUTATE" = 1 ]; then + printf " %-24s SKIP --no-mutate\n" "vt-compress-test"; SKIP=$((SKIP+1)) + else + echo " (vt-compress-test compresses partitions older than a day in var/db, and leaves them so)" + run vt-compress-test ./compress.sh --test + fi + run_expect vt-compare-kdb '16 matched, 3 differed' ./testfiles/vt-compare-kdb.sh +else + echo "stack tests skipped - nothing listening on ${IDBPORT}. Run ./start.sh first." + SKIP=$(echo $NEEDSTACK | wc -w); SKIP=$((SKIP+2)) # +vt-compress-test +vt-compare-kdb +fi + +echo "" +echo "----------------------------------------" +printf " %d passed, %d failed" "$PASS" "$FAIL" +[ "$SKIP" -gt 0 ] && printf ", %d skipped" "$SKIP" +echo "" +echo "----------------------------------------" +if [ "$FAIL" -gt 0 ]; then + echo "" + echo "failures:" + echo -e "$ROWS" | awk -F'|' -v L="$LOGDIR" '$2!="0" && $1!="" {print " "$1" (rc="$2") "L"/"$1".log"}' + echo "" + exit 1 +fi +rm -rf "$LOGDIR" +echo "" diff --git a/selftest.sh b/selftest.sh new file mode 100755 index 0000000..fffe1ef --- /dev/null +++ b/selftest.sh @@ -0,0 +1,15 @@ +#!/bin/bash +# End-to-end self test. Run it with the stack up (./start.sh). +# +# Publishes a new instrument to the tickerplant and checks it arrives at the IDB through +# the whole chain, that the partition column stays out of the files, and that plain +# appends need no rebuild. + +. "$(cd "$(dirname "$0")" && pwd)/vt-env.sh" >/dev/null 2>&1 + +if ! pgrep -f "procname idb1" >/dev/null 2>&1; then + echo "ERROR: the stack is not running - start it with ./start.sh" >&2 + exit 1 +fi + +q "${TORQAPPHOME}/code/selftest.q" < /dev/null diff --git a/start.sh b/start.sh new file mode 100755 index 0000000..dbf1b2d --- /dev/null +++ b/start.sh @@ -0,0 +1,55 @@ +#!/bin/bash +# Start the virtual-table capture stack: discovery + segmented tickerplant + WDB + feed. +# No RDB, HDB, sort process or gateway. +# +# ./start.sh start +# TORQHOME=/path ./start.sh use a different TorQ checkout + +set -e +. "$(cd "$(dirname "$0")" && pwd)/vt-env.sh" + +if [ ! -f "${TORQHOME}/torq.q" ]; then + echo "ERROR: no torq.q under TORQHOME=${TORQHOME}" >&2 + echo " set TORQHOME to your TorQ checkout, or edit vt-env.sh" >&2 + exit 1 +fi + +echo "TorQ core : ${TORQHOME}" +echo "pack : ${TORQAPPHOME}" +echo "database : ${KDBDB}" +echo "logs : ${KDBLOG}" +echo "" + +cd "$TORQHOME" +ACL="${KDBAPPCONFIG}/passwords/accesslist.txt" + +launch () { # name, proctype, extra flags, load target + echo " $1..." + q torq.q -load "$4" $KDBSTACKID \ + -proctype "$2" -procname "$1" -localtime $3 \ + "${KDBLOG}/$1.console.log" 2>&1 & +} + +launch discovery1 discovery "-U $ACL" "${KDBCODE}/processes/discovery.q" +sleep 2 +echo " stp1..." +q torq.q -load "${KDBCODE}/processes/segmentedtickerplant.q" \ + -schemafile "${TORQAPPHOME}/database.q" -tplogdir "$KDBTPLOG" $KDBSTACKID \ + -proctype segmentedtickerplant -procname stp1 -U "$ACL" -localtime \ + "${KDBLOG}/stp1.console.log" 2>&1 & +sleep 2 +launch wdb1 wdb "-U $ACL -g 1" "${KDBCODE}/processes/wdb.q" +sleep 3 +launch idb1 idb "-U $ACL -s 4" "${KDBAPPCODE}/processes/vtidb.q" +sleep 2 +launch feed1 feed "" "${KDBAPPCODE}/tick/feed.q" + +echo "" +echo "Started. To watch it work:" +echo " find ${KDBDB} -mindepth 3 -maxdepth 3 -type d | head" +echo " tail -f ${KDBLOG}/out_wdb1.log" +echo "" +echo "To query the IDB:" +echo " q -c 25 200" +echo " h:hopen \`::$((KDBBASEPORT+30)):idb:pass" +echo " h\"select n:count i by sym from trade\"" diff --git a/stop.sh b/stop.sh new file mode 100755 index 0000000..0906e07 --- /dev/null +++ b/stop.sh @@ -0,0 +1,35 @@ +#!/bin/bash +# Stop every process belonging to THIS pack. +# +# Scoped by path, not by procname alone. discovery1/stp1/wdb1/feed1/idb1 are TorQ's default +# names, so other packs on the same machine use them too - and a second copy of this pack on +# another port uses exactly these. Matching the name alone would stop all of them. Every +# process this pack launches carries $TORQAPPHOME somewhere in its command line (-load, +# -schemafile or -U), so that is what identifies ours. +. "$(cd "$(dirname "$0")" && pwd)/vt-env.sh" >/dev/null 2>&1 + +PROCS="discovery1 stp1 wdb1 feed1 idb1" + +mine () { # pids of that belong to this pack + pgrep -f "procname $1" 2>/dev/null | while read -r pid; do + [ "$pid" = "$$" ] && continue + ps -p "$pid" -o args= 2>/dev/null | grep -qF -- "$TORQAPPHOME" && echo "$pid" + done +} + +for p in $PROCS; do + pids=$(mine "$p") + [ -n "$pids" ] && kill $pids 2>/dev/null && echo " stopped $p" +done + +sleep 1 + +left="" +for p in $PROCS; do left="$left $(mine "$p")"; done +left=$(echo $left) # collapse whitespace +if [ -n "$left" ]; then + echo " WARNING: some processes still running:" + ps -o pid=,cmd= -p $left 2>/dev/null | cut -c1-120 | sed 's/^/ /' +else + echo " all stopped" +fi diff --git a/testfiles/vt-collision-test.q b/testfiles/vt-collision-test.q new file mode 100644 index 0000000..dc6892f --- /dev/null +++ b/testfiles/vt-collision-test.q @@ -0,0 +1,74 @@ +/ What happens when two instrument names sanitise to the same directory? (§2.3) +/ . +/ cd ~/TorQ-VT-Capture-Pack && . ./vt-env.sh && q testfiles/vt-collision-test.q +/ . +/ The writer builds a directory name by replacing every non-alphanumeric character with "_". +/ That mapping is NOT injective: BRK-B and BRK_B both become BRK_B. The design has always +/ noted this, describing the result as rows "interleaving". Measured, it is considerably worse +/ than interleaving, and this test pins the actual behaviour so it cannot be forgotten: +/ . +/ one instrument becomes completely unqueryable - its rows are on disk, under the other +/ instrument's name, and a query naming it returns zero +/ . +/ the other silently ABSORBS those rows - a query naming it returns more rows than were +/ ever published for it +/ . +/ Neither produces an error. A universe of plain uppercase tickers never trips this; anything +/ containing . - or / does. If yours can, hash or escape the value before it reaches the +/ partition column. +/ . +/ NOTE a line containing only "/" opens a block comment in q, so every comment line here +/ carries text after the slash. + +pass:0; fail:0; +check:{[ok;msg] $[ok; [pass+::1; -1 " PASS ",msg]; [fail+::1; -1 " FAIL ",msg]]; }; + +idbport:`$"::",string[30+"J"$getenv`KDBBASEPORT],":idb:pass"; +tpport:`$"::",getenv[`KDBBASEPORT],":feed:pass"; +h:@[hopen;idbport;{'"no reader on ",string[idbport],": ",x}]; +tp:@[hopen;tpport;{'"no tickerplant on ",string[tpport],": ",x}]; + +system "S ",string "i"$.z.t; +tag:"" sv string 4?.Q.A; +a:`$"ZZ-",tag; / differ only in a character the sanitiser destroys +b:`$"ZZ_",tag; +na:4; nb:6; +dirname:"ZZ_",tag; + +cnt:{[h;s] h "count select from trade where sym=`$\"",string[s],"\""}; + +-1 ""; +-1 " publishing ",string[na]," rows for ",(.Q.s1 a)," and ",string[nb]," for ",.Q.s1 b; +do[na; tp(".u.upd";`trade;(enlist a;enlist 11f;enlist 1i;enlist 0b;enlist " ";enlist "N";enlist`buy))]; +do[nb; tp(".u.upd";`trade;(enlist b;enlist 22f;enlist 2i;enlist 0b;enlist " ";enlist "N";enlist`sell))]; +system "sleep 4"; + +dirs:key hsym`$getenv[`KDBDB],"/",string[h".vtidb.current"],"/trade"; +made:dirs where dirs in (`$dirname;a;b); + +-1 ""; +-1 " directories created : ",.Q.s1 made; +-1 " rows for ",(.Q.s1 a)," : ",.Q.s1 cnt[h;a]; +-1 " rows for ",(.Q.s1 b)," : ",.Q.s1 cnt[h;b]; +-1 ""; + +check[1=count made; + "the two instruments collapsed into ONE directory (",(.Q.s1 made),")"]; +check[0=cnt[h;a]; + "the hyphenated name is now completely unqueryable - ",string[na]," rows published, 0 returned"]; +check[(na+nb)=cnt[h;b]; + "the surviving name absorbed them - ",string[nb]," published, ",string[cnt[h;b]]," returned"]; + +/ the rows are not lost, they are mislabelled - which is what makes it silent +p:first h "exec path from .vtidb.parts[`trade] where sym=`$\"",dirname,"\""; +check[not null p; "the merged directory is in the catalogue under the sanitised name"]; + +-1 ""; +-1 " Both answers are wrong and neither errors. This is the one failure mode in the design"; +-1 " that the writer cannot detect on its own: by the time it has a directory name, the"; +-1 " character that distinguished the two instruments is gone."; +-1 ""; +-1 " ",string[pass]," passed, ",string[fail]," failed"; +-1 ""; +hclose h; hclose tp; +exit $[fail>0;1;0] diff --git a/testfiles/vt-compare-kdb.sh b/testfiles/vt-compare-kdb.sh new file mode 100755 index 0000000..bf189f3 --- /dev/null +++ b/testfiles/vt-compare-kdb.sh @@ -0,0 +1,30 @@ +#!/bin/bash +# VT-11: do identical queries return identical answers from the virtual table and from a +# conventional date-partitioned kdb+ database holding the same data? +# +# Builds two databases from the same captured bytes - the new date+instrument format and a +# conventional date-partitioned splay - stands the conventional one up as a plain q process, +# and runs the same query battery against both. +# +# Requires the capture stack to have run at least once. Run as: +# cd ~/TorQ-VT-Capture-Pack && ./testfiles/vt-compare-kdb.sh + +set -e +cd "$(dirname "$0")/.." +. ./vt-env.sh >/dev/null 2>&1 + +SCRATCH="/tmp/vt-kdb-compare-$$" +PORT=${PORT:-6099} +mkdir -p "$SCRATCH" +trap 'pkill -f "p $PORT" 2>/dev/null; rm -rf "$SCRATCH"' EXIT + +echo "building both databases from the same captured data..." +SCRATCH="$SCRATCH" q testfiles/vt-kdb-prep.q "$SCRATCH/hdb.log" 2>&1 & +sleep 4 + +echo "" +SCRATCH="$SCRATCH" PORT=$PORT q testfiles/vt-kdb-compare.q 50"; ""); + ("where, compound"; "select from trade where sym=`AAPL, price>50";""); + ("count"; "count trade"; ""); + ("count i"; "select n:count i from trade"; ""); + ("sum"; "select sum price from trade"; ""); + ("avg"; "select avg price from trade"; ""); + ("min / max"; "select min price, max price from trade"; ""); + ("by, partition column"; "select n:count i by sym from trade"; ""); + ("by, data column"; "select n:count i by side from trade"; ""); + ("by, two columns"; "select n:count i by date,sym from trade"; ""); + ("by with aggregation"; "select avg price by sym from trade"; ""); + ("exec"; "exec price from trade"; ""); + ("exec by"; "exec avg price by sym from trade"; ""); + ("distinct"; "select distinct sym from trade"; ""); + ("fby, data column"; "select from trade where price>(avg;price) fby side"; ""); + ("fby, partition column"; "select from trade where price>(avg;price) fby sym"; + "select from (select from trade) where price>(avg;price) fby sym"); + ("dot notation, select"; "select time.minute from trade"; ""); + ("dot notation, by clause"; "select n:count i by time.minute from trade"; + "select n:count i by time.minute from select from trade"); + ("column arithmetic"; "select v:price*size from trade"; ""); + ("string / casting"; "select s:string sym from trade"; ""); + ("sublist"; "5 sublist select from trade"; ""); + ("order by"; "`price xasc select from trade"; ""); + ("in"; "select from trade where sym in `AAPL`MSFT"; ""); + ("within"; "select from trade where price within 10 90"; ""); + ("meta"; "meta trade"; "meta select from trade"); + ("cols"; "cols trade"; "cols select from trade"); + ("xasc on the object"; "`time xasc trade"; "`time xasc select from trade"); + ("xdesc on the object"; "`time xdesc trade"; "`time xdesc select from trade"); + ("update"; "update flag:1b from trade"; "update flag:1b from select from trade"); + ("delete a column"; "delete size from trade"; "delete size from select from trade"); + ("delete rows"; "delete from trade where price>50"; "delete from select from trade where price>50"); + ("index by column name"; "trade[`side]"; "(select from trade)[`side]"); + ("value"; "value trade"; "value select from trade"); + ("flip"; "flip trade"; "flip select from trade"); + ("keys"; "key trade"; "key select from trade")); + +res:{[try;iserr;p] + d:try p 1; + $[not iserr d; `direct; + 0=count p 2; `nofix; + iserr try p 2; `nofix; + `wrapper] + }[try;iserr] each probes; + +direct:where res=`direct; +wrapper:where res=`wrapper; +nofix:where res=`nofix; + +-1 ""; +-1 " probed ",string[count probes]," operations against a virtual table"; +-1 ""; +-1 " work directly ",string count direct; +-1 " need a select wrapper ",string count wrapper; +-1 " no workaround ",string count nofix; +-1 ""; +if[count wrapper; + -1 " need a wrapper:"; + {[probes;i] -1 " ",(24$probes[i;0]),probes[i;1]}[probes] each wrapper]; +if[count nofix; + -1 ""; + -1 " NO WORKAROUND:"; + {[probes;i] -1 " ",(24$probes[i;0]),probes[i;1]}[probes] each nofix]; + +-1 ""; +check[count[probes]=count[direct]+count[wrapper]+count nofix; + "the three buckets account for every operation probed (",string[count probes],")"]; +check[00;1;0] diff --git a/testfiles/vt-compress-ab.q b/testfiles/vt-compress-ab.q new file mode 100644 index 0000000..7035314 --- /dev/null +++ b/testfiles/vt-compress-ab.q @@ -0,0 +1,60 @@ +/ Does the size gate actually buy anything? (VT-15, §7.3) +/ . +/ cd ~/TorQ-VT-Capture-Pack && . ./vt-env.sh && q testfiles/vt-compress-ab.q +/ . +/ The gate skips column files that fit inside one filesystem block, on the grounds that they +/ free no disk. The open question is whether skipping them also costs less to read - a query +/ touches every column of an instrument, so leaving the small ones uncompressed might save +/ decompression work, or might be swamped by the large ones that are still compressed. +/ . +/ Run against a live stack. Reports the MINIMUM latency as well as the median: the box is +/ doing other things, and a minimum over many runs is the robust estimator here. + +n:1000; + +idbport:`$"::",string[30+"J"$getenv`KDBBASEPORT],":idb:pass"; +h:@[hopen;idbport;{'"no reader on ",string[idbport],": ",x}]; +pcol:h".vtidb.partitioncol"; +d:first h"asc distinct raze .vtidb.coverage[]"; +dir:getenv[`KDBDB],"/",string d; +inst:first h "exec ",string[pcol]," from .vtidb.parts[`trade] where date=",string d; + +files:hsym each `$system "find ",dir," -type f ! -name '.d'"; +ncomp:{[f] sum {0 /dev/null 2>&1"]; + c:sum {00; +if[not count t; -1 " no compressed files under ",root," - run ./compress.sh first"; exit 1]; + +/ what the file actually costs on disk now, versus what it cost before +t:update allocnow:alloc comp, allocwas:alloc unc from t; + +-1 ""; +-1 " ",string[count t]," compressed column files under ",root; +-1 " ",string[skipped]," uncompressed files skipped (the live partition, which minage protects)"; +-1 ""; +-1 " logical ",mb[sum t`unc]," MB -> ",mb[sum t`comp]," MB (",.Q.f[1;100*1-(sum t`comp)%sum t`unc]," % saved)"; +-1 " on disk ",mb[sum t`allocwas]," MB -> ",mb[sum t`allocnow]," MB (",.Q.f[1;100*1-(sum t`allocnow)%sum t`allocwas]," % saved)"; +-1 ""; + +/ bucket by the size the file had BEFORE compression - that is the number a capacity plan has +bucket:{[x] $[x<1024;`$" <1 kB"; x<4096;`$" 1-4 kB"; x<16384;`$" 4-16 kB"; x<65536;`$"16-64 kB";`$" >64 kB"]}; +t:update sizeband:bucket each unc from t; + +-1 " by original file size:"; +-1 ""; +show `sizeband xasc 0!select + files:count i, + logicalsaved:"F"$.Q.f[1;100*1-(sum comp)%sum unc], + disksaved:"F"$.Q.f[1;100*1-(sum allocnow)%sum allocwas], + blockswas:(sum allocwas)%fsblock, + blocksnow:(sum allocnow)%fsblock + by sizeband from t; + +-1 ""; +-1 " a file under 4 kB occupies one block before and one block after: its logical saving is"; +-1 " real but frees nothing. that band is where instrument-splitting puts most column files."; +-1 ""; +exit 0 diff --git a/testfiles/vt-compress-sizes.q b/testfiles/vt-compress-sizes.q new file mode 100644 index 0000000..ab3d76b --- /dev/null +++ b/testfiles/vt-compress-sizes.q @@ -0,0 +1,62 @@ +/ At what partition size does compression start to free real disk? (VT-15.3) +/ . +/ cd ~/TorQ-VT-Capture-Pack && . ./vt-env.sh && q testfiles/vt-compress-sizes.q +/ . +/ The measured ratio on the running stack (§7) is dominated by one fact: at ~600 rows per +/ instrument per day, most column files are smaller than a filesystem block, and a file that +/ fits in one block frees nothing when compressed. That is a property of how much data lands +/ in each directory, so it should improve as volume per instrument rises. This sweeps that. +/ . +/ Entropy is held constant across the sweep - the same real captured rows are cycled to reach +/ each row count - so the absolute ratios are optimistic, but the SHAPE of the curve, which is +/ what decides the threshold, is not affected by that. + +fsblock:4096; +alloc:{[b;x] b*ceiling x%b}[fsblock]; +mb:{.Q.f[2;x%2 xexp 20]}; + +root:getenv`KDBDB; +src:first system "find ",root," -mindepth 3 -maxdepth 3 -type d -path '*trade*'"; +if[not count src; -1 "no trade partition under ",root; exit 1]; +load hsym`$root,"/sym"; +seed:get hsym`$src,"/"; +if[not count seed; -1 "trade partition is empty - run the stack for a while first"; exit 1]; + +scratch:"/tmp/vt-cmpsize-",string .z.i; +system"rm -rf ",scratch; system"mkdir -p ",scratch; + +/ write one instrument directory holding n rows, compress it, and report both sizes +one:{[scratch;seed;n] + d:scratch,"/n",string n; + system"rm -rf ",d; system"mkdir -p ",d; + t:n#seed; + h:hsym`$d,"/"; + h set .Q.en[hsym`$scratch;t]; + f:hsym each `$system "find ",d," -type f ! -name '.d'"; + unc:hcount each f; + {[x] -19!(x; hsym`$(string x),"_z"; 16; 2; 9); system"mv ",(1_string x),"_z ",1_string x} each f; + cmp:{$[count h:-21!x; h`compressedLength; hcount x]} each f; + `rows`files`unclogical`cmplogical`uncalloc`cmpalloc! + (n; count f; sum unc; sum cmp; sum alloc unc; sum alloc cmp) + }; + +rows:100 300 1000 3000 10000 30000 100000; +r:one[scratch;seed] each rows; +t:flip r; +rnd:{0.1*"j"$10*x}; / .Q.f does not vectorise +t:update logicalsaved:rnd 100*1-cmplogical%unclogical, + disksaved:rnd 100*1-cmpalloc%uncalloc, + kbperfile:rnd (unclogical%files)%1024 from t; + +-1 ""; +-1 " source ",src; +-1 " columns ",string[count cols seed]," per instrument directory, ",string[first r@\:`files]," column files"; +-1 ""; +show select rows, kbperfile, files, uncallocKB:rnd uncalloc%1024, cmpallocKB:rnd cmpalloc%1024, logicalsaved, disksaved from t; +-1 ""; +-1 " compressed size lands on the floor - one filesystem block per column file - almost"; +-1 " immediately and stays there (cmpallocKB is flat). so disksaved is decided entirely by"; +-1 " how big the files were to begin with, which is rows per instrument per day."; +-1 ""; +system"rm -rf ",scratch; +exit 0 diff --git a/testfiles/vt-compress-test.q b/testfiles/vt-compress-test.q new file mode 100644 index 0000000..2d22cc4 --- /dev/null +++ b/testfiles/vt-compress-test.q @@ -0,0 +1,99 @@ +// VT-15.2 / VT-15.3 : does compression break a live reader, and does it pay? +// +// run against a running stack: ./compress.sh --test +// +// the reader is never told compression happened. it keeps the same handle, the same open +// views, and is not sent a rollover. if its answers change, §7's caveat is real and end of +// day must keep dropping the reader's cache - which is what makes VT-17 impossible. if they +// do not change, the caveat is wrong and end of day can be made flat. + +pass:0; fail:0; +check:{[ok;msg] $[ok; [pass+::1; -1 " PASS ",msg]; [fail+::1; -1 " FAIL ",msg]]; }; + +logical:{[d] "J"$first system "du -sk --apparent-size ",d," | cut -f1"}; +alloc:{[d] "J"$first system "du -sk ",d," | cut -f1"}; +pctsaved:{[b;a] $[0=b; 0n; 100*1-a%b]}; + +idbport:`$"::",string[30+"J"$getenv`KDBBASEPORT],":idb:pass"; +h:@[hopen;idbport;{'"no reader on ",string[idbport],": ",x}]; +pcol:h".vtidb.partitioncol"; + +// the target is the oldest complete date - compression must never touch the live one +dates:h"asc distinct raze .vtidb.coverage[]"; +if[2>count dates; -1 " need at least two dates - run the stack across a day boundary first"; exit 77]; +d:first dates; +dir:getenv[`KDBDB],"/",string d; + +// one row per instrument per table: row count and time bounds. changes if any column file +// stops resolving, or resolves to different bytes +fp:{[h;pcol;d;t] h "0!select cnt:count i, ft:first time, lt:last time by ", + string[pcol]," from ",string[t]," where date=",string d}; + +-1 ""; +-1 " reader ",string idbport; +-1 " partition ",string[d]," of ",string[count dates]," present; live is ",string last dates; +-1 ""; + +// a selective single-instrument query, the case the whole layout exists to make fast. +// compression is not free on read: every block touched has to be inflated first +inst:first h "exec ",string[pcol]," from .vtidb.parts[`trade] where date=",string d; +/ NOTE n#f[x] replicates ONE result n times - it does not run f n times. the index argument +/ is what forces a fresh application per sample +sel:{[h;pcol;d;inst;n] h ({[qry;n] `long$min {[qry;i] t:.z.p; value qry; `long$(.z.p-t)%1000}[qry] each til n}; + "select from trade where date=",string[d],", ",string[pcol],"=`",string[inst];n)}; +lat:sel[h;pcol;d;inst]; + +// start from a known state so the test is repeatable: decompress anything a previous run +// left behind. -19! with algo 0 inflates; the reader is not told, which is itself a check +files:hsym each `$system "find ",dir," -type f ! -name '.d'"; +was:sum {0 /dev/null 2>&1"; +-1 " took ",string .z.p-t0; +-1 ""; + +after:fp[h;pcol;d] each tabs; +latafter:lat 200; +la:logical dir; aa:alloc dir; +compafter:sum {0compbefore; + "files are genuinely compressed (",string[compbefore]," -> ",string[compafter]," of ",string[count files],")"]; +check[0=count system "find ",dir," -name '*_kdbtempzip'"; + "no temporary files left behind"]; +check[0 ",string[latafter]," us (", + .Q.f[1;100*-1+latafter%latbefore]," %)"; +-1 ""; +-1 " ",string[pass]," passed, ",string[fail]," failed"; +-1 ""; +hclose h; +exit $[fail>0;1;0] diff --git a/testfiles/vt-damage-test.q b/testfiles/vt-damage-test.q new file mode 100644 index 0000000..cdc4e71 --- /dev/null +++ b/testfiles/vt-damage-test.q @@ -0,0 +1,134 @@ +/ What does on-disk damage to one partition actually do? (§4.6) +/ . +/ cd ~/TorQ-VT-Capture-Pack && . ./vt-env.sh && q testfiles/vt-damage-test.q +/ . +/ §4.6 established that a MISSING table directory is served as a silently absent date. This +/ asks the next question: what about a directory that is present but damaged? Three kinds, +/ because they behave three different ways and only one of them is loud: +/ . +/ truncated column the partition attaches and silently returns FEWER ROWS - the table is +/ cut to its shortest column. no warning. this is the dangerous one. +/ note it is specific to UNCOMPRESSED columns: a compressed one has a +/ metadata header that kdb+ checks, so the same damage raises instead +/ . +/ missing .d the reader cannot type the directory, logs "skipped 1 unreadable +/ partition(s)" and leaves it out. that instrument then reads as absent +/ . +/ corrupt column any query that opens the directory errors, loudly, with the file path +/ . +/ The blast radius is the useful part: a SELECTIVE query on a healthy instrument is unaffected, +/ because the damaged directory is never opened. Whole-database queries fail - including ones +/ that do not name the damaged column, since they still have to open every directory. +/ . +/ Runs entirely on a scratch copy. Nothing here touches the live database. +/ . +/ NOTE a line containing only "/" opens a block comment in q, so every comment line here +/ carries text after the slash. + +/ enough of the TorQ framework for vtidb.q to load standalone +.lg.o:{[t;m]}; .lg.w:{[t;m] warns,:enlist m}; .lg.e:{[t;m] errs,:enlist m}; +.servers.startupdepcycles:{[t;i;c] '"no wdb in this test"}; +.servers.gethandlebytype:{[t;m] ()}; +.timer.enabled:0b; .timer.repeat:{[a;b;c;d;e]}; .proc.cp:{[] .z.P}; +warns:(); errs:(); + +pass:0; fail:0; +check:{[ok;msg] $[ok; [pass+::1; -1 " PASS ",msg]; [fail+::1; -1 " FAIL ",msg]]; }; +try:{[f;a] .[f;a;{`$"ERROR: ",x}]}; +/ NOTE the query text already says "count select …", so this must NOT count again - count of +/ an atom is 1, which made every result look like a single row +cnt:{[q] try[{[x] value x};enlist q]}; + +live:getenv`KDBDB; +s:"/tmp/vt-damage-",string .z.i; +src:first asc key[hsym`$live] where key[hsym`$live] like "[0-9][0-9][0-9][0-9].*"; +if[null src; -1 " no partitions under ",live," - start the stack first"; exit 77]; + +system"rm -rf ",s; system"mkdir -p ",s; +system"cp ",live,"/sym ",s,"/"; +system"cp -r ",live,"/",(string src)," ",s,"/2026.01.01"; + +d:s,"/2026.01.01/trade"; +insts:key hsym`$d; +if[6>count insts; -1 " need at least 6 instruments to damage three and keep healthy ones"; exit 77]; + +trunc:string insts 0; +nodotd:string insts 1; +garbage:string insts 2; +healthy:string insts 5; + +whole:cnt "count select from trade"; / before any damage, for comparison + +/ the source partition may already be compressed - vt-compress-test compresses everything +/ older than a day in var/db and leaves it that way, so a second run of the suite copies a +/ compressed partition here. that matters: a compressed column carries a metadata header +/ which kdb+ validates, so truncating one RAISES "bad meta data in file" where truncating an +/ uncompressed one short-reads silently. the silent case is the dangerous one and the one +/ under test, so rewrite the target column plain and assert against a known state. +tp:hsym`$d,"/",trunc,"/price"; +if[count -21!tp; + .z.zd:(17;0;0); / algo 0 - write it back uncompressed + (hsym`$d,"/",trunc,"/price.plain") set get tp; + system"x .z.zd"; + system"mv ",d,"/",trunc,"/price.plain ",d,"/",trunc,"/price"]; + +system"truncate -s 40 ",d,"/",trunc,"/price"; +system"rm ",d,"/",nodotd,"/.d"; +system"dd if=/dev/urandom of=",d,"/",garbage,"/time bs=200 count=1 2>/dev/null"; + +-1 ""; +-1 " scratch ",s; +-1 " damaged ",trunc," (truncated price), ",nodotd," (no .d), ",garbage," (corrupt time)"; +-1 " healthy ",healthy; +-1 ""; + +.vtidb.roots:enlist hsym`$s; +.vtidb.partitioncol:`sym; +system"l ",getenv[`KDBAPPCODE],"/processes/vtidb.q"; +.vtidb.current:2026.01.02; +.vtidb.dropcache[]; .vtidb.rebuild[]; + +attached:count .vtidb.parts`trade; +-1 " attached ",string[attached]," of ",string[count insts]," directories"; +-1 ""; + +/ --------------------------------------------------------------------------- +check[attached=count[insts]-1; + "the .d-less directory is excluded; the other two are attached"]; +check[any warns like "*unreadable partition*"; + "and its exclusion is logged, not silent"]; + +ntrunc:cnt "count select from trade where sym=`",trunc; +check[$[-7h=type ntrunc; ntrunc>0; 0b]; / $[] short-circuits, `and` does not + "KNOWN, AND SILENT: a truncated column attaches and returns ",string[ntrunc]," rows"]; +check[not any warns like "*",trunc,"*"; + " - with no warning naming it. shortest column wins, quietly"]; + +check[0=cnt "count select from trade where sym=`",nodotd; + "the excluded directory reads as absent - 0 rows, no error"]; + +check[-11h=type cnt "count select from trade where sym=`",garbage; + "a corrupt column ERRORS when its directory is opened"]; + +/ --------------------------------------------------------------------------- +-1 ""; +nhealthy:cnt "count select from trade where sym=`",healthy; +check[$[-7h=type nhealthy; nhealthy>0; 0b]; + "a selective query on a healthy instrument is unaffected (",string[nhealthy]," rows)"]; +check[(-7h=type cnt "count select from trade where date=2026.01.01, sym=`",healthy); + " - and stays unaffected with the date constrained too"]; + +check[-11h=type cnt "count select n:count i by sym from trade"; + "a whole-table query fails, because it must open the corrupt directory"]; +check[-11h=type cnt "count select from trade where price>0"; + " - even one that never names the corrupt column"]; + +-1 ""; +-1 " Partition elimination is what limits the damage: the queries this layout exists to"; +-1 " make fast are the ones that keep working. What needs watching is the truncated"; +-1 " column - it is the only one of the three that answers, and answers wrongly."; +-1 ""; +-1 " ",string[pass]," passed, ",string[fail]," failed"; +-1 ""; +system"rm -rf ",s; +exit $[fail>0;1;0] diff --git a/testfiles/vt-diskfull-test.q b/testfiles/vt-diskfull-test.q new file mode 100644 index 0000000..08d7144 --- /dev/null +++ b/testfiles/vt-diskfull-test.q @@ -0,0 +1,226 @@ +/ What happens when the disk fills up? (4.5, 5.3, 9.3) +/ . +/ cd ~/TorQ-VT-Capture-Pack && . ./vt-env.sh && q testfiles/vt-diskfull-test.q +/ . +/ This one needs a filesystem it can actually fill, so it re-runs itself inside a private mount +/ namespace with a small tmpfs in it. No root, and nothing outside that namespace is touched - +/ the mount does not even exist for other processes. If the kernel will not give us a user +/ namespace the test says so and stops rather than pretending. +/ . +/ A full disk is worth its own test because it is the one failure that hits the writer WHILE it +/ is writing, and this design has no staging area to fail into: the writer writes directly into +/ the directories readers are reading. So the questions are: +/ . +/ does the writer survive it, or does q die on a failed write +/ what is left on disk, and what does a reader make of it +/ is the data still in memory to retry, or is it gone +/ what does a retry actually do once space is free +/ . +/ The interesting answer is the third one. TorQ empties the in-memory table AFTER the upsert +/ loop, so an error that propagates leaves the data intact and the next flush retries it - but +/ the partitions written before the failure are written AGAIN, and nothing dedupes them. +/ . +/ NOTE a line containing only "/" opens a block comment in q, so every comment line here +/ carries text after the slash. + +/ --------------------------------------------------------------------------- +/ re-run inside a private mount namespace, unless we are already in one +/ --------------------------------------------------------------------------- +if[not count getenv`VTFULLDIR; + mnt:"/tmp/vt-diskfull-",string .z.i; + outf:mnt,".out"; + system"mkdir -p ",mnt; + / the child's output does not come back through system, so bash writes it to a file INSIDE + / the -c string where the redirect is honoured. the file sits outside the tmpfs, which only + / exists inside the namespace and goes away with it + cmd:"unshare -rm bash -c \"mount -t tmpfs -o size=8M tmpfs ",mnt, + " && VTFULLDIR=",mnt," ",getenv[`QCMD]," ",(string .z.f)," > ",outf," 2>&1\""; + @[system;cmd;{[e] ::}]; + out:@[read0;hsym`$outf;{[e] ()}]; + system"rm -rf ",mnt," ",outf; + if[not any {x like "*passed*"} each out; + -1 ""; + -1 " SKIPPED - could not create a private mount namespace on this kernel."; + -1 " this test needs unshare -rm to build a small filesystem it can fill."; + if[count out; -1 each out]; + -1 ""; + exit 0]; + -1 each out; + exit $[any {x like "*, 0 failed*"} each out; 0; 1]]; + +/ --------------------------------------------------------------------------- +/ from here on we are inside the namespace, on a filesystem of a few megabytes +/ --------------------------------------------------------------------------- +.lg.o:{[t;m]}; .lg.w:{[t;m] warns,:enlist m}; .lg.e:{[t;m] errs,:enlist m}; +.servers.startupdepcycles:{[t;i;c] '"no wdb in this test"}; +.servers.gethandlebytype:{[t;m] ()}; +.timer.enabled:0b; .timer.repeat:{[a;b;c;d;e]}; .proc.cp:{[] .z.P}; +warns:(); errs:(); + +pass:0; fail:0; +check:{[ok;msg] $[ok; [pass+::1; -1 " PASS ",msg]; [fail+::1; -1 " FAIL ",msg]]; }; +try:{[f;a] .[f;a;{[e] `$"ERR:",e}]}; +/ `ok is a symbol and so is a trapped error - tell them apart by the text, not the type +failed:{[x] $[-11h=type x; x like "ERR:*"; 0b]}; +saw:{[pat] $[count warns; any {[p;w] w like p}[pat] each warns; 0b]}; + +root:hsym`$getenv`VTFULLDIR; +(.Q.dd[root;`sym]) set 0#`; +d:2026.01.01; +rows:20000; +mktab:{[n] ([]time:n#.z.p; price:n?100f; side:n?`buy`sell)}; +pdir:{[root;d;i] .Q.dd[.Q.dd[.Q.dd[root;`$string d];`trade];`$"I",string i]}[root;d]; +write:{[root;pdir;i;t] try[{[root;pdir;i;t] (.Q.dd[pdir i;`]) set .Q.ens[root;t;`sym]; `ok}[root;pdir;i];enlist t]}[root;pdir]; + +-1 ""; +-1 " filesystem ",getenv`VTFULLDIR; +-1 " ",last system"df -h ",getenv`VTFULLDIR; + +/ --------------------------------------------------------------------------- +/ 1. fill it. +/ --------------------------------------------------------------------------- +-1 ""; +-1 " RUNNING OUT OF SPACE"; + +r:{[write;mktab;rows;i] write[i;mktab rows]}[write;mktab;rows] each til 60; +broke:first where failed each r; +check[not null broke; "the filesystem did fill (",string[broke]," partitions written first)"]; +check[(string r broke) like "*No space left on device*"; + "ENOSPC arrives as a normal q error naming the file, not as a signal"]; +check[4=try[{[x] x+x};enlist 2]; "the process is alive - a failed write is an error, not a death"]; +check[rows=try[{[p] count get .Q.dd[p;`]};enlist pdir 0]; + "and the partitions written before the failure are intact and readable"]; + +/ --------------------------------------------------------------------------- +/ 2. what it left behind. +/ this is a FOURTH kind of damage, on top of the three in vt-damage-test.q, and it is the +/ worst of them: .d promises columns that were never written. A lazy get accepts it without +/ complaint and even counts it, because a count reads one column. +/ --------------------------------------------------------------------------- +-1 ""; +-1 " THE HALF-WRITTEN PARTITION"; + +bad:pdir broke; +named:get .Q.dd[bad;`.d]; +present:key bad; +-1 " .d names ",.Q.s1 named; +-1 " on disk ",.Q.s1 present; +missing:named except present; +check[count missing; "the partition promises columns it does not have (",(.Q.s1 missing),")"]; + +v:try[{[p] get .Q.dd[p;`]};enlist bad]; +check[not failed v; "and it still opens cleanly - get is lazy, so nothing fails here"]; +check[rows=try[{[x] count x};enlist v]; + "it even counts correctly (",string[rows],") - a count reads only the first column"]; +check[failed try[{[x] count select from x};enlist v]; + "the failure lands on the first query that touches a missing column"]; + +/ --------------------------------------------------------------------------- +/ 3. what a reader makes of it. +/ . +/ This partition is NOT detected. get is lazy, so it attaches like any other, and the failure +/ then belongs to the whole table rather than to the directory that caused it - every partition +/ has to be opened to answer a query that does not name an instrument. +/ . +/ The reader could check each partition against its own .d before accepting it. That was built +/ and then deliberately reverted (5.7): it costs ~45% of a rescan, the transient race that +/ produces this state without a full disk is rare and heals itself within one sweep, and in the +/ permanent case it would trade a loud failure for a reader that keeps answering while silently +/ omitting an instrument. A full disk means capture has already stopped. Queries failing is the +/ correct signal, not a defect to be masked. +/ --------------------------------------------------------------------------- +-1 ""; +-1 " WHAT THE READER DOES WITH IT"; + +.vtidb.roots:enlist root; +.vtidb.partitioncol:`instrument; +warns:(); +system"l ",getenv[`KDBAPPCODE],"/processes/vtidb.q"; + +check[(`$"I",string broke) in exec instrument from .vtidb.parts`trade; + "the half-written partition ATTACHES - nothing about it looks wrong to the reader"]; +check[not saw "*cannot open*"; "nothing is logged, because nothing failed at open time"]; + +n:try[{[] count select from trade};enlist(::)]; +check[failed n; + "a whole-database query then FAILS, and the error names the missing file, not the query"]; +check[failed try[{[] count select time from trade};enlist(::)]; + " - including one that never mentions a damaged column: every partition is opened regardless"]; + +one:try[{[] count select from trade where instrument=`I0};enlist(::)]; +check[(not failed one) and rows=one; + "a SELECTIVE query on a healthy instrument still works - the blast radius is bounded by ", + "partition elimination, exactly as in vt-damage-test.q"]; +check[failed try[{[i] count select from trade where instrument=i};enlist `$"I",string broke]; + "and a selective query on the damaged one fails, which is where an operator would look"]; + +/ --------------------------------------------------------------------------- +/ 4. is the data still in memory? +/ savetablesbypart upserts every partition and only THEN empties the table, and the pack's +/ override rethrows rather than swallowing. so a failure has to leave the rows where they were. +/ --------------------------------------------------------------------------- +-1 ""; +-1 " IS THE DATA LOST?"; + +/ free enough room for a couple of partitions but not for all of them, so the save gets part +/ way through and then hits the wall - which is the case that matters. a save that fails on +/ its FIRST partition leaves nothing behind and nothing to reconcile +freeup:{[root;pdir;n] {[pdir;i] system"rm -rf ",1_string pdir i}[pdir] each n; }; +freeup[root;pdir] 0 1 2; +targets:100 101 102 103; + +buf:raze {[mktab;rows;i] update inst:i from mktab rows}[mktab;rows] each targets; +before:count buf; + +/ savetablesbypart's own order: enumerate, upsert every partition, and only THEN empty the +/ table. the pack's upsert override rethrows rather than swallowing, so the error gets out +/ before that last step - which is the whole reason the data survives +saveall:{[root;pdir;targets;t] + e:.Q.ens[root;t;`sym]; + {[root;pdir;e;i] .[{[p;x] p set x};(.Q.dd[pdir i;`];delete inst from select from e where inst=i);{[er] 'er}]}[root;pdir;e] each targets; + @[`.;`buf;0#]; + }; +res:try[{[f;targets;t] f[targets;t]; `ok}[saveall[root;pdir]];(targets;buf)]; +check[failed res; "the save fails part way through - there is not room for all of it"]; +check[before=count buf; + "the rows are STILL IN MEMORY (",string[count buf],") - the error propagates before the ", + "table is emptied, so the next flush retries them. a disk-full does not lose data"]; + +written:targets where {[pdir;i] 0 the retry re-upserts the WHOLE buffer, so those partitions get the same rows a"; +-1 " second time. nothing dedupes them. after a disk-full it is the partitions written"; +-1 " BEFORE the failure that need checking, not just the one that reported it."; + +/ --------------------------------------------------------------------------- +/ 5. recovery. +/ --------------------------------------------------------------------------- +-1 ""; +-1 " RECOVERY ONCE SPACE IS FREE"; + +/ the fill loop left every partition from the failure upwards incomplete, not just the one +/ that reported the error - clearing those is the operator's job, and the reader will go on +/ naming them until someone does it +{[pdir;i] system"rm -rf ",1_string pdir i}[pdir] each (broke+1)+til 59-broke; +freeup[root;pdir] 3 4 5,targets; +free:"J"$first system"df -k ",getenv[`VTFULLDIR]," | tail -1 | awk '{print $4}'"; +check[00;1;0] diff --git a/testfiles/vt-gap-test.q b/testfiles/vt-gap-test.q new file mode 100644 index 0000000..eaa27b3 --- /dev/null +++ b/testfiles/vt-gap-test.q @@ -0,0 +1,62 @@ +/ What happens when a partition is missing one table? +/ . +/ This is the evidence behind §4.6 and step 5 of §10. It breaks the writer's +/ fill-then-notify ordering by hand - removing a table directory from one date - and shows +/ what a reader actually does with the result. +/ . +/ Run it with the stack having captured at least one date: +/ cd ~/TorQ-VT-Capture-Pack && . ./vt-env.sh && q testfiles/vt-gap-test.q +/ . +/ NOTE a line containing only "/" opens a block comment in q, so every comment line here +/ carries text after the slash. + +/ enough of the TorQ framework for vtidb.q to load standalone +.lg.o:{[t;m]}; .lg.w:{[t;m]}; .lg.e:{[t;m]}; +.servers.startupdepcycles:{[t;i;c] '"no wdb in this test"}; +.servers.gethandlebytype:{[t;m] ()}; +.timer.enabled:0b; .timer.repeat:{[a;b;c;d;e]}; .proc.cp:{[] .z.P}; + +live:getenv`KDBDB; +scratch:"/tmp/vt-gap-test-",string .z.i; / .z.i is this process id +reader:getenv[`KDBAPPCODE],"/processes/vtidb.q"; + +/ two identical dates, so a gap can be put in either the first or the last +src:first asc key[hsym`$live] where key[hsym`$live] like "[0-9][0-9][0-9][0-9].*"; +if[null src; -1 "no partitions under ",live," - start the stack first"; exit 77]; +system"rm -rf ",scratch; +system"mkdir -p ",scratch; +system"cp ",live,"/sym ",scratch,"/"; +system"cp -r ",live,"/",(string src)," ",scratch,"/2026.01.01"; +system"cp -r ",live,"/",(string src)," ",scratch,"/2026.01.02"; +-1 "scratch database : ",scratch; + +/ the gap: remove the populated table from the SECOND date, exactly what a notify-before-fill +/ writer would leave behind +system"rm -rf ",scratch,"/2026.01.02/trade"; +-1 "removed : 2026.01.02/trade"; +-1 ""; + +.vtidb.roots:enlist hsym`$scratch; +r:@[{system"l ",x; `ok};reader;{`$"FAILED: ",x}]; + +-1 "RESULT"; +-1 (56#"-"); +-1 " reader load : ",$[r~`ok;"LOADED - no error";string r]; +if[not r~`ok; exit 1]; +-1 " tables attached : ",.Q.s1 key .vtidb.parts; +-1 " dates for trade : ",.Q.s1 asc distinct exec date from .vtidb.parts`trade; +-1 " count select from trade : ",.Q.s1 value"count select from trade"; +-1 ""; +-1 " the table answers queries. the answer is missing a whole date, with no warning."; +-1 ""; + +/ and the recovery: a restored directory needs a rebuild, which is what the sweep provides +system"cp -r ",live,"/",(string src),"/trade ",scratch,"/2026.01.02/trade"; +-1 " directory restored on disk, before rebuild : ",.Q.s1 value"count select from trade"; +.vtidb.rebuild[]; +-1 " after one rebuild (what the sweep does) : ",.Q.s1 value"count select from trade"; +-1 ""; +-1 " so the damage is bounded by the sweep interval, not permanent."; + +system"rm -rf ",scratch; +exit 0 diff --git a/testfiles/vt-inflight-test.q b/testfiles/vt-inflight-test.q new file mode 100644 index 0000000..d564f06 --- /dev/null +++ b/testfiles/vt-inflight-test.q @@ -0,0 +1,250 @@ +/ What does a query see when it arrives at the worst possible moment? (5.2, 5.3) +/ . +/ cd ~/TorQ-VT-Capture-Pack && . ./vt-env.sh && q testfiles/vt-inflight-test.q +/ . +/ Every other test in this directory starts from a world that is standing still: the data is +/ already written, the catalogue is already built, and only then does anything get asked. That +/ is not how this database is read. The whole point of the design is that the writer appends to +/ the same directories a client is querying, with no copy and no handover, so the interesting +/ question is what a query returns when it lands in the middle of something. +/ . +/ Three moments, and they fail - or do not - for different reasons: +/ . +/ mid-append the writer is extending the column files of a partition this query is +/ reading. columns are extended ONE AT A TIME, so between them the partition +/ genuinely has columns of different lengths on disk +/ . +/ mid-rebuild the reader is replacing the virtual table while a client asks for it +/ . +/ ill-formed the catalogue built, but the partitions it points at cannot be read together +/ . +/ The first is the one that ought to worry you, because nothing in the design prevents it and +/ there is no lock anywhere. The answer turns out to be the same rule that makes on-disk +/ truncation silent in vt-damage-test.q - a splayed table is cut to its SHORTEST column - only +/ here that rule is what saves it: a short read is a consistent PREFIX, never a torn row. +/ . +/ NOTE a line containing only "/" opens a block comment in q, so every comment line here +/ carries text after the slash. + +pass:0; fail:0; +check:{[ok;msg] $[ok; [pass+::1; -1 " PASS ",msg]; [fail+::1; -1 " FAIL ",msg]]; }; +try:{[f;a] .[f;a;{[e] `$"ERR:",e}]}; +iserr:{[x] -11h=type x}; + +s:"/tmp/vt-inflight-",string .z.i; +system"rm -rf ",s; system"mkdir -p ",s; + +/ --------------------------------------------------------------------------- +/ 1. the rule, established without a race. +/ a directory whose columns are different lengths is exactly what the writer leaves behind +/ between one column's append and the next. build one deliberately and read it. +/ --------------------------------------------------------------------------- +-1 ""; +-1 " RAGGED PARTITION - columns of different lengths, no race involved"; + +r1:hsym`$s,"/static"; +.Q.dd[r1;`] set ([]a:til 1000; b:2*til 1000; c:1000#`long$7); +system"truncate -s ",string[8*400]," ",(1_string .Q.dd[r1;`b]); / b now holds 400 of 1000 + +v1:get .Q.dd[r1;`]; +lens:value count each flip v1; +n1:try[{[x] count select from x};enlist v1]; +cons:try[{[x] count select from x where b<>2*a};enlist v1]; + +-1 " column lengths on disk ",.Q.s1 lens; +-1 " rows served ",.Q.s1 n1; +check[1/dev/null 2>&1 &"; +system"sleep 0.1"; + +v2:get .Q.dd[r2;`]; +one:{[try;v;i] + cnts:try[{[x] value count each flip x};enlist v]; + t:try[{[x] select from x};enlist v]; + (cnts; try[{[x] count x};enlist t]; try[{[x] count select from x where b<>2*a};enlist t]) + }[try;v2]; +res:one each til 5000; + +cs:res[;0]; +ragged:sum {$[7h=type x; 12*a - no torn rows, ever"]; +check[ns~asc ns; "and the row count a reader sees only ever goes up"]; +-1 " -> no lock is needed on the read path. a query that lands mid-append is short,"; +-1 " never wrong, and the next query sees the rest."; + +/ --------------------------------------------------------------------------- +/ 3. mid-rebuild. +/ rebuild REPLACES the global a client is querying. a client that caught it half-done would +/ see a table whose catalogue and contents disagree. this runs a real reader process with +/ rebuild on a 3ms timer while new directories keep appearing underneath it, and has a client +/ ask - in ONE message, so the answer is a single instant - for three numbers that must agree. +/ --------------------------------------------------------------------------- +-1 ""; +-1 " MID-REBUILD - a client querying while the reader replaces the table"; + +root:hsym`$s,"/live"; +system"mkdir -p ",(1_string root); +(.Q.dd[root;`sym]) set 0#`; +d:2026.01.01; +mkinst:{[root;d;i] + t:([]time:10#.z.p; price:10?100f; side:10?`buy`sell); + .Q.dd[.Q.dd[.Q.dd[.Q.dd[root;`$string d];`trade];`$"I",string i];`] set .Q.ens[root;t;`sym]; + }[root;d]; +mkinst each til 20; + +port:1+rand 2000; +port+:20000; +rq:s,"/reader.q"; +(hsym`$rq) 0: ( + ".lg.o:{[t;m]}; .lg.w:{[t;m]}; .lg.e:{[t;m]};"; + ".servers.startupdepcycles:{[t;i;c] '\"nowdb\"};"; + ".servers.gethandlebytype:{[t;m] ()};"; + ".timer.enabled:0b; .timer.repeat:{[a;b;c;d;e]}; .proc.cp:{[] .z.P};"; + ".vtidb.roots:enlist hsym`$getenv`ROOT;"; + ".vtidb.partitioncol:`instrument;"; + "system \"l \",getenv[`KDBAPPCODE],\"/processes/vtidb.q\";"; + ".vtidb.current:2026.01.01;"; + "rebuilds:0;"; + "started:.z.p;"; + "/ three numbers that MUST agree, answered in one message so they share one instant"; + "probe:{[] (count select from trade; count .vtidb.parts`trade;"; + " count distinct exec instrument from select instrument from trade)};"; + "aligned:{[] all {count[.vtidb.parts x]=count .vtidb.opened x} each key .vtidb.parts};"; + "/ the timer both rebuilds and enforces a deadline, so a stuck client cannot orphan this"; + ".z.ts:{[] .vtidb.rebuild[]; rebuilds::rebuilds+1; if[0D00:01<.z.p-started; exit 0]};"; + "system \"p \",getenv`PORT;"; + "system \"t \",getenv`TICK;"); +system"ROOT=",(1_string root)," PORT=",string[port]," TICK=3 q ",rq," ",s,"/reader.log 2>&1 &"; +system"sleep 1.5"; + +h:@[hopen;`$":localhost:",string port;0Ni]; +if[null h; -1 " could not reach the reader - see ",s,"/reader.log"; exit 1]; + +/ a churner adding directories underneath the reader, so rebuild has real work to do +cq:s,"/churn.q"; +(hsym`$cq) 0: ( + "root:hsym`$getenv`ROOT;"; + "{[root;i]"; + " t:([]time:10#.z.p; price:10?100f; side:10?`buy`sell);"; + " .Q.dd[.Q.dd[.Q.dd[.Q.dd[root;`2026.01.01];`trade];`$\"I\",string i];`] set .Q.ens[root;t;`sym];"; + " system\"sleep 0.05\";"; + " }[root] each 20+til 40;"; + "exit 0"); +system"ROOT=",(1_string root)," q ",cq," /dev/null 2>&1 &"; + +lat:(); +obs:(); +{[h;i] + t0:.z.p; + r:.[h;enlist"probe[]";{[e] `$"ERR:",e}]; + lat,::`long$(.z.p-t0)%1000; + obs,::enlist r; + } [h] each til 1500; + +bad:sum iserr each obs; +good:obs where not iserr each obs; +disagree:sum {not x[1]=x[2]} each good; +rows:{x 0} each good; +partc:{x 1} each good; + +-1 " queries ",string count obs; +-1 " rebuilds meanwhile ",string h"rebuilds"; +-1 " partitions ",(string min partc)," -> ",string max partc; +-1 " latency us median ",string med lat; +-1 " max ",string max lat; +check[0=bad; "no query errored while the table was being replaced underneath it"]; +check[0=disagree; + "no query ever saw the catalogue and the served table disagree (",string[count good]," answers)"]; +check[rows~asc rows; "row counts seen by the client only go up - no query read a half-built table"]; +check[max[partc]>min partc; "and the table really was growing while they ran (", + string[min partc]," -> ",string[max partc]," partitions)"]; +check[h"aligned[]"; "parts and opened are still row-aligned after ",(string h"rebuilds")," rebuilds"]; +/ assert on the MEDIAN, not the max. the tail here is the operating system - the worst single +/ query has been seen anywhere from 22ms to 1.2s on the same code, purely with machine load - +/ so asserting on it tests the box rather than the design +check[(med lat)<50000; + "median query latency stays inside a rebuild's cost (",string[`long$med lat]," us)"]; +-1 " the worst single query in this run was ",string[max lat]," us. that figure is the OS,"; +-1 " not the design, and it moves by an order of magnitude with load."; +-1 " -> q's main loop is the lock. rebuild is one message, a query is another, and they"; +-1 " cannot interleave. the cost of a rebuild is paid as QUEUING, which is why rebuild"; +-1 " being O(new directories) rather than O(history) matters to readers, not just to"; +-1 " the writer (8.2)."; +@[h;"exit 0";::]; +@[hclose;h;::]; + +/ --------------------------------------------------------------------------- +/ 4. the catalogue built, and the query still cannot be answered. +/ mkP does not validate what it is handed. both of these build cleanly and fail - or silently +/ mislead - only when someone asks. worth pinning down, because it says where to look when a +/ virtual table that "built fine" will not answer. +/ --------------------------------------------------------------------------- +-1 ""; +-1 " ILL-FORMED CATALOGUE - built without complaint, wrong at query time"; + +mkp:(use`kx.pq.t)`mkP; +b:hsym`$s,"/bits"; system"mkdir -p ",1_string b; +.Q.dd[.Q.dd[b;`x];`] set ([]a:til 3); +.Q.dd[.Q.dd[b;`y];`] set ([]a:3+til 3); +vx:get .Q.dd[.Q.dd[b;`x];`]; vy:get .Q.dd[.Q.dd[b;`y];`]; + +/ the same (date;instrument) twice - what two roots capturing one instrument would produce +dup:flip `date`instrument!(2#d; `AAPL`AAPL); +rd:try[{[mkp;k;v] mkp[k!v]}[mkp];(dup;(vx;vy))]; +check[not iserr rd; "duplicate (date;instrument) keys do not fail the build"]; +if[not iserr rd; + @[`.;`dupt;:;rd]; + check[6=count select from dupt; + "KNOWN LIMIT: both directories are served under one key - 6 rows, not 3. two stacks ", + "capturing the same instrument DOUBLE COUNT, silently. see 8.3"]]; + +/ partitions whose columns differ - the 9.3 state, reached by adding a column to one leaf +.Q.dd[.Q.dd[b;`y];`] set ([]a:3+til 3; extra:3#1); +vy2:get .Q.dd[.Q.dd[b;`y];`]; +mix:flip `date`instrument!(2#d; `AAPL`MSFT); +rm:try[{[mkp;k;v] mkp[k!v]}[mkp];(mix;(vx;vy2))]; +check[not iserr rm; "mismatched columns across partitions do not fail the build either"]; +if[not iserr rm; + @[`.;`mixt;:;rm]; + check[iserr try[{[] count select from mixt};()]; + "they fail at QUERY time - which is why 9.3 shows up as a broken reader, not a broken write"]; + check[iserr try[{[] count select a from mixt};()]; + " - and even selecting a column BOTH partitions have still errors"]]; + +-1 ""; +-1 " ",string[pass]," passed, ",string[fail]," failed"; +-1 ""; +system"rm -rf ",s; +exit $[fail>0;1;0] diff --git a/testfiles/vt-kdb-compare.q b/testfiles/vt-kdb-compare.q new file mode 100644 index 0000000..d5fa08c --- /dev/null +++ b/testfiles/vt-kdb-compare.q @@ -0,0 +1,136 @@ +/ VT-11: do identical queries return identical answers from the virtual table and from a +/ conventional date-partitioned kdb+ database holding the same data? + +.lg.o:{[t;m]}; .lg.w:{[t;m]}; .lg.e:{[t;m]}; +.servers.startupdepcycles:{[t;i;c] '"no wdb"}; +.servers.gethandlebytype:{[t;m] ()}; +.timer.enabled:0b; .timer.repeat:{[a;b;c;d;e]}; .proc.cp:{[] .z.P}; + +.vtidb.roots:enlist hsym`$getenv[`SCRATCH],"/vtdb"; +/ this script loads the reader directly, so it does NOT pick up appconfig/settings/idb.q - +/ set the exposed partition column explicitly to match the schema this database uses +.vtidb.partitioncol:`sym; +system"l ",getenv[`KDBAPPCODE],"/processes/vtidb.q"; +h:hopen `$"::",$[count getenv`PORT;getenv`PORT;"6099"]; + +pass:0; fail:0; ordr:0; + +/ canonical form, so row ORDER is not mistaken for a wrong answer: +/ unkey, rename the partition column to the schema's name, sort by every column +/ in-process, the virtual table hands back symbol columns as unresolved ENUMERATIONS (20h) +/ where a conventional partitioned select resolves them to symbols (11h). Values are equal +/ and over IPC both send plain symbols, so this is representation, not a wrong answer. +desym:{[t] + d:flip t; + e:where 20h=type each d; + $[count e; flip @[d;e;value]; t] + }; + +norm:{[t] + if[not type[t] in 98 99h; :t]; + if[99h=type t; t:0!t]; + / no rename needed any more: the reader now exposes the partition column under the + / schema's own name (§5.3), so both sides agree + / column order differs between the two layouts; that is not a wrong answer + t:desym t; + t:(asc cols t) xcols t; + $[count cols t; (cols t) xasc t; t] + }; + +cmp:{[lbl;vq;hq] / NOT "desc" - q keyword, gives 'nyi on application + a:@[{norm value x};vq;{(`ERROR;x)}]; + b:@[{norm h x};hq;{(`ERROR;x)}]; + exact:a~b; + / already normalised, so a mismatch here is a real difference + $[exact; + [pass+:1; -1 " PASS ",lbl]; + [fail+:1; -1 " FAIL ",lbl; + -1 " vt : ",-3!$[98h=type a;3#a;a]; + -1 " hdb: ",-3!$[98h=type b;3#b;b]]]; + }; + +-1 "comparing the virtual table against a conventional date-partitioned kdb+ database"; +-1 ""; + +cmp["total row count"; + "count select from trade"; + "count select from trade"]; + +cmp["count by date"; + "select n:count i by date from trade"; + "select n:count i by date from trade"]; + +cmp["filter on the partition column"; + "select from trade where sym=`AMD"; + "select from trade where sym=`AMD"]; + +cmp["filter on partition column + date"; + "select from trade where date=2026.01.01, sym=`AMD"; + "select from trade where date=2026.01.01, sym=`AMD"]; + +cmp["filter on a data column"; + "select from trade where side=`BUY"; + "select from trade where side=`BUY"]; + +cmp["combined partition and data filter"; + "select from trade where sym=`XAUUSD, side=`SELL"; + "select from trade where sym=`XAUUSD, side=`SELL"]; + +cmp["in on a list of instruments"; + "select from trade where sym in `AMD`MSFT`XAUUSD"; + "select from trade where sym in `AMD`MSFT`XAUUSD"]; + +cmp["sum aggregate"; + "select total:sum price from trade"; + "select total:sum price from trade"]; + +cmp["group by the partition column"; + "select n:count i, total:sum price by sym from trade"; + "select n:count i, total:sum price by sym from trade"]; + +cmp["group by date and a data column"; + "select n:count i by date, side from trade"; + "select n:count i by date, side from trade"]; + +cmp["min / max / avg"; + "select mn:min price, mx:max price, av:avg price from trade"; + "select mn:min price, mx:max price, av:avg price from trade"]; + +cmp["distinct on the partition column"; + "distinct select sym from trade"; + "distinct select sym from trade"]; + +cmp["select specific columns"; + "select time, side, price from trade where sym=`AAPL"; + "select time, side, price from trade where sym=`AAPL"]; + +cmp["time-range filter"; + "select from trade where time within (2026.01.01D00:00; 2026.01.01D23:59)"; + "select from trade where time within (2026.01.01D00:00; 2026.01.01D23:59)"]; + +cmp["dot notation on a temporal column"; + "select n:count i by time.hh from trade"; + "select n:count i by time.hh from trade"]; + +cmp["empty result - instrument that does not exist"; + "select from trade where sym=`ZZZZZZ"; + "select from trade where sym=`ZZZZZZ"]; + +cmp["weighted average"; + "select wa:size wavg price by sym from trade"; + "select wa:size wavg price by sym from trade"]; + +cmp["fby"; + "select from trade where price=(max;price) fby sym"; + "select from trade where price=(max;price) fby sym"]; + +cmp["count distinct"; + "select nd:count distinct sym from trade"; + "select nd:count distinct sym from trade"]; + +hclose h; +-1 ""; +-1 (40#"-"); +-1 " ",string[pass]," matched, ",string[fail]," differed"; +-1 (40#"-"); +exit $[fail>0;1;0]; diff --git a/testfiles/vt-kdb-prep.q b/testfiles/vt-kdb-prep.q new file mode 100644 index 0000000..5dd7e0a --- /dev/null +++ b/testfiles/vt-kdb-prep.q @@ -0,0 +1,66 @@ +/ VT-11 prep: build two databases holding IDENTICAL data. +/ 1. vtdb - the new format: date/table/instrument dirs, partition column stripped +/ 2. hdb - conventional kdb+: date-partitioned splay, sym as a real column +/ Both are derived from the same captured bytes, so any query difference is the access path. + +live:getenv`KDBDB; +scratch:getenv`SCRATCH; +vtdb:scratch,"/vtdb"; +hdb :scratch,"/hdb"; + +system"rm -rf ",vtdb," ",hdb; +system"mkdir -p ",vtdb," ",hdb; + +/ two dates, so cross-date queries are exercised. both are copies of the same captured day, +/ which is fine: both databases receive the identical duplication. +src:first asc key[hsym`$live] where key[hsym`$live] like "[0-9][0-9][0-9][0-9].*"; +system"cp ",live,"/sym ",vtdb,"/"; +system"cp -r ",live,"/",(string src)," ",vtdb,"/2026.01.01"; +system"cp -r ",live,"/",(string src)," ",vtdb,"/2026.01.02"; +-1 "vt-format database : ",vtdb; + +/ ---- build the conventional control from the same files ---- +load .Q.dd[hsym`$vtdb;`sym]; + +/ read one partition directory and put the partition columns BACK as real columns, +/ which is what a conventional layout would have stored in the first place +/ WARNING the instrument parameter must NOT be called "i" - inside q-sql, i is the row +/ index, so "sym:i" silently stores row numbers instead of the instrument +readpart:{[r;d;t;inst] + x:select from get .Q.dd[.Q.dd[.Q.dd[r;d];t];inst]; / select forces an in-memory copy + dd:"D"$string d; + update date:dd, sym:inst from x + }; + +vh:hsym`$vtdb; +dates:asc key[vh] where key[vh] like "[0-9][0-9][0-9][0-9].*"; + +gather:{[vh;t;dates] + raze raze {[vh;t;d] readpart[vh;d;t] each key .Q.dd[.Q.dd[vh;d];t]}[vh;t] each dates + }; + +tradeall:gather[vh;`trade;dates]; +quoteall:gather[vh;`quote;dates]; +-1 "control rows : trade ",(string count tradeall),", quote ",string count quoteall; + +/ ---- write it out as a conventional date-partitioned database ---- +/ Use ONE shared enumeration domain: copy the vt database's sym file, then enumerate the +/ rebuilt sym column against that same in-memory domain. Letting .Q.en manage it +/ produced column files whose indices did not match the sym file that was written, so +/ symbol columns resolved to the wrong labels. +hh:hsym`$hdb; +system"cp ",vtdb,"/sym ",hdb,"/"; + +tradeall:update sym:`sym$sym from tradeall; +quoteall:$[count quoteall; update sym:`sym$sym from quoteall; quoteall]; + +{[hh;t;data] + {[hh;t;d;data] + .Q.dd[.Q.dd[.Q.dd[hh;`$string d];t];`] set delete date from select from data where date=d; + }[hh;t;;data] each asc distinct data`date; + }[hh;;] . ' ((`trade;tradeall);(`quote;quoteall)); + +-1 "conventional hdb : ",hdb; +-1 "sym written : ",string count key hh; +-1 "hdb partitions : ",.Q.s1 key hh; +exit 0 diff --git a/testfiles/vt-limitations.q b/testfiles/vt-limitations.q new file mode 100644 index 0000000..5786f7e --- /dev/null +++ b/testfiles/vt-limitations.q @@ -0,0 +1,222 @@ +/ ============================================================================ +/ Virtual tables in kdb-x: what works, and what does not +/ ============================================================================ +/ Self-contained demonstration. Builds its own database, needs no TorQ. +/ . +/ run: +/ QHOME=~/.kx/q QLIC=~/.kx QPATH=~/.kx/mod: \ +/ ~/.kx/bin/q testfiles/vt-limitations.q +/ . +/ mp.q is Jonathon McMurray's multipart module (DataIntellect). It writes a +/ database partitioned on several dimensions and loads it back as a virtual +/ table. Everything below is about the virtual table layer underneath it, +/ kx.pq.t, which ships with the kdb-x parquet module. +/ . +/ NOTE a line containing only "/" opens a block comment in q - every comment +/ line here carries text after the slash. +/ ============================================================================ + +system"c 40 200"; +.mp:use`mp; +vt:use`kx.pq.t; + +hdr :{-1"";-1 (76#"=");-1 x;-1 (76#"=");}; +sub :{-1"";-1 "--- ",x," ",(70-count x)#"-";}; +note:{-1 " ",x;}; +try :{[d;f] -1""; -1 " ",d; r:@[f;::;{`$"THREW: ",x}]; $[-11h=type r;-1 " ",string r;show r]; }; +ok :{[d;f] v:@[{(1b;x[])};f;{(0b;x)}]; -1 (58$" ",d),$[v 0;"ok";"FAILS: ",v 1]; }; + +base:"/tmp/vtlimits"; +db:hsym`$base; +system"rm -rf ",base; +.mp.create[db;([]column:`date`sym; datatype:"DS")]; + +mk:{[d;s;n] ([]date:n#d; sym:n#s; time:n#(.z.p+til n); px:n?100f; sz:n?1000i)}; +.mp.addparts[db;`quote;.Q.en[db;raze mk[2026.08.03;;50] each `AMD`AAPL]]; +.mp.addparts[db;`trades;.Q.en[db;raze mk[2026.08.03;;50] each `AMD`AAPL]]; +.mp.loaddb[db]; + +/ ============================================================================ +hdr"PART 1 - what virtual tables do well"; +/ ============================================================================ + +sub"1.1 the partition columns are removed from the data on disk"; +note"on-disk layout is //
/"; +try["cols of one leaf splay on disk";{cols get hsym`$base,"/2026.08.03/AMD/quote"}]; +note"date and sym are NOT stored - they are implied by the directory path."; +note"that is what makes filtering on them free, and it is the single most"; +note"important design decision in the module."; + +sub"1.2 filters on partition columns prune whole directories"; +try["select from quote where sym=`AMD";{5 sublist select date,sym,px from quote where sym=`AMD}]; +note"only the AMD directory is read. AAPL is never opened."; + +sub"1.3 appends are visible with no reload, even across processes"; +note"a leaf opened as `get `:path/` (WITH a trailing slash) stays live."; +lp:hsym`$base,"/2026.08.03/AMD/quote/"; +note"rows before append : ",string count select from quote where sym=`AMD; +lp upsert .Q.en[db;select time,px,sz from mk[2026.08.03;`AMD;10]]; +note"rows after append : ",string count select from quote where sym=`AMD; +note"no reload was performed. this is what lets a writer and readers run"; +note"concurrently without coordination."; + +/ ============================================================================ +hdr"PART 2 - correctness risks"; +/ ============================================================================ +note"these produce WRONG ANSWERS rather than errors. none of them announce"; +note"themselves. this is the section that matters."; + +day:([]time:4#2026.08.03D12:00; sym:`AMD`AAPL`AMD`AAPL; px:10 20 30 40f); +l1 :([]time:2#2026.08.03D12:00; sym:2#`AMD; px:1 2f); +poison:42; / not a table: any query against it throws, proving it was read + +sub"2.1 a partition column that ALSO exists in the data stops pruning"; +note"mp avoids this by stripping the column. EXISTING kdb+ databases cannot -"; +note"a legacy date-partitioned hdb stores sym inside the table. so this is the"; +note"blocker for querying an existing estate through a virtual table."; +try["virtual col named `instrument, not in the leaf -> prunes (poison never read)"; + {vx:vt.mkP ([]date:2#2026.08.03; instrument:`AMD`AAPL)!(l1;poison); + select from vx where instrument=`AMD}]; +try["virtual col named `sym, which IS in the leaf -> does NOT prune"; + {vy:vt.mkP ([]date:2#2026.08.03; sym:`AMD`AAPL)!(l1;poison); + select from vy where sym=`AMD}]; +note"identical query, identical data. only the column NAME differs."; +note"when the name collides the constraint is pushed to every leaf instead of"; +note"being used to select leaves, so the whole database is scanned."; + +sub"2.2 mapping many partition keys onto one shared table: wrong both ways"; +note"this is how you would attach an existing date-partitioned hdb partition:"; +note"many (date,sym) keys all pointing at the same whole-day table."; +try["distinct name -> returns EVERY sym, all labelled AMD"; + {va:vt.mkP ([]date:2#2026.08.03; instrument:`AMD`AAPL)!(day;day); + select from va where instrument=`AMD}]; +try["shadowed name -> correct syms, but every row DUPLICATED per key row"; + {vb:vt.mkP ([]date:2#2026.08.03; sym:`AMD`AAPL)!(day;day); + select from vb where sym=`AMD}]; +try["no constraint at all -> 8 rows out of a 4-row table"; + {vb:vt.mkP ([]date:2#2026.08.03; sym:`AMD`AAPL)!(day;day); + select sym,px from vb}]; +note"with 10,000 instruments sharing a day partition that is a 10,000x row"; +note"amplification, silently."; +try["writing BOTH constraints by hand gives the right answer"; + {va:vt.mkP ([]date:2#2026.08.03; instrument:`AMD`AAPL)!(day;day); + select from va where instrument=`AMD, sym=`AMD}]; +note"so the engine can already compute it correctly - it just cannot be reached"; +note"from a single constraint. that is the fix we are asking KX for."; + +sub"2.3 nested partition values are silently scrambled"; +note"one key row holding a LIST of instruments - the compact way to describe"; +note"a legacy partition. mkP accepts it and then mis-attributes the rows."; +day2:([]sym:`AMD`AMD`AAPL`AAPL; px:10 20 30 40f); +try["leaf is AMD,AMD,AAPL,AAPL - compare 'instruments' against 'sym'"; + {k:([]date:2026.08.03 2026.08.04; instruments:(`AMD`AAPL;`MSFT`AUDUSD)); + vn:vt.mkP k!(day2;poison); + select date,instruments,sym,px from vn where date=2026.08.03}]; +note"rows 2 and 3 are attributed to the wrong instrument. the nested value is"; +note"cycled positionally across the rows rather than held constant. with a"; +note"3-element list against 4 rows it wraps around."; +note"the output is a well-formed table of plausible symbols. nothing indicates"; +note"it is wrong."; + +sub"2.4 new partitions are silently invisible until an explicit reload"; +note"rows before adding a new instrument : ",string count select from quote; +.mp.addparts[db;`quote;.Q.en[db;mk[2026.08.03;`MSFT;7]]]; +note"USDJPY written to the quote table on disk."; +note"rows now, no reload : ",string count select from quote; +try["select from quote where sym=`MSFT -> empty, not an error"; + {select from quote where sym=`MSFT}]; +note"appends are live (1.3) but NEW partitions are not. in a capture system a"; +note"new instrument arriving intraday is invisible until someone reloads."; + +sub"2.5 a partition present for one table but not another breaks the reload"; +note"USDJPY has quote but has not traded yet - a completely normal state for"; +note"any real feed. reloading now:"; +try[" .mp.loaddb[db]";{.mp.loaddb[db]; `reloaded}]; +note"the whole database fails to load, not just the missing table. every"; +note"partition value must exist for every table."; +note"standard kdb+ solves this with .Q.chk, which creates the empty table"; +note"directories. there is no equivalent here."; +note""; +note"creating the missing directory by hand and retrying:"; +(hsym`$base,"/2026.08.03/USDJPY/trades/") set .Q.en[db;select time,px,sz from 0#mk[2026.08.03;`MSFT;1]]; +try[" .mp.loaddb[db]";{.mp.loaddb[db]; `reloaded}]; +note"rows after reload : ",string count select from quote; + +/ ============================================================================ +hdr"PART 3 - ordinary q operations that do not work"; +/ ============================================================================ +note"virtual tables are not drop-in replacements for tables. a client library,"; +note"dashboard or analyst script written against normal kdb+ will hit these."; +-1""; + +ok["meta quote"; {meta quote}]; +ok["cols quote"; {cols quote}]; +ok["quote[`px] (index by column)";{quote`px}]; +ok["`px xasc quote (sort)"; {`px xasc quote}]; +ok["exec distinct sym from quote (virtual col)";{exec distinct sym from quote}]; +ok["update px2:2*px from quote"; {update px2:2*px from quote}]; +ok["delete px from quote"; {delete px from quote}]; +ok["select by time.minute (dot notation)";{select sum px by time.minute from quote}]; +ok["select by 5 xbar time.minute"; {select sum px by 5 xbar time.minute from quote}]; +-1""; +note"for contrast, these all work:"; +ok["count quote"; {count quote}]; +ok["select from quote where sym=`AMD"; {select from quote where sym=`AMD}]; +ok["select distinct sym from quote"; {select distinct sym from quote}]; +ok["exec px from quote (leaf col)"; {exec px from quote}]; +ok["select sum px by sym from quote"; {select sum px by sym from quote}]; +ok["aj[`sym`time;trades;quote]"; {aj[`sym`time;select from trades;select from quote]}]; +ok["lj"; {(select from trades) lj 2!select from quote}]; +ok["10 sublist quote"; {10 sublist quote}]; + +/ ============================================================================ +hdr"PART 4 - partition-pruning hints: present but unusable for symbols"; +/ ============================================================================ +note"the module supports per-partition min/max statistics, via hidden virtual"; +note"columns named 99min and 99max. they work - for numeric columns."; + +kn:flip (`date,`$("9px9min";"9px9max"))!(2#2026.08.03; 1 3f; 2 4f); +vnum:vt.mkP kn!(l1;poison); +try["numeric: where px<2.5 -> prunes, poison never read";{select from vnum where px<2.5}]; +try["numeric: where px=1.0 -> NOT pruned, = is unsupported";{select from vnum where px=1.0}]; + +ks:flip (`date,`$("9sym9min";"9sym9max"))!(2#2026.08.03; `AMD`AAPL; `AMD`AAPL); +vsym:vt.mkP ks!(l1;poison); +try["symbol : where sym>=`AMD, sym<=`AMD -> NOT pruned"; + {select from vsym where sym>=`AMD, sym<=`AMD}]; +note"the engine works out which side of the constraint is the column name by"; +note"asking which one is a symbol. for `sym>=`AMD` both sides are symbols,"; +note"so it cannot tell them apart and gives up."; +show ([]constraint:("px<2.5";"sym>=`AMD"); + bothsidessymbol:(0b;1b); + hintusable:(1b;0b)); +note"symbol columns are the commonest partition key in kdb+, so in practice the"; +note"hint mechanism is unavailable exactly where it is most wanted."; + +/ ============================================================================ +hdr"SUMMARY"; +/ ============================================================================ +-1""; +show ([] + area:("partition pruning";"live appends";"legacy data";"shared partitions"; + "nested keys";"new partitions";"q compatibility";"pruning hints"); + status:(`works;`works;`BLOCKED;`WRONG;`WRONG;`reload;`partial;`unusable); + detail:( + "free filtering on partition columns; big speedup vs splay"; + "readers see writers with no reload, cross-process"; + "a column in both the key and the data stops all pruning"; + "many keys -> one table gives mislabelled or duplicated rows"; + "nested partition values are cycled positionally, mis-attributing rows"; + "appends are live but new partitions need an explicit reload"; + "meta, cols, xasc, update, delete, dot notation all fail"; + "min/max hints exist but cannot be used on symbol columns")); + +-1""; +-1"The three marked BLOCKED/WRONG are correctness issues, not performance ones,"; +-1"and all three trace to a single behaviour: when a column name exists in both"; +-1"the virtual key and the underlying data, the constraint is applied in only"; +-1"one place instead of both. Fixing that resolves all three."; +-1""; +-1"Everything in PART 1 is available today and needs no changes."; + +exit 0 diff --git a/testfiles/vt-multistack-test.q b/testfiles/vt-multistack-test.q new file mode 100644 index 0000000..10928ec --- /dev/null +++ b/testfiles/vt-multistack-test.q @@ -0,0 +1,193 @@ +/ Can one reader serve two capture stacks? (§8.3, VT-16) +/ . +/ cd ~/TorQ-VT-Capture-Pack && . ./vt-env.sh && q testfiles/vt-multistack-test.q +/ . +/ .vtidb.roots is a list, so the scan and the virtual table already handle several trees. The +/ blocker was the enumeration domain: symbol columns are indices into a file at the root, the +/ reader loads it with `load`, and `load` binds a global named after the FILE. Two stacks that +/ both call theirs `sym` collide - one wins and the other's symbols resolve to the wrong values, +/ silently. +/ . +/ Three configurations, all of which a deployment could arrive at: +/ separate domains named `sym` and `symb` -> independent stacks, one reader +/ colliding both named `sym`, different content -> must be caught, not served +/ shared both named `sym`, same content -> stacks coordinated on one domain +/ . +/ NOTE a line containing only "/" opens a block comment in q, so every comment line here +/ carries text after the slash. + +/ enough of the TorQ framework for vtidb.q to load standalone +.lg.o:{[t;m]}; .lg.w:{[t;m]}; .lg.e:{[t;m] errs,:enlist m}; +.servers.startupdepcycles:{[t;i;c] '"no wdb in this test"}; +.servers.gethandlebytype:{[t;m] ()}; +.timer.enabled:0b; .timer.repeat:{[a;b;c;d;e]}; .proc.cp:{[] .z.P}; +errs:(); + +pass:0; fail:0; +check:{[ok;msg] $[ok; [pass+::1; -1 " PASS ",msg]; [fail+::1; -1 " FAIL ",msg]]; }; + +live:getenv`KDBDB; +scratch:"/tmp/vt-multistack-",string .z.i; +d:2026.01.01; +tab:`trade; +col:`side; / a symbol column held INSIDE the files + +src:first asc key[hsym`$live] where key[hsym`$live] like "[0-9][0-9][0-9][0-9].*"; +if[null src; -1 "no partitions under ",live," - start the stack first"; exit 77]; +load hsym`$live,"/sym"; + +/ read the live tree once and strip the enumeration: .Q.ens will not take a mapped table, +/ and each root has to be re-enumerated against its own domain anyway +deenum:{[t] flip {$[20h<=abs type x; value x; x]} each flip select from t}; +p:.Q.dd[.Q.dd[hsym`$live;src];tab]; +ls:{[deenum;p;i] (i; deenum get .Q.dd[.Q.dd[p;i];`])}[deenum;p] each key p; +if[3>count ls; -1 "need at least 3 instruments in ",string tab; exit 77]; +half:`long$0.5*count ls; + +/ split in two, because the shared-domain case has to put a symlink in place between the +/ mkdir and the first write - mkroot's rm -rf would otherwise delete it +mkdirs:{[scratch;nm] + system"rm -rf ",scratch,"/",string nm; system"mkdir -p ",scratch,"/",string nm; + hsym`$scratch,"/",string nm + }; +writeleaves:{[r;d;tab;dom;ls] + {[r;d;tab;dom;l] .Q.dd[.Q.dd[.Q.dd[.Q.dd[r;`$string d];tab];l 0];`] set .Q.ens[r;l 1;dom] + }[r;d;tab;dom] each ls; + r + }; +mkroot:{[mkdirs;writeleaves;scratch;d;tab;nm;dom;ls] + writeleaves[mkdirs[scratch;nm];d;tab;dom;ls] + }[mkdirs;writeleaves]; + +/ roots are config, not runtime state, so a change of roots is a restart in production. +/ here it means reloading the domains by hand - symchanged only watches file SIZE +useroots:{[rs] errs::(); .vtidb.roots:rs; .vtidb.loadsym[]; .vtidb.dropcache[]; .vtidb.rebuild[]; }; +q:{[s] value s}; + +system"rm -rf ",scratch; system"mkdir -p ",scratch; +a:mkroot[scratch;d;tab;`a;`sym ; half#ls]; +b:mkroot[scratch;d;tab;`b;`symb; (half _ ls)]; + +.vtidb.roots:enlist a; +system"l ",getenv[`KDBAPPCODE],"/processes/vtidb.q"; +pc:.vtidb.partitioncol; +.vtidb.current:d+1; + +-1 ""; +-1 " scratch ",scratch; +-1 " root a ",(.Q.s1 key a)," (",string[half]," instruments)"; +-1 " root b ",(.Q.s1 key b)," (",string[count[ls]-half]," instruments)"; +-1 ""; + +/ --------------------------------------------------------------------------- +-1 " SEPARATE DOMAINS - two independent stacks, one reader"; +useroots (a;b); +-1 " domains found ",.Q.s1 {last ` vs x} each .vtidb.symfiles[]; +-1 " partitions ",.Q.s1 count each .vtidb.parts; + +check[(`sym`symb)~asc {last ` vs x} each .vtidb.symfiles[]; + "both domains are discovered - the reader no longer assumes every root calls it `sym"]; +check[0=count errs; "no collision reported: the names differ, so nothing is overwritten"]; +check[count[ls]=count .vtidb.parts tab; + "every partition from both roots is attached (",string[count ls],")"]; + +r:q"select n:count i by ",string[pc]," from ",string[tab]," where date=",string d; +check[count[ls]=count r; "grouping by the PARTITION column is correct across both roots"]; +check[11h=abs type key[r]pc; " - because it comes from directory names, not from a domain"]; + +/ the expectation has to come from the leaves actually ATTACHED, not from all of ls: a value +/ that only occurs in the half that went to root b is not reachable from (a;c) +distinctin:{[col;ls] asc distinct raze {[col;l] distinct l[1] col}[col] each ls}[col]; +want:distinctin ls; +f:first want; +n1:first exec n from q"select n:count i from ",string[tab]," where date=",string[d], + ", ",string[col],"=`",string f; +n2:sum raze {[col;f;l] sum l[1][col]=f}[col;f] each ls; +check[n1=n2; "filtering on a cross-domain symbol column is correct (",string[n1]," rows)"]; + +byv:q"select n:count i by v:value ",string[col]," from ",string[tab]," where date=",string d; +check[want~asc exec v from byv; "grouping via `value` on that column is correct"]; + +/ the one thing that does NOT unify, asserted so a regression is visible +raw:q"select n:count i by ",string[col]," from ",string[tab]," where date=",string d; +check[count[raw]>count want; + "KNOWN LIMIT: grouping on the raw column splits per domain (", + string[count raw]," groups, not ",string[count want],")"]; + +/ --------------------------------------------------------------------------- +-1 ""; +-1 " COLLIDING DOMAINS - both roots call it `sym, with different contents"; +/ the second stack has to hold a symbol the first has never seen, which is what makes the +/ two domains diverge. that is the whole point: independently grown domains assign different +/ indices to the same symbols. give it one of its own +divergent:{[col;ls] @[ls;0;{[col;l] @[l;1;{[col;t] update side:`zzzonlyhere from t}[col]]}[col]]}[col; half _ ls]; +c:mkroot[scratch;d;tab;`c;`sym; divergent]; +useroots (a;c); +check[any errs like "*WRONG*"; "the unsafe configuration is reported, loudly"]; + +/ --------------------------------------------------------------------------- +/ the other supported configuration: ONE physical domain, symlinked into each root. this is +/ what "sharing a sym file" has to mean - two copies are the colliding case above, because +/ they diverge the moment either stack sees a symbol the other has not. +/ . +/ NOTE the link has to exist BEFORE the second stack writes. symlinking a domain over a tree +/ whose columns were already enumerated against a different one does not share anything - it +/ reinterprets existing indices against the wrong list, which is the collision case again. +/ so root e is built fresh, with the link in place first. +-1 ""; +-1 " SHARED DOMAIN - one file, symlinked into both roots"; +e:mkdirs[scratch;`e]; +system"ln -s ",(1_string .Q.dd[a;`sym])," ",1_string .Q.dd[e;`sym]; +writeleaves[e;d;tab;`sym; (half _ ls)]; + +check[00;1;0] diff --git a/testfiles/vt-newtable-test.q b/testfiles/vt-newtable-test.q new file mode 100644 index 0000000..65fcd86 --- /dev/null +++ b/testfiles/vt-newtable-test.q @@ -0,0 +1,126 @@ +/ Does a table that appears mid-life get picked up? (§5.3) +/ . +/ cd ~/TorQ-VT-Capture-Pack && . ./vt-env.sh && q testfiles/vt-newtable-test.q +/ . +/ The reader does not read the schema. It discovers its table list from the tree, by taking +/ `key` of a date directory - which is deliberate, so that adding a table to database.q needs +/ no change here. This checks that it works, and that the consequences are understood. +/ . +/ It looks at the LIVE partition, not at every date. Scanning all of history on every rebuild +/ cost 2.6 ms of a 4.2 ms rebuild at 250 dates and grew with retention for ever, to notice +/ something that happens once in a deployment's life. A new table appears where the writer is +/ writing, so that is where it is looked for. The cost is a real limitation and the last +/ section pins it down: a table added to a date that has already rolled needs a dropcache, +/ which is the same rule 6.1 already states for a directory added to a past date. +/ . +/ The interesting part is not discovery but COVERAGE. A table added part-way through a +/ database's life legitimately has fewer dates than the others, and §4.6's gap check compares +/ date coverage across tables to catch a partition written without one of its tables. A +/ genuinely new table therefore looks exactly like that failure, and is reported as one - +/ every rebuild, for as long as the older dates are attached. +/ . +/ NOTE a line containing only "/" opens a block comment in q, so every comment line here +/ carries text after the slash. + +/ enough of the TorQ framework for vtidb.q to load standalone +.lg.o:{[t;m]}; .lg.w:{[t;m] warns,:enlist m}; .lg.e:{[t;m] errs,:enlist m}; +.servers.startupdepcycles:{[t;i;c] '"no wdb in this test"}; +.servers.gethandlebytype:{[t;m] ()}; +.timer.enabled:0b; .timer.repeat:{[a;b;c;d;e]}; .proc.cp:{[] .z.P}; +warns:(); errs:(); + +pass:0; fail:0; +check:{[ok;msg] $[ok; [pass+::1; -1 " PASS ",msg]; [fail+::1; -1 " FAIL ",msg]]; }; + +live:getenv`KDBDB; +s:"/tmp/vt-newtable-",string .z.i; +d0:2026.01.01; d1:2026.01.02; + +src:first asc key[hsym`$live] where key[hsym`$live] like "[0-9][0-9][0-9][0-9].*"; +if[null src; -1 " no partitions under ",live," - start the stack first"; exit 77]; + +system"rm -rf ",s; system"mkdir -p ",s; +system"cp ",live,"/sym ",s,"/"; +{[live;src;s;d] system"cp -r ",live,"/",(string src)," ",s,"/",string d}[live;src;s] each d0,d1; + +.vtidb.roots:enlist hsym`$s; +.vtidb.partitioncol:`sym; +system"l ",getenv[`KDBAPPCODE],"/processes/vtidb.q"; +.vtidb.current:d1; / d1 is the partition the writer is filling +.vtidb.dropcache[]; .vtidb.rebuild[]; + +before:asc key .vtidb.parts; +-1 ""; +-1 " scratch ",s; +-1 " tables at start ",.Q.s1 before; +-1 " dates ",.Q.s1 asc distinct raze .vtidb.coverage[]; +-1 ""; + +/ --------------------------------------------------------------------------- +/ a new table appears, on the LATER date only - which is what "added mid-life" looks like +r:hsym`$s; +load .Q.dd[r;`sym]; +newtab:`signal; +insts:`AAPL`AMD`MSFT; +{[r;d1;newtab;i] + t:([]time:3#.z.p; strength:3?1f; flag:3#`live); + .Q.dd[.Q.dd[.Q.dd[.Q.dd[r;`$string d1];newtab];i];`] set .Q.en[r;t] + }[r;d1;newtab] each insts; +-1 " created ",(string d1),"/",(string newtab),"/ for ",.Q.s1 insts; + +warns:(); +.vtidb.rebuild[]; +after:asc key .vtidb.parts; + +-1 ""; +-1 " tables now ",.Q.s1 after; +-1 ""; + +check[newtab in after; "the new table is discovered from the tree, with no config change"]; +check[3=count .vtidb.parts newtab; "all three of its partitions are attached"]; +check[112h=type value newtab; "it is a virtual table like the others"]; + +n:.[{[t] count value "select from ",string t};enlist newtab;{`$"ERROR: ",x}]; +/ NOTE 3*3=n would parse as 3*(3=n). spell the expected number out. +check[9=n; "and it is queryable - ",(.Q.s1 n)," rows across 3 instruments"]; +check[(enlist d1)~asc distinct exec date from .vtidb.parts newtab; + "it covers only the date it appeared on, as it should"]; + +/ --------------------------------------------------------------------------- +-1 ""; +check[any warns like "*coverage gap*"; + "EXPECTED FALSE ALARM: 4.6's coverage check reports it as a gap"]; +{-1 " ",x} each warns where warns like "*coverage gap*"; +-1 ""; +-1 " That warning is correct by its own rules and wrong in intent. The check exists to catch"; +-1 " a partition written without one of its tables, and a table added mid-life is"; +-1 " indistinguishable from that on disk. It will repeat for as long as the older dates are"; +-1 " attached. Worth knowing before adding a table to a live database - the alternative,"; +-1 " suppressing it, would silence the failure it was built for."; +-1 ""; +/ --------------------------------------------------------------------------- +/ the limitation that buys the flat rebuild: a table appearing on a date that has already +/ rolled is not looked for, because the reader only scans the live partition +/ --------------------------------------------------------------------------- +-1 " A TABLE ADDED TO A DATE THAT HAS ALREADY ROLLED"; + +older:`archive; +{[r;d0;older;i] + t:([]time:3#.z.p; strength:3?1f; flag:3#`live); + .Q.dd[.Q.dd[.Q.dd[.Q.dd[r;`$string d0];older];i];`] set .Q.en[r;t] + }[r;d0;older] each insts; + +.vtidb.rebuild[]; .vtidb.rebuild[]; +check[not older in key .vtidb.parts; + "it is NOT discovered by a rebuild - the live partition is the only one scanned"]; +.vtidb.dropcache[]; +.vtidb.rebuild[]; +check[older in key .vtidb.parts; + "dropcache[] finds it, which is the recovery path 6.1 already prescribes for any change ", + "made to a date that has rolled"]; +-1 ""; + +-1 " ",string[pass]," passed, ",string[fail]," failed"; +-1 ""; +system"rm -rf ",s; +exit $[fail>0;1;0] diff --git a/testfiles/vt-partition-test.q b/testfiles/vt-partition-test.q new file mode 100644 index 0000000..fefd6fd --- /dev/null +++ b/testfiles/vt-partition-test.q @@ -0,0 +1,145 @@ +/ Does the writer delete the partition it is actually filling? (4.7, recovery) +/ . +/ cd ~/TorQ-VT-Capture-Pack && . ./vt-env.sh && q testfiles/vt-partition-test.q +/ . +/ A writer restart is destructive before it is restorative. TorQ deletes the current partition +/ and then rebuilds it by replaying the tickerplant log: +/ . +/ upd:.wdb.replayupd; +/ .wdb.clearwdbdata[]; / deletes savedir// +/ .wdb.startup[]; / subscribe -> replay the day's logs +/ . +/ Which makes getpartition[] load-bearing at exactly one moment: process start, before anything +/ has told the writer what date the tickerplant is on. TorQ seeds it from .proc.cd[], the +/ CALENDAR date. With a roll offset the tickerplant is on a different date - under this pack's +/ 17:00 roll they disagree from midnight until the roll - so the delete misses, the real +/ partition survives untouched, and the replay writes the whole day on top of it. Every row +/ already on disk is duplicated. Measured on the live stack: 442 duplicate rows from one +/ restart, all inside the replayed window. +/ . +/ fixpartition does correct currentpartition afterwards, from the tp log date. It is too late: +/ clearwdbdata has already run, and its corrective branch only fires when the WRONG directory +/ exists, to rename it. When the wrong date simply has no directory, nothing is cleaned. +/ . +/ NOTE a line containing only "/" opens a block comment in q, so every comment line here +/ carries text after the slash. + +pass:0; fail:0; +check:{[ok;msg] $[ok; [pass+::1; -1 " PASS ",msg]; [fail+::1; -1 " FAIL ",msg]]; }; + +/ enough of TorQ for the date logic to run standalone. the REAL eodtime.q is loaded rather +/ than a copy of its formula, so this tests what the writer actually runs - which means +/ supplying the two framework hooks timezone.q reaches for on the way in. +.proc.cd:{[] .z.d}; +.lg.o:{[t;m]}; .lg.w:{[t;m]}; .lg.e:{[t;m] -2 " ",m;}; +/ NOTE returns SYMBOLS - timezone.q does `string first ...`, and a string there would be +/ decomposed one character per element +.proc.getconfigfile:{[f] enlist `$getenv[`KDBCONFIG],"/",f}; +system"l ",getenv[`KDBCODE],"/common/timezone.q"; +system"l ",getenv[`KDBCODE],"/common/eodtime.q"; + +\d .wdb +partitiontype:`date; +/ the two definitions under test, lifted verbatim from appconfig/settings/wdb.q +startpartition:{[] + d:@[{[x] .eodtime.getday .z.p};(::);{[e] .proc.cd[]}]; + (`date^@[value;`.wdb.partitiontype;`date])$d + }; +getpartition:{[] @[value;`.wdb.currentpartition;{[e] .wdb.startpartition[]}]}; +/ what TorQ does today, for comparison +stockpartition:{[] @[value;`.wdb.currentpartition;(`date^partitiontype)$.proc.cd[]]}; +\d . + +/ set a roll offset and recompute the way a process would at startup +setroll:{[off] + .eodtime.rolltimeoffset:off; + .eodtime.d:.eodtime.getday .z.p; / the tickerplant's log date, as it computes it + }; + +-1 ""; +-1 " now ",(string .z.p)," UTC calendar date ",string .proc.cd[]; + +/ --------------------------------------------------------------------------- +/ 1. no roll configured - the default. nothing may change. +/ --------------------------------------------------------------------------- +-1 ""; +-1 " NO ROLL OFFSET - the stock configuration"; +setroll 0D00:00; +-1 " tickerplant logs to ",string .eodtime.d; +-1 " writer would seed ",string .wdb.startpartition[]; +check[.wdb.startpartition[]=.eodtime.d; + "the writer seeds the same date the tickerplant is logging to"]; +check[.wdb.startpartition[]=.eodtime.getday .z.p; " - which is the plain date, no adjustment"]; + +/ --------------------------------------------------------------------------- +/ 2. a roll offset, chosen so that right now falls in the pre-roll window whatever +/ time this test is run at. that is the window where the bug bites. +/ --------------------------------------------------------------------------- +-1 ""; +-1 " ROLL OFFSET IN FORCE - now is before today's roll"; +/ put the roll comfortably after the current time so the business date is still yesterday +/ time-of-day plus an hour, so the roll is always still ahead of us whatever time this runs +off:0D01:00+.z.p-"p"$"d"$.z.p; +setroll off; +-1 " roll offset ",string off; +-1 " tickerplant logs to ",string .eodtime.d; +-1 " calendar date ",string .proc.cd[]; +check[not .eodtime.d=.proc.cd[]; + "the two really do disagree - this is the state the pack runs in for most of the day"]; +check[.wdb.startpartition[]=.eodtime.d; + "the writer still seeds the tickerplant's date, not the calendar's"]; +check[not .wdb.stockpartition[]=.eodtime.d; + "REGRESSION GUARD: stock TorQ seeds ",(string .wdb.stockpartition[]), + " here, which is the wrong directory to delete"]; + +/ --------------------------------------------------------------------------- +/ 3. once the writer knows its partition, that answer wins - so the end-of-day +/ roll (currentpartition::pt+1) is not undone on the next flush. +/ --------------------------------------------------------------------------- +-1 ""; +-1 " AFTER THE WRITER HAS A PARTITION"; +.wdb.currentpartition:2030.01.01; +check[.wdb.getpartition[]=2030.01.01; + "getpartition returns the writer's own partition once it is set"]; +check[not .wdb.getpartition[]=.eodtime.d; " - and does not fall back to the date logic"]; +.wdb.currentpartition:2030.01.02; +check[.wdb.getpartition[]=2030.01.02; "an end-of-day roll therefore sticks"]; +![`.wdb;();0b;enlist`currentpartition]; + +/ --------------------------------------------------------------------------- +/ 4. it must not need .eodtime at all. +/ --------------------------------------------------------------------------- +-1 ""; +-1 " WITHOUT .eodtime LOADED"; +saved:.eodtime.getday; +![`.eodtime;();0b;enlist`getday]; +check[.wdb.startpartition[]=.proc.cd[]; + "falls back to the calendar date rather than failing to start"]; +.eodtime.getday:saved; +check[.wdb.startpartition[]=.eodtime.getday .z.p; "and recovers once it is back"]; + +/ --------------------------------------------------------------------------- +/ 5. the consequence, stated as the thing an operator would see: which directory +/ does a restart delete? +/ --------------------------------------------------------------------------- +-1 ""; +-1 " WHICH DIRECTORY A RESTART WOULD DELETE"; +setroll off; +s:"/tmp/vt-partition-",string .z.i; +system"rm -rf ",s; system"mkdir -p ",s,"/",string .eodtime.d; +.wdb.savedir:hsym`$s; +target:{[pt] .Q.par[.wdb.savedir;pt;`]}; +-1 " data is in ",1_string target .eodtime.d; +-1 " stock would delete ",1_string target .wdb.stockpartition[]; +-1 " this pack deletes ",1_string target .wdb.startpartition[]; +check[()~key target .wdb.stockpartition[]; + "stock targets a directory that does not exist - so it deletes NOTHING, and the replay ", + "then duplicates everything already written"]; +check[not ()~key target .wdb.startpartition[]; + "this pack targets the directory that actually holds the data, so the replay rebuilds it"]; +system"rm -rf ",s; + +-1 ""; +-1 " ",string[pass]," passed, ",string[fail]," failed"; +-1 ""; +exit $[fail>0;1;0] diff --git a/testfiles/vt-probe.q b/testfiles/vt-probe.q new file mode 100644 index 0000000..98deef8 --- /dev/null +++ b/testfiles/vt-probe.q @@ -0,0 +1,89 @@ +/ probe of kx.pq.t virtual-table semantics - evidence for docs/virtual-table-capture-pack.md +/ section 9. NOTE a line containing only "/" opens a multi-line comment block in q - every +/ comment line here must carry text after the slash. +/ run: +/ QHOME=~/.kx/q QLIC=~/.kx QPATH=~/.kx/mod ~/.kx/bin/q testfiles/vt-probe.q +/ method: leaf 2 is a "poison" value (an int, not a table). any attempt to query it +/ throws, so "THREW" proves the engine read that leaf; a clean result proves it pruned. +/ leaf 1 must be a real table - mkP introspects the first leaf for its column list. + +.pq.t:use`kx.pq.t; + +l1:([]time:2#2026.08.03D12:00; sym:2#`AMD; px:1 2f); +l2:([]time:2#2026.08.03D12:00; sym:2#`AAPL; px:3 4f); +day:([]time:4#2026.08.03D12:00; sym:`AMD`AAPL`AMD`AAPL; px:10 20 30 40f); +poison:42; + +try:{[d;f] -1 ""; -1 d; show @[f;::;{"THREW: ",x}]; }; + +-1 "=== 1. pruning on a virtual column that does NOT shadow a leaf column ==="; +vt1:.pq.t.mkP ([]date:2026.08.03 2026.08.04)!(l1;poison); +try["1a where date=2026.08.03 -> expect CLEAN (pruned)";{select from vt1 where date=2026.08.03}]; +vt2:.pq.t.mkP ([]date:2#2026.08.03; instrument:`AMD`AAPL)!(l1;poison); +try["1b where instrument=`AMD -> expect CLEAN (pruned)";{select from vt2 where instrument=`AMD}]; + +-1 ""; +-1 "=== 2. pruning on a virtual column that DOES shadow a leaf column ==="; +vt3:.pq.t.mkP ([]date:2#2026.08.03; sym:`AMD`AAPL)!(l1;poison); +try["2a where sym=`AMD -> expect THREW (no pruning, all leaves read)";{select from vt3 where sym=`AMD}]; + +-1 ""; +-1 "=== 3. correctness of the shadowed case, one leaf per instrument (greenfield) ==="; +vt4:.pq.t.mkP ([]date:2#2026.08.03; sym:`AMD`AAPL)!(l1;l2); +try["3a where sym=`AMD -> expect px 1 2 only, sym column twice";{select from vt4 where sym=`AMD}]; +try["3b select sum px by sym -> expect AMD 3, AAPL 7";{select sum px by sym from vt4}]; + +-1 ""; +-1 "=== 4. replicated links: many key rows -> ONE shared leaf (backward compat) ==="; +vt5:.pq.t.mkP ([]date:2#2026.08.03; instrument:`AMD`AAPL)!(day;day); +try["4a distinct name, where instrument=`AMD -> WRONG: returns AAPL rows too";{select from vt5 where instrument=`AMD}]; +vt6:.pq.t.mkP ([]date:2#2026.08.03; sym:`AMD`AAPL)!(day;day); +try["4b shadowed name, where sym=`AMD -> WRONG: each row duplicated per key row";{select from vt6 where sym=`AMD}]; + +-1 ""; +-1 "=== 5. min/max statistics columns, named 99min / 99max ==="; +k7:flip (`date,`$("9px9min";"9px9max"))!(2#2026.08.03; 1 3f; 2 4f); +vt7:.pq.t.mkP k7!(l1;poison); +try["5a where px<2.5 -> expect CLEAN (stats pruned leaf2)";{select from vt7 where px<2.5}]; +try["5b where px<=2.0 -> expect CLEAN";{select from vt7 where px<=2.0}]; +try["5c where px>2.5 -> expect THREW (leaf2 is the real match)";{select from vt7 where px>2.5}]; +try["5d where px=1.0 -> expect THREW (= not supported)";{select from vt7 where px=1.0}]; +try["5e where px within 1 2 -> expect THREW (within not supported)";{select from vt7 where px within 1 2}]; +try["5f no constraint -> expect THREW";{select from vt7}]; +-1 ""; +-1 "5g 9-prefixed columns are hidden from the result (see 5a output: no 9px9min column)"; + +-1 ""; +-1 "=== 6. statistics on a SYMBOL column ==="; +k8:flip (`date,`$("9sym9min";"9sym9max"))!(2#2026.08.03; `AMD`AAPL; `AMD`AAPL); +vt8:.pq.t.mkP k8!(l1;poison); +try["6a where sym>=`AMD, sym<=`AMD -> expect THREW (symbols unsupported)";{select from vt8 where sym>=`AMD, sym<=`AMD}]; +-1 ""; +-1 " why: qc identifies which argument is the column by testing which one is a symbol"; +-1 " atom, so a symbol VALUE is indistinguishable from a column name and it gives up:"; +show ([]case:`symbol`numeric; args:((`sym;`AMD);(`px;2.5)); istypesymbol:(-11h=type each (`sym;`AMD);-11h=type each (`px;2.5))); +-1 " the engine requires 10b (column identified); the symbol case gives 11b."; + + +-1 ""; +-1 "=== 7. null statistic means keep the partition ==="; +k9:flip (`date,`$("9px9min";"9px9max"))!(2#2026.08.03; 1 0n; 2 0n); +vt9:.pq.t.mkP k9!(l1;poison); +try["7a where px<2.5, leaf2 stats null -> expect THREW (null => keep => safe)";{select from vt9 where px<2.5}]; + +-1 ""; +-1 "=== 8. nested virtual key columns (backward-compat option 3) ==="; +k10:([]date:2026.08.03 2026.08.04; instruments:(`AMD`AAPL;`MSFT`AUDUSD)); +vt10:.pq.t.mkP k10!(day;poison); +try["8a can mkP be built with a nested key column? -> yes";{.pq.t.mkP k10!(day;poison); `built}]; +try["8b where instruments=`AMD -> THREW, = unsupported on nested";{select from vt10 where instruments=`AMD}]; +try["8c where any each instruments=`AMD -> prunes, but leaf unconstrained";{select from vt10 where any each instruments=`AMD}]; +-1 ""; +-1 " 8d the nested key value is RECYCLED POSITIONALLY across leaf rows."; +-1 " leaf below is AMD,AMD,AAPL,AAPL - watch the 'instruments' label:"; +day8:([]sym:`AMD`AMD`AAPL`AAPL; px:10 20 30 40f); +vt10b:.pq.t.mkP k10!(day8;poison); +show select date,instruments,sym,px from vt10b where date=2026.08.03; +-1 " rows 2 and 3 are mislabelled. with a 3-element list it wraps around."; + +exit 0 diff --git a/testfiles/vt-replay-test.q b/testfiles/vt-replay-test.q new file mode 100644 index 0000000..f56fe72 --- /dev/null +++ b/testfiles/vt-replay-test.q @@ -0,0 +1,133 @@ +/ Are the writer's overrides in force during tickerplant log replay? (§4.7) +/ . +/ cd ~/TorQ-VT-Capture-Pack && . ./vt-env.sh && q testfiles/vt-replay-test.q +/ . +/ Restarting the writer is the normal recovery path. TorQ handles it by DELETING the current +/ partition and rebuilding it from the tickerplant log, the log being the source of truth for +/ the day in flight. That replay runs inside .wdb.startup[], called at the bottom of wdb.q - +/ about a second before .proc.init[] runs the init list. +/ . +/ So an overlay installed only from the init list is NOT in force during the replay, and every +/ partition it rebuilds is written by the stock writer, which keeps the partition column in the +/ files (4.5). The database then holds a mix of 7-column and 8-column directories: the +/ mismatched-column state of 9.3, which gives silently wrong answers rather than an error. +/ . +/ This checks the invariant directly, from the writer's own log and from the tree, rather than +/ by restarting anything - a test that kills processes by name matches its own caller's command +/ line and is not worth the trouble. To exercise it for real, restart wdb1 by hand while the +/ tickerplant log has data in it, then run this. +/ . +/ Reference numbers from the run that found this, before the fix: the overrides installed at +/ log line 3573 and the replay ran at line 185. After the fix: 171 and 184. The first check +/ below is exactly that comparison. +/ . +/ NOTE a line containing only "/" opens a block comment in q, so every comment line here +/ carries text after the slash. + +pass:0; fail:0; +check:{[ok;msg] $[ok; [pass+::1; -1 " PASS ",msg]; [fail+::1; -1 " FAIL ",msg]]; }; + +db:getenv`KDBDB; +idbport:`$"::",string[30+"J"$getenv`KDBBASEPORT],":idb:pass"; + +h:@[hopen;idbport;{'"no reader on ",string[idbport],": ",x}]; +pc:h".vtidb.partitioncol"; + +/ --------------------------------------------------------------------------- +/ 1. the ordering, from the writer's own log. this is the invariant: whatever else changes, +/ the overrides have to be installed before the first replay line +/ --------------------------------------------------------------------------- +logf:first system "ls -t ",getenv[`KDBLOG],"/out_wdb1_*.log 2>/dev/null"; +if[not count logf; -1 " no wdb1 log found - is the stack up?"; exit 1]; +lines:read0 hsym`$logf; +inst:where lines like "*installing virtual-table capture overrides*"; +rep :where lines like "*replaying the log file(s)*"; + +-1 ""; +-1 " writer log ",logf; +-1 " overrides at ",.Q.s1 inst; +-1 " replay at ",.Q.s1 rep; +-1 ""; + +check[count inst; "the writer installs the capture overrides at all"]; +check[$[count rep; (first inst)0;1;0] diff --git a/testfiles/vt-restart-test.q b/testfiles/vt-restart-test.q new file mode 100644 index 0000000..136f12b --- /dev/null +++ b/testfiles/vt-restart-test.q @@ -0,0 +1,216 @@ +/ Can the reader start up in the middle of a write, and recover? (5.1, 5.3, 6.1) +/ . +/ cd ~/TorQ-VT-Capture-Pack && . ./vt-env.sh && q testfiles/vt-restart-test.q +/ . +/ A reader is restarted at whatever moment the operator restarts it, which is not a moment the +/ writer knows about. So it can arrive while a flush is half-done: a directory that has been +/ made but not filled, a partition whose columns are still being extended, a table directory +/ that exists for trade but not yet for quote. +/ . +/ None of that has to be handled perfectly - the writer will finish a moment later. What it +/ has to do is HEAL: whatever the reader could not read at startup, it must pick up on a later +/ sweep without anyone intervening. That property rests entirely on one variable. .vtidb.current +/ is the partition the writer is filling, and it decides which dates get rescanned (mutable) and +/ which are cached forever (immutable). Get it wrong and the reader stops looking at the very +/ date that is still growing - and says nothing, because from its point of view there is +/ nothing to report. +/ . +/ The last section is the one that matters. It covers the case where the reader cannot ask the +/ writer what partition it is on, which is exactly the case a restart hits: the reader comes up +/ first, or comes up alone. +/ . +/ NOTE a line containing only "/" opens a block comment in q, so every comment line here +/ carries text after the slash. + +.lg.o:{[t;m]}; .lg.w:{[t;m] warns,:enlist m}; .lg.e:{[t;m] errs,:enlist m}; +.servers.startupdepcycles:{[t;i;c] '"no wdb in this test"}; +.servers.gethandlebytype:{[t;m] ()}; +.timer.enabled:0b; .timer.repeat:{[a;b;c;d;e]}; .proc.cp:{[] .z.P}; +warns:(); errs:(); +/ warns is emptied between sections, and () like "..." is a type error rather than a false. +/ NOTE `like` takes ONE string, so a list of them has to be walked - x like p over a +/ two-element list of strings is a type error, over a one-element list it is not, which is +/ exactly the shape of bug that passes until a second warning shows up +saw:{[pat] $[count warns; any {[p;w] w like p}[pat] each warns; 0b]}; + +pass:0; fail:0; +check:{[ok;msg] $[ok; [pass+::1; -1 " PASS ",msg]; [fail+::1; -1 " FAIL ",msg]]; }; +try:{[f;a] .[f;a;{[e] `$"ERR:",e}]}; + +s:"/tmp/vt-restart-",string .z.i; +system"rm -rf ",s; system"mkdir -p ",s; +root:hsym`$s; +(.Q.dd[root;`sym]) set 0#`; + +live:2026.08.18; / the date the writer is filling +past:2026.08.17; / a date that has already rolled + +tdir:{[root;d;t;i] .Q.dd[.Q.dd[.Q.dd[root;`$string d];t];`$"I",string i]}[root]; +mk:{[root;tdir;d;t;i;n] + x:([]time:n#.z.p; price:n?100f; side:n?`buy`sell); + (.Q.dd[tdir[d;t;i];`]) set .Q.ens[root;x;`sym]; + }[root;tdir]; + +mk[past;`trade;] ./: (0 5;1 5;2 5); +mk[live;`trade;] ./: (0 5;1 5;2 5); +mk[past;`quote;] ./: (0 5;1 5;2 5); +mk[live;`quote;] ./: (0 5;1 5;2 5); + +.vtidb.roots:enlist root; +.vtidb.partitioncol:`instrument; +system"l ",getenv[`KDBAPPCODE],"/processes/vtidb.q"; + +/ a reader restart, without re-loading the file: forget everything learned and rescan +restart:{[cur] warns::(); errs::(); .vtidb.dropcache[]; .vtidb.current:cur; .vtidb.rebuild[]; }; + +-1 ""; +-1 " scratch ",s; + +/ --------------------------------------------------------------------------- +/ 1. a directory that exists but has nothing in it yet. +/ mkdir happens before the columns are written, so this is the state a reader sees if it scans +/ between the two. it must not take the process down, and it must not poison the other +/ partitions of the same table. +/ --------------------------------------------------------------------------- +-1 ""; +-1 " STARTED MID-FLUSH - a directory made, not yet filled"; + +system"mkdir -p ",1_string tdir[live;`trade;7]; +restart live; +n:.vtidb.parts`trade; +check[0=count errs; "an empty partition directory does not error the reader"]; +check[saw "*unreadable*"; "it is reported as unreadable, not passed over in silence"]; +check[6=count n; "the other 6 partitions attach normally - one bad directory is not contagious"]; +check[30=try[{[] count select from trade};enlist(::)]; "and every row of them is served"]; + +/ now the writer finishes that directory. no restart, no dropcache - just the next sweep +mk[live;`trade;7;5]; +.vtidb.rebuild[]; +check[7=count .vtidb.parts`trade; "the next rebuild picks it up - the reader heals by itself"]; +check[35=try[{[] count select from trade};enlist(::)]; "with its rows"]; + +/ --------------------------------------------------------------------------- +/ 2. a partition caught mid-append. +/ the columns are written one at a time, so a reader that starts here sees a directory whose +/ column files disagree in length. vt-inflight-test.q establishes that this reads as a short +/ but consistent prefix; what matters on the restart path is that the reader does not CACHE +/ that short answer. +/ --------------------------------------------------------------------------- +-1 ""; +-1 " STARTED MID-APPEND - a partition whose columns are different lengths"; + +pfile:.Q.dd[tdir[live;`trade;0];`price]; +full:hcount pfile; +system"truncate -s ",string[8*2]," ",1_string pfile; / price now holds 2 of 5 rows +restart live; +short:try[{[] count select from trade};enlist(::)]; +check[35>short; "the ragged partition is served short (",string[short]," of 35 rows)"]; +check[0=count errs; "and still without an error - this is the quiet one"]; + +system"truncate -s ",string[full]," ",1_string pfile; / the writer finishes the column +check[35=try[{[] count select from trade};enlist(::)]; + "when the column catches up the rows appear with NO rebuild at all - a live view has no ", + "length cached in it (5.2)"]; + +/ --------------------------------------------------------------------------- +/ 3. one table flushed, the other not. +/ savetables walks the tables in order, so between them a partition has trade and no quote. +/ 4.6 established that this is served as a silently absent date; on the restart path the +/ requirement is that it is at least SAID. +/ --------------------------------------------------------------------------- +-1 ""; +-1 " STARTED BETWEEN TABLES - trade written for a date, quote not yet"; + +mk[2026.08.19;`trade;] ./: (0 5;1 5); +restart live; +check[saw "*coverage gap*"; + "a date with trade but no quote is reported as a coverage gap (4.6)"]; +check[2026.08.19 in exec date from .vtidb.parts`trade; "trade serves the new date"]; +check[not 2026.08.19 in exec date from .vtidb.parts`quote; "quote does not - correctly, it has no data"]; + +mk[2026.08.19;`quote;] ./: (0 5;1 5); +warns:(); / NOT warns:: - at top level that defines a view +.vtidb.rebuild[]; +check[2026.08.19 in exec date from .vtidb.parts`quote; "and it heals when the writer catches up"]; +check[not saw "*coverage gap*"; "the gap stops being reported once it closes"]; + +/ --------------------------------------------------------------------------- +/ 4. the reader cannot ask the writer which partition is live. +/ . +/ THIS IS THE ONE. On restart the reader calls findwdb, and it can come back empty - the +/ writer is down, or the reader came up first, or the read of .wdb.currentpartition failed. +/ The reader then has to decide for itself which date is live, and that decision is not +/ cosmetic: an immutable date is cached and never looked at again. +/ . +/ .z.D is the wrong answer, and wrong in the normal case rather than an exotic one. This pack +/ rolls at 17:00 local (0D09:00 in GMT, see appconfig/settings/default.q), so from midnight +/ until the roll the writer is still filling YESTERDAY while .z.D already says today. A reader +/ that guesses .z.D marks the live partition immutable, caches it, and every instrument that +/ starts trading after that point is invisible - present on disk, absent from every query, with +/ nothing in the log. +/ . +/ The date on disk is knowable without asking anyone: it is the latest one there. +/ --------------------------------------------------------------------------- +-1 ""; +-1 " NO WRITER TO ASK - which date does the reader think is live?"; + +/ a second root, dated relative to today, so this reproduces the pre-roll window on any day: +/ the writer is filling YESTERDAY while .z.D already says today +r2:hsym`$s,"/preroll"; +system"mkdir -p ",1_string r2; +(.Q.dd[r2;`sym]) set 0#`; +mk2:{[r2;d;i] + x:([]time:5#.z.p; price:5?100f; side:5?`buy`sell); + (.Q.dd[.Q.dd[.Q.dd[.Q.dd[r2;`$string d];`trade];`$"I",string i];`]) set .Q.ens[r2;x;`sym]; + }[r2]; +writerpart:.z.D-1; / the partition the writer is on before the 17:00 roll +mk2[.z.D-2;] each til 2; +mk2[writerpart;] each til 2; + +/ drive the REAL startup path rather than setting state by hand - init is where the live +/ partition is decided, and findwdb below it will come back empty in this mock +.vtidb.roots:enlist r2; +.vtidb.dropcache[]; +.vtidb.current:0Nd; +warns:(); +.vtidb.init[]; + +-1 " .z.D ",string .z.D; +-1 " writer is filling ",string writerpart; +-1 " reader believes live is ",string .vtidb.current; +check[.vtidb.current=writerpart; + "started with no writer, the reader takes the live partition from DISK, not from .z.D"]; +check[not .vtidb.current=.z.D; + " - which in the pre-roll window is a different date, and the one that matters"]; + +/ the consequence, which is what an operator would actually notice - or rather, would not +was:count .vtidb.parts`trade; +mk2[writerpart;99]; +.vtidb.rebuild[]; .vtidb.rebuild[]; +check[was ", + string[count .vtidb.parts`trade],"). believing that date had rolled would CACHE it, and ", + "the instrument would sit on disk absent from every query with nothing logged"]; + +/ and it has to keep following the disk, or the same freeze returns tomorrow +mk2[.z.D;0]; +.vtidb.rebuild[]; +check[.vtidb.current=.z.D; "the live partition follows the disk forward with no rollover call"]; +was2:count .vtidb.parts`trade; +mk2[.z.D;1]; +.vtidb.rebuild[]; +check[was20;1;0] diff --git a/testfiles/vt-rollover-test.q b/testfiles/vt-rollover-test.q new file mode 100644 index 0000000..59612d5 --- /dev/null +++ b/testfiles/vt-rollover-test.q @@ -0,0 +1,102 @@ +/ Does end of day still see everything, now that it no longer rescans history? (VT-17) +/ . +/ cd ~/TorQ-VT-Capture-Pack && . ./vt-env.sh && q testfiles/vt-rollover-test.q +/ . +/ rollover used to call dropcache[], so correctness at end of day was free - every date was +/ rescanned. It now forgets only the date that just closed. That is safe only if the drop +/ happens BEFORE current moves forward: once current is the new date, the closing date reads +/ as immutable and build would reuse its catalogue as-is. +/ . +/ This builds a scratch tree, creates a directory the reader has not scanned, rolls over, and +/ checks the rows are there. Written to FAIL against the naive version of this change - simply +/ deleting the dropcache[] call - which loses that last directory silently and for good. +/ . +/ NOTE a line containing only "/" opens a block comment in q, so every comment line here +/ carries text after the slash. + +/ enough of the TorQ framework for vtidb.q to load standalone +.lg.o:{[t;m]}; .lg.w:{[t;m]}; .lg.e:{[t;m]}; +.servers.startupdepcycles:{[t;i;c] '"no wdb in this test"}; +.servers.gethandlebytype:{[t;m] ()}; +.timer.enabled:0b; .timer.repeat:{[a;b;c;d;e]}; .proc.cp:{[] .z.P}; + +pass:0; fail:0; +check:{[ok;msg] $[ok; [pass+::1; -1 " PASS ",msg]; [fail+::1; -1 " FAIL ",msg]]; }; + +live:getenv`KDBDB; +scratch:"/tmp/vt-rollover-",string .z.i; +d0:2026.01.01; d1:2026.01.02; d2:2026.01.03; + +src:first asc key[hsym`$live] where key[hsym`$live] like "[0-9][0-9][0-9][0-9].*"; +if[null src; -1 "no partitions under ",live," - start the stack first"; exit 77]; + +system"rm -rf ",scratch; system"mkdir -p ",scratch; +system"cp ",live,"/sym ",scratch,"/"; +{[live;src;scratch;d] system"cp -r ",live,"/",(string src)," ",scratch,"/",string d}[live;src;scratch] each d0,d1; + +.vtidb.roots:enlist hsym`$scratch; +system"l ",getenv[`KDBAPPCODE],"/processes/vtidb.q"; +pc:.vtidb.partitioncol; + +/ d1 is the live partition, d0 is closed history +.vtidb.current:d1; +.vtidb.dropcache[]; .vtidb.rebuild[]; + +-1 ""; +-1 " scratch ",scratch; +-1 " dates ",.Q.s1 asc distinct raze .vtidb.coverage[]; +-1 " live ",string .vtidb.current; +-1 ""; + +tabs:key .vtidb.parts; +newi:`ZZLATE; + +/ the writer's last flush of the day: a brand new instrument directory on the live date, with +/ no notification, so the reader's catalogue does not know about it yet. +/ the donor must come from the SAME table - a virtual table cannot span two column layouts, +/ and seeding one table's directory from another's gives a value error on the missing column +{[scratch;d1;newi;t] + donor:first exec path from .vtidb.parts[t] where date=d1; + system"cp -r ",(1_string donor)," ",scratch,"/",(string d1),"/",(string t),"/",string newi + }[scratch;d1;newi] each tabs; +-1 " created ",(string d1),"/*/",(string newi)," on disk, reader NOT told"; + +/ two different questions, and conflating them is a trap here: a table can be attached and +/ still have no rows (a table can be idle in a short run). check the catalogue for every +/ table, and the rows only for one that actually holds some. +attached:{[pc;newi;t] newi in .vtidb.parts[t] pc}; +rows:{[pc;newi;t] 00;1;0] diff --git a/testfiles/vt-sample-legacy.q b/testfiles/vt-sample-legacy.q new file mode 100644 index 0000000..c655266 --- /dev/null +++ b/testfiles/vt-sample-legacy.q @@ -0,0 +1,46 @@ +/ Simple sample for testing the legacy-data claims. All in memory, no disk. +/ run: \l testfiles/vt-sample-legacy.q + +vt:use`kx.pq.t; + +/ two days of a legacy hdb: one table per date, sym stored INSIDE the table +day1:([]sym:`AMD`AAPL`AMD; price:1.08 1.26 1.09; size:100 200 300); +day2:([]sym:`AMD`AAPL; price:1.10 1.25; size:400 500); + +/ the truth: 5 rows total, 3 AMD, 2 AAPL +truth:day1,day2; + +/ option 0 - key on date only (recommended) +v0:vt.mkP ([]date:2020.01.01 2020.01.02)!(day1;day2); + +/ option 1 - replicated links, one key row per (date;sym) +v1:vt.mkP ([]date:2020.01.01 2020.01.01 2020.01.02 2020.01.02; + sym :`AMD`AAPL`AMD`AAPL)!(day1;day1;day2;day2); + +/ option 2 - null / wildcard sym +v2:vt.mkP ([]date:2020.01.01 2020.01.02; sym:2#`)!(day1;day2); + +/ option 3 - nested list of syms +v3:vt.mkP ([]date:2020.01.01 2020.01.02; + sym :(`AMD`AAPL;`AMD`AAPL))!(day1;day2); + +/ option 3 but with deliberate NONSENSE in the key - proves the key is ignored +bad:vt.mkP ([]date:2020.01.01 2020.01.02; + sym :(`AAA`BBB;`CCC`DDD))!(day1;day2); + +-1 "truth: ",string[count truth]," rows, AMD=", + string[count select from truth where sym=`AMD]; +-1 ""; +-1 "count select from ... where sym=`AMD (should be 3):"; +-1 " v0 (date only) : ",string count select from v0 where sym=`AMD; +-1 " v1 (replicated) : ",string count select from v1 where sym=`AMD; +-1 " v2 (null key) : ",string count select from v2 where sym=`AMD; +-1 " v3 (nested key) : ",string count select from v3 where sym=`AMD; +-1 " bad (nonsense key) : ",string count select from bad where sym=`AMD; +-1 ""; +-1 "select sum size by sym from truth:"; show select sum size by sym from truth; +-1 "... from v1 (duplicated):"; show select sum size by sym from v1; +-1 "... from v3:"; show select sum size by sym from v3; +-1 ""; +-1 "v1 in full - the duplication:"; +show select date,sym,price,size from v1 where sym=`AMD; diff --git a/testfiles/vt-scale-test.q b/testfiles/vt-scale-test.q new file mode 100644 index 0000000..9789919 --- /dev/null +++ b/testfiles/vt-scale-test.q @@ -0,0 +1,105 @@ +/ How does the reader scale with partition count? +/ . +/ This is the evidence behind §8.2 and step 6 of §10. It builds synthetic trees of increasing +/ size, attaches the real vtidb.q to each, and measures what actually grows. +/ . +/ Run it as: cd ~/TorQ-VT-Capture-Pack && . ./vt-env.sh && q testfiles/vt-scale-test.q +/ Takes a couple of minutes and about 2 GB of scratch space in /tmp. +/ . +/ NOTE a line containing only "/" opens a block comment in q, so every comment line here +/ carries text after the slash. + +.lg.o:{[t;m]}; .lg.w:{[t;m]}; .lg.e:{[t;m]}; +.servers.startupdepcycles:{[t;i;c] '"no wdb"}; +.servers.gethandlebytype:{[t;m] ()}; +.timer.enabled:0b; .timer.repeat:{[a;b;c;d;e]}; .proc.cp:{[] .z.P}; + +pid:string .z.i; +maps:{[] "J"$first" "vs first system"wc -l /proc/",pid,"/maps" }; +fds: {[] "J"$first" "vs first system"ls /proc/",pid,"/fd | wc -l" }; +rss: {[] "J"$first" "vs first system"awk '/VmRSS/{print $2}' /proc/",pid,"/status" }; + +root:"/tmp/vt-scale-",pid; + +/ one template partition per table, matching the real schema with the partition column +/ stripped: trade 7 columns, quote 8 +mktemplate:{[r] + o:([]time:10#.z.p; price:10#100f; size:10#10i; stop:10#0b; + cond:10#" "; ex:10#"N"; side:10#`buy); + a:([]time:10#.z.p; bid:10#99f; ask:10#101f; bsize:10#10j; asize:10#10j; + mode:10#" "; ex:10#"N"; src:10#`BARX); + h:hsym`$r; + .Q.dd[.Q.dd[h;`template_trade];`] set .Q.en[h;o]; + .Q.dd[.Q.dd[h;`template_quote];`] set .Q.en[h;a]; + }; + +/ build the first date instrument by instrument, then clone the whole date - far faster +mktree:{[r;dates;insts] + system"rm -rf ",r; system"mkdir -p ",r; + mktemplate r; + d0:"2026.01.01"; + {[r;d0;t;insts] + system"mkdir -p ",r,"/",d0,"/",string t; + {[r;d0;t;i] system"cp -r ",r,"/template_",(string t)," ",r,"/",d0,"/",(string t),"/I",-4$"000",string i}[r;d0;t] each til insts; + }[r;d0;;insts] each `trade`quote; + {[r;d0;n] system"cp -r ",r,"/",d0," ",r,"/",string 2026.01.01+n}[r;d0] each 1+til dates-1; + system"rm -rf ",r,"/template_trade ",r,"/template_quote"; + }; + +measure:{[r;dates;insts] + mktree[r;dates;insts]; + m0:maps[]; f0:fds[]; r0:rss[]; + .vtidb.roots:enlist hsym`$r; + + / COLD: no cache, so every date is scanned and every directory opened + .vtidb.dropcache[]; + t0:.z.p; .vtidb.rebuild[]; cold:`long$(.z.p-t0)%1000000; + np:sum count each .vtidb.parts; + + / selective query, run twice so the reported figure is warm + q1:{count select from trade where date=2026.01.01, instrument=`I0000}; + q1[]; t0:.z.p; q1[]; sel:`long$(.z.p-t0)%1000; + + / LIVE: the production case - one date is the live partition and gets rescanned, + / every earlier date is immutable and reused + .vtidb.current:2026.01.01+dates-1; + .vtidb.dropcache[]; .vtidb.rebuild[]; + t0:.z.p; .vtidb.rebuild[]; live:`long$(.z.p-t0)%1000000; + + / ROLLOVER: end of day. used to drop the whole cache, which made this a full rescan. + / it now forgets only the date that just closed, so it should track live, not cold (VT-17) + t0:.z.p; .vtidb.rollover[2026.01.01+dates]; roll:`long$(.z.p-t0)%1000000; + + / NOTE maps/fds as a delta, RSS as an absolute: this script reuses one process across + / sizes, so a per-size RSS delta understates. For bytes-per-directory, run a fresh + / process per size - measured that way it is a consistent 891 B/dir. + -1 " ",(-9$string np),(-8$string maps[]-m0),(-6$string fds[]-f0), + (-10$string rss[]),(-9$string cold),(-9$string live),(-9$string roll),-9$string sel; + }; + +/ the reader has to be loaded once, with a root that exists +mktree[root;1;1]; +.vtidb.roots:enlist hsym`$root; +/ this script builds its own synthetic tree, so pin the exposed name to what its queries use +/ rather than inheriting the deployment's setting +.vtidb.partitioncol:`instrument; +system"l ",getenv[`KDBAPPCODE],"/processes/vtidb.q"; + +-1 ""; +-1 " ",(-9$"dirs"),(-8$"maps+"),(-6$"fds+"),(-10$"rss kB"), + (-9$"cold ms"),(-9$"live ms"),(-9$"roll ms"),-9$"sel us"; +-1 " ",(-9$"")," ",65#"-"; +measure[root;10;20]; +measure[root;25;50]; +measure[root;50;100]; +measure[root;100;100]; +measure[root;200;100]; + +system"rm -rf ",root; +-1 ""; +-1 " cold = full rescan of every date (startup, or a manual dropcache)."; +-1 " live = the production case: rescan only the live partition, reuse immutable history."; +-1 " roll = end of day. it rescans the one date that just closed and keeps everything older,"; +-1 " so it tracks live rather than cold, and does not grow with retention (VT-17)."; +-1 " mappings and file descriptors do not grow: a trailing-slash open does not mmap."; +exit 0 diff --git a/testfiles/vt-sym-concurrency.q b/testfiles/vt-sym-concurrency.q new file mode 100644 index 0000000..0d42004 --- /dev/null +++ b/testfiles/vt-sym-concurrency.q @@ -0,0 +1,79 @@ +/ Is one enumeration domain safe for several concurrent writers? (§8.3.1, VT-16) +/ . +/ cd ~/TorQ-VT-Capture-Pack && . ./vt-env.sh && q testfiles/vt-sym-concurrency.q +/ . +/ Sharing one domain across capture stacks was rejected in an earlier draft partly on the +/ grounds that two writers appending to one sym file risk corrupting it. That deserved +/ measuring rather than asserting, because it is the difference between "a deployment choice" +/ and "a thing you must not do". +/ . +/ The enumeration primitive is `path?syms` on a FILE HANDLE, which is what .Q.en calls. This +/ runs several processes through it against one file, on an overlapping vocabulary, and checks +/ the two things that would break a database: a domain that gained duplicates, and an index +/ handed to a writer that no longer resolves to the symbol it was given for - which would mean +/ column files already on disk are now wrong. +/ . +/ NOTE a line containing only "/" opens a block comment in q, so every comment line here +/ carries text after the slash. + +writers:6; +rounds:300; + +d:"/tmp/vt-symconc-",string .z.i; +system"rm -rf ",d; system"mkdir -p ",d; +sp:d,"/sym"; +(hsym`$sp) set 0#`; + +/ the child, written out rather than shipped separately so this stays one file +child:d,"/child.q"; +(hsym`$child) 0: ( + "p:hsym`$getenv`SYMPATH;"; + "id:\"J\"$getenv`WID;"; + "rounds:\"J\"$getenv`ROUNDS;"; + "system \"S \",string 1000+id; / every q process starts on the same seed"; + "vocab:`$\"S\",/:string til 400; / shared with every other writer"; + "mine:`$\"W\",string[id],\"_\",/:string til 100;"; + "seen:()!();"; + "{[p;vocab;mine;i]"; + " s:(neg[40]?vocab),neg[10]?mine;"; + " seen[s]::p?s; / the enumeration primitive .Q.en uses"; + " }[p;vocab;mine] each til rounds;"; + "(hsym`$getenv[`OUTPATH]) set seen;"; + "exit 0"); + +-1 ""; +-1 " ",string[writers]," concurrent writers x ",string[rounds]," enumeration calls, one shared domain file"; + +{[sp;d;rounds;child;i] + system"SYMPATH=",sp," WID=",string[i]," ROUNDS=",string[rounds], + " OUTPATH=",d,"/out",string[i]," q ",child," ",d,"/log",string[i]," 2>&1 &" + }[sp;d;rounds;child] each til writers; + +/ wait for every child to land its result +wait:{[d;n] $[n=count key[hsym`$d] where key[hsym`$d] like "out*"; ::; [system"sleep 0.3"; .z.s[d;n]]]}; +wait[d;writers]; +system"sleep 0.5"; + +final:get hsym`$sp; +outs:{[d;i] get hsym`$d,"/out",string i}[d] each til writers; +pairs:(!/)(raze key each outs; raze value each outs); +dups:count[final]-count distinct final; +bad:where not final[value pairs]=key pairs; + +-1 ""; +-1 " final domain size ",string count final; +-1 " duplicate entries ",string dups; +-1 " indices handed out ",string count pairs; +-1 " now resolving WRONG ",string count bad; +if[count bad; -1 " e.g. ",.Q.s1 3#key[pairs] bad]; +-1 ""; +-1 $[(0=dups) and 0=count bad; + " SAFE - the primitive locks. every index still resolves to the symbol it was issued for."; + " UNSAFE - the file lost or reordered entries under concurrency."]; +-1 ""; +-1 " what this does NOT say: two stacks can only share a domain if they share one PHYSICAL"; +-1 " file (shared storage, symlinked into each root). two copies diverge on the first new"; +-1 " symbol either stack sees, and that is the silently-wrong configuration. see 8.3.1."; +-1 ""; +system"rm -rf ",d; +exit $[(0=dups) and 0=count bad; 0; 1] diff --git a/testfiles/vt-symdomain-test.q b/testfiles/vt-symdomain-test.q new file mode 100644 index 0000000..66e88b4 --- /dev/null +++ b/testfiles/vt-symdomain-test.q @@ -0,0 +1,85 @@ +/ Does a new symbol VALUE reach the reader, and how fast? (§5.4) +/ . +/ cd ~/TorQ-VT-Capture-Pack && . ./vt-env.sh && q testfiles/vt-symdomain-test.q +/ . +/ Two different things travel at two different speeds, and conflating them hides a real gap. +/ . +/ rows appended to a directory the reader already holds are visible IMMEDIATELY, with no +/ rebuild and no notification - that is what the trailing-slash live view buys (5.2) +/ . +/ a symbol VALUE that has never been seen before is a different matter. it is an index into +/ the enumeration domain, and the reader holds that domain in memory. the writer appends the +/ new entry to the domain file, but creates no directory - so nothing is announced (4.1 is +/ edge-triggered on directories). until the reader reloads the domain, the rows are there but +/ that column reads as NULL. no error. +/ . +/ Before the domain got its own timer, the window was the 30s rebuild sweep. This checks it is +/ now bounded by the writer's flush interval instead. +/ . +/ NOTE a line containing only "/" opens a block comment in q, so every comment line here +/ carries text after the slash. + +budget:0D00:00:05; / generous: flush interval + symsweep, both ~1s + +pass:0; fail:0; +check:{[ok;msg] $[ok; [pass+::1; -1 " PASS ",msg]; [fail+::1; -1 " FAIL ",msg]]; }; + +idbport:`$"::",string[30+"J"$getenv`KDBBASEPORT],":idb:pass"; +tpport:`$"::",getenv[`KDBBASEPORT],":feed:pass"; +h:@[hopen;idbport;{'"no reader on ",string[idbport],": ",x}]; +tp:@[hopen;tpport;{'"no tickerplant on ",string[tpport],": ",x}]; + +system "S ",string "i"$.z.t; / every q process starts on the same seed +tabs:h"key .vtidb.parts"; +if[not `trade in tabs; -1 " no trade table yet - let the stack capture for a few seconds"; exit 1]; +inst:first h"exec sym from .vtidb.parts`trade"; +if[null inst; -1 " no partitions yet"; exit 77]; + +pc:h".vtidb.partitioncol"; +newside:`$"ZS","" sv string 5?.Q.A; / a data symbol column value never seen before +rows:{[h;inst] h"count select from trade where sym=`",string inst}; +sides:{[h;inst] (h"select distinct side from trade where sym=`",string inst)`side}; + +n0:rows[h;inst]; +s0:h"count sym"; + +-1 ""; +-1 " existing partition : ",.Q.s1 inst; +/ NOTE the parentheses matter: .Q.s1 is unary, so ".Q.s1 x," y"" parses as .Q.s1 (x,"y") +-1 " new value : ",(.Q.s1 newside)," - a value in `side, a column INSIDE the files"; +-1 " rows / domain : ",string[n0]," / ",string s0; +-1 ""; + +t0:.z.p; +do[3; tp(".u.upd";`trade;(enlist inst;enlist 99f;enlist 1i;enlist 0b;enlist " ";enlist "N";enlist newside))]; + +/ 1. the rows must arrive through the live view, with nothing told to the reader +arrived:{[rows;h;inst;n0;t0] + $[n00;1;0] diff --git a/testfiles/vt-tprestart-test.q b/testfiles/vt-tprestart-test.q new file mode 100644 index 0000000..c9dbfb0 --- /dev/null +++ b/testfiles/vt-tprestart-test.q @@ -0,0 +1,81 @@ +/ Is the stack actually capturing, and is the writer still subscribed? (§4.8) +/ . +/ cd ~/TorQ-VT-Capture-Pack && . ./vt-env.sh && q testfiles/vt-tprestart-test.q +/ . +/ A liveness check, and the reason it exists is worth reading before you need it. +/ . +/ Restart the tickerplant and the stack does not fully recover on its own. Every process stays +/ up, the writer keeps logging "enumerated trade table" once a second, the reader answers +/ queries - and nothing new is captured, indefinitely. Measured: still stalled ten minutes +/ later, well past the five-minute .servers RETRY. +/ . +/ Two separate causes, one fixed and one not: +/ . +/ the feed cached its tickerplant handle at startup. a cached handle dies with the +/ tickerplant, and .servers reconnecting afterwards updates its own table, not a copy +/ somebody took at load time. FIXED - code/tick/feed.q now resolves the handle per publish +/ . +/ the writer does not re-subscribe. TorQ defines .wdb.notpconnected[] for exactly this +/ condition and then never calls it - the predicate exists, nothing invokes it. NOT FIXED: +/ re-subscribing also re-runs the partition delete-and-replay, so wiring it to a timer needs +/ more care than it looks. The operational answer is to restart the writer, which replays +/ the tickerplant log and loses nothing - verified, 435 rows to 2005 on restart +/ . +/ So: after a tickerplant restart, restart the writer. This test tells you whether you need to. +/ . +/ NOTE a line containing only "/" opens a block comment in q, so every comment line here +/ carries text after the slash. + +window:0D00:00:08; / long enough for several 1s flushes + +pass:0; fail:0; +check:{[ok;msg] $[ok; [pass+::1; -1 " PASS ",msg]; [fail+::1; -1 " FAIL ",msg]]; }; + +idbport:`$"::",string[30+"J"$getenv`KDBBASEPORT],":idb:pass"; +wdbport:`$"::",string[5+"J"$getenv`KDBBASEPORT],":wdb:pass"; +h:@[hopen;idbport;{'"no reader on ",string[idbport],": ",x}]; +w:@[hopen;wdbport;{'"no writer on ",string[wdbport],": ",x}]; + +-1 ""; + +/ --------------------------------------------------------------------------- +/ 1. the writer's subscription. +/ NOTE not via .wdb.notpconnected[] - it reads `tickerplanttypes` unqualified, which only +/ resolves when the calling context is already .wdb, so over IPC it raises a value error. +/ .sub.SUBSCRIPTIONS is a root-namespace table and asks the same question safely. +nsub:.[{[w] w"count select from .sub.SUBSCRIPTIONS where active"};enlist w;{`$"ERROR: ",x}]; +check[(-7h=type nsub) and nsub>0; + "the writer holds an active tickerplant subscription (",(.Q.s1 nsub),")"]; +if[not (-7h=type nsub) and nsub>0; + -1 " -> a tickerplant restart drops this and nothing re-establishes it."; + -1 " restart the writer; it replays the log and loses nothing."]; + +/ --------------------------------------------------------------------------- +/ 2. and the thing that actually matters: are rows arriving on disk? +/ NOTE the trailing ignored argument keeps this a FUNCTION - {[a;b]…}[x;y] is fully applied +/ and evaluates on the spot, so tot[] would hand back a cached number and measure nothing +tabs:h"key .vtidb.parts"; +tot:{[h;tabs;i] sum {[h;t] h"count select from ",string t}[h] each tabs}[h;tabs]; +a:tot 0; +system "sleep ",string `long$window%0D00:00:01; +b:tot 0; + +-1 ""; +-1 " rows over ",(string window)," : ",(string a)," -> ",string b; +-1 ""; +check[b>a; "the database is growing - the whole chain is live"]; + +/ --------------------------------------------------------------------------- +/ 3. the feed must not be holding a stale handle. it publishes through a lookup, not a +/ cached global, so a tickerplant restart cannot strand it +feedsrc:read0 hsym`$getenv[`KDBAPPCODE],"/tick/feed.q"; +check[not any feedsrc like "h:.servers.gethandlebytype*"; + "the feed resolves its tickerplant handle per publish, not once at startup"]; +check[any feedsrc like "*tphandle:*"; + " - via tphandle[], so .servers reconnection is picked up automatically"]; + +-1 ""; +-1 " ",string[pass]," passed, ",string[fail]," failed"; +-1 ""; +hclose h; hclose w; +exit $[fail>0;1;0] diff --git a/testfiles/vt-wdbrestart-test.q b/testfiles/vt-wdbrestart-test.q new file mode 100644 index 0000000..d9e94ee --- /dev/null +++ b/testfiles/vt-wdbrestart-test.q @@ -0,0 +1,171 @@ +/ What does a reader serve while the writer is rebuilding the day? (4.7, 5.3, recovery) +/ . +/ cd ~/TorQ-VT-Capture-Pack && . ./vt-env.sh && q testfiles/vt-wdbrestart-test.q +/ . +/ A writer restart is not a quiet event on disk. It DELETES the whole live date directory and +/ then rebuilds it by replaying the tickerplant log - which for a full day is hundreds of +/ milliseconds of directories reappearing one at a time. The reader is not told any of this. It +/ is holding a catalogue of live views that point straight into the directory that just went +/ away, and its sweep will fire somewhere in the middle. +/ . +/ So there are three distinct states a query can land in, and they are not the same: +/ . +/ deleted, not yet rescanned the catalogue still lists partitions that no longer exist +/ deleted, rescanned the reader knows they are gone +/ half rebuilt some partitions are back, some are not, one may be mid-write +/ . +/ The question that matters is not whether a query can fail - it obviously can, the data is +/ genuinely absent for a moment - but whether it can come back WRONG. A short answer that +/ completes itself is a different thing from a plausible answer that is quietly missing rows. +/ . +/ NOTE a line containing only "/" opens a block comment in q, so every comment line here +/ carries text after the slash. + +.lg.o:{[t;m]}; .lg.w:{[t;m] warns,:enlist m}; .lg.e:{[t;m] errs,:enlist m}; +.servers.startupdepcycles:{[t;i;c] '"no wdb in this test"}; +.servers.gethandlebytype:{[t;m] ()}; +.timer.enabled:0b; .timer.repeat:{[a;b;c;d;e]}; .proc.cp:{[] .z.P}; +warns:(); errs:(); + +pass:0; fail:0; +check:{[ok;msg] $[ok; [pass+::1; -1 " PASS ",msg]; [fail+::1; -1 " FAIL ",msg]]; }; +try:{[f;a] .[f;a;{[e] `$"ERR:",e}]}; +failed:{[x] $[-11h=type x; x like "ERR:*"; 0b]}; +q:{[s] try[{[x] value x};enlist s]}; + +s:"/tmp/vt-wdbrestart-",string .z.i; +system"rm -rf ",s; system"mkdir -p ",s; +root:hsym`$s; +(.Q.dd[root;`sym]) set 0#`; +d:2026.01.02; +past:2026.01.01; +insts:`$"I",/:string til 12; +rows:50; + +mk:{[root;rows;dt;t;i] + x:([]time:rows#.z.p; price:rows?100f; side:rows?`buy`sell); + (.Q.dd[.Q.dd[.Q.dd[.Q.dd[root;`$string dt];t];i];`]) set .Q.ens[root;x;`sym]; + }[root;rows]; + +/ yesterday, closed and immutable - a writer restart must never touch it +mk[past;`trade;] each insts; +mk[past;`quote;] each insts; +/ today, the live partition +mk[d;`trade;] each insts; +mk[d;`quote;] each insts; + +.vtidb.roots:enlist root; +.vtidb.partitioncol:`instrument; +system"l ",getenv[`KDBAPPCODE],"/processes/vtidb.q"; +.vtidb.current:d; +.vtidb.dropcache[]; .vtidb.rebuild[]; + +full:count[insts]*rows; +-1 ""; +-1 " ",string[count insts]," instruments x ",string[rows]," rows, on ",(string past)," and ",string d; +-1 " baseline today ",string q"count select from trade where date=",string d; + +check[full=q"count select from trade where date=",string d; "baseline is correct"]; +check[(2*full)=q"count select from trade"; "and both dates are attached"]; + +/ --------------------------------------------------------------------------- +/ 1. the moment after clearwdbdata: the directory is gone, the catalogue is not. +/ --------------------------------------------------------------------------- +-1 ""; +-1 " DELETED, NOT YET RESCANNED - the catalogue points at directories that are gone"; +system"rm -rf ",s,"/",string d; + +n:q"count select from trade where date=",string d; +-1 " today ",$[failed n; string n; string n]; +y:q"count select from trade where date=",string past; +-1 " yesterday ",$[failed y; string y; string y]; + +check[failed n; "a query for the deleted date FAILS - loudly, naming the missing file"]; +check[(not failed y) and full=y; + "but yesterday is untouched and still exact - the blast radius is the live date only"]; +check[failed q"count select from trade"; + "a whole-database query fails too, because it has to open every partition"]; +-1 " -> this is the state that matters most, and it errors rather than under-reporting."; +-1 " a client sees an exception, not a plausible number that is quietly short."; + +/ --------------------------------------------------------------------------- +/ 2. the sweep runs while the directory is still empty. +/ --------------------------------------------------------------------------- +-1 ""; +-1 " DELETED, RESCANNED - the sweep has caught up with the deletion"; +warns:(); errs:(); +.vtidb.rebuild[]; +check[not failed q"count select from trade"; "queries answer again once the sweep has run"]; +check[full=q"count select from trade"; "and return exactly yesterday, with today absent"]; +check[0=count .vtidb.parts[`trade] where .vtidb.parts[`trade][`date]=d; + "the deleted date is dropped from the catalogue entirely"]; +check[not any {x like "*coverage gap*"} each warns; + "4.6's coverage check stays silent - and correctly so: it compares dates ACROSS tables, and a ", + "writer restart removes the date from every table at once. a symmetric loss is invisible to it"]; + +/ --------------------------------------------------------------------------- +/ 3. the replay puts partitions back, one at a time, with the sweep firing in between. +/ --------------------------------------------------------------------------- +-1 ""; +-1 " HALF REBUILT - the sweep fires while the replay is still running"; +seen:(); +{[mk;d;insts;i] + mk[d;`trade;insts i]; + mk[d;`quote;insts i]; + .vtidb.rebuild[]; / the sweep, landing mid-replay + r:q"count select from trade"; + seen,:enlist (i; $[failed r; -1; r]); + }[mk;d;insts] each til count insts; + +got:{x 1} each seen; +-1 " rows seen as partitions came back: ",.Q.s1 got; +check[not any got=-1; "no query errored while the replay was in progress"]; +check[got~asc got; "every answer was larger than the last - monotonic, never going backwards"]; +check[all got>=full; "and never below yesterday's total, which was never at risk"]; +check[(2*full)=last got; "the last one is the complete database again"]; + +/ --------------------------------------------------------------------------- +/ 4. the sharper case: the sweep lands while a partition is MID-WRITE, with .d +/ already on disk and its columns not. this is the state 5.7 documents, reached +/ here the way a replay would reach it rather than by a full disk. +/ --------------------------------------------------------------------------- +-1 ""; +-1 " MID-WRITE - .d is on disk, the columns are not yet"; +half:.Q.dd[.Q.dd[.Q.dd[root;`$string d];`trade];`HALF]; +system"mkdir -p ",1_string half; +(.Q.dd[half;`.d]) set `time`price`side; / promises three columns, has none +warns:(); errs:(); +.vtidb.rebuild[]; +r:q"count select from trade"; +check[failed r; + "a whole-database query fails while that partition is mid-write - the reader cannot ", + "tell it apart from a permanently damaged one (5.7)"]; +sel:q"count select from trade where date=",(string d),", instrument=`I0"; +check[(not failed sel) and rows=sel; + "a SELECTIVE query on a healthy instrument is unaffected - partition elimination again"]; + +/ and it clears itself the moment the writer finishes that directory +mk[d;`trade;`HALF]; +check[not failed q"count select from trade"; + "and it resolves as soon as the columns land, with no rebuild and no intervention"]; + +-1 ""; +-1 " WHAT THIS ADDS UP TO"; +-1 ""; +-1 " Only the FIRST state is loud. Once the sweep has run, the reader is serving a database"; +-1 " that is genuinely missing today - and it answers, with a number that looks perfectly"; +-1 " reasonable. The same is true all the way through the replay: 650 rows when the true"; +-1 " figure is 1200 is not an error, it is a short answer, and nothing marks it as one."; +-1 ""; +-1 " So a writer restart opens a window - the replay time plus up to one sweep interval -"; +-1 " in which whole-database aggregates under-report silently. Selective queries on"; +-1 " instruments already rebuilt are exact throughout, and history is never at risk."; +-1 ""; +-1 " That is a property of restarting the writer, not a defect in the reader: the rows really"; +-1 " are absent from disk while the replay runs. It is worth knowing before scheduling a"; +-1 " writer restart underneath something that reports numbers to people."; +-1 ""; +-1 " ",string[pass]," passed, ",string[fail]," failed"; +-1 ""; +system"rm -rf ",s; +exit $[fail>0;1;0] diff --git a/vt-env.sh b/vt-env.sh new file mode 100755 index 0000000..1b554d9 --- /dev/null +++ b/vt-env.sh @@ -0,0 +1,66 @@ +#!/bin/bash +# Environment for the TorQ Virtual-Table Capture Pack. +# +# This is an application overlay: it supplies config and code that layer on top of a +# TorQ checkout, which supplies the framework and process code. +# +# Only TORQHOME below should ever need editing. Everything else derives from it and +# from the location of this file, so the pack can be cloned anywhere. + +# --- the two roots ----------------------------------------------------------- +# TorQ core: the checkout that contains torq.q, code/ and config/. There is no sensible +# default, so either export TORQHOME before sourcing this file or fill in the path below. +# start.sh and compress.sh both check it and stop with a clear message if it is wrong. +export TORQHOME="${TORQHOME:-}" + +# warn when sourced by hand - the test headers say ". ./vt-env.sh && q testfiles/.q", and +# without this an unset TORQHOME turns KDBCODE into "/code" and the failure is a load error +# deep inside a test rather than anything pointing back here. +if [ ! -f "${TORQHOME}/torq.q" ]; then + echo "vt-env.sh: WARNING - no torq.q under TORQHOME=${TORQHOME:-}" >&2 + echo "vt-env.sh: set TORQHOME to your TorQ checkout, or edit this file" >&2 +fi + +# this pack, resolved from the location of this script - never hardcode it +if [ -n "${BASH_SOURCE[0]}" ]; then + _VTDIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)" +else + _VTDIR="$(cd "$(dirname "$0")" && pwd)" +fi +export TORQAPPHOME="${_VTDIR}" +export TORQDATAHOME="${TORQDATAHOME:-${_VTDIR}/var}" # runtime state; data/ holds the feed's sample csv + +# --- TorQ core --------------------------------------------------------------- +export KDBCONFIG="${TORQHOME}/config" +export KDBCODE="${TORQHOME}/code" +export KDBLIB="${TORQHOME}/lib" +export KDBHTML="${TORQHOME}/html" + +# --- this pack --------------------------------------------------------------- +export KDBAPPCONFIG="${TORQAPPHOME}/appconfig" +export KDBAPPCODE="${TORQAPPHOME}/code" +export TORQPROCESSES="${KDBAPPCONFIG}/process.csv" + +# --- data and logs ----------------------------------------------------------- +# ONE directory for the database. No separate wdb/hdb areas: the writer writes where +# the readers read, and nothing moves at end of day. KDBHDB and KDBWDB are kept as +# aliases because TorQ core and the stock settings read them by name. +export KDBDB="${TORQDATAHOME}/db" +export KDBWDB="${KDBDB}" +export KDBHDB="${KDBDB}" +export KDBLOG="${TORQDATAHOME}/logs" +export KDBTPLOG="${TORQDATAHOME}/tplogs" + +# --- kdb-x ------------------------------------------------------------------- +# these are usually absent from the shell profile. without QLIC q reports +# "license error: no license loaded"; without QPATH `use` cannot resolve modules. +export QHOME="${QHOME:-$HOME/.kx/q}" +export QLIC="${QLIC:-$HOME/.kx}" +export QPATH="${QPATH:-$HOME/.kx/mod}" +export QCMD="${QCMD:-q}" +export RLWRAP="${RLWRAP:-rlwrap}" +export QCON="${QCON:-qcon}" + +export KDBBASEPORT="${KDBBASEPORT:-6000}" + +mkdir -p "${KDBDB}" "${KDBLOG}" "${KDBTPLOG}"