Repository navigation
feat(coverage): build the coverage table and probe map in the app's JVM - #194
Draft
slayerjain wants to merge 14 commits into
Draft
slayerjain wants to merge 14 commits into
slayerjain wants to merge 14 commits into
Conversation
slayerjain
force-pushed
the
feat/jcov-buildtable
branch
from
November 19, 2026 19:42
7b21804 to
b449cc6
Compare
Keploy's per-test code coverage needs the build table of the app's classes: packages, classes, methods, their lines and branch edges. Only the app's JVM holds the class bytes in every run mode, so keploy-sdk builds it there. - The table goes only when Keploy asks for it with the control-socket command TABLE, after its first START is answered ACK. The agent answers ACK at once and starts the work on a daemon thread of its own; the control path never analyzes, encodes or sends, and START and END never ask for the table. An older Keploy never sends TABLE, so it never gets a frame; an older SDK answers it IGNORED. A TABLE while a send runs has that send run once more after it, so a new receiver can ask again. A TABLE that fails to start (no thread, or any error on the control path) is answered IGNORED and logged at WARNING, and the control server goes on serving START and END. - The analysis competes with the app for CPU: the thread's priority is only a hint, which HotSpot on Linux ignores by default. When the index is not loaded yet, its first load runs on this thread and holds the index's lock, which an END that needs the index waits on; an END would otherwise load it itself. - JaCoCo's analyzer runs over the indexed class bytes. One record per top-level class, its nested, inner, anonymous, lambda and Scala module classes folded in by name (the simple name cut at the first $ after its first character). A class is named by its bytes, the name JaCoCo's runtime gives its probes, wherever the index found the file. A method has one code leaf per line (n = JaCoCo's instructions on that line in that method) and one leaf per branch edge. A method with no line numbers has no leaf and is left out; the INFO line counts the class files built without line numbers and says to compile them with -g. - Records are in the ingest contract's order (packages segment by segment, then id; methods by first line, then id) and written as Go's encoding/json writes them with HTML escaping off. The build id is jv1- and the first 32 hex of the sha256 over the sorted JaCoCo class ids of every indexed class; a record's hash is the same sum over its class files' ids. - A class JaCoCo cannot analyze makes its record not_measured with reason analyzer_failed and no methods. A record larger than an inflated 8 MiB chunk is not_measured with reason record_too_large. Keploy's encoder also refuses a record whose compressed form passes a chunk; the CLI is the authority on that, so the stream's sha256 is the CLI's only while it re-marks no record. - The records go to Keploy as TABLE frames on the data socket, one frame per connection, each at most 1 MiB: KTB1, a 4-byte length, a header line (build id, seq, last; the last one adds the record count and the sha256 of the whole stream), then the next bytes of the NDJSON stream. With no app class the stream is one empty last frame with build "". The SDK shuts its output after each frame and waits for OK; a receiver that closes without it ends the stream, which the SDK logs at WARNING with the frame and the reason, and the table goes again only when Keploy asks again. - KeployDedupAgent gains only the TABLE branch in the unix control socket's command handler. The TCP transport does not send a table. Two golden rows check the table byte for byte against the Go encoder's output for the same classes, with the hashes and the build id computed in Go from the class bytes: the java-dedup sample's classes, and a generated set for the names, packages and escapes the sample lacks. A control-socket row runs START and END on an index of the sample's classes, checks that neither asked for the table, then sends TABLE and gets that golden in one frame. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
Keploy gets each test's coverage as JaCoCo probe hits per class, while the build table counts lines and branch edges (its leaves). To turn hits into leaves without the class files, Keploy needs, per class, which leaves a hit of each probe covers. keploy-sdk now computes that map in the app's JVM, with the same JaCoCo analysis steps as the table, and sends it right after the table, in the same answer to TABLE. - A probe covers more than its own line: JaCoCo marks every instruction on the path that ends at the probe, and the branch edges along it. So each probe maps to a list of leaves, and a test's leaves are the union of its hit probes' leaves. - The map is JaCoCo's own result, not a copy of its rules. JaCoCo's public API gives only per-line sums, which lose which branch edge of a line is covered, and two tests that cover different edges of one line must count two. A small class in JaCoCo's package (org.jacoco.core.internal.analysis.KeployBranchCoverage) runs JaCoCo's package-private analysis steps unchanged (the instruction graph, the filters, merges and branch replacements) and reads each branch through JaCoCo's own replaceBranches. It checks every line's sums against the coverage JaCoCo computes in the same pass and throws on a difference. keploy-sdk.jar relocates org.jacoco.core, and this class moves with it. - A method's probes never reach another method, so one analysis maps the k-th probe of every method: the map takes one analysis of a class file more than its largest method has probes, on top of the table's. Each one reads the whole class file, so a class with one very large method costs seconds of CPU on the analysis thread. - The map is a second stream of TABLE frames after the table's last frame: "part":"probes", seq from 0, the last frame with the record count and the sha256 of the stream. One record per class name of each measured class: its name, its JaCoCo class id, its unit, and for each probe the covered leaves as ranges of indices into the unit's leaves. Indices are relative to the unit, so a unit Keploy marks not measured only drops its records. The map is never uploaded. - A test's hits name a class by its name only, so a name has one record. The index can find one name under two paths: the same bytes count once, and two class files that differ share the record when their probes map alike (the record carries the first one's id). - The map is JaCoCo 0.8.15's probe layout, that of the agent keploy-sdk.jar starts. It is not for the hits of a JaCoCo agent the app's command attached (the index record names that case): another version can place the same number of probes differently. - A class whose probes cannot be mapped exactly is sent as not measured (analyzer_failed) in the table itself, so the table and the map always agree; that includes two class files of one name whose methods or probes differ. A class that is not measured has no map record, and the INFO summary counts the classes that failed this way. - Lines below 1 have no leaf (javac never writes line 0; bytecode tools can): a method keeps its other lines in the table and the map. Rows: for every class file of the java-dedup sample, of a Java 8 class of code shapes (&& on one line, loops with break and continue, an int switch with shared and fall-through cases, a String and an enum switch, try/finally, try-with-resources, synchronized, a lambda, an assert) and of a Java 21 one (records, a sealed type, pattern switches with guards and record patterns, an exhaustive enum switch, a String switch with yield, instanceof patterns), the leaves the map gives no probe, each probe alone, every pair of probes and all of them equal what JaCoCo's public Analyzer counts for that set, line by line (code leaf hit, branch leaves hit, branch leaves), and each record has as many probes as JaCoCo's instrumenter gives the class. The java-dedup map is a golden for the Go decoder, and the control-socket row checks it arrives after the table on a real socket. Rows cover one class file found twice, two class files of one name that map alike and that differ (moved line, missing method, probes in another order), and a method with line 0. Only BuildTable.java changes in the agent; KeployDedupAgent.java is untouched. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
Keploy files the build table under the build id the INDEX record named, so the two ids must be one value over one class set. Until now they were two copies of the same algorithm: IndexReport.buildId for the INDEX record and BuildTable's own classIdsSum for the table. A change to one copy would have left Keploy looking for a table that never arrives. - IndexReport.classIdsSum is now the only implementation: the sha256, as 64 hex, over the class ids sorted as unsigned numbers, de-duplicated, each as 16 hex, joined by newlines. IndexReport.buildId is "jv1-" and its first 32 hex, or "" when there is no class. - BuildTable.write takes its class ids with IndexReport.classId and its build id with IndexReport.buildId, and a unit's hash is IndexReport.classIdsSum over that unit's ids. BuildTable's copy and its sha256Hex helper are removed (tests now take the sha256 digest themselves). - Both ids are taken over CoverageIndex.entries(), every class the index holds, whether a class reader can read it or not. Every entry has its bytes (the dir, jar and derived readers never store an entry without them), so the table's skip of an entry with no bytes never drops one the INDEX record counted. The table bytes are unchanged: the Go encoder's golden for the java-dedup sample still matches byte for byte. Rows: - ControlSocketTest.theIndexRecordAndTheTableNameOneBuild: the agent's index is the java-dedup sample plus one file no class reader reads; the INDEX record's build, and the build in every table and probe-map frame sent for TABLE, equal the id the test computes itself (sorted 16-hex ids, sha256), which differs from the sample's alone. - IndexReportTest.theUnitHashIsTheWholeSumAndTheBuildIdItsFirstHalf: the sum of the ids 1, 0xff and 0xffffffffffffffff, given out of order and with a repeat, is the full sha256 the Go side's build id is cut from. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
The probe map analyzed each class once per probe round: a class with a method of K probes was parsed, sanitized, flow-analyzed and filtered K times, whole. KeployBranchCoverage now analyzes a class once. For each method it keeps the filters' output, then replays only that method's instructions through JaCoCo's own probe adapter, analyzer, instruction builder and coverage calculator, once per probe, with that probe alone executed. The result is still JaCoCo's analysis, not a copy of its rules, and a replay that meets other probes than the class's analysis did fails the class (not_measured, analyzer_failed). The filters now run once per class, in class order, exactly as for the table. A method with no line (a class built without -g) has no leaf to map, so it is not replayed. A class now costs, summed over its methods, each method's probes times that method's size. One harness, run before and after on the same machine (JDK 21, heap capped at 1 GiB), on a generated app of 50,000 classes (one 15-branch method each): the table and probe map took 51.4 s of wall time and 58.3 s of process CPU before, 29.1 s and 33.5 s after. BuildTableBudgetTest measures it in the test JVM on JDK 8, 17 and 21, heap capped at 1 GiB, table and probe map: - 50,000 generated classes: 28.6-31.2 s wall, 27.4-30.3 s on the table's thread, 35.7-38.1 s of process CPU (GC 0.6-1.0 s). At most 100-114 MB of heap stayed in use after any collection. Output: 55 MB of table and 21 MB of probe map. With the heap capped at 256 MB (JDK 21): 33-36 s wall, 41-64 s of process CPU (GC 1.9 s). - One class with a single 2000-branch method: 12.1-15.1 s; with 3500 branches (JDK 21), 57.4 s. No frame goes out while one class's map is computed, so such a class pauses the stream that long. - JaCoCo core's jar (210 classes): 0.5-0.6 s. The analysis runs on its own daemon thread after Keploy asks for it, and START and END are answered while it runs. Also: - A TABLE that arrives while a send runs now runs the newest request after it, not the first one again. Each request names the index and data socket it was made for, so after the agent's stop() and start() during a send, the rerun used the old index and the old socket. A send that throws no longer drops the request made while it ran. - BuildTable's logging can no longer throw. java.util.logging does not catch what a Handler throws (an app's logging bridge), so a log call that threw could fail a send, and one in the send loop left it marked running with no thread: every later TABLE was ACKed and never sent. - keploy-sdk's tests run with a 1 GiB heap (the argLine property; -DargLine overrides it). Left to the JVM, the heap was a quarter of the machine's RAM. In a 4 GiB container where JDK 8 sizes it from the host's RAM, the new rows let its collector grow the heap until the kernel killed the test JVM. - IndexReport.hex is the one lowercase-hex encoder (the build id and a frame's sha256); BuildTable's copy is removed. - BuildTable.write no longer skips an index entry without bytes: the agent never passes one, and the INDEX record has no such skip. - BuildTableTest's receiver keeps each frame before it answers OK, as Keploy does; it answered first, so a row could count one frame short. Rows: - ProbeMapTest.eachProbesReplayIsTheClassAnalyzedWithThatProbeExecuted: on JaCoCo core's and Gson's jars, each probe's replay equals the class analyzed whole with that round's probes executed, for every method and probe. - ProbeMapTest.aMethodTheCallerDeclinesIsNotReplayed: a method the caller declines gets no replay; the class pass still analyzes it. - BuildTableTest.theRunAfterASendIsTheNewestRequests: two requests during a send run the newest once (it ran the first one again). - BuildTableTest.aRequestMadeWhileASendFailsStillRuns: a request made during a send that throws still runs (it was dropped). - BuildTableTest.aLogHandlerThatThrowsNeitherFailsASendNorWedgesSends: with a log handler that throws, send does not throw, and a request after a failed send, and after a sent table, still runs. - ControlSocketTest.startAndEndAreAnsweredWhileTheTableIsAnalyzed: with a 5000-class index, START/END pairs are answered while the table is analyzed and sent; TABLE's own answer and the slowest pair each take under 5 s. - BuildTableBudgetTest: measures and prints the cost on a generated app, a one-method class and a real library; every class is measured. -Dkeploy.budget.classes and -Dkeploy.budget.branches set the sizes. - The java-dedup table and probe-map goldens are unchanged. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
The app's JVM builds the coverage build table and probe map when Keploy asks (TABLE). The table already went out record by record, but every probe-map record stayed in the heap until the table's last frame, as the map streams after the table: about 25 MB more live heap for 50,000 classes, and more for classes with more probes. An app with a tight -Xmx (a k8s pod) could run out of heap while the table was built. Each measured class's probe records now go, as they are made, to a temp file in java.io.tmpdir (owner-only, opened without following a link, unlinked once open on Unix, so a JVM that dies mid-send leaves none and one that dies before the open leaves an empty file), which is streamed back as the probe map's frames after the table's last frame. The wire is unchanged. Each class is dropped once written and the grouping's lookups end before the analysis, so a send holds the class list, one frame and one class's analysis. A tmpdir the app cannot write ends the send before any frame, with a WARNING naming the dir: Keploy gets no table for that build and asks again on its next run. When TABLE arrives the SDK reads the free heap: the heap cap the JVM reports (a little under -Xmx with the Serial and Parallel collectors) less the heap in use. That is what the JVM's newest collection left in the heap pools (the collectors' last GcInfo), or the heap in use now when that is smaller, when no collection ran yet or when the runtime does not report one. The SDK never runs a collection. With less free than 256 MiB it answers IGNORED proto=2 reason=low_heap heap_free_mb=F heap_needed_mb=N logs a WARNING and sends nothing, so Keploy can mark the run not measured with that reason. KEPLOY_COVERAGE_TABLE_MIN_HEAP_MB, else -Dkeploy.coverage.table.min.heap.mb, sets the bound in MiB; 0 turns the check off. A Keploy that does not read the reason sees IGNORED, as from an SDK that sends no table. Measured on 50,000 generated classes, against the parent commit in the same run: the live heap at each frame (after a full collection) is 18-22 MB above the heap in use before the send (the classes' bytes, the list of them the send is handed and the harness), from 44-47 MB (JDK 8 and 21); what is left is the send's own entry for each class, about 0.4 KB per class. Time is unchanged: 25.3-26.2 s wall, 24.7-25.4 s on the analysis thread at -Xmx1g (JDK 8/17/21). With -Xmx96m it takes 44 s, from 71 s, and with -Xmx80m it completes in 48 s where the parent ran out of heap (JDK 21). Rows: a measured class's probe records reach their sink just before its own table record; the map waits in a temp file that is gone during and after the send, and both parts still match the java-dedup goldens; a tmpdir that cannot be written sends no frame and names it; the temp file is never opened through a link; the low_heap answer at and around the bound, with 0 and no cap; the env, property and default of the bound; the heap in use is the newest collection's, heap pools only, or the heap in use now when smaller; and through the control socket, a TABLE with too little heap is answered low_heap, starts no send and sends no frame (it failed first: ACK). The other TABLE rows turn the bound off, so the test JVM's -Xmx does not decide them. The budget row also reports the heap delta, the pools' peak and, with -Dkeploy.budget.live=true, the live set at each frame. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
Each probe's leaves are now read off the class's one JaCoCo pass. JaCoCo
marks an executed probe's instruction and walks back through each
instruction's predecessor (the one edge into it that carries no probe;
JaCoCo puts a probe on every edge into an instruction that more than one
edge reaches), stopping after an instruction it marked already; its
calculator then merges and replaces branches as the filters said.
KeployBranchCoverage takes that walk over the edges the pass's builder
wired and applies the pass's filter output to what it marked. A method
costs its size plus its probes' walks, with no object per probe. Before,
each probe ran JaCoCo's builder and calculator over its whole method
again.
The walks are short in an if/else chain, whose statements join at labels
two edges reach. In a chain of early returns ("if (x == k) return k;"
lines, or a run of guard throws) each probe walks back over every check
before it and covers each one, so the map itself is quadratic: about
4.4 MB at 1,000 lines and 43 MB at 3,000, and at 4,500 it ran a 256 MiB
heap out of memory. A unit's probe map is now bounded at 8 MiB
(its records with their newlines, the table record's bound) and 64 Mi
walk steps. Past either, the unit is sent not_measured record_too_large
with no probe record and is counted in the send's log line, and the map
is given up as soon as it is past, so 3,000 and 4,500 lines now take
0.07-0.2 s of CPU and 40-51 MiB of allocation. The map's text is ASCII
in one buffer and copied once into an exact-size record. Two class files
of one name in the index are both charged against the one bound, so such
a unit is declined once its two maps together pass 8 MiB, though its one
record would fit (counted, never unbounded).
A method whose flow has another shape (an instruction that two edges
without a probe reach, or a replaced branch that reads another replaced
instruction) throws, so its class is not measured (analyzer_failed), as
a mismatch with JaCoCo's coverage did before.
The pass reads an instruction's branches only when it is covered (an
uncovered instruction has none; the table's pass executes no probe), and
keeps each instruction's branch numbers in one sorted array, not a set
per instruction.
The low_heap bound's default is 64 MiB plus 1 KiB per class the index
holds (113 MiB at 50,000 classes). 256 MiB declined every 1 GiB pod run
with no -Xmx. The count is the index's only when it is read already, so
the answer reads no class file and waits on no lock; an index not read
yet is bounded by the 64 MiB base, and read on the send's thread.
Measured on 50,000 generated classes (JDK 21 and 8, G1 and Parallel)
before the map's bound was added, which only takes copies out of the
map's path, so these are upper bounds: the send allocates 6,142-6,915
MiB, 125-141 KiB a class, against 90,611-98,076 MiB (about 1.9 MiB a
class) before, and takes 4.3-5.8 s of CPU against 29-42 s. A method of
3,000 if/else branches maps in about 0.2 s of CPU.
The heap the JVM commits while the table is built still follows the
analysis's allocation rate, as it did before (up to 657 MiB more with G1
on JDK 8 in a 4 GiB pod): the analysis is not paced, and this commit
does not bound that growth.
Rows: the probe oracle (each probe's map against JaCoCo's analysis of
the class with that probe executed) now covers JaCoCo core, Gson, the
java-dedup, code-shape and Java 21 fixtures, a 300-branch generated
class and an 80-return chain; the budget rows bound the allocation per
class (256 KiB), a 3,000-branch method's CPU (1 s), and a chain of
1,000, 3,000 and 4,500 early returns (mapped, then record_too_large,
each under 1 s of CPU and 64 MiB of allocation); the map's bytes and
steps bounds have rows at both edges; the heap bound has rows at both
edges of its default and through the TABLE answer with an index count
given and with none (-1).
Signed-off-by: slayerjain <shubhamkjain@outlook.com>
The build id hashed the class ids alone, so an SDK release that writes a
different table or probe map for unchanged classes named the same build
as the older SDK did. Keploy keeps one table per build, so every later
save of that build was refused (409 build_table_mismatch) until the app
itself was rebuilt.
The id is now "jv1-" and the first 32 hex of the sha256 over the ASCII
bytes
"keploy-java-table " F "\n" "jacoco " V "\n" ID_1 "\n" ... ID_n
with no newline after the last id, where F is the table's format
(BuildTable.TABLE_FORMAT, 1) and V the release of the JaCoCo core whose
analyzer builds the table: the one keploy-sdk.jar carries renamed, read
from that core's own JaCoCo.VERSION less its build qualifier ("0.8.15"),
not the measuring agent's, since an attached agent of another version
only records the probes. The ids are as before: CRC64, sorted unsigned,
de-duplicated, 16 lowercase hex. These are the bytes the contract gives
Keploy's codecoverage.JavaBuildID. F covers the bounds that decide a
class's records (the record, probe-map bytes and probe-map steps
bounds) and the reason a unit past one is not measured, so a release
that changes one raises it. The frames that carry the records are not
in F: their header names its own version.
The INDEX record and the table still take the id from one function, and
a unit's hash is unchanged: the sum over that unit's class ids alone.
V is read once, at the first build id, from a holder of its own, so
the class-id helpers a testcase's END runs before its ACK never load
JaCoCo's version. If the renamed core cannot read its version resource,
V is "" (logged once) and the INDEX record and the table still name one
build. The smoke test, which CI runs on JDK 8, 17 and 21 against the
built jar, now fails a jar whose analyzer names a JaCoCo release other
than its org.jacoco.core's Maven version, read both as the SDK loads
from the class path and from the jar alone in a loader of its own. The
README says that an SDK upgrade that changes the table format or the
analyzer's JaCoCo gives the same classes a new build id.
Rows: the build id's three golden vectors (ids 1, 0xff and
0xffffffffffffffff with F=1 and V="0.8.15", with F=2, and with
V="0.8.14"), which differ from each other and from the id over the ids
alone; the SDK's F is 1 and its V the core jar's pom.properties version;
a JaCoCo version less its qualifier; the class-id helpers never load the
analyzer's JaCoCo version; and a tripwire that pins F=1 together with
the bounds, the step count of a 200-return chain, the sha256 of the
record each cap writes (a unit past the probe-map steps, and records
past a 1600-byte chunk) and of the golden table and probe-map records,
so a change to any of them, its reason code included, fails until F is
decided. The independent build id checks of the INDEX record and of the
table now hash the same bytes, and the java-dedup and edge goldens are
regenerated from an oracle that also reproduces their old ids-only
values. Their table and probe-map records are unchanged.
Signed-off-by: slayerjain <shubhamkjain@outlook.com>
JaCoCo analyzed every class file twice while the build table was made: once through its public Analyzer, for the table record's methods, and once through KeployBranchCoverage, for the probe map. The second pass already runs the same steps (ClassAnalyzer's instruction graph, filters and MethodCoverageCalculator), so the first was pure cost: about 30% of what the send allocates and of its CPU. Now one pass gives both. KeployBranchCoverage hands each method's JaCoCo coverage (Method.coverage, computed as ClassAnalyzer computes the coverage it adds to its class) to the caller before the method's probes are walked, and BuildTable makes the table methods from it. A probe's ranges are kept relative to its method and get the method's base when the unit's records are written, since the base is known only once all of the unit's methods are ordered. The map budget counts the text before the bases are added (never more than the records hold), then each whole record in its place, so the bound holds as before. The table and probe-map records are byte for byte what they were: the java-dedup and edge goldens and the records the build-id tripwire pins pass unchanged. What changes: - At 50,000 generated classes (G1 and Parallel; 1 and 4 GiB pods) the send's thread allocates 84-96 KiB a class (4,145-4,697 MiB in all; JDK 8, 17 and 21), from 123-134 KiB (6,021-6,577 MiB; JDK 8 and 21), in 2.5-3.4 s of CPU, from 3.5-4.8 s. The committed heap does not drop (G1 in 4 GiB pods grows it by 222-518 MiB, from 254-420): the collectors size it to the allocation rate, not to the garbage's total. - A check of the map that fails (KeployBranchCoverage.NotMapped, or one of BuildTable's) counts as not mapped, as before; anything else the pass throws counts as JaCoCo failing to read the class (before, an error in the map's own pass counted as not mapped). Both are analyzer_failed. An error while a table method is made from its coverage now fails only its unit, as not mapped; before, it ended the whole send. - A unit's class files are now all analyzed first, in name order, and the first analysis that fails decides; only then are its probe-map records made and charged to the budget. Before, every file's JaCoCo read came first, then each file's map and record in turn. So a unit with two failures of different kinds can now be given the other one's reason: analyzer_failed instead of record_too_large, or the reverse. Either way it is not measured and has no probe record. - A second class file of one name is charged its text against the budget while it is analyzed, not its record, so the smallest bound that maps a unit of two such copies is a few hundred bytes lower. - While a unit's records are made, only the class file being written keeps its ranges; a record whose text is not the length counted for it fails its unit as not mapped instead of being sent. Rows: - aGeneratedAppsTableIsMeasured bounds the allocation at 112 KiB a class, below what two analyses took (123-141 KiB in every run so far); on JDK 21 it fails on the two-pass code (135,287 bytes a class at its default 2,000 classes) and passes on this one on JDK 8, 17 and 21. - theTableNeedsNoSecondAnalysisOfAClass: no class of BuildTable's names JaCoCo's public Analyzer (the two-pass BuildTable does). - eachTableMethodIsWhatJacocosAnalyzerCounts: on JaCoCo core's and Gson's classes and every fixture, with no probe executed and with all of them, each method's coverage from the one pass equals what the Analyzer counts (the same methods in the same order, every counter, line by line). It fails when the pass skips the filters. - aClassJacocoCannotAnalyzeIsNotMeasuredAndSoIsItsUnit now also adds a class JaCoCo's instrumenter wrote, and counts none of the three refusals as a map that failed (it fails when they are). Signed-off-by: slayerjain <shubhamkjain@outlook.com>
At 50,000 classes the build table's send allocates 4.2-4.8 GB in 2.5-6.3 s, nearly all of it short-lived. G1 and Parallel answer that rate by committing more heap, and the app keeps that heap after the send: unpaced, G1 on JDK 21 grew a 4 GiB pod's heap by 175-343 MiB. Two changes: - The send's thread is paced by the collections (BuildTable.Pacer). After each unit it owes 200 times the collection time since the send began (the app's collections and its own), less the time since, and sleeps that off, so collecting takes about 0.5% of the send's time. A JVM that does not collect is never paced. The sleeps come from an allowance of 0.8 ms per analyzed class file, of which at most 1 s is saved up: no one sleep is over a second, frames keep coming (so Keploy's wait for the next frame never runs out), and the sleeps add up to at most 0.8 ms a class file (40 s at 50,000). No sleep is taken that would end the table past 50 s at 50,000 class files (1 ms a class file past that), reckoned as the time so far plus the rest of the class files at the rate the work went so far, so on a slow or busy CPU the pacing gives way to the table's 60 s bound. A clock or collector bean that fails ends the pacing, and the rest is sent unpaced. An interrupt ends the send, with a WARNING that names it, and the interrupt is kept; before, the next write to the probe map's temp file failed and the WARNING blamed java.io.tmpdir. The send's INFO line now says how long it was paced. - FrameWriter fills one frame buffer, reused for every frame, and the socket writes it in place. Before, each 1 MiB frame was allocated twice (the body's copy and the frame), as G1 humongous objects. FrameSender gains send(b, off, len); its default copies the bytes, for a sender that keeps them. At 50,000 generated classes (G1 and Parallel; 1 and 4 GiB pods with no -Xmx; JDK 8, 17 and 21), three runs paced against two of the same code with the pacer off, the committed heap grows over what it was just after a full collection by: - JDK 21: 19-46 MiB (unpaced: G1 in a 4 GiB pod 175-343 MiB). - JDK 17 and 8, on Parallel and on G1 in a 1 GiB pod: 11-92 MiB. - G1 in a 4 GiB pod: 13-166 MiB on JDK 17, and 420-421 MiB on JDK 8 in every run (unpaced 422-707). The collection shrank G1's heap (JDK 8: 632 to 120 MiB) and its pause raised G1's pause-time ratio, so G1 grows the heap back while the table is sent, paced or not. Never past -Xmx. The table takes 10.3-46.1 s, from 2.5-6.3 s, in 2.4-6.7 s of thread CPU; the largest gap between two frames is 2.25 s. Where the pacing does not cut the growth it still costs that time: Parallel on JDK 8 takes 28-46 s, from 3.6-4.1 s, for about the same growth. Rows: - BuildTablePacerTest (16 rows, on a clock the rows drive): no sleep without a collection; a collection is slept off to 200 times its time, less the work since; the sleeps over any run of units stay within a second plus the allowance of their class files; a quiet stretch saves up at most a second; an overrun sleep is charged, and a sleep the clock did not see is charged what it asked; the sleeps give way to the deadline, told the class files or not; a table whose work alone fills its deadline is not paced; an interrupt ends the send and is kept; a failing collector bean ends the pacing; write paces after each unit by its class files and tells the pacer how many come. - anInterruptEndsTheSendAndIsSaidSo: an interrupt before the send, or after the table's last frame (when it closes the probe map's real temp file), ends the send with a WARNING that names it, the interrupt kept and no temp file left. - aGeneratedAppsTableIsMeasured asserts the growth over the heap just after its full collection that the paced send reaches: under 128 MiB, but under 192 MiB on G1 with a heap cap of 512 MiB or more on JDK 9-20 and under 448 MiB on JDK 8; the table within 60 s (scaled past 50,000 classes); and no frame more than 10 s after the one before. It fails with the pacer off (G1, JDK 21, 4 GiB pod: 175 and 343 MiB). Its sink now reads each frame in place, counting a match split over two frames: a String of a 1 MiB frame is itself a humongous object, and grew the heap the row measures. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
An interrupt ends a build table send, and the interrupt is kept for its thread. Before, Sends ran the request made during that send on the same thread with the interrupt still set. That request names the agent's newest index, which may not be read yet. On JDK 8 a class file's read fails on an interrupted thread (ClosedByInterruptException), so the index would keep only the classes it got, for the JVM's life (a wrong build id). The request then ended at its first unit with a second WARNING. - Sends hands the request made during an interrupted send to a new thread, uninterrupted, and ends the interrupted one. Sends stays running while that thread starts, so a request made meanwhile waits to run after it. If that thread cannot start, the request is dropped (and any request made while it tried) with the same WARNING as a first thread that cannot start, and the next request starts a thread. - send() checks the interrupt before it reads any class: on an interrupted thread it reads none, sends no frame, and logs the WARNING that names the interrupt. - Pacer takes the analysis's rate from begin(), after the index's read, not from the send's start. Before, a slow index read counted as the analysis's work, so the rate was overestimated and the first units went unpaced. The 50 s deadline still counts from the send's start, the index's read included. Rows: a request made while a send is interrupted runs on a new, uninterrupted thread; one made while that thread starts starts no thread of its own and runs after the first on that thread; one whose new thread cannot start is dropped with a WARNING, as is one made while it tried, and the next request runs; a send on an interrupted thread reads no class; after a 10 s read the first wait comes as soon as the allowance allows and the waits end by the deadline. Each fails on the code before this change. The Pacer's interrupt row clears the flag in a finally, so a failing row cannot leave it set for the rows after it. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
A native keploy test stops the app at each test set's end. A build table send still running then ended with no line in the app's log, so a run whose table never arrived could not tell why from the app's side. Two ways the SDK puts the table off were silent too. - An exit notice: while a send runs, a shutdown hook is held for it and removed when the send ends. If the JVM exits first, the hook prints one line to System.err: how many class files were analyzed of how many (or that the class files were still being read), the frames sent, the time since the send began and how much of it was paced. It writes to System.err, not java.util.logging, because shutdown hooks run at once and in no order, and the logging handlers may already be closed. A JVM that is killed (SIGKILL) or halted runs no hook and prints nothing. - A TABLE ask made while a send runs is logged at INFO, once per ask: it is sent again when that send ends, and an earlier ask that waited for it is dropped (the newest names Keploy's current socket). - A paced send logs one INFO line at its first wait: the collection time since the send began, the wait per millisecond of it, the most the waits may add up to for its class files, and the deadline no wait may push the table's end past. The Pacer now counts every analyzed class file, also when the pacing is off, so the notice can say how far the send got. Rows: a JVM that exits mid-send prints the notice exactly once; a JVM whose send ends before its exit prints none (each in its own JVM); an ask made during a send logs one INFO line each, the second naming the dropped ask, and only the newest ask runs; a paced send logs one line however many times it waits; a send that never waits logs none. The first, third and fourth fail on the code before this change. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
The build table's send was paced by owing 200 times all the collection
time since it began, the app's own collections included, and its waits
were bounded by 0.8 ms a class file (40 s at 50,000) and a 50 s
deadline. A busy app collects past that 0.5% goal for its own sake
(2.7% of the time while it allocates 400 MB/s), so every send in a busy
app waited out its whole allowance. A native keploy test restarts the
app at each test set, and each new JVM starts the send over: a
50,000-class app under test never got its table to Keploy within a set,
so the whole run had no coverage.
- The waits are drawn from the send's own work time, not from its class
files: each unit's wall time since the one before (its socket writes
included, its waits left out; the first unit's includes the index's
read) is what the thread may wait, at most a second of it saved up.
All the waits add up to at most the send's own work (plus one sleep's
overrun), so a paced table takes at most about twice as long as
unpaced, however much the app collects. No one wait is over a second,
as before, so frames keep coming.
- No wait may push the table's end past 20 s at 50,000 class files
(0.4 ms a class file past that), down from 50 s.
- The paced line says what is owed when the first wait is taken ("so it
owes N ms now (each ms of collection adds 200, and each ms the send
works or waits takes one off, never below 0)") and the bound.
- The exit notice says what it knows when the exit starts ("the JVM is
exiting while its build table is being sent. So far: ..."): the send's
daemon thread may still finish while the other hooks run. Its hook
thread is made inside the try, so a notice that fails never ends the
send, and a JVM already exiting when a send starts prints the line at
once.
- When the thread for a request that waited out an interrupted send
fails to start, the WARNING says that request is not sent.
Rows:
- A busy app's table: two threads allocate 400 MB/s, keeping a ring of
arrays, while the table is sent. Its waits stay within the send's own
work time (plus 0.5 s for one sleep's overrun), the table is whole,
within 60 s, with no frame more than 10 s after the one before. 5,000
classes by default, 50,000 with -Dkeploy.budget.classes. It fails on
the code before this change: at 50,000 classes on JDK 21 with -Xmx4g
and G1 the send took 43,498 ms, 39,991 of them paced, for 3,507 ms of
work; at 5,000 classes, 3,990 ms paced for 655 ms of work.
- The Pacer rows are recast for the work bound: the waits add up to at
most the work, and over any run of units to at most a second plus
those units' time; a unit may wait its own time, not its class files;
the deadline is 20 s; the Pacer counts its class files paced or not.
- A send removes its exit hook, after a whole send and after one whose
socket fails (it fails if the hook is left registered). The hook the
row reads is kept per thread, so the app keeps no send's buffers past
the thread that sent it. The paced line names the owed time, after an
index read longer than what the first unit owes, so the floor is hit
and nothing carries over. The rows read only their own thread's log
lines.
Measured at 50,000 generated classes, one run each, on JDK 21, 17 and 8,
G1 and Parallel, in 1 and 4 GiB pods (-XX:MaxRAM, no -Xmx):
- A quiet app: the table took 5.0-8.1 s (2.5-4.0 s of it paced; thread
CPU 2.4-3.7 s; the longest gap between frames 0.74-1.21 s). Committed
heap growth over the heap after a full collection, the row's bound in
brackets: JDK 21 G1 20 and 59 MiB (128), Parallel 54 and 23 (128);
JDK 17 G1 11 (128) and 145 (192), Parallel 33 and 32 (128); JDK 8 G1
79 (128) and 300 (448), Parallel 47 and 32 (128), in 1 and 4 GiB pods.
- A busy app (the same cells, plus -Xmx4g G1 on each JDK): the table
took 4.9-8.5 s, 2.4-4.2 s of it paced, each send's waits within its
own 2.5-4.3 s of work. The committed heap grew 0-280 MiB in the pods
and 0-816 MiB at -Xmx4g over the send's start; that is the app's own
allocation as much as the send's, and it is not asserted.
Signed-off-by: slayerjain <shubhamkjain@outlook.com>
A class whose probe map passes its bound (8 MiB of records, or 64 Mi walk steps) has no probe map, so no test's leaves can be named in it. The table sent it as not_measured record_too_large, the code for a table record past 8 MiB, so Keploy could not tell a record too large from a class too complex to map. It is now not_measured with the contract's code unit_too_complex. It has no methods, as any not_measured record, so Keploy leaves its lines out of the totals and never counts them as missed; the rest of the build is measured as usual. - record_too_large is now only a table record past 8 MiB with its newline. The send's INFO line counts the two apart: "N not measured as too complex to map (unit_too_complex: a probe map over 8 MiB or 64 Mi steps; ...)". - The table format stays 1: no released SDK writes a table, so the build-id row's pinned hash of the capped record is updated alone, as that row says to do before the first release. - A RuntimeException in Keploy's own steps of the class's one analysis (setting a method's instructions from JaCoCo's graph, and walking each probe; both under one wrap) is now NotMapped, with its cause. Before, an index error there was logged as "JaCoCo could not analyze" the class and was not counted with the probe maps that failed, though the class is the same analyzer_failed either way. JaCoCo's own steps (its instruction graph, filters and coverage calculator, with the hooks Keploy runs inside them) still throw as they do, and what the caller's callbacks (wants, and each probe) throw passes as it is, so a map past its bound stays unit_too_complex. A row that a throwing log handler fails no send counted every record published on the build table's logger, and expected two. A collection during its first send adds the paced line (INFO), so it failed once with three. It now counts the two sends' outcome lines, from its own thread. Rows: a probe map past its bytes or steps is unit_too_complex with no probe record and not record_too_large; the send's line counts it apart; a record past its bound is record_too_large and not unit_too_complex; an index error in the probe walk is NotMapped naming the class and method, while what the callbacks throw passes as it is. Each failed on the previous code except two: the record-too-large row is a guard (the previous code had no unit_too_complex to count), and the throwing-handler row is the fix of a row that failed by timing. In the NotMapped row only the walk's part failed there; its callback parts guard against wrapping them, and since the walk and the setting of instructions share one wrap, removing that wrap fails the row. The early-returns budget row now counts unit_too_complex. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
A class whose probe map passes its bound (8 MiB of records, or 64 Mi
walk steps) is sent as not_measured record_too_large again, as it was
before the previous commit, which sent it as unit_too_complex. Keploy's
readers (the CLI and the server's table ingest) accept only
analyzer_failed and record_too_large as a table reason, and refuse the
whole table over a record with any other code. So one class too
complex to map cost the run its coverage and dedup; with
record_too_large it costs that class only.
- The send's INFO line still counts such a class apart: "N not
measured as too complex to map (record_too_large: a probe map over
8 MiB or 64 Mi steps; the FINE log names each)". A table record past
8 MiB is counted as before ("not measured as too large").
- Keploy's reason text for record_too_large says the class is too
large; in this release it also covers a class too complex to map.
- Such a class's record is byte for byte what table format 1 wrote
before the previous commit: the build-id row's pinned hash of the
capped record is back to its earlier value.
- The rest of the previous commit stays: a RuntimeException in
Keploy's own steps of a class's analysis is NotMapped, and the
throwing-handler row counts only its own sends' outcome lines.
Rows: the build-id row (the capped record is record_too_large, its
pinned hash back), the bounds row (record_too_large at each bound,
counted as too complex to map and not as too large), the send's line
row (the class counted as too complex to map under record_too_large,
with no unit_too_complex and no "not measured as too large"), and the
early-returns budget row (counts record_too_large on the wire). Each
failed on the previous commit.
Signed-off-by: slayerjain <shubhamkjain@outlook.com>
slayerjain
force-pushed
the
feat/jcov-buildtable
branch
from
November 20, 2026 12:26
b449cc6 to
7d8d216
Compare
slayerjain
changed the base branch from
feat/java-dedup-app-classes
to
feat/dedup-sdk-220
November 20, 2026 12:26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this branch does today
Stacked on #193 (which is stacked on #192).
Keploy's per-test code coverage for Java needs the app's build table (packages, classes, methods, their lines and branch edges) and, for each class, which of those leaves a hit of each JaCoCo probe covers. Only the app's JVM holds the class bytes in every run mode, so keploy-sdk builds both there, and no class file leaves the app.
When it runs
TABLE, which Keploy sends once its first START is answered. The SDK answersACK proto=2at once and works on a daemon thread of its own: START and END never analyze, encode or send. An older Keploy never sendsTABLE, so it never gets a frame; an older SDK answers itIGNORED.TABLEwhile a send runs is logged at INFO and has that send run once more after it, so a new receiver can ask again. A request made during a send that an interrupt ended runs on a new thread.IGNORED proto=2 reason=low_heap heap_free_mb=F heap_needed_mb=N, logs a WARNING and sends nothing, so Keploy can mark the run not measured with that reason.KEPLOY_COVERAGE_TABLE_MIN_HEAP_MB(else-Dkeploy.coverage.table.min.heap.mb) sets the bound in MiB; 0 turns the check off.TABLE. The TCP transport sends no table.The table
One JaCoCo pass per class file gives both the table and the probe map. The analyzer is the JaCoCo core keploy-sdk.jar carries (0.8.15, relocated).
One record per top-level class, with its nested, inner, anonymous, lambda and Scala module classes folded in by name. A method has one code leaf per line and one leaf per branch edge.
Records are in the ingest contract's order and written as Keploy's Go encoder writes them. Golden rows check the bytes against the Go encoder's output on the java-dedup sample's classes and on generated classes.
The build id is
jv1-and the first 32 hex of the sha256 over"keploy-java-table " F "\n" "jacoco " V "\n"followed by the sorted class ids (JaCoCo's CRC64, unsigned order, de-duplicated, 16 lowercase hex), joined by newlines. F is the table format (1) and V the analyzer's JaCoCo release. The INDEX record takes its build id from the same function, so the two always match, and an SDK release that writes a different table for the same classes names a different build.A unit is
not_measured, with no methods and no probe record, when:analyzer_failed: JaCoCo cannot analyze it, or its probes cannot be mapped exactly (this includes two class files of one name whose methods or probes differ);record_too_large: its table record is over 8 MiB, or it is too complex to map: its probe map is over 8 MiB or 64 Mi walk steps (a method of about 3,000 early returns, as generated parsers have). The SDK's INFO line counts the two apart.Keploy leaves such a unit's lines out of the totals and never counts them as missed. The rest of the build is measured as usual.
The probe map
On the wire
TABLEframes on the data socket, one frame per connection, each at most 1 MiB:KTB1, a 4-byte big-endian length, a JSON header line ({"v":1,"part":"table","build":"jv1-...","seq":k,"last":false}; the last frame addsrecordsand the stream'ssha256), then the next bytes of the NDJSON stream. The probe map follows the table's last frame as a second stream,"part":"probes", seq from 0.OK. A receiver that closes without it ends the send, with a WARNING naming the frame, and the table goes again only when Keploy asks again.Memory and CPU
java.io.tmpdir, unlinked once open on Unix, and is streamed after the table. A send holds the class list, one frame and one class's analysis. A tmpdir the app cannot write ends the send before any frame, with a WARNING naming it.Logs
-g).Limits (fail closed)
keploy testrestarts the app at each test set; a set that ends before its table arrives is not measured (its tests are kept), and the next JVM sends the table again.-Xmx, so this matters only for a pod whose-Xmxis larger than its memory limit.record_too_large, as Keploy's readers (the CLI and the server) accept onlyanalyzer_failedandrecord_too_largeas a table reason. Keploy's reason text for it says the class is too large to record: in this release that text also covers a class too complex to map.Part of keploy/enterprise#3435 (code coverage and dynamic dedup for large Java apps). Draft: this branch is updated at milestones while the work lands; it is marked ready for review when its checks pass. Do not merge before then.