Repository navigation
feat(dedup): java-sdk 2.2.0: faster, steadier per-test coverage - #193
Draft
slayerjain wants to merge 14 commits into
Draft
slayerjain wants to merge 14 commits into
slayerjain wants to merge 14 commits into
Conversation
slayerjain
force-pushed
the
feat/dedup-sdk-220
branch
from
November 19, 2026 19:50
4f4a68b to
29d0b60
Compare
Every END built a new set of the app's class names from the index, to keep only the app's classes from JaCoCo's dump. The index never changes once it is loaded, so the set is now built once. In a profile with 50k generated app classes, building it again took 1.4 to 2.2 ms of each test. KEPLOY_JAVA_DEDUP_DIAGNOSTICS now logs one line per START and END: how long the command took, and how long each step did. A START has warmup and reset. An END has the JaCoCo dump, the parse, the filter to the app's classes, the JSON and the send, plus the check for new classes (and the class-zip frame) on TCP. A step only takes the time when it ends; the line is formatted once the command is done, so no step is charged with formatting. A run on the real app then says where its per-test time goes. With diagnostics off nothing is logged and no step is timed. The warmup's log line now says how long the warmup took. The README lists both the diagnostics and the warmup switches. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
With JaCoCo's agent in the app's JVM, each test took three passes over the probe arrays of every class JaCoCo had seen, library classes too: the START's reset, then the END's dump and its reset. On large apps those passes are most of the SDK's time per test. A START and an END now make one pass each, over the probe arrays of the app's classes only, read in place from JaCoCo's runtime: - The arrays are kept sorted by class name and added to only when JaCoCo's store has grown (a class was loaded); a pass with no class loaded since does not walk the store. - The pass is split across min(4, cores / 2) threads, the caller's included, once there are 2048 classes or more for each; the others are daemon threads that stop after a minute idle. A part no thread can be made for is scanned on the caller's thread. - A probe read as hit is noted and cleared in the same step, so a hit made while the pass runs is in that pass or the next one. JaCoCo's dump then reset loses a hit made between the two. - What a START clears ran between two tests (a health check, a scheduled job, the warmup's static initializers): it is background, never counted in a test's hits. With diagnostics on, the START logs how many classes and probes it was. It is not sent to Keploy: no frame of the current protocol carries it. - Library classes are neither read nor cleared. JaCoCo's runtime is reached by reflection, through the types of its fields, as the agent shades them under a package of its own. When JaCoCo is reached over TCP, or its agent is not laid out that way, the SDK uses JaCoCo's reset and dump, as before. When a pass fails, the SDK uses them for the rest of the run. A pass that failed before it cleared a probe leaves the test's hits to JaCoCo's dump. One that failed after (an Error in a part, such as out of memory) has lost hits, so its END is not taken; it waits for every part to end first. Of two classes of one name (loaded twice, from other bytes), the hits sent are those of the one the coverage manifest names when that one ran, else those of the lowest id, as JaCoCo's dump sends a name's hits whichever copy made them. The diagnostics line names the new step `scan`. The smoke test checks that the real JaCoCo agent is read in place, both one the command attaches and the one keploy-sdk.jar starts itself. Rows: a hit is reported once and cleared; background never reaches the next test; a class loaded after a pass is scanned by the next; order and completeness across four parts; a writer racing the passes has each of its 16384 hits reported exactly once, over at least two passes for each 512 writes; a failed part ends the pass only once every part has; a part the pool cannot take is scanned by the caller; an Error reading JaCoCo's classes clears nothing; agents not laid out as JaCoCo's are refused; and, through the control socket, the background is left out, a class loaded twice is reported once, as the copy the manifest names when it ran and as the other copy when only that one ran, a pass that lost hits leaves its END not taken, an Error reading JaCoCo's classes or an agent's fields keeps the server and falls back, and a failed pass falls back to JaCoCo's dump and reset for good. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
The first START runs the static initializers of the app's own classes,
so that no test is charged with their lines. It looked the classes up
through the control thread's context loader, the system loader and the
SDK's own loader. A Spring Boot fat jar's classes are reachable only
through the loader its launcher makes, so the warmup initialized none
of them ("initialized=0") and the first tests were charged with them.
It also ran on the START's own thread, so the START was answered only
once every class was initialized, however long that took; Keploy's k8s
collector gives up on an answer after 30 s by default.
The warmup now:
- finds the app's loaders: those that defined the app's classes already
loaded (from premain's Instrumentation), the one that defined the most
first; then the live threads' context loaders, the oldest thread first
(a launcher sets its main thread's to the app's loader), read from the
root thread group, which pauses no thread; then the system loader and
the SDK's own. A class already loaded is initialized in the loader
that loaded it; of a class loaded in two loaders, in the one that
defined more of the app's classes. A class not loaded yet goes on to
the next loader only when a loader cannot find it: one that a loader
found but could not link or initialize is left to lazy init, never
loaded again in another loader, where its static initializer would
run twice. Each class is initialized with the thread's context loader
set to the loader it goes through, as on the app's own threads.
- runs on a thread of its own, which also reads the app's classes when
premain has not. The first START waits for it for at most 10 s, then
goes on, and the warmup stops after the class it is initializing. The
WARNING then names that class (or the step before the classes). Its
static initializer goes on while the tests run, and each line it runs
from then on counts in the test running at the time (between two
tests, in none). The classes it did not reach initialize when
the app first uses them, as with the warmup off, and the log says how
many ("not reached=N"). If the thread cannot start (a pids limit), the
START goes on with no warmup and says so.
Fat jars are now warmed up for the first time: every app class not
loaded yet is loaded, instrumented by JaCoCo and initialized at the
first START, as other apps' classes already were. On an app with many
classes it never uses, that is more metaspace than the app alone would
take. KEPLOY_JAVA_DEDUP_WARMUP_DISABLED=true turns the warmup off.
It still runs only when the index is the app's own classes, and never
initializes a library class itself. When premain has not read the app's
classes (a JaCoCo agent attached separately), the first START's scan
still waits for that read, as it needs the class set.
Rows: a fat jar's classes warmed up through its launcher's loader;
premain's Instrumentation finding the loader of the app's loaded
classes with no thread holding it; a class the app loaded in a second
loader initialized there; of two copies of a class, the copy of the
loader that defined more initialized, whichever the JVM lists first; a
class whose static initializer throws run once, not again in a second
loader that sees it; the loaders' order; every live thread seen from
the root group; a slow static initializer holding the first START only
for the wait, the WARNING naming it, the warmup stopped after it and a
second START not waiting; a warmup thread that cannot start leaving the
START and the control server working; the WARNING naming each step; an
empty class list asking the JVM nothing.
Signed-off-by: slayerjain <shubhamkjain@outlook.com>
What a START clears ran between two tests: a health check, a scheduled
job, a cache refresh. It never counts in a test's hits, but the same
threads also run while a test does, so their classes' probes land in
some tests and not others from one run to the next. Keploy can leave
those probes out of what it compares only if it knows them.
Keploy now asks with the line BACKGROUND on the control socket. The SDK
answers ACK when it reads JaCoCo's probes in place and a receiver is on
the data socket, else IGNORED, as an older SDK answers a line it does
not know. From then on, each START that clears hits made since the last
test's END (an END answered ACK) sends one record on the data socket,
right after that START is answered, after the INDEX record when that
receiver does not have it yet:
{"kind":"background","v":1,"build":"<the INDEX's build id>",
"after":"<test that ended>","before":"<test that starts>",
"executedLinesByFile":{class:[probe...]}}
plus "lost":N when the records of the N windows before it could not be
made or sent (each is tried once; one WARNING while the sends fail).
The probes are under a test's record's key and in its shape, so Keploy
reads both alike. Of two copies of one class, the copy the coverage
manifest names, else the lowest id.
- Not background, never sent: what ran before the first test (the app's
start, the warmup), and what ran after an END that was not taken, or
after a START whose END never came (a test's own hits).
- A test's own work that goes on after its END (an async task it
started) lands in the next window and is reported as background; the
record names both tests so that Keploy can weigh that.
- The ask holds for one run of the app and one receiver: an app started
again (native keploy test does so for each test set) and a receiver
bound anew on the data socket's path (the next docker-compose test
set) get none until Keploy asks again, so Keploy asks after each INDEX
record. A Keploy that never asks gets none, short of a new receiver
bound within one file-system clock tick of the one that asked
(receivers are told apart by the socket file's inode and modification
time, and a socket file's modification time is its bind time).
- If the in-place read fails after the ask, the records stop (the read's
WARNING says so): Keploy then leaves fewer probes out, never more.
- No BACKGROUND on the TCP transport used in Kubernetes.
KEPLOY_JAVA_SCAN=off (any case) turns the in-place read off: each test
takes JaCoCo's reset and dump, as when JaCoCo is reached over TCP, and
BACKGROUND is answered IGNORED.
Rows (InProcessScanTest): a window's hits reach a receiver that asked
once, and never a test; a class loaded twice is reported as a test's
record would; no record for a receiver that did not ask; none after a
START with no END, an END for another test (with or without a START
before it), or an END whose record could not be sent (skipped as root);
a receiver bound anew gets none until it asks, and the INDEX record
before its first window; with no receiver BACKGROUND is IGNORED and a
window goes nowhere; records that could not be sent are counted in the
next, said once, kept across a re-ask by the same receiver, and not
carried to a new receiver (skipped as root); with the scan off each
test takes JaCoCo's dump and BACKGROUND is IGNORED; any other value
keeps the scan.
Signed-off-by: slayerjain <shubhamkjain@outlook.com>
Keploy reads a test's probes against the classes the SDK indexed: the
class files it read from the app's dirs and jars. When a class loader
that weaves the bytes it defines (load-time weaving), or an agent
listed before JaCoCo's that does not ask to retransform, changed a
class before JaCoCo saw it, or the JVM loaded a multi-release jar's
versioned copy, JaCoCo's probes are those of other bytes, and their
indices do not belong to the indexed class. Only the SDK holds both
ids: JaCoCo's (ExecutionData's id) and the CRC64 of the indexed bytes.
A test's record now names each such class after its probes:
{"id":"<test>","executedLinesByFile":{class:[probe...]},
"classMismatch":[class...]}
The key is left out when there is none, so a test that ran one copy of
each class, the one the SDK indexed, sends the same record as before,
and receivers that do not know the key ignore it (Keploy's Go receivers
decode it with encoding/json). The CRC64 of a class's indexed bytes is
worked out, and kept, the first time a record reports the class.
A BACKGROUND record names such classes in a classMismatch of its own,
left out when empty: Keploy takes background out of a test's hits
probe by probe, so it must know which probes are not the indexed
class's.
Of two copies of one class (loaded twice, from other bytes) that a test
ran, the record now reports the copy the SDK indexed, else the copy the
coverage manifest names, else the lowest id compared as an unsigned
number (before: the manifest's, else the lowest id compared as a signed
long). A BACKGROUND record chooses alike. JaCoCo's dump path
(KEPLOY_JAVA_SCAN=off, or JaCoCo reached over TCP) now goes through the
same choice: before, of two copies it kept whichever the dump listed
last, and added the first to the manifest.
Every class id the SDK writes (the coverage manifest's id) is now 16
lowercase hex digits, zero-padded, as the INDEX record's build id
already hashes them. Older SDKs wrote Long.toHexString's unpadded form,
so readers parse the id as a hex number.
Rows: ids as 16 hex digits; the manifest's id over the TCP transport,
with classMismatch in its COV; a class JaCoCo saw other bytes of named,
and none for a test that ran only indexed classes; of two copies, the
indexed one over the manifest's and over a lower id, the manifest's
over a lower id (also when its id has fewer than 16 hex digits), the
lower id compared unsigned; JaCoCo's dump path choosing alike and
leaving library classes out; a BACKGROUND record naming a copy JaCoCo
saw other bytes of, and none for the indexed copy; the record's shape
with and without the key. The Java agent smoke test now fails a record
that names any class when the app ran only its own, and runs a test in
which only another copy of a class runs (other source, its own class
loader), whose record must name that class alone: on the in-place read
with the JaCoCo agent the command attaches and the one keploy-sdk.jar
starts, and on JaCoCo's dump (KEPLOY_JAVA_SCAN=off).
Signed-off-by: slayerjain <shubhamkjain@outlook.com>
slayerjain
force-pushed
the
feat/dedup-sdk-220
branch
from
November 19, 2026 23:29
af7d94d to
9d04275
Compare
keploy-sdk.jar carries JaCoCo (EPL-2.0), the ASM it depends on (BSD 3-Clause), Gson and junixsocket (Apache 2.0). Its META-INF/THIRD-PARTY-NOTICES.txt named them, but the jar held neither license text the notices relied on: not the Apache License 2.0, which keploy-sdk, Gson and junixsocket are under and whose section 4(a) asks for a copy with each redistribution, nor the EPL-2.0, named only by its URL. junixsocket's NOTICE attribution was missing too. - META-INF/LICENSE is the repo's LICENSE (Apache 2.0), copied in at build time, so the jar's and the repo's cannot differ. - META-INF/LICENSE-EPL-2.0.txt is the EPL-2.0 as eclipse.org publishes it. - THIRD-PARTY-NOTICES.txt points to both. It now says that keploy-sdk's own code is under the Apache 2.0 and the third-party software stays under its own license, where keploy-sdk's source is, and the build that renames the packages; names org.jacoco.agent's AgentJar class, which the jar carries outside jacocoagent.jar and under its own name; adds org.jacoco.core's sources on Maven Central to JaCoCo's source pointer; gives Gson's and junixsocket's source tags, Gson's copyright line and junixsocket's NOTICE text; and says which about.html covers which JaCoCo part. - org.jacoco.core and org.jacoco.agent both hold an about.html at their root. The shade plugin kept the first it met, core's, and dropped the other with a warning. It now leaves out org.jacoco.agent's, the same file as the one inside jacocoagent.jar, which the jar carries whole, so which one lands no longer rests on the dependencies' order. Rows: ThirdPartyNoticesTest checks that the notices name the versions the build resolves (JaCoCo's core and agent, with AgentJar; ASM with asm-commons and asm-tree; Gson; junixsocket), so a dependency bumped without its notice fails CI's unit tests; that they quote the attributions the licenses ask for (the copyright lines, ASM's three conditions, the start and the damages sentence of its disclaimer, and the first and last lines of junixsocket's NOTICE); and that both license texts sit next to the notices. scripts/smoke-javaagent.sh checks that the built jar holds the repo's LICENSE, the EPL-2.0 text and the notices as the sources hold them, and org.jacoco.core's about.html; that org.jacoco.agent's about.html is the one inside jacocoagent.jar; and that each part of the jar maps onto a part the notices name, so a new package, or a dependency from another project, shaded in without its notice fails. Another artifact of ASM or junixsocket maps onto its project's part, whose license the notices already carry. The README gets a License section. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
A server sends the response before it is done with the request: a filter's code after the chain (a request log, an authentication filter's cleanup) runs after Keploy has the response and has sent END. Under load END's pass ran first, that code landed between two tests, and the next START cleared it as background, so the same test got different coverage from run to run. The SDK now counts the requests of Tomcat (CoyoteAdapter.service and asyncDispatch), Jetty (HttpChannel.handle, and on Jetty 12 and 12.1 the servlet channel's handle, matched by name) and Undertow (Connectors.executeRootHandler). As each class loads, that method becomes one that counts the request in and out of RequestGate around the server's own code, which moves to a private keploy$request$<name>. A class whose loader does not see the agent's own RequestGate is left unchanged, and so is any class loaded before the agent started. The JaCoCo keploy-sdk.jar starts adds its transformer first, so it ids the server's class as the index read it, and a server in a named module is counted too (the JVM lets a module an agent changed read the class path). END waits until every request that came in since the test's START is done, for at most KEPLOY_JAVA_END_WAIT_MS (200 ms by default, 10 s at most; 0 turns the wait and the class change off). A request already done costs END one read; one from before the START, or one that comes in once END has come (a probe, a scrape), is never waited for. Past the bound, a WARNING names the test, its record ends with "windowCut":true (after classMismatch, left out when false), and the next START opens no BACKGROUND window, since the cut test's late hits are in it. Limits: code on the app's own threads (an async task) is not counted, nor are Netty and WebFlux; the log says once when END does not wait, and why. A JaCoCo agent attached after keploy-sdk.jar (the agents in JAVA_TOOL_OPTIONS start before the command line's) sees the changed server class, so the README's examples list JaCoCo first. Rows: RequestGate's count and wait; the server entries on stand-ins of each server's class (arguments of every type, static, synchronized, a pre-Java 6 class, abstract/native/bridge/synthetic methods, interfaces, loaders that do not see the counter); END's wait, cut and no-window on the unix socket and on TCP; and real Tomcat 9, Jetty 9.4 and Undertow 2.2 in JVMs of their own with the agent, where a filter's code that runs after the response is in the test's record, missing with the wait off, and cut past the bound; Tomcat on the module path; and the embedded JaCoCo measuring Tomcat's own class with no classMismatch for it. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
keploy-sdk.jar changes the method each server runs a request in (Tomcat's CoyoteAdapter, Jetty's HttpChannel or servlet channel, Undertow's Connectors) with a transformer that could not retransform, so the JVM ran it in the order the agents were attached. A JaCoCo agent attached after keploy-sdk.jar, or keploy-sdk.jar in JAVA_TOOL_OPTIONS with JaCoCo on the command line, then instrumented and ided the changed class, and every test's record named it in classMismatch whenever the classes Keploy measures include the server's (an uber jar, or every class on the class path). The requests were counted in every order. The transformer can now retransform (Can-Retransform-Classes: true in the jar's manifest). The JVM runs such transformers after every one that cannot, JaCoCo's among them, so JaCoCo instruments and ids the server's own class in any order. When another agent retransforms a class the SDK changed as it loaded, the JVM passes the class as it was before the SDK's change; on a redefine it passes the redefiner's bytes. The SDK changes either again, since the JVM refuses a class that drops a method, unless the bytes already carry the change (a copy taken from the class as the SDK changed it), which the JVM then takes as they are. A redefine with the server's own bytes, which the JVM refused before, now succeeds when no other agent changed the class as it loaded (JaCoCo does not redo its own change on a redefine). A class loaded before the SDK started is left as it is, since the JVM refuses a method added to a loaded class too. A jar that carries the SDK without the manifest entry still counts the requests, as before, and when the JVM has a JaCoCo agent of its own it says at WARNING that one attached after it is then named in classMismatch. KEPLOY_JAVA_END_WAIT_MS now caps at 2000 ms instead of 10000: a longer wait per test costs more than the SDK's overhead target. The JVM now calls the SDK's transformer in a second pass over each class it loads: one call into the agent with a copy of the class's bytes, then one map lookup. With KEPLOY_JAVA_END_WAIT_MS=0 no class gets that pass, but the JVM still holds the retransform capability the manifest asks for, as with any agent that can retransform. Rows, on a real Tomcat in a JVM of its own: JaCoCo's own agent attached before keploy-sdk.jar, after it, and after it with keploy-sdk.jar in JAVA_TOOL_OPTIONS, each measuring CoyoteAdapter with no classMismatch for it (the two later orders named it before this change); a jar without Can-Retransform-Classes counting, with its WARNING when JaCoCo is attached after it and none with the SDK's own JaCoCo; another agent's retransform of CoyoteAdapter succeeding and the requests still counted; a redefine of CoyoteAdapter with a dump of it as the SDK changed it succeeding, the requests still counted; and a retransform of a CoyoteAdapter loaded ahead of the SDK succeeding, the server left uncounted. Unit rows: the change made again on a retransform for the class changed as it loaded, and for no other; a retransform it cannot make again refused with a WARNING; bytes that already carry the change taken as they are on a redefine (changed twice, the JVM refuses the class) and counted as they are when a class loads with them; the 2000 ms cap. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
The rows that attach JaCoCo's own agent after keploy-sdk.jar write an agent jar of their own, so none read the manifest the release ships: a keploy-sdk.jar built without Can-Retransform-Classes: true passed every row, while a JaCoCo agent attached after it would name the server's class in classMismatch. scripts/smoke-javaagent.sh now runs the built jar on a real embedded Tomcat 9.0.98, with JaCoCo's own agent attached after it and the index holding Tomcat's classes, as an uber jar's does. Its servlet runs more code after the response once the test's END waits for the request. The case fails unless the INDEX record says the attached JaCoCo measures and names Tomcat's jar among its roots, the test's record measures CoyoteAdapter and that later code and names no class in classMismatch, the log says the SDK counts Tomcat's requests, the END's diagnostics show the wait for them, and no WARNING names Can-Retransform-Classes. A failed check ends the harness's JVM with 1 (Tomcat's threads are not daemons), its sockets and HTTP reads time out, and the case is bounded at 180 s, so a regression fails the job instead of hanging it. With the manifest entry set to false, the case fails on JDK 21 and on JDK 8. ServerEntriesTest wraps JaCoCo's instrumented copy of a stand-in of CoyoteAdapter at class versions 52, 55, 61 and 65, as far as the JVM can define them: JaCoCo keeps its probes in a field at 52 and reads them through a dynamic constant from 55 on. The wrapped copy loads through the verifier, counts its request, and reports its probes under the CRC64 of the class's original bytes, the id the index holds, not the id of the wrapped bytes. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
The verify job had no timeout, so a hung build, unit test or smoke case held its runner for GitHub's default of 6 hours, up to three times over across the JDK 8, 17 and 21 legs. Timed on 4 CPUs (the size of a GitHub-hosted runner) with an empty Maven repository per JDK, the cache-miss worst case, the job's three steps as written take 116 to 121 seconds per JDK: build 25-29 s, unit tests 60-63 s, smoke 28-30 s. The smoke already stops its Tomcat case after 180 s. A job that reaches 10 minutes, about five times the slowest measured run, is stuck rather than slow, and now fails there. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
keploy-sdk.jar carried org.jacoco.agent's AgentJar class under its own name and jacocoagent.jar at the jar's root, and EmbeddedJacoco read the agent jar through AgentJar. An app that ships its own org.jacoco.agent met the SDK's copy in either order: - the app's jar ahead of keploy-sdk.jar on the class path (a Jib launch): EmbeddedJacoco's AgentJar was the app's, and the SDK started the app's JaCoCo. With org.jacoco.agent 0.7.9 on JDK 21 that agent failed to start and every test's coverage was empty. - the app's jar in a loader that asks the system class loader first (Spring Boot's BOOT-INF/lib): the app's AgentJar was keploy-sdk.jar's, and read keploy-sdk.jar's jacocoagent.jar, not the app's. org.jacoco.agent is now a test dependency, not shaded in. The build unpacks only its jacocoagent.jar, byte for byte, to io/keploy/dedup/jacocoagent.jar, which EmbeddedJacoco reads from the loader that loaded keploy-sdk and names in an IllegalStateException when it is missing. keploy-sdk.jar holds none of that artifact's classes, and its about.html no longer needs a shade filter. The build unpacks it under target/jacocoagent and copies it into target/classes only when it is missing there or older, so a cleared target/classes gets it back and a build without clean leaves keploy-sdk.jar as it was. THIRD-PARTY-NOTICES.txt names the jar's new place and drops AgentJar. Rows: the carried jar is org.jacoco.agent's, byte for byte, and none sits at the root of keploy-sdk's classes; with an app's own jacocoagent.jar ahead of keploy-sdk, EmbeddedJacoco still reads the SDK's; a loader without it is the named error. The notices name the jar's place and no AgentJar. scripts/smoke-javaagent.sh checks the carried jar against org.jacoco.agent's, maps it as a part of its own, and adds org.jacoco.agent 0.7.9 to the app's own libraries: ahead of keploy-sdk.jar, the INDEX record must name JaCoCo 0.8.15; in the app's own parent-first loader, AgentJar must load from the app's jar and read the app's jacocoagent.jar. Both smoke cases fail on the jar before this change. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
keploy-sdk and keploy-junit5 run on Java 8, and both are released from JDK 17 with source and target 1.8. Those set the language level and the class file version, not the API the code is checked against: javac on JDK 17 compiles against JDK 17's own classes. A call to a method Java 8 lacks (java.util.List.of()) then fails on Java 8 only, in the released jar, with a NoSuchMethodError. CI's JDK 8 legs build on JDK 8 and catch that one, but not a Java 8 method that JDK 9 overrode with a narrower return type (ByteBuffer.position(int), flip()): that call compiles on every JDK, and on JDK 17 it refers to ByteBuffer's own position(int), which returns ByteBuffer and which Java 8 does not have. No CI leg runs a jar built on JDK 17 on Java 8. Each pom (the root's, for keploy-sdk, and keploy-junit5's, a project of its own) gets a profile, active on JDK 9 and later, that sets maven.compiler.release to 8: javac then checks main and test code against Java 8's API, and a call resolves as it would on JDK 8. The release profile does not turn it off (only activeByDefault profiles yield to -P). JDK 8's javac has no --release, and needs none: there the build is unchanged. Today's code has no such call: on JDK 17 both projects' main classes come out the same, instruction for instruction, with the profile and without it. Java8ApiTest, in keploy-sdk's rows and keploy-junit5's, checks that a call to ByteBuffer.position(int) refers to Buffer's method, as on Java 8. On JDK 17 and 21 it fails with the profile off, and on JDK 17 with release 9. Checked on JDK 8, 17 and 21, for both projects: the tree compiles to class file version 52; a planted java.util.List.of() fails to compile, with and without -P release; a planted ByteBuffer.position(0) compiles to ByteBuffer.position:(I)Ljava/nio/Buffer;, Buffer's method. On JDK 17 and 21 with the profile off, both compile, and the second to ByteBuffer.position:(I)Ljava/nio/ByteBuffer;, which Java 8 lacks. On JDK 17 the release builds of both projects (javadoc jars included, signing skipped) pass, and Java 8 runs scripts/smoke-javaagent.sh on the keploy-sdk jar built there. keploy-junit5's rows pass on JDK 8 with JUnit 5.10.5, JDK 17 with 5.14.4 and JDK 21 with JUnit 6.1.3. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
The java8-api profile sets maven.compiler.release to 8, which only maven-compiler-plugin 3.6 and later read. Neither the root pom nor keploy-sdk's named a version of that plugin, so keploy-sdk was compiled with the one the running Maven binds by default: 3.15.0 on the Maven of CI's runners today, 3.13.0 on Maven 3.9.9, and 3.1 on Maven 3.8 and older. 3.1 has no release parameter and ignores the property without a warning. A keploy-sdk jar built there on JDK 9 or later with -DskipTests could again carry a call Java 8 lacks, and nothing failed. With the tests on, Java8ApiTest fails, so CI would catch it; a package build would not. The root pom now pins 3.15.0 in its pluginManagement, which keploy-sdk inherits. That is the version CI's runners, which also build the release, bind today, so CI's and the release's compile do not change. keploy-junit5's pom keeps its own pin, 3.13.0. Other builds of keploy-sdk move to 3.15.0: from 3.1 on Maven 3.8, which is the fix checked below, and from 3.13.0 on Maven 3.9.9, where its main and test classes come out byte for byte the same from both on JDK 8, 17 and 21. A second row in each Java8ApiTest reads the pom (keploy-sdk's, then the root's; keploy-junit5's own) and fails when it names no compiler plugin version, or one before 3.6 such as 3.1 or 3.5.1. It also fails when a pom it reads is not the one it expects: run outside Maven from another directory, it would read another project's pom and could pass on that one's pin. On Maven 3.9 nothing else fails without the pin: the default binding hides it. Checked with Maven 3.8.8 on JDK 17 and 21. Without the pin, keploy-sdk compiles with compiler 3.1: a java.util.List.of() planted in its main code builds with -DskipTests, a planted ByteBuffer.position(0) refers to ByteBuffer's own position(int), which Java 8 lacks, and both Java8ApiTest rows fail. With the pin, compiler 3.15.0: the first fails to compile, the second refers to Buffer's method, and the rows pass. keploy-junit5 builds with 3.13.0 on Maven 3.8.8 as before. java-agent.yml's matrix (JDK 8, 17 and 21, KEPLOY_PROVEN unset and =1, the smoke), junit5-sdk.yml's three legs and release.yml's verify on JDK 17 pass. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
release.yml builds keploy-sdk with the release profile and smoke-tests the jar of that clean build, then deploys to Maven Central from a second pass over the same target/, with the profile and no clean. If that pass changes the jar's bytes, Central gets a jar other than the one tested, and CI does not notice: the Java Agent job never packaged twice, and release.yml states the rule only in a comment. The job now builds as release.yml does: with the release profile, and with signing skipped by -Dgpg.skip, since CI holds no release key. It records the sha256 of keploy-sdk's jar and of its sources jar. Its last step runs that build again without clean and checks both digests. Shade writes the jar anew on every pass, from the same entries and times, so what the check holds is the bytes. The unit-test step between them stops at the test phase, but what it changes in target/classes would reach the jar in that last pass, so it is checked too. The javadoc jar is left out: every build makes it anew, with the time in its pages, as release.yml already says of it. The two new steps run with `shell: bash`, so a command that fails inside a pipe fails the step. Checked: on JDK 8, 17 and 21 every step of the job passes, the unit tests with KEPLOY_PROVEN unset and =1, and both jars keep their digests through both unit-test passes, the smoke and the build without clean. On JDK 17 the unit tests pass and the new step fails in two cases: with copy-jacocoagent set to overwrite the file on every build, and with a plugin under keploy-sdk's release profile that touches a file in its classes on every build. The second passes a check run without the release profile. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What this branch does today
Stacked on #192 (it includes that PR's INDEX record commit).
Per-test speed
KEPLOY_JAVA_DEDUP_DIAGNOSTICS=1logs one line per START and END: how long the command took, and each step (scan, ordump/parse/filter, thenjsonandsend).Warmup (the static initializers of the app's own classes, run once on the first START)
-javaagent's Instrumentation), the one that defined the most first; then the live threads' context loaders, the oldest thread first (a launcher sets its main thread's to the app's loader), read from the root thread group, which pauses no thread; then the system loader. Before, it used only the control thread's context loader and the system loader, so on a Spring Boot fat jar, whose classes only its launcher's loader can load, it initialized none of them and the first tests were charged with their lines.KEPLOY_JAVA_DEDUP_WARMUP_DISABLED=trueturns the warmup off.Background hits
BACKGROUNDon the control socket. The SDK answers ACK when it reads the probes in place and Keploy's receiver is on the data socket, else IGNORED (as an older SDK answers a line it does not know). From then on, each START that clears hits made since the last test's END (ACKed) sends one record on the data socket, right after that START is answered (after the INDEX record when that receiver does not have it yet):{"kind":"background","v":1,"build":"<the INDEX's build id>","after":"<test that ended>","before":"<test that starts>","executedLinesByFile":{class:[probe...]}}, the probes under a test's record's key and in its shape (with itsclassMismatch, left out when empty), plus"lost":Nwhen the records of the N windows before it could not be made or sent.keploy testdoes so for each test set) and a receiver bound anew (the next docker-compose test set) get none until Keploy asks again, so Keploy asks after each INDEX record. A Keploy that never asks gets none, short of a new receiver bound within one file-system clock tick of the one that asked (receivers are told apart by the socket file's inode and modification time, and a socket file's modification time is its bind time).afterandbeforeso that Keploy can weigh that.KEPLOY_JAVA_SCAN=offturns the in-place read off: each test takes JaCoCo's reset and dump, and BACKGROUND is answered IGNORED.A test's requests are done before its END reads its hits
CoyoteAdapter.serviceandasyncDispatch), Jetty (HttpChannel.handle; on Jetty 12 and 12.1 the servlet channel'shandle, matched by name) and Undertow (Connectors.executeRootHandler), Spring Boot's three servers. Each of those methods becomes one that counts the request in and out around the server's own code, which moves to a privatekeploy$request$<name>beside it. Only classes loaded after the agent starts change (the JVM refuses a method added to a loaded class), and only when their class loader sees the agent's own copy of the counter. The SDK's transformer is retransform-capable, so the JVM runs it after JaCoCo's, which is not, whatever the-javaagentorder: a JaCoCo agent attached before or after keploy-sdk.jar, or inJAVA_TOOL_OPTIONS, instruments and ids the server's own class, soclassMismatchnever names it for that. When another agent retransforms or redefines a class the SDK changed, the SDK changes it again (the JVM would refuse the change otherwise), unless the new bytes already carry the change (a copy taken from the class as the SDK changed it), which the JVM then takes as they are; a server class loaded before the SDK started is left as it is. A jar that carries the SDK withoutCan-Retransform-Classes: truestill counts the requests, in the order its transformer was added, and, when the JVM has a JaCoCo agent of its own, says at WARNING that a JaCoCo attached after it is then named inclassMismatch. A server in a named module (its jars on the module path) is counted too: the JVM lets a module whose class an agent changed read the class path. The JaCoCo keploy-sdk.jar starts adds its transformer before this one, so it instruments, and ids, the server's class as the index read it.KEPLOY_JAVA_END_WAIT_MS(200 ms by default; a value above 2000 waits 2000 ms, said once at WARNING: a longer wait would cost each test more than the overhead target). A request already done costs END one read. A request that came in before the START (a stream, the request of a test cut before), or once END has come (a readiness probe, a metrics scrape, or a call the test's own code makes after its response), is never waited for."windowCut":true(afterclassMismatch, left out when false, as the receiver side agreed), and what the next START clears is not sent as a BACKGROUND window, since the cut test's own late hits are in it. The record still holds what was read at the cut. A Keploy that reads the flag writes that test as not measured; an older Keploy judges the hits the record holds.KEPLOY_JAVA_END_WAIT_MS=0turns the wait off and leaves the servers' classes unchanged.Failure behaviour (fail closed)
Class ids, and classes JaCoCo saw changed
"classMismatch":[class, ...]after its probes, each class whose hits come from bytes other than those the SDK indexed: a class loader that weaves the bytes it defines (load-time weaving) or an agent listed before JaCoCo's that does not ask to retransform changed it before JaCoCo saw it, or the JVM loaded a multi-release jar's versioned copy. JaCoCo's id for it is not the CRC64 of the indexed bytes, so its probe indices do not belong to the indexed class. The key is left out when there is none; Keploy's receivers decode the record with encoding/json and ignore a key they do not know. The CRC64 of a class's indexed bytes is worked out the first time a test reports the class.classMismatchof its own (Keploy takes background out probe by probe), and so does JaCoCo's dump path (KEPLOY_JAVA_SCAN=off, JaCoCo over TCP), which before kept whichever copy the dump listed last.id) is 16 lowercase hex digits, zero-padded. Before 2.2.0 it was unpadded, so readers parse it as a hex number.Fixes
Coverage could count lines that did not run, for an app that loads one class from two sources (two class loaders, a woven copy, a multi-release jar's versioned copy): a test's record carried the probes of whichever copy JaCoCo's dump listed last, and Keploy read them against the class the SDK indexed. 2.2.0 sends the indexed copy's probes when the test ran it, and names in
classMismatcha class whose probes come from other bytes.The same test could get different coverage from run to run on a loaded machine: code a request ran after its response was sent (a servlet filter's code after the chain) was missed by the test's END and dropped as background by the next START. 2.2.0 waits for the test's requests to finish on Tomcat, Jetty and Undertow.
An app that ships its own
org.jacoco.agentlost its own JaCoCo, or the SDK lost its own, with feat(dedup): find the Java app's own classes and start JaCoCo on them #192's keploy-sdk.jar, which carried JaCoCo'sAgentJarunder its own name andjacocoagent.jarat the jar's root: with the app's jar ahead of keploy-sdk.jar on the class path (a Jib launch), the SDK started the app's JaCoCo version (JaCoCo 0.7.9 failed to start on JDK 21, and every test's coverage was empty); in a Spring Boot fat jar, whose loader asks the class path first, the app'sAgentJarwas the SDK's and read the SDK'sjacocoagent.jar. The jar now carriesjacocoagent.jaralone, unchanged, atio/keploy/dedup/jacocoagent.jar, and none of that artifact's classes, so each gets its own. No release carried that jar: the SDK starts JaCoCo itself since 2.2.0.Java 8 API in builds on newer JDKs
NoSuchMethodError. A call to a Java 8 method that JDK 9 overrode with a narrower return type (ByteBuffer.position(int),flip()) is the case CI's JDK 8 leg cannot catch: it compiles on every JDK, and on JDK 17 refers to the method JDK 9 added. Each project's pom now setsmaven.compiler.releaseto 8 in a profile active on JDK 9 and later (the release profile keeps it), so javac checks main and test code against Java 8's API and such a call fails the build or resolves as on Java 8. JDK 8 builds are unchanged. Today's code has no such call: on JDK 17 both projects' main classes come out the same, instruction for instruction, with the profile and without it.Licenses in keploy-sdk.jar
META-INF/LICENSE(Apache 2.0, the repo's), JaCoCo's EPL-2.0 text (META-INF/LICENSE-EPL-2.0.txt), andMETA-INF/THIRD-PARTY-NOTICES.txt, which names every part of the jar that is not keploy-sdk's own code in the version the jar carries: JaCoCo (EPL-2.0, and where its source is: jacocoagent.jar, unchanged atio/keploy/dedup/jacocoagent.jar, and the renamed org.jacoco.core classes), the ASM it depends on (BSD 3-Clause, with its text), Gson and junixsocket (Apache 2.0, with Gson's copyright line and junixsocket's NOTICE). Theabout.htmlat the jar's root is org.jacoco.core's, and no longer depends on the order of the dependencies.<licenses>entries as a choice between them, not as all of them applying, and keploy-sdk's own code is not offered under the EPL-2.0 or the BSD 3-Clause License.META-INF/THIRD-PARTY-NOTICES.txtis the record of what the jar carries under which license.Limits of this branch today
classMismatchin Kubernetes: the SDK's CoverageReporter and k8s-proxy's coverage union do not read it. When the coverage manifest names the indexed copy of a class and a test ran only another copy, they credit that copy's probes to the indexed class.Tests
mvn -B -pl keploy-sdk -am test: 211 rows (three skipped when run as root, and on JDK 8 the module-path row) pass on JDK 8, 17 and 21, withKEPLOY_PROVENunset and=1. New rows cover: a hit reported once and cleared; background never reaching the next test; a class loaded after a pass; order and completeness across four parts; a writer racing the passes, each of its 16384 hits reported exactly once; a failed part ending the pass only once every part has; a part the pool cannot take scanned by the caller; an Error reading JaCoCo's classes clearing nothing; agents not laid out as JaCoCo's refused; and, through the control socket, the background left out, a class loaded twice, a pass that lost hits, and the fallback to JaCoCo's dump and reset. Warmup rows: a fat jar's classes warmed up through its launcher's loader; the-javaagent's Instrumentation finding the loader of the app's loaded classes with no thread holding it; a class the app loaded in a second loader initialized there; of two copies of a class, the copy of the loader that defined more initialized, whichever the JVM lists first; a class whose static initializer throws run once, not again in a second loader; the loaders' order; every live thread seen from the root group; a slow static initializer holding the first START only for the wait, the WARNING naming it, the warmup stopped after it, and a second START not waiting; a warmup thread that cannot start leaving the START and the control server working; the WARNING's steps; an empty class list asking the JVM nothing. BACKGROUND rows: a window's hits sent once to a receiver that asked, never into a test; a class loaded twice reported as a test's record would; no record for a receiver that did not ask, after a START with no END, after an END for another test (with or without a START before it), or after an END whose record could not be sent; a receiver bound anew getting none until it asks, and the INDEX record before its first window; no receiver answered IGNORED and a window with none going nowhere; records that could not be sent counted in the next (said once, kept across a re-ask, not carried to a new receiver);KEPLOY_JAVA_SCAN=offtaking JaCoCo's dump and answering IGNORED, and any other value keeping the scan. Class id rows: ids as 16 hex digits; the manifest's id over the TCP transport, withclassMismatchin its COV; a class JaCoCo saw other bytes of named, and none for a test that ran only indexed classes; of two copies, the indexed one over the manifest's and over a lower id, the manifest's over a lower id (also when its id has fewer than 16 hex digits), the lower id compared unsigned; JaCoCo's dump path choosing alike and leaving library classes out; a BACKGROUND record naming a copy JaCoCo saw other bytes of, and none for the indexed copy; the record's shape with and without the key. License rows: the notices name the versions the build resolves (JaCoCo's core and agent, and noAgentJar; ASM with asm-commons and asm-tree; Gson; junixsocket), so a dependency bumped without its notice fails CI's unit tests; they quote the attributions the licenses ask for (JaCoCo's, Gson's and ASM's copyright lines, ASM's three conditions, the start and the damages sentence of its disclaimer, and the first and last lines of junixsocket's NOTICE); and both license texts sit next to them. keploy-sdk'sio/keploy/dedup/jacocoagent.jaris org.jacoco.agent's, byte for byte, and none sits at the root of keploy-sdk's classes; with an app's ownjacocoagent.jarahead of keploy-sdk, the SDK still reads its own; a loader without it is a named error.KEPLOY_JAVA_END_WAIT_MSread and capped. Real servers, each in a JVM of its own with the agent: Tomcat 9.0.98, Jetty 9.4.57 and Undertow 2.2.37 with the wait on (the filter's tail in the record, and the diagnostics line showing the wait), Tomcat with the wait off (the tail missing: the loss this fixes) and past the bound (windowCut); Tomcat's jars on the module path (JDK 17 and 21); and the index holding Tomcat's jar, where the embedded JaCoCo, and JaCoCo's own agent attached before keploy-sdk.jar, after it, or after it with keploy-sdk.jar inJAVA_TOOL_OPTIONS, measureCoyoteAdapterand the record does not name it inclassMismatch(the two later orders named it before this change); a jar withoutCan-Retransform-Classesstill counting, with its WARNING when JaCoCo's own agent is attached after it, and with none when the SDK starts its own JaCoCo; another agent retransformingCoyoteAdapteronce Tomcat runs, which succeeds and keeps the requests counted, redefining it with a dump of it as the SDK changed it (as a hot-swap tool does), which succeeds and keeps them counted, and retransforming it after loading it ahead of the SDK, which succeeds and leaves it uncounted. Not run on a real server: Tomcat 8.5, 10.1 and 11 and Jetty 10 to 12.1, matched by class and method name (Jetty 12.0's and 12.1's servlet channels checked against their jars). These rows run the SDK from the test class path; the smoke below runs the shaded keploy-sdk.jar on Tomcat.Java8ApiTest, in keploy-sdk's rows and keploy-junit5's: a call toByteBuffer.position(int)must refer toBuffer's method, as on Java 8. On JDK 17 and 21 it fails with the profile off, and on JDK 17 with release 9. A second row fails when the pom names no compiler plugin version, or one before 3.6 (the pin dropped, or set to 3.1 or 3.5.1), and when a pom it reads is not the one it expects (a run outside Maven from another directory).java.util.List.of()fails to compile, also with-P release, and compiles on JDK 17 and 21 with the profile off. On JDK 17 the release builds of both (javadoc jars included, signing skipped) pass, and Java 8 runsscripts/smoke-javaagent.shon the keploy-sdk jar built there. keploy-junit5's rows pass on JDK 8 with JUnit 5.10.5, JDK 17 with 5.14.4 and JDK 21 with 6.1.3.java.util.List.of()builds with tests skipped, and a plantedByteBuffer.position(0)refers to the method Java 8 lacks; with it, the first fails to compile and the second refers toBuffer's.scripts/smoke-javaagent.shchecks that the real JaCoCo runtime is read in place, both the agent the command attaches and the one keploy-sdk.jar starts itself. On real JaCoCo it also fails a record that names any class inclassMismatchwhen the app ran only its own classes, and runs a test in which only another copy of a class runs (built from other source, in a class loader of its own), whose record must name that class alone: on the in-place read with both JaCoCo agents, and on JaCoCo's dump (KEPLOY_JAVA_SCAN=off). It runs the built keploy-sdk.jar on a real Tomcat 9.0.98 with JaCoCo's own agent attached after it and the index holding Tomcat's classes: the record must measureCoyoteAdapter, and the servlet's code that runs after the response while END waits, and name no class inclassMismatch, which fails for a jar built withoutCan-Retransform-Classes: true.scripts/smoke-javaagent.shalso checks that the built jar holds the repo'sLICENSE, the EPL-2.0 text and the notices as the sources hold them, and org.jacoco.core'sabout.html; that itsio/keploy/dedup/jacocoagent.jaris org.jacoco.agent's, byte for byte, and that org.jacoco.agent'sabout.html, which the jar leaves out with the rest of that artifact, is the one inside it; and that each part of the jar maps onto a part the notices name, so a new package, or a dependency from another project, shaded in without its notice fails. Another artifact of ASM or junixsocket maps onto its project's part, whose license the notices already carry.scripts/smoke-javaagent.shruns an app with its own ASM, Gson, JaCoCo core and JaCoCo agent (org.jacoco.agent0.7.9), other versions than the jar's: with them ahead of keploy-sdk.jar on the class path, the INDEX record must name the JaCoCo 0.8.15 the SDK started; in a loader of the app's own that asks the class path first, as Spring Boot's does, each must load from the app's jars and the app'sAgentJarmust read the app's ownjacocoagent.jar. Both fail on the jar before this change.Part of keploy/enterprise#3435 (code coverage + dynamic dedup for large Java apps). Draft: this branch is updated at milestones while the work lands; it is marked ready for review when its checks pass. Do not merge before then.