Repository navigation
feat(dedup): find the Java app's own classes and start JaCoCo on them - #192
Draft
slayerjain wants to merge 5 commits into
Draft
slayerjain wants to merge 5 commits into
slayerjain wants to merge 5 commits into
Conversation
With no KEPLOY_JAVA_CLASS_DIRS, a Jib image (java -cp @jib-classpath-file Main) gave the agent no class dirs, so it indexed the whole classpath: on Apache Fineract 112k classes, about 7k of them the app's. A shaded jar was indexed whole. Library code in every fingerprint made duplicate tests look different (dedup kept 41% of Fineract's tests), and the warmup over all of it ran for 54 s inside the first START, past the time Keploy waits for its answer. The agent now derives the app's classes from how the JVM was launched. The dir or jar that holds the main class is the app's own and counts whole, so generated code outside its package still counts. Other jars give only the classes under the main class's package root: its package cut to three parts, or a longer Maven groupId that holds it (a shorter one never widens it). - java -cp ... Main (Jib, or any classpath): the classpath's dirs and the jar that holds the main class, and the classes under the root in its other jars. - A Spring Boot jar or war, or the dir it was extracted to: BOOT-INF/classes, and the classes under the root in BOOT-INF/lib's jars. Each lib is read in place from the jar mapped read-only, by its central directory, so only those classes are inflated and no lib is copied into the app's heap. A thin jar counts by its Start-Class. - A Vert.x app: its Main-Verticle, or the verticle its command runs, is the main class. - Any other jar: all of it, or only the classes under the root when it is shaded (it holds other projects' pom.properties), and the classes under the root in the jars its manifest's Class-Path names. A jar shaded of libraries with no pom.properties counts whole, as before. A launch it cannot read is not guessed at: a main class that is not in the jar or classpath launched (a launcher stub, as Quarkus and Maven's test forks run), a known server, framework or test runner launcher (Tomcat, Jetty, WildFly, Play, Helidon MP, Karaf, Felix, Equinox, Maven, Gradle), or java -m. A launcher not on that list is taken for the app. KEPLOY_JAVA_CLASS_DIRS and the build's target/classes still come first. When the derivation finds no class, the agent still falls back to the whole jar, then the whole classpath, but now logs that at WARNING with the reason and the fix, and skips the warmup: the warmup runs only over the classes found as the app's, as a library's static initializer can start threads or attach an agent. An index left empty is said at WARNING too, as no test's coverage is measured then. A dir it cannot read is passed over instead of failing the whole index, and a symlink to a class file is followed. The README says which classes count as the app's. Signed-off-by: slayerjain <shubhamkjain@outlook.com>
A JaCoCo agent that the command attaches instruments every class the JVM
loads. On a large app nearly all of those are library classes, so most
of the instrumentation time at startup goes to code that dedup then
filters out of every fingerprint. JaCoCo reads includes= only from its
own agent options, in its premain, so nothing outside the JVM can set it
from the classes the SDK finds from the launch.
keploy-sdk.jar now carries jacocoagent.jar (org.jacoco:org.jacoco.agent
0.8.15, unchanged) and, when the JVM has no JaCoCo agent of its own,
starts it in premain, before the app's main runs. It does so after the
control socket is bound (or, on TCP, the client made) and before the
worker that reads commands starts, so a START that comes early waits
for JaCoCo instead of resetting one not yet started:
- it reads the app's class index there (the first START read it before);
- it writes jacocoagent.jar into a new temp dir only this user can read,
appends it to the system classpath as the JVM does a -javaagent jar,
and runs its premain with output=none and includes= the index's
classes: the first three parts of each package ("org.acme.web.*"),
none under another, and the exact name of a class in a package of
fewer than two parts. Every indexed class matches, and the coverage
dump is already cut to the index, so nothing dedup reads is lost;
- when the index is not the app's own classes alone (the whole jar or
the whole classpath), it passes no includes=, so JaCoCo instruments
every class, as an attached one does;
- when the index is empty, it starts no JaCoCo and says so at WARNING:
no class could pass the index's cut;
- if JaCoCo cannot start, it says so at WARNING and the app runs with no
coverage measured.
With KEPLOY_JAVA_DEDUP_DISABLED=true, or when the socket cannot be
bound, the SDK starts no JaCoCo.
A JaCoCo agent the command attaches, before or after keploy-sdk.jar, on
the command line or in JAVA_TOOL_OPTIONS, is left to run alone, said at
INFO. One before it is on the system classpath already; one after it is
not until its own premain runs, so it is found by its jar among the
JVM's arguments. A second JaCoCo agent would fail the JVM's start on
Java 9 and later (it defines java.lang.$JaCoCo again), and on Java 8
log an error for every class it loads. So k8s, where the proxy attaches
JaCoCo, runs as it did, with no index read before main.
Limits of this change: the index is read before main, so startup grows
by that read (on the whole-classpath fallback, the whole classpath);
the temp dir is removed when the JVM exits, except on Windows, where
the JVM holds the jar open, and a JVM killed with SIGKILL leaves it
behind.
META-INF/THIRD-PARTY-NOTICES.txt names JaCoCo (EPL-2.0, with where its
source is) and ASM (with its BSD 3-Clause text), and the other software
the jar carries. The README now runs the app with keploy-sdk.jar alone,
and says to remove keploy-sdk from the app's dependencies.
jacoco.version sets both JaCoCo artifacts.
Rows: EmbeddedJacocoTest (includes= takes in every indexed class and no
library, none for a fallback index, no JaCoCo for an empty one, the
options, detection on the classpath and among the arguments, and six
JVMs of their own: the SDK alone, JaCoCo after it on the command line
and in JAVA_TOOL_OPTIONS, JaCoCo before it in JAVA_TOOL_OPTIONS, dedup
turned off, and a temp dir JaCoCo cannot start from). The smoke script
runs two modes with keploy-sdk.jar alone.
Signed-off-by: slayerjain <shubhamkjain@outlook.com>
This was referenced Nov 19, 2026
Keploy could not tell whether dedup ran on the app's own classes or fell
back to the whole jar or classpath: the SDK said so only in the app's
log. When it fell back, dedup kept far more tests than it should, and
the Keploy run looked normal.
Before the first test's coverage, the SDK now sends Keploy one INDEX
record on /tmp/coverage_data.sock. It is a JSON object on its own
connection, framed as each test's coverage record is, and it starts
with {"kind":"index",. It holds:
- source: override (KEPLOY_JAVA_CLASS_DIRS), derived (from the launch),
whole-jar, classpath (the app's classes could not be told from its
libraries) or none;
- launch: how the JVM was launched, or why the app's classes could not
be told apart;
- roots: the dirs and jars the classes were read from, up to 64
(rootCount gives the full count); in a Spring Boot jar these are
app.jar!BOOT-INF/classes and app.jar!BOOT-INF/lib/<lib>.jar;
- packages: the app's package root when it was derived and has two
parts or more, else empty;
- classes: how many classes were indexed;
- build: "jv1-" and the first 32 hex of the sha256 over the sorted,
unsigned, de-duplicated JaCoCo class ids (CRC64 of each class's
bytes, 16 lowercase hex, joined by newlines), the id Keploy gives the
same build; "" when no class was found;
- includes and jacoco: the includes= of the JaCoCo agent keploy-sdk.jar
started (embedded), or that the command attached its own (attached),
that none runs (none), or that the SDK was not started from premain
(unchecked).
It carries no class list.
The worker sends it right after the control socket is bound and before
it reads any command, when the data socket's file is there (Keploy's
receiver makes it); a START sent meanwhile waits in the socket's
backlog. With no socket file there it sends nothing and does not read
the index for it, so a JVM that runs with no Keploy does no extra work.
A file left by a Keploy that died makes that send fail at once; it is
tried again before the next test's coverage. The SDK
sends it again before the next test's coverage when the receiver on
the data socket is a new one (a new run, a docker-compose test set's
receiver, a restarted Keploy, or none there at the first send). It
tells receivers apart by the socket file's identity and the time it
was made. A send that fails never fails the test's END.
source is override only when the dirs and jars KEPLOY_JAVA_CLASS_DIRS
names hold a class. When they hold none, the SDK now says so at
WARNING and reports what it found instead, for example the build
output under the working dir, as derived. A launch that lists more
than five dirs now says how many more there are.
Limits of this change:
- On the TCP transport used in Kubernetes no INDEX record is sent.
- The order is the send order: Keploy reads each connection on its own,
so it may handle a test's record before the INDEX sent just ahead of
it. A receiver replaced while the SDK sends can get the INDEX twice.
- A Keploy that does not know the record stores it as a coverage record
with an empty test id, then logs "Received malformed test ID, skipping
dedup file update" at ERROR, once for each start of the app and again
for each new receiver: on `keploy test`, which starts the app again
for every test set, and on docker compose alike, that is once per test
set. Its dedupData.yaml does not change. In a run where no test's
coverage arrives, for example when the record says no class was found,
that Keploy also logs that dedupData.yaml was generated although it
writes none.
Tests: IndexReportTest checks the build id against the value Keploy's
own test gives for the same ids, in any order and repeated, and the
record's shape. AppIndexTest checks the record for derived dirs, the
build output under the working dir, an override, an override with no
class, a Spring Boot jar with no source tree, the whole jar, the whole
classpath, and no class, and that a JVM with no receiver does not read
its index for it. ControlSocketTest checks that the record reaches a
receiver once before any command, that a receiver bound again on the
same path gets it before its first coverage record, and that a socket
file's new time, or another file with the same time, tells a new
receiver apart. Three forked EmbeddedJacocoTest rows check includes= and
jacoco for the JaCoCo agent the SDK started, for one the command
attached, and none when it could not start. scripts/smoke-javaagent.sh,
which CI runs on JDK 8, 17 and 21, now takes the INDEX record as the
first on the data socket, then the test's coverage, and checks its
source and jacoco for each launch.
Signed-off-by: slayerjain <shubhamkjain@outlook.com>
keploy-sdk.jar is a -javaagent jar, so the JVM puts it on the app's
system class path. It carried JaCoCo's core, the ASM that core needs, and
Gson under their own package names, so they met the app's own copies:
- Spring Boot's launcher loads BOOT-INF/lib through a loader that asks the
system class loader first. An app that ships its own ASM, Gson or JaCoCo
core got keploy-sdk's copy instead, in keploy-sdk's version: a
NoSuchMethodError or LinkageError in the user's app once it calls an API
that version lacks.
- A launch that puts the app's jars first (Jib's -cp /app/libs/*) gave
keploy-sdk the app's version instead. With an older JaCoCo core there,
keploy-sdk sent no INDEX record at all, and said nothing about it.
The shade plugin now relocates org.jacoco.core, org.objectweb.asm and
com.google.gson under io.keploy.shaded. junixsocket keeps its names: its
native library is bound to them. The jacocoagent.jar that keploy-sdk
starts already carries its own renamed runtime.
The shade plugin also rewrites every string constant that starts with a
relocated package, and the list of keploy-sdk's own classes, which are
never fingerprinted as the app's, named org/objectweb/asm/ and
com/google/gson/. Left as they were, the jar would have fingerprinted its
own renamed JaCoCo core. The list now names keploy-sdk.jar's classes as
the jar holds them: io/keploy/shaded/, the JaCoCo agent under org/jacoco/,
and all of junixsocket under org/newsclub/. That last one fixes an older
gap: junixsocket's org/newsclub/lib class was not on the list, so an
override that named keploy-sdk.jar fingerprinted it and passed it to
JaCoCo's includes=. An app's own ASM or Gson is now one of the app's
libraries, like Jackson, and is fingerprinted when the index takes in
libraries (the whole jar or the whole class path).
THIRD-PARTY-NOTICES.txt says which packages are renamed and to what.
Tests:
- AppIndexTest: an override that takes in keploy-sdk.jar's classes
(its renamed libraries, the JaCoCo agent, both junixsocket packages)
indexes the app's own classes and its own ASM and Gson only.
- smoke-javaagent.sh, with keploy-sdk.jar alone, against the app's own
ASM 5.2, Gson 2.11.0 and JaCoCo core 0.7.9:
- those jars on the class path before keploy-sdk.jar: the INDEX record
and the test's coverage still arrive;
- those jars in a loader under the system class loader, as Spring Boot
loads BOOT-INF/lib: the app gets its own ASM, Gson and JaCoCo core;
- KEPLOY_JAVA_CLASS_DIRS that names keploy-sdk.jar too: the INDEX
record counts the app's classes alone.
On the jar from before this change, all three cases fail.
Signed-off-by: slayerjain <shubhamkjain@outlook.com>
Only the JVM knows which keploy-sdk.jar it runs and which JaCoCo agent
measures the app's classes, and Keploy needs both to say what produced a
test's coverage. The INDEX record now carries them, after its other
fields:
- sdkVersion: the Implementation-Version of keploy-sdk.jar's manifest,
which the build sets to the project's version. It is read only when
that manifest's Implementation-Title is keploy-sdk's own, so an app
jar that shades or copies the SDK's classes never lends its version.
It is "" when not known: the SDK's classes come from a build's class
dir, or from a jar that is not keploy-sdk.jar.
- jacocoVersion: the version the measuring JaCoCo agent's jar names
("0.8.15"). That is the agent keploy-sdk.jar carries and started
(jacoco embedded), or the one the command or JAVA_TOOL_OPTIONS
attached (attached), read from the jar that holds its runtime, on the
system class path or named by a -javaagent argument. It is "" when
nothing measures the classes, the SDK was not started by premain, the
jar names no version or cannot be read, or the runtime is in a class
dir or in a jar nested in another.
The record's v stays 1: both fields are new and come last, and a reader
that does not know them skips them. EmbeddedJacoco.attached now returns
the attached agent's version, or null when there is none, in place of a
boolean. A jar that holds JaCoCo's runtime counts as attached whatever
its manifest says, as before.
Tests: IndexReportTest checks both fields and their place at the end of
the record, and versionOf for a class loaded from keploy-sdk's jar with
and without a version, from a jar with another title or none, and from
a class dir. EmbeddedJacocoTest checks the version attached returns for
an agent on the class path and among the arguments, "" for a JaCoCo
runtime in a jar that names no version, in a class dir, or in a jar
nested in another, and the versions in the INDEX of forked JVMs: the
SDK alone, JaCoCo attached after it and before it, and a JaCoCo that
could not start. scripts/smoke-javaagent.sh checks, for every launch
that sends an INDEX, that it names keploy-sdk.jar's version and the
version of the JaCoCo agent that measures the app (the 0.8.12 agent the
script attaches, or the one keploy-sdk.jar carries), each as its jar's
manifest names it.
Signed-off-by: slayerjain <shubhamkjain@outlook.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
On a large Java app, dynamic dedup failed on its defaults:
KEPLOY_JAVA_CLASS_DIRS, a Jib image (java -cp @jib-classpath-file Main) gave the SDK no class dirs, so it indexed the whole classpath. On Apache Fineract that is 112k classes, about 7k of them the app's. A shaded jar was indexed whole.includes=only from its own agent options, so Keploy could not narrow it from outside the JVM: only the agent's own options can.What changes
The app's own classes, found from the launch, with no configuration and no source code.
java -cp ... Main(Jib, or any classpath): the classpath's dirs and the jar that holds the main class, plus the classes under the main class's package root in the other jars.BOOT-INF/classes, plus the classes under the root inBOOT-INF/lib. Each lib is read in place, so only those classes are inflated.pom.properties).java -m) is not guessed at. It falls back to the whole jar, then the whole classpath, and says so at WARNING with the fix.KEPLOY_JAVA_CLASS_DIRSand the build'starget/classesstill come first.keploy-sdk.jar starts JaCoCo itself, on those classes only.
jacocoagent.jar(JaCoCo 0.8.15). When the JVM has no JaCoCo agent of its own, the premain starts it withincludes=the app's packages, before the app's main runs.-javaagent:keploy-sdk.jaralone.JAVA_TOOL_OPTIONSattaches is left to run alone, and this is said at INFO.META-INF/THIRD-PARTY-NOTICES.txtnames JaCoCo (EPL-2.0, with where its source is), ASM (BSD 3-Clause) and the other software the jar carries.One INDEX record to Keploy. Before the first test's coverage, on
/tmp/coverage_data.sock, framed like a coverage record and starting with{"kind":"index",, the SDK sends:source: override, derived, whole-jar, classpath or none;launch,roots(up to 64),rootCount,packagesandclasses;build: the build id over the indexed classes' JaCoCo class ids;includesandjacoco: embedded, attached, none or unchecked.It carries no class list. It is sent once per app start, and again for each new receiver (a new run, a docker-compose test set, a restarted Keploy). A failed send never fails a test's END. With no receiver there, the SDK sends nothing and reads no index for it.
ASM, Gson and JaCoCo core are renamed in keploy-sdk.jar (
io.keploy.shaded.*). keploy-sdk.jar is on the system class path as a -javaagent, so its copies used to meet the app's own:BOOT-INF/lib) got keploy-sdk's version, and aNoSuchMethodErrororLinkageError;Limits of this change
org.jacoco.agent.AgentJarand the/jacocoagent.jarresource, which keep their names. An app whose class path putsorg.jacoco:org.jacoco.agent(the build-time artifact) ahead of keploy-sdk.jar gets that version started instead. A version too old for the app's class files instruments none of them, so no test's coverage is measured and dedup keeps every test. A JaCoCo agent the app attaches itself is not affected: the SDK then starts none.Tests
CI's java-agent workflow (JDK 8, 17 and 21) passes on this head: 92 tests each, 1 skipped.
AppClassesTest,AppIndexTest: derivation for Jib, Spring Boot (jar, war, extracted dir, thin jar), shaded and plain jars, Vert.x, launchers that are not guessed at, the override, the build output, and a jar-only app with no source tree; the INDEX record for a derived index, the build output, an override (with and without a class), the whole jar, the whole classpath and no class.EmbeddedJacocoTest:includes=, detection of an attached JaCoCo agent on the class path and among the arguments, and forked JVMs for the SDK alone, JaCoCo attached after it and inJAVA_TOOL_OPTIONS, dedup turned off, and a JaCoCo that cannot start.IndexReportTest: the build id against Keploy's value for the same ids, and the record's shape.ControlSocketTest: the INDEX reaches a receiver once before any command, and again for a new receiver.scripts/smoke-javaagent.sh, with keploy-sdk.jar alone:Part of keploy/enterprise#3435. Draft: updated at milestones; marked ready for review when its checks pass.