From c882b650af7cff7c5a1016715000f31d15c6cde6 Mon Sep 17 00:00:00 2001 From: slayerjain Date: Thu, 19 Nov 2026 23:21:51 +0530 Subject: [PATCH 01/14] feat(dedup): send the app's coverage build table when Keploy asks Keploy's per-test code coverage needs the build table of the app's classes: packages, classes, methods, their lines and branch edges. Only the app's JVM holds the class bytes in every run mode, so keploy-sdk builds it there. - The table goes only when Keploy asks for it with the control-socket command TABLE, after its first START is answered ACK. The agent answers ACK at once and starts the work on a daemon thread of its own; the control path never analyzes, encodes or sends, and START and END never ask for the table. An older Keploy never sends TABLE, so it never gets a frame; an older SDK answers it IGNORED. A TABLE while a send runs has that send run once more after it, so a new receiver can ask again. A TABLE that fails to start (no thread, or any error on the control path) is answered IGNORED and logged at WARNING, and the control server goes on serving START and END. - The analysis competes with the app for CPU: the thread's priority is only a hint, which HotSpot on Linux ignores by default. When the index is not loaded yet, its first load runs on this thread and holds the index's lock, which an END that needs the index waits on; an END would otherwise load it itself. - JaCoCo's analyzer runs over the indexed class bytes. One record per top-level class, its nested, inner, anonymous, lambda and Scala module classes folded in by name (the simple name cut at the first $ after its first character). A class is named by its bytes, the name JaCoCo's runtime gives its probes, wherever the index found the file. A method has one code leaf per line (n = JaCoCo's instructions on that line in that method) and one leaf per branch edge. A method with no line numbers has no leaf and is left out; the INFO line counts the class files built without line numbers and says to compile them with -g. - Records are in the ingest contract's order (packages segment by segment, then id; methods by first line, then id) and written as Go's encoding/json writes them with HTML escaping off. The build id is jv1- and the first 32 hex of the sha256 over the sorted JaCoCo class ids of every indexed class; a record's hash is the same sum over its class files' ids. - A class JaCoCo cannot analyze makes its record not_measured with reason analyzer_failed and no methods. A record larger than an inflated 8 MiB chunk is not_measured with reason record_too_large. Keploy's encoder also refuses a record whose compressed form passes a chunk; the CLI is the authority on that, so the stream's sha256 is the CLI's only while it re-marks no record. - The records go to Keploy as TABLE frames on the data socket, one frame per connection, each at most 1 MiB: KTB1, a 4-byte length, a header line (build id, seq, last; the last one adds the record count and the sha256 of the whole stream), then the next bytes of the NDJSON stream. With no app class the stream is one empty last frame with build "". The SDK shuts its output after each frame and waits for OK; a receiver that closes without it ends the stream, which the SDK logs at WARNING with the frame and the reason, and the table goes again only when Keploy asks again. - KeployDedupAgent gains only the TABLE branch in the unix control socket's command handler. The TCP transport does not send a table. Two golden rows check the table byte for byte against the Go encoder's output for the same classes, with the hashes and the build id computed in Go from the class bytes: the java-dedup sample's classes, and a generated set for the names, packages and escapes the sample lacks. A control-socket row runs START and END on an index of the sample's classes, checks that neither asked for the table, then sends TABLE and gets that golden in one frame. Signed-off-by: slayerjain --- .../main/java/io/keploy/dedup/BuildTable.java | 1029 +++++++++++++++++ .../io/keploy/dedup/KeployDedupAgent.java | 12 + .../java/io/keploy/dedup/BuildTableTest.java | 769 ++++++++++++ .../io/keploy/dedup/ControlSocketTest.java | 77 +- .../io/keploy/dedup/buildtable/edge.build | 1 + .../keploy/dedup/buildtable/edge.table.ndjson | 15 + .../fixture-java-dedup-table.ndjson | 4 + .../keploy/dedup/buildtable/java-dedup.build | 1 + .../dedup/buildtable/java-dedup.table.ndjson | 5 + .../javadedup/DedupController.class.bin | Bin 0 -> 15569 bytes .../javadedup/JavaDedupApplication.class.bin | Bin 0 -> 698 bytes .../samples/javadedup/model/Item.class.bin | Bin 0 -> 1186 bytes .../samples/javadedup/model/Product.class.bin | Bin 0 -> 1869 bytes .../samples/javadedup/model/User.class.bin | Bin 0 -> 889 bytes 14 files changed, 1909 insertions(+), 4 deletions(-) create mode 100644 keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java create mode 100644 keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java create mode 100644 keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/edge.build create mode 100644 keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/edge.table.ndjson create mode 100644 keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/fixture-java-dedup-table.ndjson create mode 100644 keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup.build create mode 100644 keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup.table.ndjson create mode 100644 keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup/io/keploy/samples/javadedup/DedupController.class.bin create mode 100644 keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup/io/keploy/samples/javadedup/JavaDedupApplication.class.bin create mode 100644 keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup/io/keploy/samples/javadedup/model/Item.class.bin create mode 100644 keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup/io/keploy/samples/javadedup/model/Product.class.bin create mode 100644 keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup/io/keploy/samples/javadedup/model/User.class.bin diff --git a/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java b/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java new file mode 100644 index 00000000..d9ff5217 --- /dev/null +++ b/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java @@ -0,0 +1,1029 @@ +package io.keploy.dedup; + +import org.jacoco.core.analysis.Analyzer; +import org.jacoco.core.analysis.IClassCoverage; +import org.jacoco.core.analysis.ICoverageVisitor; +import org.jacoco.core.analysis.ILine; +import org.jacoco.core.analysis.IMethodCoverage; +import org.jacoco.core.analysis.ISourceNode; +import org.jacoco.core.data.ExecutionDataStore; +import org.jacoco.core.internal.data.CRC64; +import org.newsclub.net.unix.AFUNIXSocket; +import org.newsclub.net.unix.AFUNIXSocketAddress; +import org.objectweb.asm.ClassReader; +import org.objectweb.asm.ClassVisitor; +import org.objectweb.asm.MethodVisitor; +import org.objectweb.asm.Opcodes; + +import java.io.BufferedReader; +import java.io.ByteArrayOutputStream; +import java.io.File; +import java.io.IOException; +import java.io.InputStreamReader; +import java.io.OutputStream; +import java.nio.charset.StandardCharsets; +import java.security.MessageDigest; +import java.security.NoSuchAlgorithmException; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.Comparator; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.ThreadFactory; +import java.util.function.Function; +import java.util.function.Supplier; +import java.util.logging.Level; +import java.util.logging.Logger; + +/** + * BuildTable is the build table of the code-coverage ingest contract (schema 1, section 3.3) for the app's + * classes: one record per top-level class (its nested, inner, anonymous, lambda and Scala module classes + * folded in by name), methods with JaCoCo's per-method lines and branch edges, in the contract's order and + * written byte for byte as Keploy's Go encoder writes them (enterprise pkg/service/codecoverage). + * + *

Keploy asks for it with the control-socket command {@code TABLE}, which it sends once its first START + * is answered ACK. The agent answers ACK at once and never works on the table on the control path: JaCoCo's + * analyzer runs over the indexed class bytes on a daemon thread of its own, which streams the records to + * Keploy as TABLE frames on the data socket. An older SDK answers the command IGNORED, and an older Keploy + * never sends it, so it never gets a frame. A TABLE while a send runs has that send run once more after it. + * + *

A TABLE frame is one connection to the data socket: the 4 bytes {@code KTB1}, a 4-byte big-endian + * length n, then n bytes (n plus 8 is at most {@link #MAX_FRAME_BYTES}): a header JSON line, then the next + * bytes of the table's NDJSON stream (a record may run on into the next frame). The header is + * {@code {"v":1,"part":"table","build":"jv1-...","seq":k,"last":false}}; the last frame also carries + * {@code "records"} (the record count) and {@code "sha256"} (hex, over the whole NDJSON stream). With no app + * class the stream is one frame: build "", last, 0 records. After each frame the SDK shuts its side's output + * and waits for {@code OK} and a newline; a receiver that closes without it ends the stream, and the table + * goes again only when Keploy sends TABLE again. + */ +final class BuildTable { + + private static final Logger LOGGER = Logger.getLogger(BuildTable.class.getName()); + + /** The control-socket command Keploy sends for the build table. */ + static final String COMMAND = "TABLE"; + /** A TABLE frame's size cap, magic and length included. */ + static final int MAX_FRAME_BYTES = 1 << 20; + /** A record (with its newline) larger than a whole inflated chunk can never be sent: record_too_large. */ + static final int MAX_RECORD_BYTES = 8 << 20; + /** The first 4 bytes of a TABLE frame. */ + static final String MAGIC = "KTB1"; + + private static final int CONNECT_TIMEOUT_MILLIS = 5_000; + private static final int REPLY_TIMEOUT_MILLIS = 30_000; + private static final int CONNECT_ATTEMPTS = 3; + // Room left in a frame for the magic, the length and the header line. + private static final int HEADER_ROOM = 1024; + + // The JVM's table sends, one at a time. + private static final Sends JVM = new Sends(BuildTable::thread); + + private BuildTable() { + } + + /** One class file the index read: its VM name (org/acme/Foo$Bar) and its bytes. */ + static final class ClassFile { + final String name; + final byte[] bytes; + + ClassFile(String name, byte[] bytes) { + this.name = name; + this.bytes = bytes; + } + } + + /** files maps index entries to class files, so the agent's command hook stays one call. */ + static List files(List entries, Function name, Function bytes) { + List out = new ArrayList<>(entries.size()); + for (T e : entries) { + out.add(new ClassFile(name.apply(e), bytes.apply(e))); + } + return out; + } + + /** + * request answers Keploy's TABLE command: true (ACK) once the table's send is started on its own thread, + * or, when a send runs already, once that send is set to run once more after it. It never blocks. + * When no thread can start it answers false (IGNORED) and does not throw; the agent's command hook catches + * anything else. The classes are read on that thread. + */ + static boolean request(Supplier> classes, File dataSocket) { + return JVM.request(() -> send(classes, new SocketSender(dataSocket))); + } + + /** requests is how many times the JVM's table was asked for, answered or not: only TABLE asks. */ + static int requests() { + return JVM.requests(); + } + + static Thread thread(Runnable r) { + Thread t = new Thread(r, "keploy-build-table"); + t.setDaemon(true); + // A hint only: HotSpot on Linux runs every Java thread at one OS priority unless told otherwise. + t.setPriority(Thread.MIN_PRIORITY); + return t; + } + + /** + * Sends runs one send at a time; a request while one runs has it run once more after it. It assumes one + * caller at a time (the control server handles one command at a time): a reset below clears {@code again}, + * which would drop a request that raced in from a second caller. + */ + static final class Sends { + private final ThreadFactory threads; + private boolean running; + private boolean again; + private int requests; + + Sends(ThreadFactory threads) { + this.threads = threads; + } + + synchronized int requests() { + return requests; + } + + boolean request(Runnable send) { + synchronized (this) { + requests++; + if (running) { + again = true; + return true; + } + running = true; + } + try { + threads.newThread(() -> loop(send)).start(); + return true; + } catch (Throwable t) { + // Called on the control path: a thread that cannot start (a pids limit, a security manager) + // must not end the control server. Keploy reads IGNORED as no table from this SDK. + synchronized (this) { + running = false; + again = false; + } + LOGGER.log(Level.WARNING, "Keploy coverage: the build table's thread did not start", t); + return false; + } + } + + private void loop(Runnable send) { + try { + while (true) { + send.run(); + synchronized (this) { + if (!again) { + running = false; + return; + } + again = false; + } + } + } catch (Throwable t) { + synchronized (this) { + running = false; + again = false; + } + LOGGER.log(Level.WARNING, "Keploy coverage: the build table was not sent", t); + } + } + } + + /** + * send analyzes the classes and streams the table through sender. It logs what it sent, or why it stopped, + * and never throws. + */ + static void send(Supplier> classes, FrameSender sender) { + try { + long cpu0 = threadCpuNanos(); + long t0 = System.nanoTime(); + FrameWriter frames = new FrameWriter(sender); + Summary s = write(classes.get(), frames); + frames.finish(s.build, s.records); + StringBuilder m = new StringBuilder("Keploy coverage: build table "); + m.append(s.build.isEmpty() ? "(no app classes)" : s.build).append(" sent (").append(s.records) + .append(" classes, ").append(s.notMeasured).append(" not measured"); + if (s.noLines > 0) { + m.append(", ").append(s.noLines) + .append(" class files without line numbers, so without lines: compile them with -g"); + } + if (s.unnamed > 0) { + m.append(", ").append(s.unnamed).append(" index entries without a class name left out"); + } + m.append("; ").append(frames.seq).append(" frames, ").append((System.nanoTime() - t0) / 1_000_000) + .append(" ms, cpu ").append((threadCpuNanos() - cpu0) / 1_000_000).append(" ms)"); + LOGGER.log(Level.INFO, m.toString()); + } catch (IOException e) { + LOGGER.log(Level.WARNING, "Keploy coverage: the build table was not sent: " + e.getMessage() + + "; it goes again when Keploy asks for it again"); + } catch (Throwable t) { + // Never let the table take the app down: Keploy names the coverage it did not get. + LOGGER.log(Level.WARNING, "Keploy coverage: the build table was not sent", t); + } + } + + private static long threadCpuNanos() { + try { + return java.lang.management.ManagementFactory.getThreadMXBean().getCurrentThreadCpuTime(); + } catch (Throwable t) { + return 0; + } + } + + // ---------------------------------------------------------------- the table + + /** + * What write wrote: the build id, the record count, how many records are not measured, how many class + * files have code but no line numbers, and how many index entries were left out as no class has a name. + */ + static final class Summary { + final String build; + final int records; + final int notMeasured; + final int noLines; + final int unnamed; + + Summary(String build, int records, int notMeasured, int noLines, int unnamed) { + this.build = build; + this.records = records; + this.notMeasured = notMeasured; + this.noLines = noLines; + this.unnamed = unnamed; + } + } + + /** Where write puts each record (UTF-8 JSON, no newline), in the table's order. */ + interface RecordSink { + void record(String build, byte[] json) throws IOException; + } + + /** + * write analyzes the classes and hands each table record to sink, in the contract's order. With no class + * bytes the build id is "" and there is no record. + */ + static Summary write(List classes, RecordSink sink) throws IOException { + return write(classes, sink, MAX_RECORD_BYTES); + } + + static Summary write(List classes, RecordSink sink, int maxRecord) throws IOException { + Map byId = new HashMap<>(); + long[] ids = new long[classes.size()]; + int n = 0; + int unnamed = 0; + for (ClassFile in : classes) { + if (in == null || in.bytes == null) { + continue; + } + // Every indexed class is in the build id, as in the index record's. + long id = CRC64.classId(in.bytes); + ids[n++] = id; + ClassFile f = new ClassFile(vmName(in), in.bytes); + Unit u = f.name == null ? null : Unit.of(f.name); + if (u == null || u.name.isEmpty()) { + // No unit has an empty name (Go refuses the whole table for one): a file named "a/.class". + unnamed++; + continue; + } + Unit had = byId.get(u.id); + if (had == null) { + byId.put(u.id, u); + had = u; + } + had.files.add(new UnitFile(f, id)); + } + if (n == 0) { + return new Summary("", 0, 0, 0, unnamed); + } + String build = "jv1-" + classIdsSum(Arrays.copyOf(ids, n)).substring(0, 32); + List units = new ArrayList<>(byId.values()); + Collections.sort(units, UNIT_ORDER); + int notMeasured = 0; + int noLines = 0; + for (Unit u : units) { + byte[] rec = u.record(); + if (rec.length + 1 > maxRecord) { + rec = u.notMeasured("record_too_large"); + u.failed = true; + } + if (u.failed) { + notMeasured++; + } else { + noLines += u.noLines; + } + sink.record(build, rec); + } + return new Summary(build, units.size(), notMeasured, noLines, unnamed); + } + + /** + * vmName is the class's own name, from its bytes: the name JaCoCo's runtime gives its probes, wherever + * the index found the file. When the bytes are not a class ASM reads, it is the index's name. + */ + static String vmName(ClassFile f) { + try { + String name = new ClassReader(f.bytes).getClassName(); + return name != null ? name : f.name; + } catch (RuntimeException e) { + return f.name; + } + } + + /** classIdsSum is sha256, as hex, over the sorted, de-duplicated class ids, each %016x, joined by \n. */ + static String classIdsSum(long[] ids) { + long[] sorted = ids.clone(); + // Go sorts them as uint64. + Long[] boxed = new Long[sorted.length]; + for (int i = 0; i < sorted.length; i++) { + boxed[i] = sorted[i]; + } + Arrays.sort(boxed, Long::compareUnsigned); + StringBuilder sb = new StringBuilder(boxed.length * 17); + Long prev = null; + for (Long id : boxed) { + if (prev != null && prev.longValue() == id.longValue()) { + continue; + } + if (prev != null) { + sb.append('\n'); + } + String hex = Long.toHexString(id); + for (int i = hex.length(); i < 16; i++) { + sb.append('0'); + } + sb.append(hex); + prev = id; + } + return sha256Hex(sb.toString().getBytes(StandardCharsets.UTF_8)); + } + + static String sha256Hex(byte[] b) { + MessageDigest md; + try { + md = MessageDigest.getInstance("SHA-256"); + } catch (NoSuchAlgorithmException e) { + throw new IllegalStateException(e); + } + return hex(md.digest(b)); + } + + static String hex(byte[] b) { + StringBuilder sb = new StringBuilder(b.length * 2); + for (byte x : b) { + sb.append(Character.forDigit((x >> 4) & 0xf, 16)).append(Character.forDigit(x & 0xf, 16)); + } + return sb.toString(); + } + + /** Units by package path, segment by segment ('/' sorts first), then by id: Go's comparePkg order. */ + static final Comparator UNIT_ORDER = (a, b) -> { + int c = comparePkg(a.pkg, b.pkg); + return c != 0 ? c : compareCodePoints(a.id, b.id); + }; + + /** comparePkg is codecoverage.comparePkg over code points (the order of their UTF-8 bytes). */ + static int comparePkg(String a, String b) { + int i = 0; + int j = 0; + while (i < a.length() && j < b.length()) { + int ca = a.codePointAt(i); + int cb = b.codePointAt(j); + if (ca != cb) { + if (ca == '/') { + return -1; + } + if (cb == '/') { + return 1; + } + return Integer.compare(ca, cb); + } + i += Character.charCount(ca); + j += Character.charCount(cb); + } + return Integer.compare(a.length() - i, b.length() - j); + } + + /** compareCodePoints orders strings as Go's strings.Compare orders their UTF-8 bytes. */ + static int compareCodePoints(String a, String b) { + int i = 0; + int j = 0; + while (i < a.length() && j < b.length()) { + int ca = a.codePointAt(i); + int cb = b.codePointAt(j); + if (ca != cb) { + return Integer.compare(ca, cb); + } + i += Character.charCount(ca); + j += Character.charCount(cb); + } + return Integer.compare(a.length() - i, b.length() - j); + } + + private static final class UnitFile { + final ClassFile file; + final long classId; + + UnitFile(ClassFile file, long classId) { + this.file = file; + this.classId = classId; + } + } + + /** A table record: a top-level class and the class files folded into it by name. */ + static final class Unit { + final String pkg; + final String name; + final String id; + final List files = new ArrayList<>(1); + boolean failed; + // Class files with code but no line numbers: their methods have no leaf, so they are left out. + int noLines; + + private Unit(String pkg, String name) { + this.pkg = pkg; + this.name = name; + this.id = "c:" + (pkg.isEmpty() ? name : pkg + "/" + name); + } + + /** of finds a class file's unit by name: its simple name cut at the first $ after its first character. */ + static Unit of(String vmName) { + int slash = vmName.lastIndexOf('/'); + String pkg = slash < 0 ? "" : vmName.substring(0, slash); + String simple = vmName.substring(slash + 1); + int cut = simple.indexOf('$', 1); + return new Unit(pkg, cut < 0 ? simple : simple.substring(0, cut)); + } + + String path() { + return pkg.isEmpty() ? name : pkg + "/" + name; + } + + String hash() { + long[] ids = new long[files.size()]; + for (int i = 0; i < ids.length; i++) { + ids[i] = files.get(i).classId; + } + return classIdsSum(ids); + } + + byte[] notMeasured(String code) { + StringBuilder sb = head(sourcePath(null)); + sb.append(",\"state\":\"not_measured\",\"reason\":{\"code\":"); + Json.string(sb, code); + sb.append("},\"methods\":[]}"); + return sb.toString().getBytes(StandardCharsets.UTF_8); + } + + private StringBuilder head(String path) { + StringBuilder sb = new StringBuilder(256); + sb.append("{\"kind\":\"class\",\"id\":"); + Json.string(sb, id); + sb.append(",\"pkg\":"); + Json.string(sb, pkg); + sb.append(",\"name\":"); + Json.string(sb, name); + sb.append(",\"path\":"); + Json.string(sb, path); + sb.append(",\"hash\":"); + Json.string(sb, hash()); + return sb; + } + + /** + * sourcePath is the build path of the unit's source: the package path and the SourceFile of its + * first class file (by name, so the top-level class when it is there) whose SourceFile is a bare, + * non-empty file name; "" when none has one. + */ + private String sourcePath(Map infos) { + List sorted = sortedFiles(); + for (UnitFile f : sorted) { + ClassInfo info = infos != null ? infos.get(f.file.name) : ClassInfo.read(f.file.bytes); + String sf = info == null ? null : info.sourceFile; + if (sf != null && !sf.isEmpty() && sf.indexOf('/') < 0) { + return pkg.isEmpty() ? sf : pkg + "/" + sf; + } + } + return ""; + } + + private List sortedFiles() { + List sorted = new ArrayList<>(files); + Collections.sort(sorted, (a, b) -> compareCodePoints(a.file.name, b.file.name)); + return sorted; + } + + /** record analyzes the unit's class files and writes its record; any class JaCoCo refuses fails it. */ + byte[] record() { + Map infos = new HashMap<>(); + List methods = new ArrayList<>(); + Set methodIds = new HashSet<>(); + for (UnitFile f : sortedFiles()) { + ClassInfo info = ClassInfo.read(f.file.bytes); + infos.put(f.file.name, info); + List got; + try { + got = analyze(f.file); + } catch (Exception | LinkageError e) { + LOGGER.log(Level.FINE, "Keploy coverage: JaCoCo could not analyze " + f.file.name, e); + failed = true; + return notMeasured("analyzer_failed"); + } + for (IClassCoverage cc : got) { + if (cc.getFirstLine() == ISourceNode.UNKNOWN_LINE + && cc.getInstructionCounter().getTotalCount() > 0) { + noLines++; + } + String nested = f.file.name.substring(path().length()); + for (IMethodCoverage mc : cc.getMethods()) { + Method m = Method.of(f.file.name, nested, name, mc, info); + if (m != null && methodIds.add(m.id)) { + methods.add(m); + } + } + } + } + Collections.sort(methods, (a, b) -> { + int c = Integer.compare(a.line, b.line); + return c != 0 ? c : compareCodePoints(a.id, b.id); + }); + StringBuilder sb = head(sourcePath(infos)); + sb.append(",\"state\":\"ok\",\"methods\":["); + for (int i = 0; i < methods.size(); i++) { + if (i > 0) { + sb.append(','); + } + methods.get(i).json(sb); + } + sb.append("]}"); + return sb.toString().getBytes(StandardCharsets.UTF_8); + } + } + + /** analyze runs JaCoCo's analyzer over one class file with no execution data. */ + static List analyze(ClassFile f) throws IOException { + final List out = new ArrayList<>(1); + Analyzer analyzer = new Analyzer(new ExecutionDataStore(), new ICoverageVisitor() { + @Override + public void visitCoverage(IClassCoverage coverage) { + out.add(coverage); + } + }); + analyzer.analyzeClass(f.bytes, f.name); + return out; + } + + /** What a cheap ASM pass (no code) gives: the SourceFile and which methods take varargs. */ + static final class ClassInfo { + String sourceFile; + final Set varargs = new HashSet<>(); + + static ClassInfo read(byte[] bytes) { + final ClassInfo info = new ClassInfo(); + try { + new ClassReader(bytes).accept(new ClassVisitor(Opcodes.ASM9) { + @Override + public void visitSource(String source, String debug) { + info.sourceFile = source; + } + + @Override + public MethodVisitor visitMethod(int access, String name, String desc, String sig, String[] ex) { + if ((access & Opcodes.ACC_VARARGS) != 0) { + info.varargs.add(name + desc); + } + return null; + } + }, ClassReader.SKIP_CODE | ClassReader.SKIP_FRAMES); + } catch (RuntimeException e) { + return null; + } + return info; + } + } + + /** A method of a unit: its id, display name, first line and leaves in the contract's order. */ + static final class Method { + final String id; + final String display; + final int line; + final int[] leaves; // flat: line, count, kind per leaf + + private Method(String id, String display, int line, int[] leaves) { + this.id = id; + this.display = display; + this.line = line; + this.leaves = leaves; + } + + /** + * of makes the method's record from JaCoCo's per-method lines: one code leaf per line with + * instructions (n = their count), then one branch leaf per branch edge on that line. A method with no + * line (a class built without line numbers) has no leaf to show and is left out, as the contract's + * method count leaves out a method with no code leaf. + */ + static Method of(String vmClass, String nested, String unitName, IMethodCoverage mc, ClassInfo info) { + int first = mc.getFirstLine(); + int last = mc.getLastLine(); + if (first < 1 || last < first) { + return null; + } + int[] buf = new int[16]; + int k = 0; + for (int l = first; l <= last; l++) { + ILine line = mc.getLine(l); + int ic = line.getInstructionCounter().getTotalCount(); + if (ic == 0) { + continue; + } + int bc = line.getBranchCounter().getTotalCount(); + int need = k + 3 * (1 + bc); + if (need > buf.length) { + buf = Arrays.copyOf(buf, Math.max(need, buf.length * 2)); + } + buf[k++] = l; + buf[k++] = ic; + buf[k++] = 0; + for (int b = 0; b < bc; b++) { + buf[k++] = l; + buf[k++] = 1; + buf[k++] = 1; + } + } + boolean varargs = info != null && info.varargs.contains(mc.getName() + mc.getDesc()); + return new Method("m:" + vmClass + "#" + mc.getName() + mc.getDesc(), + Names.method(unitName, nested, mc.getName(), mc.getDesc(), varargs), first, + Arrays.copyOf(buf, k)); + } + + void json(StringBuilder sb) { + sb.append("{\"id\":"); + Json.string(sb, id); + sb.append(",\"name\":"); + Json.string(sb, display); + sb.append(",\"line\":").append(line).append(",\"leaves\":["); + for (int i = 0; i < leaves.length; i += 3) { + if (i > 0) { + sb.append(','); + } + int l = leaves[i]; + sb.append('[').append(l).append(",0,").append(l).append(",0,").append(leaves[i + 1]).append(',') + .append(leaves[i + 2]).append(']'); + } + sb.append("]}"); + } + } + + /** Display names of methods (API-CONTRACT 2.3). */ + static final class Names { + private Names() { + } + + /** + * method names a method for people: "bar(int)", a nested class's "Inner.bar(int)", a constructor as + * its class's nested name ("Foo(String)", "Inner(int)", "Inner.Deep(long)"), the static initializer as + * "static {}", a lambda lambda$bar$0 as "bar lambda #0"; a varargs method's last parameter as + * "Object...". + */ + static String method(String unitName, String nested, String name, String desc, boolean varargs) { + List parts = new ArrayList<>(); + for (String p : nested.split("\\$")) { + if (!p.isEmpty()) { + parts.add(p); + } + } + String prefix = parts.isEmpty() ? "" : join(parts, ".") + "."; + if (name.equals("")) { + String cls = parts.isEmpty() ? unitName : join(parts, "."); + return cls + "(" + params(desc, varargs) + ")"; + } + if (name.equals("")) { + return prefix + "static {}"; + } + if (name.startsWith("lambda$")) { + int d = name.lastIndexOf('$'); + String num = name.substring(d + 1); + if (d > "lambda$".length() && !num.isEmpty() && isDigits(num)) { + return prefix + name.substring("lambda$".length(), d) + " lambda #" + num; + } + } + return prefix + name + "(" + params(desc, varargs) + ")"; + } + + private static boolean isDigits(String s) { + for (int i = 0; i < s.length(); i++) { + if (s.charAt(i) < '0' || s.charAt(i) > '9') { + return false; + } + } + return true; + } + + private static String join(List parts, String sep) { + StringBuilder sb = new StringBuilder(); + for (String p : parts) { + if (sb.length() > 0) { + sb.append(sep); + } + sb.append(p); + } + return sb.toString(); + } + + /** params lists the descriptor's parameter types by simple name, ", "-joined. */ + static String params(String desc, boolean varargs) { + List out = new ArrayList<>(); + int i = desc.indexOf('(') + 1; + int end = desc.indexOf(')'); + if (i <= 0 || end < i) { + return ""; + } + while (i < end) { + int dims = 0; + while (i < end && desc.charAt(i) == '[') { + dims++; + i++; + } + if (i >= end) { + break; + } + String t; + char c = desc.charAt(i); + if (c == 'L') { + int semi = desc.indexOf(';', i); + if (semi < 0 || semi > end) { + break; + } + String cls = desc.substring(i + 1, semi); + t = cls.substring(cls.lastIndexOf('/') + 1).replace('$', '.'); + i = semi + 1; + } else { + t = primitive(c); + i++; + } + StringBuilder sb = new StringBuilder(t); + for (int d = 0; d < dims; d++) { + sb.append("[]"); + } + out.add(sb.toString()); + } + if (varargs && !out.isEmpty()) { + String lastParam = out.get(out.size() - 1); + if (lastParam.endsWith("[]")) { + out.set(out.size() - 1, lastParam.substring(0, lastParam.length() - 2) + "..."); + } + } + return join(out, ", "); + } + + private static String primitive(char c) { + switch (c) { + case 'Z': + return "boolean"; + case 'B': + return "byte"; + case 'C': + return "char"; + case 'S': + return "short"; + case 'I': + return "int"; + case 'J': + return "long"; + case 'F': + return "float"; + case 'D': + return "double"; + case 'V': + return "void"; + default: + return String.valueOf(c); + } + } + } + + /** JSON strings as Go's encoding/json writes them with HTML escaping off. */ + static final class Json { + private Json() { + } + + private static final char[] HEX = "0123456789abcdef".toCharArray(); + + static void string(StringBuilder sb, String s) { + sb.append('"'); + for (int i = 0; i < s.length(); i++) { + char c = s.charAt(i); + switch (c) { + case '"': + sb.append("\\\""); + continue; + case '\\': + sb.append("\\\\"); + continue; + case '\b': + sb.append("\\b"); + continue; + case '\f': + sb.append("\\f"); + continue; + case '\n': + sb.append("\\n"); + continue; + case '\r': + sb.append("\\r"); + continue; + case '\t': + sb.append("\\t"); + continue; + case '\u2028': + sb.append("\\u2028"); + continue; + case '\u2029': + sb.append("\\u2029"); + continue; + default: + break; + } + if (c < 0x20) { + sb.append("\\u00").append(HEX[c >> 4]).append(HEX[c & 0xf]); + } else if (Character.isHighSurrogate(c) && i + 1 < s.length() + && Character.isLowSurrogate(s.charAt(i + 1))) { + sb.append(c).append(s.charAt(++i)); + } else if (Character.isSurrogate(c)) { + // Not UTF-8: Go reads it as U+FFFD, which it writes as is. + sb.append('\ufffd'); + } else { + sb.append(c); + } + } + sb.append('"'); + } + } + + // ---------------------------------------------------------------- frames + + /** Sends one whole frame; it throws when the receiver did not take it. */ + interface FrameSender { + void send(byte[] frame) throws IOException; + } + + /** + * FrameWriter cuts the NDJSON stream of records into TABLE frames of at most MAX_FRAME_BYTES each and + * sends them in order, one at a time. + */ + static final class FrameWriter implements RecordSink { + private final FrameSender sender; + private final int bodyCap; + private final ByteArrayOutputStream body = new ByteArrayOutputStream(); + private final MessageDigest digest; + int seq; + private String build; + + FrameWriter(FrameSender sender) { + this(sender, MAX_FRAME_BYTES); + } + + FrameWriter(FrameSender sender, int maxFrame) { + this.sender = sender; + this.bodyCap = maxFrame - HEADER_ROOM; + if (bodyCap < 1) { + throw new IllegalArgumentException("frame cap " + maxFrame + " leaves no room for a body"); + } + try { + this.digest = MessageDigest.getInstance("SHA-256"); + } catch (NoSuchAlgorithmException e) { + throw new IllegalStateException(e); + } + } + + @Override + public void record(String build, byte[] json) throws IOException { + this.build = build; + digest.update(json); + digest.update((byte) '\n'); + write(json, 0, json.length); + write(new byte[] {'\n'}, 0, 1); + } + + private void write(byte[] b, int off, int len) throws IOException { + while (len > 0) { + int room = bodyCap - body.size(); + if (room == 0) { + flush(false, -1); + continue; + } + int take = Math.min(room, len); + body.write(b, off, take); + off += take; + len -= take; + } + } + + /** finish sends the last frame, with the record count and the stream's sha256. */ + void finish(String build, int records) throws IOException { + this.build = build; + flush(true, records); + } + + private void flush(boolean last, int records) throws IOException { + StringBuilder h = new StringBuilder(160); + h.append("{\"v\":1,\"part\":\"table\",\"build\":"); + Json.string(h, build); + h.append(",\"seq\":").append(seq).append(",\"last\":").append(last); + if (last) { + h.append(",\"records\":").append(records).append(",\"sha256\":\"").append(hex(digest.digest())) + .append('"'); + } + h.append("}\n"); + byte[] header = h.toString().getBytes(StandardCharsets.UTF_8); + byte[] payload = body.toByteArray(); + int n = header.length + payload.length; + byte[] frame = new byte[8 + n]; + System.arraycopy(MAGIC.getBytes(StandardCharsets.US_ASCII), 0, frame, 0, 4); + frame[4] = (byte) (n >>> 24); + frame[5] = (byte) (n >>> 16); + frame[6] = (byte) (n >>> 8); + frame[7] = (byte) n; + System.arraycopy(header, 0, frame, 8, header.length); + System.arraycopy(payload, 0, frame, 8 + header.length, payload.length); + try { + sender.send(frame); + } catch (IOException e) { + throw new IOException("TABLE frame " + seq + ": " + e.getMessage(), e); + } + body.reset(); + seq++; + } + } + + /** + * SocketSender sends each frame on its own connection to the data socket, shuts its output so a receiver + * reading to the end of the connection is never left waiting, and waits for OK. + */ + static final class SocketSender implements FrameSender { + private final File socket; + + SocketSender(File socket) { + this.socket = socket; + } + + @Override + public void send(byte[] frame) throws IOException { + AFUNIXSocket s = connect(); + try { + s.setSoTimeout(REPLY_TIMEOUT_MILLIS); + OutputStream out = s.getOutputStream(); + out.write(frame); + out.flush(); + // Through the channel: junixsocket's Socket.shutdownOutput says "not connected" after connect. + s.getChannel().shutdownOutput(); + BufferedReader in = new BufferedReader(new InputStreamReader(s.getInputStream(), + StandardCharsets.UTF_8)); + String reply = in.readLine(); + if (!"OK".equals(reply)) { + throw new IOException(reply == null ? "Keploy closed the connection without OK" + : "Keploy answered " + reply + ", not OK"); + } + } finally { + closeQuietly(s); + } + } + + private AFUNIXSocket connect() throws IOException { + IOException lastErr = null; + for (int attempt = 0; attempt < CONNECT_ATTEMPTS; attempt++) { + if (attempt > 0) { + sleep(1000L * attempt); + } + AFUNIXSocket s = AFUNIXSocket.newInstance(); + try { + s.connect(AFUNIXSocketAddress.of(socket), CONNECT_TIMEOUT_MILLIS); + return s; + } catch (IOException e) { + lastErr = e; + closeQuietly(s); + } + } + throw new IOException("no Keploy listens on the data socket " + socket + " (" + lastErr + ")", + lastErr); + } + + private static void sleep(long millis) { + try { + Thread.sleep(millis); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + } + + private static void closeQuietly(AFUNIXSocket s) { + try { + s.close(); + } catch (IOException ignored) { + // nothing to do + } + } + } +} diff --git a/keploy-sdk/src/main/java/io/keploy/dedup/KeployDedupAgent.java b/keploy-sdk/src/main/java/io/keploy/dedup/KeployDedupAgent.java index 0e13ce83..b39d02a0 100644 --- a/keploy-sdk/src/main/java/io/keploy/dedup/KeployDedupAgent.java +++ b/keploy-sdk/src/main/java/io/keploy/dedup/KeployDedupAgent.java @@ -807,6 +807,18 @@ private void handle(Socket socket) { return; } + if (BuildTable.COMMAND.equals(line.trim())) { // the build table: ACK, then sent on its own thread + boolean asked; + try { + asked = BuildTable.request(() -> BuildTable.files(collector.coverageIndex.entries(), + e -> e.className, e -> e.bytes), new File(dataSocketPath)); + } catch (Throwable t) { // thrown past here, it would end the accept loop: IGNORED instead + log(Level.WARNING, "Keploy coverage: the build table was not started", t); + asked = false; + } + writeReply(commandSocket.getOutputStream(), asked ? UNIX_REPLY_TAKEN : UNIX_REPLY_IGNORED); + return; + } boolean taken; try { taken = run(line.trim()); diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java new file mode 100644 index 00000000..317bbac1 --- /dev/null +++ b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java @@ -0,0 +1,769 @@ +package io.keploy.dedup; + +import com.google.gson.JsonArray; +import com.google.gson.JsonElement; +import com.google.gson.JsonObject; +import com.google.gson.JsonParser; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; +import org.jacoco.core.internal.data.CRC64; +import org.objectweb.asm.ClassWriter; +import org.objectweb.asm.Label; +import org.objectweb.asm.MethodVisitor; +import org.objectweb.asm.Opcodes; +import org.newsclub.net.unix.AFUNIXServerSocket; +import org.newsclub.net.unix.AFUNIXSocketAddress; + +import java.io.ByteArrayOutputStream; +import java.io.DataInputStream; +import java.io.File; +import java.io.IOException; +import java.io.InputStream; +import java.net.Socket; +import java.net.URISyntaxException; +import java.net.URL; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.stream.Collectors; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.logging.Handler; +import java.util.logging.Level; +import java.util.logging.LogRecord; +import java.util.logging.Logger; +import java.util.stream.Stream; + +import static org.junit.Assert.assertArrayEquals; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +/** + * Rows for the build table: the java-dedup sample's classes (compiled once with javac --release 8 -g from + * keploy/samples-java java-dedup, kept as .class.bin) encode to the golden table byte for byte. The golden + * (java-dedup.table.ndjson, java-dedup.build) is the Go encoder's output (enterprise pkg/service/codecoverage + * NewTable + EncodeTable) over these records, with the hashes and the build id computed in Go from the class + * bytes, so the order, the JSON form, the fold and the ids are Go's, not this class's. So is edge.table.ndjson, + * over a set of classes generated here ({@link #edge}) for the names, packages and escapes the sample lacks. + * + *

To regenerate a golden: run these rows with -Dbuildtable.dump=DIR (each golden row writes DIR/NAME.ndjson + * and the class files as DIR/NAME/VM-NAME.class.bin), then in an enterprise checkout read each record with + * codecoverage's Unit JSON, recompute its hash and the build id from the class files with JaCoCo's CRC64, + * NewTable(LanguageJava, build, units) and EncodeTable, and write the result back over the golden here. + */ +public class BuildTableTest { + + private static final String RES = "/io/keploy/dedup/buildtable/"; + + @Rule + public TemporaryFolder tmp = new TemporaryFolder(); + + static List javaDedup() throws IOException, URISyntaxException { + URL root = BuildTableTest.class.getResource(RES + "java-dedup"); + assertNotNull("java-dedup class files", root); + Path dir = Paths.get(root.toURI()); + List out = new ArrayList<>(); + try (Stream walk = Files.walk(dir)) { + for (Path p : walk.filter(p -> p.toString().endsWith(".class.bin")).sorted().collect(Collectors.toList())) { + String rel = dir.relativize(p).toString().replace(File.separatorChar, '/'); + out.add(new BuildTable.ClassFile(rel.substring(0, rel.length() - ".class.bin".length()), + Files.readAllBytes(p))); + } + } + assertEquals(5, out.size()); + return out; + } + + static byte[] resource(String name) throws IOException { + try (InputStream in = BuildTableTest.class.getResourceAsStream(RES + name)) { + assertNotNull(name, in); + ByteArrayOutputStream b = new ByteArrayOutputStream(); + byte[] buf = new byte[8192]; + for (int n; (n = in.read(buf)) > 0; ) { + b.write(buf, 0, n); + } + return b.toByteArray(); + } + } + + /** Collects the records write hands out, as the NDJSON stream. */ + static final class Collect implements BuildTable.RecordSink { + final ByteArrayOutputStream ndjson = new ByteArrayOutputStream(); + final List records = new ArrayList<>(); + String build; + + @Override + public void record(String build, byte[] json) { + this.build = build; + records.add(new String(json, StandardCharsets.UTF_8)); + ndjson.write(json, 0, json.length); + ndjson.write('\n'); + } + } + + static Collect table(List files) throws IOException { + Collect c = new Collect(); + assertNotNull(BuildTable.write(files, c)); + return c; + } + + /** dump writes a golden row's output and class files under -Dbuildtable.dump, to regenerate its golden. */ + static void dump(String name, List files, Collect c) throws IOException { + String dir = System.getProperty("buildtable.dump"); + if (dir == null) { + return; + } + Files.createDirectories(Paths.get(dir)); + Files.write(Paths.get(dir, name + ".ndjson"), c.ndjson.toByteArray()); + for (BuildTable.ClassFile f : files) { + Path p = Paths.get(dir, name, f.name + ".class.bin"); + Files.createDirectories(p.getParent()); + Files.write(p, f.bytes); + } + } + + @Test + public void javaDedupTableIsTheGoEncodersByteForByte() throws Exception { + List files = javaDedup(); + Collect c = table(files); + dump("java-dedup", files, c); + assertEquals(new String(resource("java-dedup.build"), StandardCharsets.UTF_8).trim(), c.build); + assertEquals(new String(resource("java-dedup.table.ndjson"), StandardCharsets.UTF_8), + new String(c.ndjson.toByteArray(), StandardCharsets.UTF_8)); + // The same classes in another order give the same bytes. + List reversed = new ArrayList<>(files); + java.util.Collections.reverse(reversed); + assertArrayEquals(c.ndjson.toByteArray(), table(reversed).ndjson.toByteArray()); + } + + static Map byId(List lines) { + Map out = new LinkedHashMap<>(); + for (String l : lines) { + if (l.trim().isEmpty()) { + continue; + } + JsonObject o = JsonParser.parseString(l).getAsJsonObject(); + out.put(o.get("id").getAsString(), o); + } + return out; + } + + static Map methods(JsonObject unit) { + Map out = new LinkedHashMap<>(); + for (JsonElement m : unit.getAsJsonArray("methods")) { + out.put(m.getAsJsonObject().get("id").getAsString(), m.getAsJsonObject()); + } + return out; + } + + // per line: [code leaves, branch edges] + static Map> lineShape(JsonObject method) { + Map> out = new LinkedHashMap<>(); + for (JsonElement e : method.getAsJsonArray("leaves")) { + JsonArray l = e.getAsJsonArray(); + List s = out.computeIfAbsent(l.get(0).getAsInt(), k -> new ArrayList<>(Arrays.asList(0, 0))); + s.set(l.get(5).getAsInt(), s.get(l.get(5).getAsInt()) + 1); + } + return out; + } + + /** + * The contract fixture (fixtures/java-dedup) is hand-made: its leaf counts (n = 3), hashes and the model + * classes' lines are illustrative, it lists only some of DedupController's methods and not Product, and + * its simpleMessage takes jakarta's HttpServletRequest where the sample imports javax's. What it holds as + * real must agree: every fixture class is a record with its id, pkg, name and path; every fixture method + * of DedupController is a method with the same display name and first line; and the two methods with + * branches have the fixture's lines and branch edges. + */ + @Test + public void javaDedupTableAgreesWithTheContractFixture() throws Exception { + Map ours = byId(table(javaDedup()).records); + Map fixture = byId(Arrays.asList( + new String(resource("fixture-java-dedup-table.ndjson"), StandardCharsets.UTF_8).split("\n"))); + assertEquals(4, fixture.size()); + for (JsonObject f : fixture.values()) { + String id = f.get("id").getAsString(); + JsonObject o = ours.get(id); + assertNotNull(id, o); + for (String k : new String[] {"kind", "pkg", "name", "path", "state"}) { + assertEquals(id + " " + k, f.get(k), o.get(k)); + } + } + String ctl = "c:io/keploy/samples/javadedup/DedupController"; + Map fm = methods(fixture.get(ctl)); + Map om = methods(ours.get(ctl)); + for (Map.Entry e : fm.entrySet()) { + String mid = e.getKey().replace("jakarta/servlet", "javax/servlet"); + JsonObject o = om.get(mid); + assertNotNull(mid, o); + assertEquals(mid, e.getValue().get("name"), o.get("name")); + assertEquals(mid, e.getValue().get("line"), o.get("line")); + } + for (String m : new String[] {"messageForPath(Ljava/lang/String;)Ljava/lang/String;", + "response([Ljava/lang/Object;)Ljava/util/Map;"}) { + String mid = "m:io/keploy/samples/javadedup/DedupController#" + m; + assertEquals(mid, lineShape(fm.get(mid)), lineShape(om.get(mid))); + } + } + + static byte[] patchedMajor(byte[] bytes, int major) { + byte[] b = bytes.clone(); + b[6] = (byte) (major >> 8); + b[7] = (byte) major; + return b; + } + + @Test + public void aClassJacocoCannotAnalyzeIsNotMeasuredAndSoIsItsUnit() throws Exception { + List files = new ArrayList<>(javaDedup()); + BuildTable.ClassFile item = null; + for (BuildTable.ClassFile f : files) { + if (f.name.endsWith("/model/Item")) { + item = f; + } + } + assertNotNull(item); + // A nested class of Item no ASM in this SDK reads (class file version 99), and a class that is + // not a class file at all. + files.add(new BuildTable.ClassFile(item.name + "$Later", patchedMajor(item.bytes, 99))); + files.add(new BuildTable.ClassFile("io/keploy/samples/javadedup/Junk", "not a class".getBytes(StandardCharsets.UTF_8))); + Map ours = byId(table(files).records); + Map golden = byId(Arrays.asList( + new String(resource("java-dedup.table.ndjson"), StandardCharsets.UTF_8).split("\n"))); + + JsonObject it = ours.get("c:io/keploy/samples/javadedup/model/Item"); + assertEquals("not_measured", it.get("state").getAsString()); + assertEquals("{\"code\":\"analyzer_failed\"}", it.get("reason").toString()); + assertEquals(0, it.getAsJsonArray("methods").size()); + assertEquals("its own class file still names the path", "io/keploy/samples/javadedup/model/Item.java", + it.get("path").getAsString()); + assertFalse("the hash takes the new class file's id", golden.get(it.get("id").getAsString()) + .get("hash").equals(it.get("hash"))); + + JsonObject junk = ours.get("c:io/keploy/samples/javadedup/Junk"); + assertEquals("not_measured", junk.get("state").getAsString()); + assertEquals("", junk.get("path").getAsString()); + assertEquals("[]", junk.get("methods").toString()); + + // The other classes are as they were. + for (String id : golden.keySet()) { + if (!id.endsWith("/model/Item")) { + assertEquals(id, golden.get(id), ours.get(id)); + } + } + } + + @Test + public void aRecordLargerThanAChunkInflatesToIsRecordTooLarge() throws Exception { + Collect c = new Collect(); + BuildTable.Summary s = BuildTable.write(javaDedup(), c, 1600); + assertEquals(5, s.records); + int tooLarge = 0; + for (String r : c.records) { + assertTrue(r, r.length() + 1 <= 1600); + if (r.contains("\"record_too_large\"")) { + tooLarge++; + assertTrue(r, r.contains("\"state\":\"not_measured\"") && r.endsWith("\"methods\":[]}")); + } + } + // DedupController and Product are the two records past 1600 bytes. + assertEquals(2, tooLarge); + assertEquals(2, s.notMeasured); + } + + /** Reads frames back: each is KTB1, a length, a header line and the next bytes of the stream. */ + static final class Frames implements BuildTable.FrameSender { + final List frames = new ArrayList<>(); + + @Override + public void send(byte[] frame) { + frames.add(frame); + } + } + + @Test + public void framesCarryTheStreamInOrderAndNeverPassTheCap() throws Exception { + Collect c = table(javaDedup()); + for (int cap : new int[] {1100, 1500, BuildTable.MAX_FRAME_BYTES}) { + Frames sent = new Frames(); + BuildTable.FrameWriter w = new BuildTable.FrameWriter(sent, cap); + BuildTable.Summary s = BuildTable.write(javaDedup(), w); + w.finish(s.build, s.records); + ByteArrayOutputStream stream = new ByteArrayOutputStream(); + for (int i = 0; i < sent.frames.size(); i++) { + byte[] f = sent.frames.get(i); + assertTrue("frame " + i + " is " + f.length + " bytes", f.length <= cap); + assertEquals(BuildTable.MAGIC, new String(f, 0, 4, StandardCharsets.US_ASCII)); + DataInputStream in = new DataInputStream(new java.io.ByteArrayInputStream(f, 4, f.length - 4)); + int n = in.readInt(); + assertEquals(f.length - 8, n); + int nl = 8; + while (f[nl] != '\n') { + nl++; + } + JsonObject h = JsonParser.parseString(new String(f, 8, nl - 8, StandardCharsets.UTF_8)).getAsJsonObject(); + assertEquals(1, h.get("v").getAsInt()); + assertEquals("table", h.get("part").getAsString()); + assertEquals(c.build, h.get("build").getAsString()); + assertEquals(i, h.get("seq").getAsInt()); + boolean last = i == sent.frames.size() - 1; + assertEquals(last, h.get("last").getAsBoolean()); + if (last) { + assertEquals(5, h.get("records").getAsInt()); + assertEquals(BuildTable.sha256Hex(c.ndjson.toByteArray()), h.get("sha256").getAsString()); + } else { + assertFalse(h.has("records")); + } + stream.write(f, nl + 1, f.length - nl - 1); + } + if (cap < 2000) { + assertTrue(sent.frames.size() > 5); + } else { + assertEquals(1, sent.frames.size()); + } + assertArrayEquals(c.ndjson.toByteArray(), stream.toByteArray()); + } + } + + @Test + public void theGoJsonFormIsKept() { + StringBuilder sb = new StringBuilder(); + BuildTable.Json.string(sb, "#&\"\\\b\f\n\r\t\u0001\u001f\u007f

é😀\uD800x"); + assertEquals("\"#&\\\"\\\\\\b\\f\\n\\r\\t\\u0001\\u001f\u007f\\u2028\\u2029é😀�x\"", + sb.toString()); + assertTrue(BuildTable.comparePkg("a", "a/b") < 0); + assertTrue(BuildTable.comparePkg("a/b", "a/b/c") < 0); + assertTrue(BuildTable.comparePkg("a/b/c", "a-x") < 0); + assertTrue(BuildTable.comparePkg("", "a") < 0); + // Code point order, as Go orders UTF-8 bytes: U+FF5E sorts before U+1F600. + assertTrue(BuildTable.compareCodePoints("~", "😀") < 0); + } + + @Test + public void namesFollowTheContract() { + assertEquals("Foo(String, int[])", BuildTable.Names.method("Foo", "", "", "(Ljava/lang/String;[I)V", false)); + assertEquals("Inner.Deep(long)", BuildTable.Names.method("Foo", "$Inner$Deep", "", "(J)V", false)); + assertEquals("Inner.bar(int)", BuildTable.Names.method("Foo", "$Inner", "bar", "(I)V", false)); + assertEquals("static {}", BuildTable.Names.method("Foo", "", "", "()V", false)); + assertEquals("bar lambda #0", BuildTable.Names.method("Foo", "", "lambda$bar$0", "(Ljava/lang/Object;)V", false)); + assertEquals("response(Object...)", BuildTable.Names.method("Foo", "", "response", "([Ljava/lang/Object;)V", true)); + assertEquals("of(Map.Entry, byte[][])", BuildTable.Names.method("Foo", "", "of", "(Ljava/util/Map$Entry;[[B)V", false)); + assertEquals("apply()", BuildTable.Names.method("Foo", "$", "apply", "()V", false)); + } + + static String header(byte[] frame) { + int nl = 8; + while (frame[nl] != '\n') { + nl++; + } + return new String(frame, 8, nl - 8, StandardCharsets.UTF_8); + } + + @Test + public void noClassesIsOneEmptyLastFrame() throws Exception { + BuildTable.Summary s = BuildTable.write(new ArrayList(), new Collect()); + assertEquals("", s.build); + assertEquals(0, s.records); + Frames f = new Frames(); + BuildTable.send(ArrayList::new, f); + assertEquals(1, f.frames.size()); + assertEquals("{\"v\":1,\"part\":\"table\",\"build\":\"\",\"seq\":0,\"last\":true,\"records\":0,\"sha256\":" + + "\"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\"}", header(f.frames.get(0))); + } + + // ---------------------------------------------------------------- an edge set, against Go + + /** + * edgeClass generates a class: a constructor, run (varargs, an if and a switch), a lambda, a method on + * run's first line, two methods with odd names (a quote, #, $ and U+2028) and a static initializer on line 1; with lines false, none + * of them has line numbers. + */ + static byte[] edgeClass(String name, String source, boolean lines, int base) { + ClassWriter cw = new ClassWriter(ClassWriter.COMPUTE_FRAMES | ClassWriter.COMPUTE_MAXS); + cw.visit(Opcodes.V1_8, Opcodes.ACC_PUBLIC | Opcodes.ACC_SUPER, name, null, "java/lang/Object", null); + if (source != null) { + cw.visitSource(source, null); + } + MethodVisitor mv = cw.visitMethod(Opcodes.ACC_PUBLIC, "", "()V", null, null); + mv.visitCode(); + Label l0 = new Label(); + mv.visitLabel(l0); + if (lines) { + mv.visitLineNumber(base, l0); + } + mv.visitVarInsn(Opcodes.ALOAD, 0); + mv.visitMethodInsn(Opcodes.INVOKESPECIAL, "java/lang/Object", "", "()V", false); + mv.visitInsn(Opcodes.RETURN); + mv.visitMaxs(0, 0); + mv.visitEnd(); + int line = base + 1; + for (String m : new String[] {"run", "lambda$run$0", "same", "we\"ird#na$me", "a\u2028b"}) { + boolean run = m.equals("run"); + mv = cw.visitMethod(Opcodes.ACC_PUBLIC | Opcodes.ACC_STATIC | (run ? Opcodes.ACC_VARARGS : 0), m, + run ? "(I[Ljava/lang/String;)I" : "(I)I", null, null); + mv.visitCode(); + int first = m.equals("same") ? base + 1 : line; + Label a = new Label(); + mv.visitLabel(a); + if (lines) { + mv.visitLineNumber(first, a); + } + mv.visitVarInsn(Opcodes.ILOAD, 0); + Label els = new Label(); + mv.visitJumpInsn(Opcodes.IFEQ, els); + Label b = new Label(); + mv.visitLabel(b); + if (lines) { + mv.visitLineNumber(first + 1, b); + } + mv.visitInsn(Opcodes.ICONST_1); + mv.visitInsn(Opcodes.IRETURN); + mv.visitLabel(els); + Label c = new Label(); + mv.visitLabel(c); + if (lines) { + mv.visitLineNumber(first + 2, c); + } + mv.visitVarInsn(Opcodes.ILOAD, 0); + Label d1 = new Label(); + Label d2 = new Label(); + Label d3 = new Label(); + Label dflt = new Label(); + mv.visitTableSwitchInsn(1, 3, dflt, d1, d2, d3); + mv.visitLabel(d1); + mv.visitInsn(Opcodes.ICONST_2); + mv.visitInsn(Opcodes.IRETURN); + mv.visitLabel(d2); + mv.visitInsn(Opcodes.ICONST_3); + mv.visitInsn(Opcodes.IRETURN); + mv.visitLabel(d3); + mv.visitInsn(Opcodes.ICONST_4); + mv.visitInsn(Opcodes.IRETURN); + mv.visitLabel(dflt); + mv.visitInsn(Opcodes.ICONST_0); + mv.visitInsn(Opcodes.IRETURN); + mv.visitMaxs(0, 0); + mv.visitEnd(); + line += 10; + } + mv = cw.visitMethod(Opcodes.ACC_STATIC, "", "()V", null, null); + mv.visitCode(); + Label z = new Label(); + mv.visitLabel(z); + if (lines) { + mv.visitLineNumber(1, z); + } + mv.visitInsn(Opcodes.RETURN); + mv.visitMaxs(0, 0); + mv.visitEnd(); + cw.visitEnd(); + return cw.toByteArray(); + } + + /** The edge set: VM name and SourceFile (null: none). */ + static final String[][] EDGE = { + {"a/b/Foo", "Foo.java"}, {"a/b/Foo$Bar", "Foo.java"}, {"a/b/Foo$1", "Foo.java"}, {"a/b/Foo$", "Foo.scala"}, + {"a/b/Foo$Bar$Deep", null}, {"a-x/Baz", "Baz.java"}, {"a/Qux", "sub/Qux.java"}, {"a/b/c/Deep", ""}, + {"Default", "Default.java"}, {"$Lead", "Lead.java"}, {"$Lead$In", "Lead.java"}, + {"a/ü/Ünï", "Ünï.java"}, {"a/b/Z~", "Z.java"}, {"a/b/Z😀", "Z.java"}, + {"a/b/Q\"uo\\te<&>", "Q\"uo\\te.java"}, {"a/b/Ctl\u0001\u001f\u007f", "C
.java"}, + {"a/b/L
S
", "L.java"}, {"a/b/NoLines", "NoLines.java"}, {"a/b/Foo$Bar$", null}, + {"a/b/Foo$$anon$1", "Foo.scala"}, {"a/b/module-info", null}, {"a/b/package-info", "package-info.java"}, + }; + + static List edge() { + List out = new ArrayList<>(); + int base = 10; + for (String[] e : EDGE) { + out.add(new BuildTable.ClassFile(e[0], edgeClass(e[0], e[1], !e[0].endsWith("NoLines"), base))); + base += 7; + } + return out; + } + + @Test + public void anEdgeSetIsTheGoEncodersByteForByte() throws Exception { + List files = edge(); + Collect c = table(files); + dump("edge", files, c); + assertEquals(new String(resource("edge.build"), StandardCharsets.UTF_8).trim(), c.build); + assertEquals(new String(resource("edge.table.ndjson"), StandardCharsets.UTF_8), + new String(c.ndjson.toByteArray(), StandardCharsets.UTF_8)); + BuildTable.Summary s = BuildTable.write(files, new Collect()); + assertEquals(15, s.records); + assertEquals("NoLines has code and no line numbers", 1, s.noLines); + assertEquals(0, s.notMeasured); + } + + @Test + public void aClassIsNamedByItsBytesNotByWhereTheIndexFoundIt() throws Exception { + byte[] foo = edgeClass("com/acme/Foo", "Foo.java", true, 10); + byte[] in = edgeClass("com/acme/Foo$In", "Foo.java", true, 20); + byte[] junk = "not a class".getBytes(StandardCharsets.UTF_8); + Collect c = new Collect(); + BuildTable.Summary s = BuildTable.write(Arrays.asList(new BuildTable.ClassFile("classes/com/acme/Foo", foo), + new BuildTable.ClassFile("a/", in), new BuildTable.ClassFile("a/", junk)), c); + assertEquals(1, s.records); + assertEquals("not a class, and its index name names no class", 1, s.unnamed); + JsonObject u = JsonParser.parseString(c.records.get(0)).getAsJsonObject(); + assertEquals("c:com/acme/Foo", u.get("id").getAsString()); + assertEquals("com/acme", u.get("pkg").getAsString()); + assertEquals("com/acme/Foo.java", u.get("path").getAsString()); + assertEquals(BuildTable.classIdsSum(new long[] {CRC64.classId(foo), CRC64.classId(in)}), + u.get("hash").getAsString()); + // Every indexed class is in the build id, as in the index record's, named or not. + assertEquals("jv1-" + BuildTable.classIdsSum(new long[] {CRC64.classId(foo), CRC64.classId(in), + CRC64.classId(junk)}).substring(0, 32), s.build); + assertTrue(c.records.get(0), c.records.get(0).contains("\"id\":\"m:com/acme/Foo$In#run(I[Ljava/lang/String;)I\"")); + } + + // ---------------------------------------------------------------- the data socket + + enum Mode { + /** reads the whole frame and answers OK */ + OK, + /** reads at most 512 bytes and closes, as the JSON decoder of an older Keploy does */ + READ_512_AND_CLOSE, + /** reads to the end of the connection and closes without an answer */ + READ_TO_EOF + } + + /** A receiver on a unix socket: what it read on each connection, and its answer per mode. */ + static final class Receiver implements AutoCloseable { + final AFUNIXServerSocket server; + final List got = new ArrayList<>(); + + Receiver(File path, Mode mode) throws IOException { + server = AFUNIXServerSocket.newInstance(); + server.bind(AFUNIXSocketAddress.of(path)); + Thread thread = new Thread(() -> { + while (!server.isClosed()) { + try (Socket s = server.accept()) { + s.setSoTimeout(15_000); + InputStream in = s.getInputStream(); + byte[] read; + if (mode == Mode.OK) { + DataInputStream d = new DataInputStream(in); + byte[] magic = new byte[4]; + d.readFully(magic); + byte[] body = new byte[d.readInt()]; + d.readFully(body); + read = body; + s.getOutputStream().write("OK\n".getBytes(StandardCharsets.UTF_8)); + s.getOutputStream().flush(); + } else if (mode == Mode.READ_512_AND_CLOSE) { + byte[] b = new byte[512]; + int n = in.read(b); + read = Arrays.copyOf(b, Math.max(n, 0)); + } else { + ByteArrayOutputStream all = new ByteArrayOutputStream(); + byte[] b = new byte[8192]; + for (int n; (n = in.read(b)) > 0; ) { + all.write(b, 0, n); + } + read = all.toByteArray(); + } + synchronized (got) { + got.add(read); + } + } catch (IOException ignored) { + // closed, or the sender went away + } + } + }, "test-table-receiver"); + thread.setDaemon(true); + thread.start(); + } + + int count() { + synchronized (got) { + return got.size(); + } + } + + @Override + public void close() throws IOException { + server.close(); + } + } + + /** Logs collects what BuildTable logs. */ + static final class Logs extends Handler implements AutoCloseable { + final List records = new java.util.concurrent.CopyOnWriteArrayList<>(); + private final Logger logger = Logger.getLogger(BuildTable.class.getName()); + + Logs() { + logger.addHandler(this); + } + + @Override + public void publish(LogRecord record) { + records.add(record); + } + + @Override + public void flush() { + } + + @Override + public void close() { + logger.removeHandler(this); + } + } + + static List javaDedupUnchecked() { + try { + return javaDedup(); + } catch (Exception e) { + throw new IllegalStateException(e); + } + } + + @Test(timeout = 30_000) + public void theSocketSenderSendsEveryFrameEachAnsweredOk() throws Exception { + File sock = new File(tmp.getRoot(), "data.sock"); + try (Receiver r = new Receiver(sock, Mode.OK)) { + BuildTable.FrameWriter w = new BuildTable.FrameWriter(new BuildTable.SocketSender(sock), 1500); + BuildTable.Summary s = BuildTable.write(javaDedup(), w); + w.finish(s.build, s.records); + assertTrue(w.seq > 1); + assertEquals(w.seq, r.count()); + } + } + + @Test(timeout = 30_000) + public void aReceiverThatClosesWithoutOkEndsTheStreamAtOnceAndNothingThrows() throws Exception { + for (Mode mode : new Mode[] {Mode.READ_512_AND_CLOSE, Mode.READ_TO_EOF}) { + File sock = new File(tmp.getRoot(), mode + ".sock"); + try (Receiver r = new Receiver(sock, mode); Logs logs = new Logs()) { + long t0 = System.nanoTime(); + BuildTable.FrameWriter w = new BuildTable.FrameWriter(new BuildTable.SocketSender(sock), 1500); + try { + BuildTable.Summary s = BuildTable.write(javaDedup(), w); + w.finish(s.build, s.records); + fail(mode + ": no OK, so the stream stops"); + } catch (IOException e) { + assertTrue(mode + ": " + e.getMessage(), e.getMessage().startsWith("TABLE frame 0: ")); + } + assertEquals(mode.toString(), 0, w.seq); + assertTrue(mode + ": the sender shut its output, so no read waited for the reply timeout", + System.nanoTime() - t0 < TimeUnit.SECONDS.toNanos(10)); + if (mode == Mode.READ_TO_EOF) { + while (r.count() == 0) { + Thread.sleep(10); + } + synchronized (r.got) { + assertEquals(1, r.got.size()); + byte[] f = r.got.get(0); + assertEquals(BuildTable.MAGIC, new String(f, 0, 4, StandardCharsets.US_ASCII)); + int n = ((f[4] & 0xff) << 24) | ((f[5] & 0xff) << 16) | ((f[6] & 0xff) << 8) | (f[7] & 0xff); + assertEquals("the whole frame, then the end of the connection", 8 + n, f.length); + } + } + // send never throws: it says why at WARNING, with no stack trace. + BuildTable.send(BuildTableTest::javaDedupUnchecked, new BuildTable.SocketSender(sock)); + LogRecord last = logs.records.get(logs.records.size() - 1); + assertEquals(Level.WARNING, last.getLevel()); + assertTrue(last.getMessage(), last.getMessage().contains("the build table was not sent: TABLE frame 0: ")); + assertNull(last.getThrown()); + } + } + } + + @Test(timeout = 30_000) + public void noReceiverIsSaidSo() throws Exception { + try (Logs logs = new Logs()) { + BuildTable.send(BuildTableTest::javaDedupUnchecked, + new BuildTable.SocketSender(new File(tmp.getRoot(), "none.sock"))); + LogRecord last = logs.records.get(logs.records.size() - 1); + assertTrue(last.getMessage(), last.getMessage().contains("TABLE frame 0: no Keploy listens on the data socket")); + } + } + + @Test(timeout = 30_000) + public void aRequestedTableArrivesWithTheBuildId() throws Exception { + File sock = new File(tmp.getRoot(), "table.sock"); + try (Receiver r = new Receiver(sock, Mode.OK)) { + BuildTable.Sends sends = new BuildTable.Sends(BuildTable::thread); + assertTrue(sends.request(() -> BuildTable.send(BuildTableTest::javaDedupUnchecked, + new BuildTable.SocketSender(sock)))); + String last = null; + while (last == null) { + synchronized (r.got) { + for (byte[] b : r.got) { + String h = new String(b, StandardCharsets.UTF_8); + if (h.contains("\"last\":true")) { + last = h.substring(0, h.indexOf('\n')); + } + } + } + Thread.sleep(20); + } + String build = new String(resource("java-dedup.build"), StandardCharsets.UTF_8).trim(); + assertTrue(last, last.startsWith("{\"v\":1,\"part\":\"table\",\"build\":\"" + build + "\",")); + assertTrue(last, last.contains("\"records\":5,")); + } + } + + @Test(timeout = 30_000) + public void aRequestDuringASendRunsItOnceMore() throws Exception { + CountDownLatch inFirst = new CountDownLatch(1); + CountDownLatch release = new CountDownLatch(1); + AtomicInteger runs = new AtomicInteger(); + Runnable send = () -> { + if (runs.incrementAndGet() == 1) { + inFirst.countDown(); + try { + release.await(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + } + }; + BuildTable.Sends sends = new BuildTable.Sends(BuildTable::thread); + assertTrue(sends.request(send)); + assertTrue(inFirst.await(10, TimeUnit.SECONDS)); + assertTrue(sends.request(send)); + assertTrue(sends.request(send)); + release.countDown(); + while (runs.get() < 2) { + Thread.sleep(10); + } + Thread.sleep(300); + assertEquals("two requests during a send: one more run", 2, runs.get()); + assertTrue(sends.request(send)); + while (runs.get() < 3) { + Thread.sleep(10); + } + } + + @Test + public void aThreadThatCannotStartIsIgnoredNeverThrown() throws Exception { + AtomicInteger made = new AtomicInteger(); + BuildTable.Sends sends = new BuildTable.Sends(r -> { + if (made.incrementAndGet() == 1) { + throw new OutOfMemoryError("unable to create native thread"); + } + return BuildTable.thread(r); + }); + CountDownLatch ran = new CountDownLatch(1); + try (Logs logs = new Logs()) { + assertFalse(sends.request(ran::countDown)); + assertEquals(Level.WARNING, logs.records.get(logs.records.size() - 1).getLevel()); + } + assertTrue("nothing is left running: the next request starts", sends.request(ran::countDown)); + assertTrue(ran.await(10, TimeUnit.SECONDS)); + } +} diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/ControlSocketTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/ControlSocketTest.java index c365d74b..e4a8fd0e 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/ControlSocketTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/ControlSocketTest.java @@ -19,6 +19,8 @@ import java.nio.file.Files; import java.nio.file.attribute.FileTime; import java.nio.file.attribute.PosixFilePermission; +import java.security.MessageDigest; +import java.security.NoSuchAlgorithmException; import java.util.List; import java.util.concurrent.BlockingQueue; import java.util.concurrent.CopyOnWriteArrayList; @@ -52,6 +54,7 @@ public class ControlSocketTest { private static final String JACOCO_PORT_PROPERTY = "keploy.jacoco.port"; private static final String WARMUP_DISABLED_PROPERTY = "keploy.java.dedup.warmup.disabled"; + private static final String CLASS_DIRS_PROPERTY = "keploy.java.class.dirs"; private static final String TAKEN = "ACK proto=2\n"; private static final String IGNORED = "IGNORED proto=2\n"; @@ -107,19 +110,24 @@ public void tearDown() throws Exception { KeployDedupAgent.dataSocketPath = savedDataPath; System.clearProperty(JACOCO_PORT_PROPERTY); System.clearProperty(WARMUP_DISABLED_PROPERTY); + System.clearProperty(CLASS_DIRS_PROPERTY); if (jacoco != null) { jacoco.close(); } if (data != null) { data.close(); } - File[] left = dir.listFiles(); + delete(dir); + } + + private static void delete(File f) { + File[] left = f.listFiles(); if (left != null) { - for (File f : left) { - f.delete(); + for (File c : left) { + delete(c); } } - dir.delete(); + f.delete(); } @Test @@ -170,6 +178,67 @@ public void aTestStartedAndEndedIsAnsweredTakenAndItsCoverageIsSent() throws Exc assertTrue(payload, payload.contains("\"id\":\"test-set-0/test-1\"")); } + @Test(timeout = 60000) + public void theBuildTableGoesOnlyWhenKeployAsksForIt() throws Exception { + // The index is the java-dedup sample's classes, so the table is the Go encoder's golden for them. + File classes = new File(dir, "classes"); + for (BuildTable.ClassFile f : BuildTableTest.javaDedup()) { + File out = new File(classes, f.name + ".class"); + assertTrue(out.getParentFile().isDirectory() || out.getParentFile().mkdirs()); + Files.write(out.toPath(), f.bytes); + } + System.setProperty(CLASS_DIRS_PROPERTY, classes.getPath()); + startWithJacoco(); + data = new DataReceiver(KeployDedupAgent.dataSocketPath); + int asked = BuildTable.requests(); + assertEquals(TAKEN, send("START test-set-0/test-1")); + assertEquals(TAKEN, send("END test-set-0/test-1")); + String payload = data.next(); + assertNotNull("the END's coverage reached the data socket", payload); + assertTrue(payload, payload.contains("\"id\":\"test-set-0/test-1\"")); + // Commands are handled one at a time, so once END is answered all START and END asked for is asked. + // An older Keploy never sends TABLE: it never gets a frame. + assertEquals("START and END never ask for the table", asked, BuildTable.requests()); + assertNull("no table frame without TABLE", data.poll()); + + assertEquals("answered at once; the table goes on its own thread", TAKEN, send("TABLE")); + assertEquals(asked + 1, BuildTable.requests()); + String frame = data.next(); + assertNotNull("the TABLE's frame reached the data socket", frame); + assertTrue(frame, frame.startsWith("KTB1")); + int header = frame.indexOf("{\"v\":1,"); + int body = frame.indexOf('\n', header) + 1; + assertTrue(frame, header > 0 && body > header); + byte[] golden = BuildTableTest.resource("java-dedup.table.ndjson"); + String build = new String(BuildTableTest.resource("java-dedup.build"), StandardCharsets.UTF_8).trim(); + assertEquals("{\"v\":1,\"part\":\"table\",\"build\":\"" + build + "\",\"seq\":0,\"last\":true," + + "\"records\":5,\"sha256\":\"" + sha256(golden) + "\"}\n", frame.substring(header, body)); + assertEquals(new String(golden, StandardCharsets.UTF_8), frame.substring(body)); + } + + @Test(timeout = 60000) + public void aTableThatFailsToStartIsIgnoredAndTheServerGoesOn() throws Exception { + startWithJacoco(); + String dataPath = KeployDedupAgent.dataSocketPath; + data = new DataReceiver(dataPath); + KeployDedupAgent.dataSocketPath = null; // new File(null) throws on the control thread + assertEquals(IGNORED, send("TABLE")); + KeployDedupAgent.dataSocketPath = dataPath; + assertEquals(TAKEN, send("START test-set-0/test-1")); + assertEquals(TAKEN, send("END test-set-0/test-1")); + String payload = data.next(); + assertNotNull("the END's coverage reached the data socket", payload); + assertTrue(payload, payload.contains("\"id\":\"test-set-0/test-1\"")); + } + + private static String sha256(byte[] b) throws NoSuchAlgorithmException { + StringBuilder hex = new StringBuilder(); + for (byte x : MessageDigest.getInstance("SHA-256").digest(b)) { + hex.append(String.format("%02x", x)); + } + return hex.toString(); + } + @Test public void anEndForAnotherTestIsIgnoredAndTheTestRunningGoesOn() throws Exception { startWithJacoco(); diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/edge.build b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/edge.build new file mode 100644 index 00000000..350796a9 --- /dev/null +++ b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/edge.build @@ -0,0 +1 @@ +jv1-19348879a23c5d3452cc7da8f15505f9 diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/edge.table.ndjson b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/edge.table.ndjson new file mode 100644 index 00000000..8b23c5c9 --- /dev/null +++ b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/edge.table.ndjson @@ -0,0 +1,15 @@ +{"kind":"class","id":"c:$Lead","pkg":"","name":"$Lead","path":"Lead.java","hash":"dc3c222023b867f1112019d50ceba20066ece22df39651cddf4058ec3d3c8f25","state":"ok","methods":[{"id":"m:$Lead#()V","name":"static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:$Lead$In#()V","name":"In.static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:$Lead#()V","name":"$Lead()","line":73,"leaves":[[73,0,73,0,3,0]]},{"id":"m:$Lead#run(I[Ljava/lang/String;)I","name":"run(int, String...)","line":74,"leaves":[[74,0,74,0,2,0],[74,0,74,0,1,1],[74,0,74,0,1,1],[75,0,75,0,2,0],[76,0,76,0,10,0],[76,0,76,0,1,1],[76,0,76,0,1,1],[76,0,76,0,1,1],[76,0,76,0,1,1]]},{"id":"m:$Lead#same(I)I","name":"same(int)","line":74,"leaves":[[74,0,74,0,2,0],[74,0,74,0,1,1],[74,0,74,0,1,1],[75,0,75,0,2,0],[76,0,76,0,10,0],[76,0,76,0,1,1],[76,0,76,0,1,1],[76,0,76,0,1,1],[76,0,76,0,1,1]]},{"id":"m:$Lead$In#()V","name":"In()","line":80,"leaves":[[80,0,80,0,3,0]]},{"id":"m:$Lead$In#run(I[Ljava/lang/String;)I","name":"In.run(int, String...)","line":81,"leaves":[[81,0,81,0,2,0],[81,0,81,0,1,1],[81,0,81,0,1,1],[82,0,82,0,2,0],[83,0,83,0,10,0],[83,0,83,0,1,1],[83,0,83,0,1,1],[83,0,83,0,1,1],[83,0,83,0,1,1]]},{"id":"m:$Lead$In#same(I)I","name":"In.same(int)","line":81,"leaves":[[81,0,81,0,2,0],[81,0,81,0,1,1],[81,0,81,0,1,1],[82,0,82,0,2,0],[83,0,83,0,10,0],[83,0,83,0,1,1],[83,0,83,0,1,1],[83,0,83,0,1,1],[83,0,83,0,1,1]]},{"id":"m:$Lead#lambda$run$0(I)I","name":"run lambda #0","line":84,"leaves":[[84,0,84,0,2,0],[84,0,84,0,1,1],[84,0,84,0,1,1],[85,0,85,0,2,0],[86,0,86,0,10,0],[86,0,86,0,1,1],[86,0,86,0,1,1],[86,0,86,0,1,1],[86,0,86,0,1,1]]},{"id":"m:$Lead$In#lambda$run$0(I)I","name":"In.run lambda #0","line":91,"leaves":[[91,0,91,0,2,0],[91,0,91,0,1,1],[91,0,91,0,1,1],[92,0,92,0,2,0],[93,0,93,0,10,0],[93,0,93,0,1,1],[93,0,93,0,1,1],[93,0,93,0,1,1],[93,0,93,0,1,1]]},{"id":"m:$Lead#we\"ird#na$me(I)I","name":"we\"ird#na$me(int)","line":104,"leaves":[[104,0,104,0,2,0],[104,0,104,0,1,1],[104,0,104,0,1,1],[105,0,105,0,2,0],[106,0,106,0,10,0],[106,0,106,0,1,1],[106,0,106,0,1,1],[106,0,106,0,1,1],[106,0,106,0,1,1]]},{"id":"m:$Lead$In#we\"ird#na$me(I)I","name":"In.we\"ird#na$me(int)","line":111,"leaves":[[111,0,111,0,2,0],[111,0,111,0,1,1],[111,0,111,0,1,1],[112,0,112,0,2,0],[113,0,113,0,10,0],[113,0,113,0,1,1],[113,0,113,0,1,1],[113,0,113,0,1,1],[113,0,113,0,1,1]]},{"id":"m:$Lead#a\u2028b(I)I","name":"a\u2028b(int)","line":114,"leaves":[[114,0,114,0,2,0],[114,0,114,0,1,1],[114,0,114,0,1,1],[115,0,115,0,2,0],[116,0,116,0,10,0],[116,0,116,0,1,1],[116,0,116,0,1,1],[116,0,116,0,1,1],[116,0,116,0,1,1]]},{"id":"m:$Lead$In#a\u2028b(I)I","name":"In.a\u2028b(int)","line":121,"leaves":[[121,0,121,0,2,0],[121,0,121,0,1,1],[121,0,121,0,1,1],[122,0,122,0,2,0],[123,0,123,0,10,0],[123,0,123,0,1,1],[123,0,123,0,1,1],[123,0,123,0,1,1],[123,0,123,0,1,1]]}]} +{"kind":"class","id":"c:Default","pkg":"","name":"Default","path":"Default.java","hash":"2608d1b58304af9071c418e3653d575c90f4ea734b06f55997566719ab39a2ed","state":"ok","methods":[{"id":"m:Default#()V","name":"static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:Default#()V","name":"Default()","line":66,"leaves":[[66,0,66,0,3,0]]},{"id":"m:Default#run(I[Ljava/lang/String;)I","name":"run(int, String...)","line":67,"leaves":[[67,0,67,0,2,0],[67,0,67,0,1,1],[67,0,67,0,1,1],[68,0,68,0,2,0],[69,0,69,0,10,0],[69,0,69,0,1,1],[69,0,69,0,1,1],[69,0,69,0,1,1],[69,0,69,0,1,1]]},{"id":"m:Default#same(I)I","name":"same(int)","line":67,"leaves":[[67,0,67,0,2,0],[67,0,67,0,1,1],[67,0,67,0,1,1],[68,0,68,0,2,0],[69,0,69,0,10,0],[69,0,69,0,1,1],[69,0,69,0,1,1],[69,0,69,0,1,1],[69,0,69,0,1,1]]},{"id":"m:Default#lambda$run$0(I)I","name":"run lambda #0","line":77,"leaves":[[77,0,77,0,2,0],[77,0,77,0,1,1],[77,0,77,0,1,1],[78,0,78,0,2,0],[79,0,79,0,10,0],[79,0,79,0,1,1],[79,0,79,0,1,1],[79,0,79,0,1,1],[79,0,79,0,1,1]]},{"id":"m:Default#we\"ird#na$me(I)I","name":"we\"ird#na$me(int)","line":97,"leaves":[[97,0,97,0,2,0],[97,0,97,0,1,1],[97,0,97,0,1,1],[98,0,98,0,2,0],[99,0,99,0,10,0],[99,0,99,0,1,1],[99,0,99,0,1,1],[99,0,99,0,1,1],[99,0,99,0,1,1]]},{"id":"m:Default#a\u2028b(I)I","name":"a\u2028b(int)","line":107,"leaves":[[107,0,107,0,2,0],[107,0,107,0,1,1],[107,0,107,0,1,1],[108,0,108,0,2,0],[109,0,109,0,10,0],[109,0,109,0,1,1],[109,0,109,0,1,1],[109,0,109,0,1,1],[109,0,109,0,1,1]]}]} +{"kind":"class","id":"c:a/Qux","pkg":"a","name":"Qux","path":"","hash":"623ebc9845cb84b69f7686e6f8406e26165945d737cb511f1c9926548a48b9a0","state":"ok","methods":[{"id":"m:a/Qux#()V","name":"static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:a/Qux#()V","name":"Qux()","line":52,"leaves":[[52,0,52,0,3,0]]},{"id":"m:a/Qux#run(I[Ljava/lang/String;)I","name":"run(int, String...)","line":53,"leaves":[[53,0,53,0,2,0],[53,0,53,0,1,1],[53,0,53,0,1,1],[54,0,54,0,2,0],[55,0,55,0,10,0],[55,0,55,0,1,1],[55,0,55,0,1,1],[55,0,55,0,1,1],[55,0,55,0,1,1]]},{"id":"m:a/Qux#same(I)I","name":"same(int)","line":53,"leaves":[[53,0,53,0,2,0],[53,0,53,0,1,1],[53,0,53,0,1,1],[54,0,54,0,2,0],[55,0,55,0,10,0],[55,0,55,0,1,1],[55,0,55,0,1,1],[55,0,55,0,1,1],[55,0,55,0,1,1]]},{"id":"m:a/Qux#lambda$run$0(I)I","name":"run lambda #0","line":63,"leaves":[[63,0,63,0,2,0],[63,0,63,0,1,1],[63,0,63,0,1,1],[64,0,64,0,2,0],[65,0,65,0,10,0],[65,0,65,0,1,1],[65,0,65,0,1,1],[65,0,65,0,1,1],[65,0,65,0,1,1]]},{"id":"m:a/Qux#we\"ird#na$me(I)I","name":"we\"ird#na$me(int)","line":83,"leaves":[[83,0,83,0,2,0],[83,0,83,0,1,1],[83,0,83,0,1,1],[84,0,84,0,2,0],[85,0,85,0,10,0],[85,0,85,0,1,1],[85,0,85,0,1,1],[85,0,85,0,1,1],[85,0,85,0,1,1]]},{"id":"m:a/Qux#a\u2028b(I)I","name":"a\u2028b(int)","line":93,"leaves":[[93,0,93,0,2,0],[93,0,93,0,1,1],[93,0,93,0,1,1],[94,0,94,0,2,0],[95,0,95,0,10,0],[95,0,95,0,1,1],[95,0,95,0,1,1],[95,0,95,0,1,1],[95,0,95,0,1,1]]}]} +{"kind":"class","id":"c:a/b/Ctl\u0001\u001f","pkg":"a/b","name":"Ctl\u0001\u001f","path":"a/b/C\u2028.java","hash":"5b96177df3b932214ab696a4b7eda4f0cb0fc447721ad05bdfbdba5f4c932fe7","state":"ok","methods":[{"id":"m:a/b/Ctl\u0001\u001f#()V","name":"static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:a/b/Ctl\u0001\u001f#()V","name":"Ctl\u0001\u001f()","line":115,"leaves":[[115,0,115,0,3,0]]},{"id":"m:a/b/Ctl\u0001\u001f#run(I[Ljava/lang/String;)I","name":"run(int, String...)","line":116,"leaves":[[116,0,116,0,2,0],[116,0,116,0,1,1],[116,0,116,0,1,1],[117,0,117,0,2,0],[118,0,118,0,10,0],[118,0,118,0,1,1],[118,0,118,0,1,1],[118,0,118,0,1,1],[118,0,118,0,1,1]]},{"id":"m:a/b/Ctl\u0001\u001f#same(I)I","name":"same(int)","line":116,"leaves":[[116,0,116,0,2,0],[116,0,116,0,1,1],[116,0,116,0,1,1],[117,0,117,0,2,0],[118,0,118,0,10,0],[118,0,118,0,1,1],[118,0,118,0,1,1],[118,0,118,0,1,1],[118,0,118,0,1,1]]},{"id":"m:a/b/Ctl\u0001\u001f#lambda$run$0(I)I","name":"run lambda #0","line":126,"leaves":[[126,0,126,0,2,0],[126,0,126,0,1,1],[126,0,126,0,1,1],[127,0,127,0,2,0],[128,0,128,0,10,0],[128,0,128,0,1,1],[128,0,128,0,1,1],[128,0,128,0,1,1],[128,0,128,0,1,1]]},{"id":"m:a/b/Ctl\u0001\u001f#we\"ird#na$me(I)I","name":"we\"ird#na$me(int)","line":146,"leaves":[[146,0,146,0,2,0],[146,0,146,0,1,1],[146,0,146,0,1,1],[147,0,147,0,2,0],[148,0,148,0,10,0],[148,0,148,0,1,1],[148,0,148,0,1,1],[148,0,148,0,1,1],[148,0,148,0,1,1]]},{"id":"m:a/b/Ctl\u0001\u001f#a\u2028b(I)I","name":"a\u2028b(int)","line":156,"leaves":[[156,0,156,0,2,0],[156,0,156,0,1,1],[156,0,156,0,1,1],[157,0,157,0,2,0],[158,0,158,0,10,0],[158,0,158,0,1,1],[158,0,158,0,1,1],[158,0,158,0,1,1],[158,0,158,0,1,1]]}]} +{"kind":"class","id":"c:a/b/Foo","pkg":"a/b","name":"Foo","path":"a/b/Foo.java","hash":"313aeac8b1344fd6831d47fff4a1ad5fc283f46fd9b1157d8e340858ac49b26e","state":"ok","methods":[{"id":"m:a/b/Foo#()V","name":"static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:a/b/Foo$#()V","name":"static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:a/b/Foo$$anon$1#()V","name":"anon.1.static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:a/b/Foo$1#()V","name":"1.static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:a/b/Foo$Bar#()V","name":"Bar.static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:a/b/Foo$Bar$#()V","name":"Bar.static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:a/b/Foo$Bar$Deep#()V","name":"Bar.Deep.static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:a/b/Foo#()V","name":"Foo()","line":10,"leaves":[[10,0,10,0,3,0]]},{"id":"m:a/b/Foo#run(I[Ljava/lang/String;)I","name":"run(int, String...)","line":11,"leaves":[[11,0,11,0,2,0],[11,0,11,0,1,1],[11,0,11,0,1,1],[12,0,12,0,2,0],[13,0,13,0,10,0],[13,0,13,0,1,1],[13,0,13,0,1,1],[13,0,13,0,1,1],[13,0,13,0,1,1]]},{"id":"m:a/b/Foo#same(I)I","name":"same(int)","line":11,"leaves":[[11,0,11,0,2,0],[11,0,11,0,1,1],[11,0,11,0,1,1],[12,0,12,0,2,0],[13,0,13,0,10,0],[13,0,13,0,1,1],[13,0,13,0,1,1],[13,0,13,0,1,1],[13,0,13,0,1,1]]},{"id":"m:a/b/Foo$Bar#()V","name":"Bar()","line":17,"leaves":[[17,0,17,0,3,0]]},{"id":"m:a/b/Foo$Bar#run(I[Ljava/lang/String;)I","name":"Bar.run(int, String...)","line":18,"leaves":[[18,0,18,0,2,0],[18,0,18,0,1,1],[18,0,18,0,1,1],[19,0,19,0,2,0],[20,0,20,0,10,0],[20,0,20,0,1,1],[20,0,20,0,1,1],[20,0,20,0,1,1],[20,0,20,0,1,1]]},{"id":"m:a/b/Foo$Bar#same(I)I","name":"Bar.same(int)","line":18,"leaves":[[18,0,18,0,2,0],[18,0,18,0,1,1],[18,0,18,0,1,1],[19,0,19,0,2,0],[20,0,20,0,10,0],[20,0,20,0,1,1],[20,0,20,0,1,1],[20,0,20,0,1,1],[20,0,20,0,1,1]]},{"id":"m:a/b/Foo#lambda$run$0(I)I","name":"run lambda #0","line":21,"leaves":[[21,0,21,0,2,0],[21,0,21,0,1,1],[21,0,21,0,1,1],[22,0,22,0,2,0],[23,0,23,0,10,0],[23,0,23,0,1,1],[23,0,23,0,1,1],[23,0,23,0,1,1],[23,0,23,0,1,1]]},{"id":"m:a/b/Foo$1#()V","name":"1()","line":24,"leaves":[[24,0,24,0,3,0]]},{"id":"m:a/b/Foo$1#run(I[Ljava/lang/String;)I","name":"1.run(int, String...)","line":25,"leaves":[[25,0,25,0,2,0],[25,0,25,0,1,1],[25,0,25,0,1,1],[26,0,26,0,2,0],[27,0,27,0,10,0],[27,0,27,0,1,1],[27,0,27,0,1,1],[27,0,27,0,1,1],[27,0,27,0,1,1]]},{"id":"m:a/b/Foo$1#same(I)I","name":"1.same(int)","line":25,"leaves":[[25,0,25,0,2,0],[25,0,25,0,1,1],[25,0,25,0,1,1],[26,0,26,0,2,0],[27,0,27,0,10,0],[27,0,27,0,1,1],[27,0,27,0,1,1],[27,0,27,0,1,1],[27,0,27,0,1,1]]},{"id":"m:a/b/Foo$Bar#lambda$run$0(I)I","name":"Bar.run lambda #0","line":28,"leaves":[[28,0,28,0,2,0],[28,0,28,0,1,1],[28,0,28,0,1,1],[29,0,29,0,2,0],[30,0,30,0,10,0],[30,0,30,0,1,1],[30,0,30,0,1,1],[30,0,30,0,1,1],[30,0,30,0,1,1]]},{"id":"m:a/b/Foo$#()V","name":"Foo()","line":31,"leaves":[[31,0,31,0,3,0]]},{"id":"m:a/b/Foo$#run(I[Ljava/lang/String;)I","name":"run(int, String...)","line":32,"leaves":[[32,0,32,0,2,0],[32,0,32,0,1,1],[32,0,32,0,1,1],[33,0,33,0,2,0],[34,0,34,0,10,0],[34,0,34,0,1,1],[34,0,34,0,1,1],[34,0,34,0,1,1],[34,0,34,0,1,1]]},{"id":"m:a/b/Foo$#same(I)I","name":"same(int)","line":32,"leaves":[[32,0,32,0,2,0],[32,0,32,0,1,1],[32,0,32,0,1,1],[33,0,33,0,2,0],[34,0,34,0,10,0],[34,0,34,0,1,1],[34,0,34,0,1,1],[34,0,34,0,1,1],[34,0,34,0,1,1]]},{"id":"m:a/b/Foo$1#lambda$run$0(I)I","name":"1.run lambda #0","line":35,"leaves":[[35,0,35,0,2,0],[35,0,35,0,1,1],[35,0,35,0,1,1],[36,0,36,0,2,0],[37,0,37,0,10,0],[37,0,37,0,1,1],[37,0,37,0,1,1],[37,0,37,0,1,1],[37,0,37,0,1,1]]},{"id":"m:a/b/Foo$Bar$Deep#()V","name":"Bar.Deep()","line":38,"leaves":[[38,0,38,0,3,0]]},{"id":"m:a/b/Foo$Bar$Deep#run(I[Ljava/lang/String;)I","name":"Bar.Deep.run(int, String...)","line":39,"leaves":[[39,0,39,0,2,0],[39,0,39,0,1,1],[39,0,39,0,1,1],[40,0,40,0,2,0],[41,0,41,0,10,0],[41,0,41,0,1,1],[41,0,41,0,1,1],[41,0,41,0,1,1],[41,0,41,0,1,1]]},{"id":"m:a/b/Foo$Bar$Deep#same(I)I","name":"Bar.Deep.same(int)","line":39,"leaves":[[39,0,39,0,2,0],[39,0,39,0,1,1],[39,0,39,0,1,1],[40,0,40,0,2,0],[41,0,41,0,10,0],[41,0,41,0,1,1],[41,0,41,0,1,1],[41,0,41,0,1,1],[41,0,41,0,1,1]]},{"id":"m:a/b/Foo#we\"ird#na$me(I)I","name":"we\"ird#na$me(int)","line":41,"leaves":[[41,0,41,0,2,0],[41,0,41,0,1,1],[41,0,41,0,1,1],[42,0,42,0,2,0],[43,0,43,0,10,0],[43,0,43,0,1,1],[43,0,43,0,1,1],[43,0,43,0,1,1],[43,0,43,0,1,1]]},{"id":"m:a/b/Foo$#lambda$run$0(I)I","name":"run lambda #0","line":42,"leaves":[[42,0,42,0,2,0],[42,0,42,0,1,1],[42,0,42,0,1,1],[43,0,43,0,2,0],[44,0,44,0,10,0],[44,0,44,0,1,1],[44,0,44,0,1,1],[44,0,44,0,1,1],[44,0,44,0,1,1]]},{"id":"m:a/b/Foo$Bar#we\"ird#na$me(I)I","name":"Bar.we\"ird#na$me(int)","line":48,"leaves":[[48,0,48,0,2,0],[48,0,48,0,1,1],[48,0,48,0,1,1],[49,0,49,0,2,0],[50,0,50,0,10,0],[50,0,50,0,1,1],[50,0,50,0,1,1],[50,0,50,0,1,1],[50,0,50,0,1,1]]},{"id":"m:a/b/Foo$Bar$Deep#lambda$run$0(I)I","name":"Bar.Deep.run lambda #0","line":49,"leaves":[[49,0,49,0,2,0],[49,0,49,0,1,1],[49,0,49,0,1,1],[50,0,50,0,2,0],[51,0,51,0,10,0],[51,0,51,0,1,1],[51,0,51,0,1,1],[51,0,51,0,1,1],[51,0,51,0,1,1]]},{"id":"m:a/b/Foo#a\u2028b(I)I","name":"a\u2028b(int)","line":51,"leaves":[[51,0,51,0,2,0],[51,0,51,0,1,1],[51,0,51,0,1,1],[52,0,52,0,2,0],[53,0,53,0,10,0],[53,0,53,0,1,1],[53,0,53,0,1,1],[53,0,53,0,1,1],[53,0,53,0,1,1]]},{"id":"m:a/b/Foo$1#we\"ird#na$me(I)I","name":"1.we\"ird#na$me(int)","line":55,"leaves":[[55,0,55,0,2,0],[55,0,55,0,1,1],[55,0,55,0,1,1],[56,0,56,0,2,0],[57,0,57,0,10,0],[57,0,57,0,1,1],[57,0,57,0,1,1],[57,0,57,0,1,1],[57,0,57,0,1,1]]},{"id":"m:a/b/Foo$Bar#a\u2028b(I)I","name":"Bar.a\u2028b(int)","line":58,"leaves":[[58,0,58,0,2,0],[58,0,58,0,1,1],[58,0,58,0,1,1],[59,0,59,0,2,0],[60,0,60,0,10,0],[60,0,60,0,1,1],[60,0,60,0,1,1],[60,0,60,0,1,1],[60,0,60,0,1,1]]},{"id":"m:a/b/Foo$#we\"ird#na$me(I)I","name":"we\"ird#na$me(int)","line":62,"leaves":[[62,0,62,0,2,0],[62,0,62,0,1,1],[62,0,62,0,1,1],[63,0,63,0,2,0],[64,0,64,0,10,0],[64,0,64,0,1,1],[64,0,64,0,1,1],[64,0,64,0,1,1],[64,0,64,0,1,1]]},{"id":"m:a/b/Foo$1#a\u2028b(I)I","name":"1.a\u2028b(int)","line":65,"leaves":[[65,0,65,0,2,0],[65,0,65,0,1,1],[65,0,65,0,1,1],[66,0,66,0,2,0],[67,0,67,0,10,0],[67,0,67,0,1,1],[67,0,67,0,1,1],[67,0,67,0,1,1],[67,0,67,0,1,1]]},{"id":"m:a/b/Foo$Bar$Deep#we\"ird#na$me(I)I","name":"Bar.Deep.we\"ird#na$me(int)","line":69,"leaves":[[69,0,69,0,2,0],[69,0,69,0,1,1],[69,0,69,0,1,1],[70,0,70,0,2,0],[71,0,71,0,10,0],[71,0,71,0,1,1],[71,0,71,0,1,1],[71,0,71,0,1,1],[71,0,71,0,1,1]]},{"id":"m:a/b/Foo$#a\u2028b(I)I","name":"a\u2028b(int)","line":72,"leaves":[[72,0,72,0,2,0],[72,0,72,0,1,1],[72,0,72,0,1,1],[73,0,73,0,2,0],[74,0,74,0,10,0],[74,0,74,0,1,1],[74,0,74,0,1,1],[74,0,74,0,1,1],[74,0,74,0,1,1]]},{"id":"m:a/b/Foo$Bar$Deep#a\u2028b(I)I","name":"Bar.Deep.a\u2028b(int)","line":79,"leaves":[[79,0,79,0,2,0],[79,0,79,0,1,1],[79,0,79,0,1,1],[80,0,80,0,2,0],[81,0,81,0,10,0],[81,0,81,0,1,1],[81,0,81,0,1,1],[81,0,81,0,1,1],[81,0,81,0,1,1]]},{"id":"m:a/b/Foo$Bar$#()V","name":"Bar()","line":136,"leaves":[[136,0,136,0,3,0]]},{"id":"m:a/b/Foo$Bar$#run(I[Ljava/lang/String;)I","name":"Bar.run(int, String...)","line":137,"leaves":[[137,0,137,0,2,0],[137,0,137,0,1,1],[137,0,137,0,1,1],[138,0,138,0,2,0],[139,0,139,0,10,0],[139,0,139,0,1,1],[139,0,139,0,1,1],[139,0,139,0,1,1],[139,0,139,0,1,1]]},{"id":"m:a/b/Foo$Bar$#same(I)I","name":"Bar.same(int)","line":137,"leaves":[[137,0,137,0,2,0],[137,0,137,0,1,1],[137,0,137,0,1,1],[138,0,138,0,2,0],[139,0,139,0,10,0],[139,0,139,0,1,1],[139,0,139,0,1,1],[139,0,139,0,1,1],[139,0,139,0,1,1]]},{"id":"m:a/b/Foo$$anon$1#()V","name":"anon.1()","line":143,"leaves":[[143,0,143,0,3,0]]},{"id":"m:a/b/Foo$$anon$1#run(I[Ljava/lang/String;)I","name":"anon.1.run(int, String...)","line":144,"leaves":[[144,0,144,0,2,0],[144,0,144,0,1,1],[144,0,144,0,1,1],[145,0,145,0,2,0],[146,0,146,0,10,0],[146,0,146,0,1,1],[146,0,146,0,1,1],[146,0,146,0,1,1],[146,0,146,0,1,1]]},{"id":"m:a/b/Foo$$anon$1#same(I)I","name":"anon.1.same(int)","line":144,"leaves":[[144,0,144,0,2,0],[144,0,144,0,1,1],[144,0,144,0,1,1],[145,0,145,0,2,0],[146,0,146,0,10,0],[146,0,146,0,1,1],[146,0,146,0,1,1],[146,0,146,0,1,1],[146,0,146,0,1,1]]},{"id":"m:a/b/Foo$Bar$#lambda$run$0(I)I","name":"Bar.run lambda #0","line":147,"leaves":[[147,0,147,0,2,0],[147,0,147,0,1,1],[147,0,147,0,1,1],[148,0,148,0,2,0],[149,0,149,0,10,0],[149,0,149,0,1,1],[149,0,149,0,1,1],[149,0,149,0,1,1],[149,0,149,0,1,1]]},{"id":"m:a/b/Foo$$anon$1#lambda$run$0(I)I","name":"anon.1.run lambda #0","line":154,"leaves":[[154,0,154,0,2,0],[154,0,154,0,1,1],[154,0,154,0,1,1],[155,0,155,0,2,0],[156,0,156,0,10,0],[156,0,156,0,1,1],[156,0,156,0,1,1],[156,0,156,0,1,1],[156,0,156,0,1,1]]},{"id":"m:a/b/Foo$Bar$#we\"ird#na$me(I)I","name":"Bar.we\"ird#na$me(int)","line":167,"leaves":[[167,0,167,0,2,0],[167,0,167,0,1,1],[167,0,167,0,1,1],[168,0,168,0,2,0],[169,0,169,0,10,0],[169,0,169,0,1,1],[169,0,169,0,1,1],[169,0,169,0,1,1],[169,0,169,0,1,1]]},{"id":"m:a/b/Foo$$anon$1#we\"ird#na$me(I)I","name":"anon.1.we\"ird#na$me(int)","line":174,"leaves":[[174,0,174,0,2,0],[174,0,174,0,1,1],[174,0,174,0,1,1],[175,0,175,0,2,0],[176,0,176,0,10,0],[176,0,176,0,1,1],[176,0,176,0,1,1],[176,0,176,0,1,1],[176,0,176,0,1,1]]},{"id":"m:a/b/Foo$Bar$#a\u2028b(I)I","name":"Bar.a\u2028b(int)","line":177,"leaves":[[177,0,177,0,2,0],[177,0,177,0,1,1],[177,0,177,0,1,1],[178,0,178,0,2,0],[179,0,179,0,10,0],[179,0,179,0,1,1],[179,0,179,0,1,1],[179,0,179,0,1,1],[179,0,179,0,1,1]]},{"id":"m:a/b/Foo$$anon$1#a\u2028b(I)I","name":"anon.1.a\u2028b(int)","line":184,"leaves":[[184,0,184,0,2,0],[184,0,184,0,1,1],[184,0,184,0,1,1],[185,0,185,0,2,0],[186,0,186,0,10,0],[186,0,186,0,1,1],[186,0,186,0,1,1],[186,0,186,0,1,1],[186,0,186,0,1,1]]}]} +{"kind":"class","id":"c:a/b/L\u2028S\u2029","pkg":"a/b","name":"L\u2028S\u2029","path":"a/b/L.java","hash":"251d9cff82aeb2b359aeaccf0a1e7016751ab5c4cb0844602bd8a4887b453f8a","state":"ok","methods":[{"id":"m:a/b/L\u2028S\u2029#()V","name":"static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:a/b/L\u2028S\u2029#()V","name":"L\u2028S\u2029()","line":122,"leaves":[[122,0,122,0,3,0]]},{"id":"m:a/b/L\u2028S\u2029#run(I[Ljava/lang/String;)I","name":"run(int, String...)","line":123,"leaves":[[123,0,123,0,2,0],[123,0,123,0,1,1],[123,0,123,0,1,1],[124,0,124,0,2,0],[125,0,125,0,10,0],[125,0,125,0,1,1],[125,0,125,0,1,1],[125,0,125,0,1,1],[125,0,125,0,1,1]]},{"id":"m:a/b/L\u2028S\u2029#same(I)I","name":"same(int)","line":123,"leaves":[[123,0,123,0,2,0],[123,0,123,0,1,1],[123,0,123,0,1,1],[124,0,124,0,2,0],[125,0,125,0,10,0],[125,0,125,0,1,1],[125,0,125,0,1,1],[125,0,125,0,1,1],[125,0,125,0,1,1]]},{"id":"m:a/b/L\u2028S\u2029#lambda$run$0(I)I","name":"run lambda #0","line":133,"leaves":[[133,0,133,0,2,0],[133,0,133,0,1,1],[133,0,133,0,1,1],[134,0,134,0,2,0],[135,0,135,0,10,0],[135,0,135,0,1,1],[135,0,135,0,1,1],[135,0,135,0,1,1],[135,0,135,0,1,1]]},{"id":"m:a/b/L\u2028S\u2029#we\"ird#na$me(I)I","name":"we\"ird#na$me(int)","line":153,"leaves":[[153,0,153,0,2,0],[153,0,153,0,1,1],[153,0,153,0,1,1],[154,0,154,0,2,0],[155,0,155,0,10,0],[155,0,155,0,1,1],[155,0,155,0,1,1],[155,0,155,0,1,1],[155,0,155,0,1,1]]},{"id":"m:a/b/L\u2028S\u2029#a\u2028b(I)I","name":"a\u2028b(int)","line":163,"leaves":[[163,0,163,0,2,0],[163,0,163,0,1,1],[163,0,163,0,1,1],[164,0,164,0,2,0],[165,0,165,0,10,0],[165,0,165,0,1,1],[165,0,165,0,1,1],[165,0,165,0,1,1],[165,0,165,0,1,1]]}]} +{"kind":"class","id":"c:a/b/NoLines","pkg":"a/b","name":"NoLines","path":"a/b/NoLines.java","hash":"25d1deee75fbd65a4d742d7e65f7c93ab9d0b39859f709a53df3e98111f1a70e","state":"ok","methods":[]} +{"kind":"class","id":"c:a/b/Q\"uo\\te<&>","pkg":"a/b","name":"Q\"uo\\te<&>","path":"a/b/Q\"uo\\te.java","hash":"ae57d97bcbf4241c5a0e02ea8a1a9287528b3ac8bfe539141d10878fb35e4794","state":"ok","methods":[{"id":"m:a/b/Q\"uo\\te<&>#()V","name":"static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:a/b/Q\"uo\\te<&>#()V","name":"Q\"uo\\te<&>()","line":108,"leaves":[[108,0,108,0,3,0]]},{"id":"m:a/b/Q\"uo\\te<&>#run(I[Ljava/lang/String;)I","name":"run(int, String...)","line":109,"leaves":[[109,0,109,0,2,0],[109,0,109,0,1,1],[109,0,109,0,1,1],[110,0,110,0,2,0],[111,0,111,0,10,0],[111,0,111,0,1,1],[111,0,111,0,1,1],[111,0,111,0,1,1],[111,0,111,0,1,1]]},{"id":"m:a/b/Q\"uo\\te<&>#same(I)I","name":"same(int)","line":109,"leaves":[[109,0,109,0,2,0],[109,0,109,0,1,1],[109,0,109,0,1,1],[110,0,110,0,2,0],[111,0,111,0,10,0],[111,0,111,0,1,1],[111,0,111,0,1,1],[111,0,111,0,1,1],[111,0,111,0,1,1]]},{"id":"m:a/b/Q\"uo\\te<&>#lambda$run$0(I)I","name":"run lambda #0","line":119,"leaves":[[119,0,119,0,2,0],[119,0,119,0,1,1],[119,0,119,0,1,1],[120,0,120,0,2,0],[121,0,121,0,10,0],[121,0,121,0,1,1],[121,0,121,0,1,1],[121,0,121,0,1,1],[121,0,121,0,1,1]]},{"id":"m:a/b/Q\"uo\\te<&>#we\"ird#na$me(I)I","name":"we\"ird#na$me(int)","line":139,"leaves":[[139,0,139,0,2,0],[139,0,139,0,1,1],[139,0,139,0,1,1],[140,0,140,0,2,0],[141,0,141,0,10,0],[141,0,141,0,1,1],[141,0,141,0,1,1],[141,0,141,0,1,1],[141,0,141,0,1,1]]},{"id":"m:a/b/Q\"uo\\te<&>#a\u2028b(I)I","name":"a\u2028b(int)","line":149,"leaves":[[149,0,149,0,2,0],[149,0,149,0,1,1],[149,0,149,0,1,1],[150,0,150,0,2,0],[151,0,151,0,10,0],[151,0,151,0,1,1],[151,0,151,0,1,1],[151,0,151,0,1,1],[151,0,151,0,1,1]]}]} +{"kind":"class","id":"c:a/b/Z~","pkg":"a/b","name":"Z~","path":"a/b/Z.java","hash":"9f34a97e2d2c5f4d2d321437cba3c738d2f85b785929d0e2a2f4e8b561413dda","state":"ok","methods":[{"id":"m:a/b/Z~#()V","name":"static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:a/b/Z~#()V","name":"Z~()","line":94,"leaves":[[94,0,94,0,3,0]]},{"id":"m:a/b/Z~#run(I[Ljava/lang/String;)I","name":"run(int, String...)","line":95,"leaves":[[95,0,95,0,2,0],[95,0,95,0,1,1],[95,0,95,0,1,1],[96,0,96,0,2,0],[97,0,97,0,10,0],[97,0,97,0,1,1],[97,0,97,0,1,1],[97,0,97,0,1,1],[97,0,97,0,1,1]]},{"id":"m:a/b/Z~#same(I)I","name":"same(int)","line":95,"leaves":[[95,0,95,0,2,0],[95,0,95,0,1,1],[95,0,95,0,1,1],[96,0,96,0,2,0],[97,0,97,0,10,0],[97,0,97,0,1,1],[97,0,97,0,1,1],[97,0,97,0,1,1],[97,0,97,0,1,1]]},{"id":"m:a/b/Z~#lambda$run$0(I)I","name":"run lambda #0","line":105,"leaves":[[105,0,105,0,2,0],[105,0,105,0,1,1],[105,0,105,0,1,1],[106,0,106,0,2,0],[107,0,107,0,10,0],[107,0,107,0,1,1],[107,0,107,0,1,1],[107,0,107,0,1,1],[107,0,107,0,1,1]]},{"id":"m:a/b/Z~#we\"ird#na$me(I)I","name":"we\"ird#na$me(int)","line":125,"leaves":[[125,0,125,0,2,0],[125,0,125,0,1,1],[125,0,125,0,1,1],[126,0,126,0,2,0],[127,0,127,0,10,0],[127,0,127,0,1,1],[127,0,127,0,1,1],[127,0,127,0,1,1],[127,0,127,0,1,1]]},{"id":"m:a/b/Z~#a\u2028b(I)I","name":"a\u2028b(int)","line":135,"leaves":[[135,0,135,0,2,0],[135,0,135,0,1,1],[135,0,135,0,1,1],[136,0,136,0,2,0],[137,0,137,0,10,0],[137,0,137,0,1,1],[137,0,137,0,1,1],[137,0,137,0,1,1],[137,0,137,0,1,1]]}]} +{"kind":"class","id":"c:a/b/Z😀","pkg":"a/b","name":"Z😀","path":"a/b/Z.java","hash":"52292e7aaf05aeb62ddf6484e25618bb828638ad0ab9fcb89bc1811a5d502b34","state":"ok","methods":[{"id":"m:a/b/Z😀#()V","name":"static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:a/b/Z😀#()V","name":"Z😀()","line":101,"leaves":[[101,0,101,0,3,0]]},{"id":"m:a/b/Z😀#run(I[Ljava/lang/String;)I","name":"run(int, String...)","line":102,"leaves":[[102,0,102,0,2,0],[102,0,102,0,1,1],[102,0,102,0,1,1],[103,0,103,0,2,0],[104,0,104,0,10,0],[104,0,104,0,1,1],[104,0,104,0,1,1],[104,0,104,0,1,1],[104,0,104,0,1,1]]},{"id":"m:a/b/Z😀#same(I)I","name":"same(int)","line":102,"leaves":[[102,0,102,0,2,0],[102,0,102,0,1,1],[102,0,102,0,1,1],[103,0,103,0,2,0],[104,0,104,0,10,0],[104,0,104,0,1,1],[104,0,104,0,1,1],[104,0,104,0,1,1],[104,0,104,0,1,1]]},{"id":"m:a/b/Z😀#lambda$run$0(I)I","name":"run lambda #0","line":112,"leaves":[[112,0,112,0,2,0],[112,0,112,0,1,1],[112,0,112,0,1,1],[113,0,113,0,2,0],[114,0,114,0,10,0],[114,0,114,0,1,1],[114,0,114,0,1,1],[114,0,114,0,1,1],[114,0,114,0,1,1]]},{"id":"m:a/b/Z😀#we\"ird#na$me(I)I","name":"we\"ird#na$me(int)","line":132,"leaves":[[132,0,132,0,2,0],[132,0,132,0,1,1],[132,0,132,0,1,1],[133,0,133,0,2,0],[134,0,134,0,10,0],[134,0,134,0,1,1],[134,0,134,0,1,1],[134,0,134,0,1,1],[134,0,134,0,1,1]]},{"id":"m:a/b/Z😀#a\u2028b(I)I","name":"a\u2028b(int)","line":142,"leaves":[[142,0,142,0,2,0],[142,0,142,0,1,1],[142,0,142,0,1,1],[143,0,143,0,2,0],[144,0,144,0,10,0],[144,0,144,0,1,1],[144,0,144,0,1,1],[144,0,144,0,1,1],[144,0,144,0,1,1]]}]} +{"kind":"class","id":"c:a/b/module-info","pkg":"a/b","name":"module-info","path":"","hash":"f2d4c49e23f2b4b46332871c40cded4fb31cd369d48f2de0563fa3a15fe1247b","state":"ok","methods":[{"id":"m:a/b/module-info#()V","name":"static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:a/b/module-info#()V","name":"module-info()","line":150,"leaves":[[150,0,150,0,3,0]]},{"id":"m:a/b/module-info#run(I[Ljava/lang/String;)I","name":"run(int, String...)","line":151,"leaves":[[151,0,151,0,2,0],[151,0,151,0,1,1],[151,0,151,0,1,1],[152,0,152,0,2,0],[153,0,153,0,10,0],[153,0,153,0,1,1],[153,0,153,0,1,1],[153,0,153,0,1,1],[153,0,153,0,1,1]]},{"id":"m:a/b/module-info#same(I)I","name":"same(int)","line":151,"leaves":[[151,0,151,0,2,0],[151,0,151,0,1,1],[151,0,151,0,1,1],[152,0,152,0,2,0],[153,0,153,0,10,0],[153,0,153,0,1,1],[153,0,153,0,1,1],[153,0,153,0,1,1],[153,0,153,0,1,1]]},{"id":"m:a/b/module-info#lambda$run$0(I)I","name":"run lambda #0","line":161,"leaves":[[161,0,161,0,2,0],[161,0,161,0,1,1],[161,0,161,0,1,1],[162,0,162,0,2,0],[163,0,163,0,10,0],[163,0,163,0,1,1],[163,0,163,0,1,1],[163,0,163,0,1,1],[163,0,163,0,1,1]]},{"id":"m:a/b/module-info#we\"ird#na$me(I)I","name":"we\"ird#na$me(int)","line":181,"leaves":[[181,0,181,0,2,0],[181,0,181,0,1,1],[181,0,181,0,1,1],[182,0,182,0,2,0],[183,0,183,0,10,0],[183,0,183,0,1,1],[183,0,183,0,1,1],[183,0,183,0,1,1],[183,0,183,0,1,1]]},{"id":"m:a/b/module-info#a\u2028b(I)I","name":"a\u2028b(int)","line":191,"leaves":[[191,0,191,0,2,0],[191,0,191,0,1,1],[191,0,191,0,1,1],[192,0,192,0,2,0],[193,0,193,0,10,0],[193,0,193,0,1,1],[193,0,193,0,1,1],[193,0,193,0,1,1],[193,0,193,0,1,1]]}]} +{"kind":"class","id":"c:a/b/package-info","pkg":"a/b","name":"package-info","path":"a/b/package-info.java","hash":"ed7f74fba3fe4572cda03001178ca3138337e83f015b90c7a54e8282d80b7651","state":"ok","methods":[]} +{"kind":"class","id":"c:a/b/c/Deep","pkg":"a/b/c","name":"Deep","path":"","hash":"13d1610117f52711d1289994ce274a66e9a44ee4fe5d1501d8a69ff58d3a29ea","state":"ok","methods":[{"id":"m:a/b/c/Deep#()V","name":"static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:a/b/c/Deep#()V","name":"Deep()","line":59,"leaves":[[59,0,59,0,3,0]]},{"id":"m:a/b/c/Deep#run(I[Ljava/lang/String;)I","name":"run(int, String...)","line":60,"leaves":[[60,0,60,0,2,0],[60,0,60,0,1,1],[60,0,60,0,1,1],[61,0,61,0,2,0],[62,0,62,0,10,0],[62,0,62,0,1,1],[62,0,62,0,1,1],[62,0,62,0,1,1],[62,0,62,0,1,1]]},{"id":"m:a/b/c/Deep#same(I)I","name":"same(int)","line":60,"leaves":[[60,0,60,0,2,0],[60,0,60,0,1,1],[60,0,60,0,1,1],[61,0,61,0,2,0],[62,0,62,0,10,0],[62,0,62,0,1,1],[62,0,62,0,1,1],[62,0,62,0,1,1],[62,0,62,0,1,1]]},{"id":"m:a/b/c/Deep#lambda$run$0(I)I","name":"run lambda #0","line":70,"leaves":[[70,0,70,0,2,0],[70,0,70,0,1,1],[70,0,70,0,1,1],[71,0,71,0,2,0],[72,0,72,0,10,0],[72,0,72,0,1,1],[72,0,72,0,1,1],[72,0,72,0,1,1],[72,0,72,0,1,1]]},{"id":"m:a/b/c/Deep#we\"ird#na$me(I)I","name":"we\"ird#na$me(int)","line":90,"leaves":[[90,0,90,0,2,0],[90,0,90,0,1,1],[90,0,90,0,1,1],[91,0,91,0,2,0],[92,0,92,0,10,0],[92,0,92,0,1,1],[92,0,92,0,1,1],[92,0,92,0,1,1],[92,0,92,0,1,1]]},{"id":"m:a/b/c/Deep#a\u2028b(I)I","name":"a\u2028b(int)","line":100,"leaves":[[100,0,100,0,2,0],[100,0,100,0,1,1],[100,0,100,0,1,1],[101,0,101,0,2,0],[102,0,102,0,10,0],[102,0,102,0,1,1],[102,0,102,0,1,1],[102,0,102,0,1,1],[102,0,102,0,1,1]]}]} +{"kind":"class","id":"c:a/ü/Ünï","pkg":"a/ü","name":"Ünï","path":"a/ü/Ünï.java","hash":"2c745e3098b938c18e3c19b7adeca112a933d1e527fed86837be10080991f137","state":"ok","methods":[{"id":"m:a/ü/Ünï#()V","name":"static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:a/ü/Ünï#()V","name":"Ünï()","line":87,"leaves":[[87,0,87,0,3,0]]},{"id":"m:a/ü/Ünï#run(I[Ljava/lang/String;)I","name":"run(int, String...)","line":88,"leaves":[[88,0,88,0,2,0],[88,0,88,0,1,1],[88,0,88,0,1,1],[89,0,89,0,2,0],[90,0,90,0,10,0],[90,0,90,0,1,1],[90,0,90,0,1,1],[90,0,90,0,1,1],[90,0,90,0,1,1]]},{"id":"m:a/ü/Ünï#same(I)I","name":"same(int)","line":88,"leaves":[[88,0,88,0,2,0],[88,0,88,0,1,1],[88,0,88,0,1,1],[89,0,89,0,2,0],[90,0,90,0,10,0],[90,0,90,0,1,1],[90,0,90,0,1,1],[90,0,90,0,1,1],[90,0,90,0,1,1]]},{"id":"m:a/ü/Ünï#lambda$run$0(I)I","name":"run lambda #0","line":98,"leaves":[[98,0,98,0,2,0],[98,0,98,0,1,1],[98,0,98,0,1,1],[99,0,99,0,2,0],[100,0,100,0,10,0],[100,0,100,0,1,1],[100,0,100,0,1,1],[100,0,100,0,1,1],[100,0,100,0,1,1]]},{"id":"m:a/ü/Ünï#we\"ird#na$me(I)I","name":"we\"ird#na$me(int)","line":118,"leaves":[[118,0,118,0,2,0],[118,0,118,0,1,1],[118,0,118,0,1,1],[119,0,119,0,2,0],[120,0,120,0,10,0],[120,0,120,0,1,1],[120,0,120,0,1,1],[120,0,120,0,1,1],[120,0,120,0,1,1]]},{"id":"m:a/ü/Ünï#a\u2028b(I)I","name":"a\u2028b(int)","line":128,"leaves":[[128,0,128,0,2,0],[128,0,128,0,1,1],[128,0,128,0,1,1],[129,0,129,0,2,0],[130,0,130,0,10,0],[130,0,130,0,1,1],[130,0,130,0,1,1],[130,0,130,0,1,1],[130,0,130,0,1,1]]}]} +{"kind":"class","id":"c:a-x/Baz","pkg":"a-x","name":"Baz","path":"a-x/Baz.java","hash":"9ac2b9e989ea27c93330ec7f1b7d7ba81c253b4b7a3da606e2866e2c3693c72d","state":"ok","methods":[{"id":"m:a-x/Baz#()V","name":"static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:a-x/Baz#()V","name":"Baz()","line":45,"leaves":[[45,0,45,0,3,0]]},{"id":"m:a-x/Baz#run(I[Ljava/lang/String;)I","name":"run(int, String...)","line":46,"leaves":[[46,0,46,0,2,0],[46,0,46,0,1,1],[46,0,46,0,1,1],[47,0,47,0,2,0],[48,0,48,0,10,0],[48,0,48,0,1,1],[48,0,48,0,1,1],[48,0,48,0,1,1],[48,0,48,0,1,1]]},{"id":"m:a-x/Baz#same(I)I","name":"same(int)","line":46,"leaves":[[46,0,46,0,2,0],[46,0,46,0,1,1],[46,0,46,0,1,1],[47,0,47,0,2,0],[48,0,48,0,10,0],[48,0,48,0,1,1],[48,0,48,0,1,1],[48,0,48,0,1,1],[48,0,48,0,1,1]]},{"id":"m:a-x/Baz#lambda$run$0(I)I","name":"run lambda #0","line":56,"leaves":[[56,0,56,0,2,0],[56,0,56,0,1,1],[56,0,56,0,1,1],[57,0,57,0,2,0],[58,0,58,0,10,0],[58,0,58,0,1,1],[58,0,58,0,1,1],[58,0,58,0,1,1],[58,0,58,0,1,1]]},{"id":"m:a-x/Baz#we\"ird#na$me(I)I","name":"we\"ird#na$me(int)","line":76,"leaves":[[76,0,76,0,2,0],[76,0,76,0,1,1],[76,0,76,0,1,1],[77,0,77,0,2,0],[78,0,78,0,10,0],[78,0,78,0,1,1],[78,0,78,0,1,1],[78,0,78,0,1,1],[78,0,78,0,1,1]]},{"id":"m:a-x/Baz#a\u2028b(I)I","name":"a\u2028b(int)","line":86,"leaves":[[86,0,86,0,2,0],[86,0,86,0,1,1],[86,0,86,0,1,1],[87,0,87,0,2,0],[88,0,88,0,10,0],[88,0,88,0,1,1],[88,0,88,0,1,1],[88,0,88,0,1,1],[88,0,88,0,1,1]]}]} diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/fixture-java-dedup-table.ndjson b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/fixture-java-dedup-table.ndjson new file mode 100644 index 00000000..3f68ee15 --- /dev/null +++ b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/fixture-java-dedup-table.ndjson @@ -0,0 +1,4 @@ +{"kind":"class","id":"c:io/keploy/samples/javadedup/DedupController","pkg":"io/keploy/samples/javadedup","name":"DedupController","path":"io/keploy/samples/javadedup/DedupController.java","hash":"7c03b81f9f20db7531ed647244af5fa640a6faa6c1b804e039b0d67782aa7371","state":"ok","methods":[{"id":"m:io/keploy/samples/javadedup/DedupController#()V","name":"DedupController()","line":33,"leaves":[[33,0,33,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#root()Ljava/util/Map;","name":"root()","line":37,"leaves":[[37,0,37,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#hello(Ljava/lang/String;)Ljava/lang/String;","name":"hello(String)","line":42,"leaves":[[42,0,42,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#random()Ljava/util/Map;","name":"random()","line":47,"leaves":[[47,0,47,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#welcome(Ljava/lang/String;)Ljava/util/Map;","name":"welcome(String)","line":52,"leaves":[[52,0,52,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#createUser(Lio/keploy/samples/javadedup/model/User;)Ljava/util/Map;","name":"createUser(User)","line":57,"leaves":[[57,0,57,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#items()Ljava/util/List;","name":"items()","line":62,"leaves":[[62,0,62,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#updateItem(Ljava/lang/String;Lio/keploy/samples/javadedup/model/Item;)Ljava/util/Map;","name":"updateItem(String, Item)","line":67,"leaves":[[67,0,67,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#deleteItem(Ljava/lang/String;)Ljava/util/Map;","name":"deleteItem(String)","line":72,"leaves":[[72,0,72,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#simpleMessage(Ljakarta/servlet/http/HttpServletRequest;)Ljava/util/Map;","name":"simpleMessage(HttpServletRequest)","line":81,"leaves":[[81,0,81,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#ping()Ljava/lang/String;","name":"ping()","line":86,"leaves":[[86,0,86,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#status()Ljava/util/Map;","name":"status()","line":91,"leaves":[[91,0,91,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#greeting(Ljava/lang/String;)Ljava/lang/String;","name":"greeting(String)","line":285,"leaves":[[285,0,285,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#itemList()Ljava/util/List;","name":"itemList()","line":289,"leaves":[[289,0,289,0,3,0],[290,0,290,0,3,0],[291,0,291,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#messageForPath(Ljava/lang/String;)Ljava/lang/String;","name":"messageForPath(String)","line":306,"leaves":[[306,0,306,0,3,0],[307,0,307,0,3,0],[307,0,307,0,1,1],[307,0,307,0,1,1],[308,0,308,0,3,0],[310,0,310,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#response([Ljava/lang/Object;)Ljava/util/Map;","name":"response(Object...)","line":314,"leaves":[[314,0,314,0,3,0],[315,0,315,0,3,0],[315,0,315,0,1,1],[315,0,315,0,1,1],[316,0,316,0,3,0],[318,0,318,0,3,0]]}]} +{"kind":"class","id":"c:io/keploy/samples/javadedup/JavaDedupApplication","pkg":"io/keploy/samples/javadedup","name":"JavaDedupApplication","path":"io/keploy/samples/javadedup/JavaDedupApplication.java","hash":"85cfab44164d4f24ca778bfac41594bb52fd1bf2d42c75532c048d25ebb61372","state":"ok","methods":[{"id":"m:io/keploy/samples/javadedup/JavaDedupApplication#()V","name":"JavaDedupApplication()","line":8,"leaves":[[8,0,8,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/JavaDedupApplication#main([Ljava/lang/String;)V","name":"main(String[])","line":10,"leaves":[[10,0,10,0,3,0]]}]} +{"kind":"class","id":"c:io/keploy/samples/javadedup/model/Item","pkg":"io/keploy/samples/javadedup/model","name":"Item","path":"io/keploy/samples/javadedup/model/Item.java","hash":"2e4af0ca3be34b61f048b8ad0440406cb107142896d18cb80d9557f496173ba4","state":"ok","methods":[{"id":"m:io/keploy/samples/javadedup/model/Item#()V","name":"Item()","line":5,"leaves":[[5,0,5,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/model/Item#getId()Ljava/lang/String;","name":"getId()","line":10,"leaves":[[10,0,10,0,3,0]]}]} +{"kind":"class","id":"c:io/keploy/samples/javadedup/model/User","pkg":"io/keploy/samples/javadedup/model","name":"User","path":"io/keploy/samples/javadedup/model/User.java","hash":"2fd7360ebf831da22c293cfaf9f29408521d03343cf5fb6ef06608d39e521188","state":"ok","methods":[{"id":"m:io/keploy/samples/javadedup/model/User#()V","name":"User()","line":5,"leaves":[[5,0,5,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/model/User#getName()Ljava/lang/String;","name":"getName()","line":9,"leaves":[[9,0,9,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/model/User#setName(Ljava/lang/String;)V","name":"setName(String)","line":13,"leaves":[[13,0,13,0,3,0]]}]} diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup.build b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup.build new file mode 100644 index 00000000..a32ff64c --- /dev/null +++ b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup.build @@ -0,0 +1 @@ +jv1-41cdf8ddf2cd4e64008b050d972818b2 diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup.table.ndjson b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup.table.ndjson new file mode 100644 index 00000000..9991c36c --- /dev/null +++ b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup.table.ndjson @@ -0,0 +1,5 @@ +{"kind":"class","id":"c:io/keploy/samples/javadedup/DedupController","pkg":"io/keploy/samples/javadedup","name":"DedupController","path":"io/keploy/samples/javadedup/DedupController.java","hash":"33497ea96394e86dd8e0ed484fcf8441bd4205353303837f097938abd377c03a","state":"ok","methods":[{"id":"m:io/keploy/samples/javadedup/DedupController#()V","name":"DedupController()","line":33,"leaves":[[33,0,33,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#root()Ljava/util/Map;","name":"root()","line":37,"leaves":[[37,0,37,0,13,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#hello(Ljava/lang/String;)Ljava/lang/String;","name":"hello(String)","line":42,"leaves":[[42,0,42,0,4,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#random()Ljava/util/Map;","name":"random()","line":47,"leaves":[[47,0,47,0,19,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#welcome(Ljava/lang/String;)Ljava/util/Map;","name":"welcome(String)","line":52,"leaves":[[52,0,52,0,20,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#createUser(Lio/keploy/samples/javadedup/model/User;)Ljava/util/Map;","name":"createUser(User)","line":57,"leaves":[[57,0,57,0,21,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#items()Ljava/util/List;","name":"items()","line":62,"leaves":[[62,0,62,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#updateItem(Ljava/lang/String;Lio/keploy/samples/javadedup/model/Item;)Ljava/util/Map;","name":"updateItem(String, Item)","line":67,"leaves":[[67,0,67,0,29,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#deleteItem(Ljava/lang/String;)Ljava/util/Map;","name":"deleteItem(String)","line":72,"leaves":[[72,0,72,0,21,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#simpleMessage(Ljavax/servlet/http/HttpServletRequest;)Ljava/util/Map;","name":"simpleMessage(HttpServletRequest)","line":81,"leaves":[[81,0,81,0,16,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#ping()Ljava/lang/String;","name":"ping()","line":86,"leaves":[[86,0,86,0,2,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#status()Ljava/util/Map;","name":"status()","line":91,"leaves":[[91,0,91,0,21,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#namedStatus(Ljava/lang/String;)Ljava/util/Map;","name":"namedStatus(String)","line":96,"leaves":[[96,0,96,0,23,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#healthz()Ljava/util/Map;","name":"healthz()","line":101,"leaves":[[101,0,101,0,14,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#info()Ljava/util/Map;","name":"info()","line":106,"leaves":[[106,0,106,0,21,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#timestamp()Ljava/util/Map;","name":"timestamp()","line":111,"leaves":[[111,0,111,0,14,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#products()Ljava/util/List;","name":"products()","line":116,"leaves":[[116,0,116,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#createProduct(Lio/keploy/samples/javadedup/model/Product;)Lorg/springframework/http/ResponseEntity;","name":"createProduct(Product)","line":121,"leaves":[[121,0,121,0,25,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#product(Ljava/lang/String;)Ljava/util/Map;","name":"product(String)","line":126,"leaves":[[126,0,126,0,32,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#updateProduct(Ljava/lang/String;Lio/keploy/samples/javadedup/model/Product;)Ljava/util/Map;","name":"updateProduct(String, Product)","line":131,"leaves":[[131,0,131,0,30,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#deleteProduct(Ljava/lang/String;)Ljava/util/Map;","name":"deleteProduct(String)","line":136,"leaves":[[136,0,136,0,22,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#patchProduct(Ljava/lang/String;Ljava/util/Map;)Ljava/util/Map;","name":"patchProduct(String, Map)","line":141,"leaves":[[141,0,141,0,30,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#post(Ljava/lang/String;Ljava/lang/String;)Ljava/util/Map;","name":"post(String, String)","line":146,"leaves":[[146,0,146,0,29,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#search(Ljava/lang/String;Ljava/lang/String;)Ljava/util/Map;","name":"search(String, String)","line":152,"leaves":[[152,0,152,0,21,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#file(Ljavax/servlet/http/HttpServletRequest;)Ljava/util/Map;","name":"file(HttpServletRequest)","line":157,"leaves":[[157,0,157,0,10,0],[158,0,158,0,6,0],[159,0,159,0,13,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#html()Ljava/lang/String;","name":"html()","line":164,"leaves":[[164,0,164,0,2,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#xml()Ljava/lang/String;","name":"xml()","line":169,"leaves":[[169,0,169,0,2,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#redirect()Lorg/springframework/http/ResponseEntity;","name":"redirect()","line":174,"leaves":[[174,0,174,0,4,0],[175,0,175,0,4,0],[176,0,176,0,6,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#config(Ljava/util/Map;)Ljava/util/Map;","name":"config(Map)","line":181,"leaves":[[181,0,181,0,21,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#resourceOptions()Lorg/springframework/http/ResponseEntity;","name":"resourceOptions()","line":186,"leaves":[[186,0,186,0,4,0],[187,0,187,0,4,0],[188,0,188,0,6,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#apiV1Data()Ljava/util/Map;","name":"apiV1Data()","line":193,"leaves":[[193,0,193,0,22,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#apiV1Users()Ljava/util/Map;","name":"apiV1Users()","line":198,"leaves":[[198,0,198,0,32,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#createApiV1User()Lorg/springframework/http/ResponseEntity;","name":"createApiV1User()","line":203,"leaves":[[203,0,203,0,26,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#apiV2Data()Ljava/util/Map;","name":"apiV2Data()","line":208,"leaves":[[208,0,208,0,22,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#apiV2Users()Ljava/util/Map;","name":"apiV2Users()","line":213,"leaves":[[213,0,213,0,13,0],[214,0,214,0,13,0],[215,0,215,0,32,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#createApiV2User()Lorg/springframework/http/ResponseEntity;","name":"createApiV2User()","line":220,"leaves":[[220,0,220,0,26,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#logs()Ljava/util/Map;","name":"logs()","line":225,"leaves":[[225,0,225,0,22,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#metrics()Ljava/util/Map;","name":"metrics()","line":230,"leaves":[[230,0,230,0,21,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#reboot()Lorg/springframework/http/ResponseEntity;","name":"reboot()","line":235,"leaves":[[235,0,235,0,17,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#proxy()Ljava/util/Map;","name":"proxy()","line":240,"leaves":[[240,0,240,0,13,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#legacy()Lorg/springframework/http/ResponseEntity;","name":"legacy()","line":245,"leaves":[[245,0,245,0,17,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#secureData()Lorg/springframework/http/ResponseEntity;","name":"secureData()","line":250,"leaves":[[250,0,250,0,17,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#adminPanel()Lorg/springframework/http/ResponseEntity;","name":"adminPanel()","line":255,"leaves":[[255,0,255,0,17,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#longPoll()Ljava/util/Map;","name":"longPoll()","line":260,"leaves":[[260,0,260,0,2,0],[261,0,261,0,13,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#password(Ljava/lang/String;)Ljava/util/Map;","name":"password(String)","line":266,"leaves":[[266,0,266,0,22,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#profile(Ljava/lang/String;)Ljava/util/Map;","name":"profile(String)","line":271,"leaves":[[271,0,271,0,21,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#events()Lorg/springframework/http/ResponseEntity;","name":"events()","line":276,"leaves":[[276,0,276,0,17,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#sessionInfo()Ljava/util/Map;","name":"sessionInfo()","line":281,"leaves":[[281,0,281,0,22,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#greeting(Ljava/lang/String;)Ljava/lang/String;","name":"greeting(String)","line":285,"leaves":[[285,0,285,0,11,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#itemList()Ljava/util/List;","name":"itemList()","line":289,"leaves":[[289,0,289,0,40,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#productList()Ljava/util/List;","name":"productList()","line":297,"leaves":[[297,0,297,0,21,0],[299,0,299,0,20,0],[301,0,301,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#messageForPath(Ljava/lang/String;)Ljava/lang/String;","name":"messageForPath(String)","line":306,"leaves":[[306,0,306,0,8,0],[307,0,307,0,4,0],[307,0,307,0,1,1],[307,0,307,0,1,1],[308,0,308,0,2,0],[310,0,310,0,15,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#response([Ljava/lang/Object;)Ljava/util/Map;","name":"response(Object...)","line":314,"leaves":[[314,0,314,0,4,0],[315,0,315,0,8,0],[315,0,315,0,1,1],[315,0,315,0,1,1],[316,0,316,0,12,0],[318,0,318,0,2,0]]}]} +{"kind":"class","id":"c:io/keploy/samples/javadedup/JavaDedupApplication","pkg":"io/keploy/samples/javadedup","name":"JavaDedupApplication","path":"io/keploy/samples/javadedup/JavaDedupApplication.java","hash":"59ddb82bf4da76309565d9805e05fd73a16d60f2672511efa4d793141be708ad","state":"ok","methods":[{"id":"m:io/keploy/samples/javadedup/JavaDedupApplication#()V","name":"JavaDedupApplication()","line":7,"leaves":[[7,0,7,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/JavaDedupApplication#main([Ljava/lang/String;)V","name":"main(String[])","line":10,"leaves":[[10,0,10,0,4,0],[11,0,11,0,1,0]]}]} +{"kind":"class","id":"c:io/keploy/samples/javadedup/model/Item","pkg":"io/keploy/samples/javadedup/model","name":"Item","path":"io/keploy/samples/javadedup/model/Item.java","hash":"29bf729dbb3da977244066732cac7307006f81cd719613004b6449768d357706","state":"ok","methods":[{"id":"m:io/keploy/samples/javadedup/model/Item#()V","name":"Item()","line":11,"leaves":[[11,0,11,0,2,0],[12,0,12,0,1,0]]},{"id":"m:io/keploy/samples/javadedup/model/Item#(Ljava/lang/String;Ljava/lang/String;Ljava/math/BigDecimal;)V","name":"Item(String, String, BigDecimal)","line":14,"leaves":[[14,0,14,0,2,0],[15,0,15,0,3,0],[16,0,16,0,3,0],[17,0,17,0,3,0],[18,0,18,0,1,0]]},{"id":"m:io/keploy/samples/javadedup/model/Item#getId()Ljava/lang/String;","name":"getId()","line":21,"leaves":[[21,0,21,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/model/Item#setId(Ljava/lang/String;)V","name":"setId(String)","line":25,"leaves":[[25,0,25,0,3,0],[26,0,26,0,1,0]]},{"id":"m:io/keploy/samples/javadedup/model/Item#getName()Ljava/lang/String;","name":"getName()","line":29,"leaves":[[29,0,29,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/model/Item#setName(Ljava/lang/String;)V","name":"setName(String)","line":33,"leaves":[[33,0,33,0,3,0],[34,0,34,0,1,0]]},{"id":"m:io/keploy/samples/javadedup/model/Item#getPrice()Ljava/math/BigDecimal;","name":"getPrice()","line":37,"leaves":[[37,0,37,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/model/Item#setPrice(Ljava/math/BigDecimal;)V","name":"setPrice(BigDecimal)","line":41,"leaves":[[41,0,41,0,3,0],[42,0,42,0,1,0]]}]} +{"kind":"class","id":"c:io/keploy/samples/javadedup/model/Product","pkg":"io/keploy/samples/javadedup/model","name":"Product","path":"io/keploy/samples/javadedup/model/Product.java","hash":"6195ac4af3e554fa4ed4374e1e8124d6b194beecd32412c5424988b4c56f0310","state":"ok","methods":[{"id":"m:io/keploy/samples/javadedup/model/Product#()V","name":"Product()","line":15,"leaves":[[15,0,15,0,2,0],[16,0,16,0,1,0]]},{"id":"m:io/keploy/samples/javadedup/model/Product#(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/util/List;)V","name":"Product(String, String, String, List)","line":18,"leaves":[[18,0,18,0,2,0],[19,0,19,0,3,0],[20,0,20,0,3,0],[21,0,21,0,3,0],[22,0,22,0,3,0],[23,0,23,0,1,0]]},{"id":"m:io/keploy/samples/javadedup/model/Product#getProductId()Ljava/lang/String;","name":"getProductId()","line":26,"leaves":[[26,0,26,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/model/Product#setProductId(Ljava/lang/String;)V","name":"setProductId(String)","line":30,"leaves":[[30,0,30,0,3,0],[31,0,31,0,1,0]]},{"id":"m:io/keploy/samples/javadedup/model/Product#getName()Ljava/lang/String;","name":"getName()","line":34,"leaves":[[34,0,34,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/model/Product#setName(Ljava/lang/String;)V","name":"setName(String)","line":38,"leaves":[[38,0,38,0,3,0],[39,0,39,0,1,0]]},{"id":"m:io/keploy/samples/javadedup/model/Product#getDescription()Ljava/lang/String;","name":"getDescription()","line":42,"leaves":[[42,0,42,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/model/Product#setDescription(Ljava/lang/String;)V","name":"setDescription(String)","line":46,"leaves":[[46,0,46,0,3,0],[47,0,47,0,1,0]]},{"id":"m:io/keploy/samples/javadedup/model/Product#getTags()Ljava/util/List;","name":"getTags()","line":50,"leaves":[[50,0,50,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/model/Product#setTags(Ljava/util/List;)V","name":"setTags(List)","line":54,"leaves":[[54,0,54,0,3,0],[55,0,55,0,1,0]]}]} +{"kind":"class","id":"c:io/keploy/samples/javadedup/model/User","pkg":"io/keploy/samples/javadedup/model","name":"User","path":"io/keploy/samples/javadedup/model/User.java","hash":"b591fb83b5aa4cfc6e71b286daeecc93fc81ab930644a2087d0bee3a3a03d2df","state":"ok","methods":[{"id":"m:io/keploy/samples/javadedup/model/User#()V","name":"User()","line":8,"leaves":[[8,0,8,0,2,0],[9,0,9,0,1,0]]},{"id":"m:io/keploy/samples/javadedup/model/User#(Ljava/lang/String;Ljava/lang/String;)V","name":"User(String, String)","line":11,"leaves":[[11,0,11,0,2,0],[12,0,12,0,3,0],[13,0,13,0,3,0],[14,0,14,0,1,0]]},{"id":"m:io/keploy/samples/javadedup/model/User#getName()Ljava/lang/String;","name":"getName()","line":17,"leaves":[[17,0,17,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/model/User#setName(Ljava/lang/String;)V","name":"setName(String)","line":21,"leaves":[[21,0,21,0,3,0],[22,0,22,0,1,0]]},{"id":"m:io/keploy/samples/javadedup/model/User#getEmail()Ljava/lang/String;","name":"getEmail()","line":25,"leaves":[[25,0,25,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/model/User#setEmail(Ljava/lang/String;)V","name":"setEmail(String)","line":29,"leaves":[[29,0,29,0,3,0],[30,0,30,0,1,0]]}]} diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup/io/keploy/samples/javadedup/DedupController.class.bin b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup/io/keploy/samples/javadedup/DedupController.class.bin new file mode 100644 index 0000000000000000000000000000000000000000..80e47e737f91878b37b1ab0eebfc305be18e1b72 GIT binary patch literal 15569 zcmcIr2YeLA)qit3=@xOK=qSOo0~?GAT95%7kZ}aaAR{D#glyS~@k+On&YW(~w<|z) zY$r}HPH}qg-ANoLk+6xK-g~jrdz{{ToZ|2QW@h(Rrv)AP<>QC$w#>|X@BjMj-e>-E z@BKuyC3a~o#i%Ml@g!AKji3eBTgNRkXXQuCp26$wAx}_ETQ;Bdb_uH5(9my?fh(?O zdE>4@wSr>Kv06$}U4rH%X+A9w)RcA1V|F3uOqi}UR>;||$%|&}%y_}v%b$CkyjOH` zIlCyxDB5np$-B0o3pQNSRZf0qgSs^CWpn01tFV)a)h8)Mi!gVjXxm;kKO$)ThS1br zuZWBOR4I(UWY7{p)yJ*exSgO!`QpmO4_o<+Ggb{$_+Mi!Eu|F<%1S};4OcYu^M%z( zO4AxaiM)N%>&ye3_=e7g&RRO3E=bV2B(0|l1uZNIfid<7iwwQ-29#a}PtV2LY-phKh9uS$YXZ!YL&;)RXU08GICVG5=ZcfrI zbgQ8A&$z18s|-{?X3yb{_P!3-&bqD&CO{gGp2*&Ro1h(GH&Q`yXGM!}w0ts@twlM? z*85aQk%#aGJsq~xE8}DOkV49!XF_cNXb1t|*>F7>0LEZt=I-psUVA7zX5|ce4hC)C z-m-mrEj^2#$4ow-l@f|6+Dn=K3t)ymlm@*BXTEzG^b$VPPGiu^;FjrvRrInJT(~sq z70i?64Z+l3B~9J+40;V99YPd`@fh?vSY_X6)=lHTl~yr7U2t5l#h^F9mt5N_4vj+X zM~B%6--HEo*|Dt0zWJ6Uy_Fqv9vqI-0^+K2tGB~c&EYJvM=jk+?@Z9UlJsu6%MaP1 zppJn#J3r!$$_xc(y_4=v(tGKB2$1ghpevIBBQ!1U29azM#EZ)B?aWc8XwWGPZyVjb z%a>|@-@&dn6Bi7+SJ2ircI;hktjAr~JEQqFQy#Xdw75$-dYh?F+e|q;K_5VhJFD^S zw~-}_5co6)_z%JfIY8|&%@M~L$=NN4^|kb2`bdI4nxv1>$Kk=!MDhq_2vYz;MPos0 zqs&2Bh#m|RKTV%Z(x>Rt@Kx9Lx||`)L+}7qJ=q`G(9b^i*(80A9d6;lp8k%#M-O!z zKG@#f(cRZIz|Qgow!<$9>fUg6*U%zX?m%CLdSGe8*^!b-eif3c6kyN;u%`B0&N*Sw z*I~i?I{KQ^hkAPP*>k9`v!}b4<@{!nzD3`LeOMXf#S0_AiSizn`nyT`9!nkTIbhHa zfK<*Nv4$qn9ElA25s0&LQ40M8hgPmIimN|`n+)2XRZBmoUnJ<4N%|H2+BaO8@!E?; zYr=&Tma7uty4gw}og;q>l{3LoSo}Rav0zQ)91GFqj|iN3`-J2!4QE5_FzCzP&^VW5SsDZ#^_Lx(gFa)Mlw{G2baJL~) zL@w=}0Escs8gv{l&GlxMO8{sQFbVPLS;ZqBv~hCoqR=UMKt zGy(zYY7gLUgxm$ob&*Untnsu|pdm~sgbz9Ch|O#xMMuUqaS_zf($Zpxt?(@95jX$= z8>ISuMLv-QrpaVDz(T&5_tu0$O83KiyWCxO=ax)It6N3&c$`B~ssxRIOb`k=0Tf^+) z`T1L*v}2$%n>Im4)OE``hMbMV@QQo}DfLR2Suwra@jT=h@mOfC9nix|26J|LP+o>w zu7!T=A(-@aSnOC9iPz58iX&o>aY0?zc@DUQvgA&MRH@)*j8GeDet{B-!v$Buwvlq( z^qx_xh{nYbC=(Z}6E{8Kj2B_k>{%{2JqAbdY!n7!j6K^KMD9fP6;4tVxJ#Owvs`pZ z8T(|-HSaiiImAnfad8~8;7(Q!k#Ji5qKmACs||4yW4fKRo^~Vhy0}U9P*KU8 ziQk+Qw{Sr**K>}b4=V1lTstA2;I~s+{`@gJv)^(@(NWflTd9g?J_%K~cyd`Q>9aC3 zKg^}SiKiyT)5O!Eu);XX<4eMt@e4&6V?+J)w60VS0*-s&CkWnM*}UC7J~n6<``9XA zBAk|U{Z=u{=lWvYL$!teuIp^3S=MR8dE-R{S5EKcJvG8oZ+0Y)p1uf@7nk2}n<;$g zik4v)2rN51jz~FX_h;QKxNXnp9n{~5C17?-WL{z|`d%0-`DJnm2I?b~7rU={y{=xYdB)O=D(T#msOq>Z1>KYKn4~0B&Oj zM<-Mhj#ebBf6d@bsH6P1Y*fX78)iVfa*=djE0KXn$&5X0jpw|6-V_1Y`&0*9t4hfu z$RUEZZ#b)|sUXioee3p%3anr8R6$289{b+{wzYD_qkk!xVYRH`QTkUrU!%O+$xHwQ znP}hzZPR>k2CEKWTvs{fGR9kmJ8klvLDZ*D0 zx)DW>B4gd`YY4hlp+Ec3GF4X$wPCv0V?+i9V)78#hqqi42-*|!a*ZhyJ73B>gRpoc z6ceRoQQe-y1NEf2>bgENFkT<3rB95a(LgbV>%k!=t2JxYT%Q1QrH2f0FTyWPkH)eamEAP(3bXod=CbsM=2p2vDLAU?E zFr@9#&m0w+Cr8D30eIT6>r`|M5X7d$jLP90BrbCtQ1t$%3sxv9EJFpXK(3_RmkUl? zqqv&Z<41&?`3f2zb_lm*ddWFa@XOAv(}z~f_ARsWRpC-)il}gwpvr3_lzT(m6zJ-6 z>lrx@cUEO=0m2$?y*tsl*YGwdA6)a_tW>dpt7LiNnl~xBH=9frr;0jKU5(mlRPMH%CGqVG^ck^)D=AS%b?Yiu?NBPjJu*UV;15x?~$TSJM%2w`S&*(c_X1> z0%5@Wooq(O<`T0tYAaGuTI)0Lo0&$CD;pACTz3bduoSdWWeZg}i?!!RgSy$F`00`6@lO^K2d>Xz-jB>}#z;a5>3Hc3Y>h+ZtxKRkK?? zoGtPbM0gzZ0jxCtt;$8G@h|7VOR%L1!IGvccR@{cif6&tPaR^3{TsoR8nsc%sZY=G zFW~W!a}xDe4d+C8pf;@8KKvu>@ZrIq?aQLh0$DO++590ZZ^M0|7Ctiz>eR^LG3OyX zw}*y0P7c}1x)L#LORg&ASAL*fERGkjr5X(4(upT;EXdi0zhfyVC1H_y8)VczCOZA_ zRkd*mk>qA2@JMhSnCDUn&n@I$Wv49J>JZoDsV?2(NRJ3!b*TS_j=`e!?g?*c2|O%G zMZ9ew=!)5P;;~SLW&^2vr5|1{qwAuw9hnx!4DG@>_@*nLk*QZFcG4O=ZH>vd6}BRf zQ4i#zZ=4-3Bli!Hf^gezW%vlr1K zO=br;XWnZXo2F=C`O1xQ~=? z%d;5roTjA%i)h&tEw7%WRi|ielz3)HytT86_X<#brFa#^C5l&pSr+dCpXvrlwQ*pI zn(H@ClG%j6NxCSC?v{}77X-pzpoLcezd8bh*GLeO48m*i_c~X+9$#+|Z`3>;f@Ybg z6^#j{S>~z!;@*Mk`t7{~HBC{xY!C6WBH(2O-tTx5)>gc{IpF0jfwJFPQpnq+Lf%fh zwC2OsxwBmHyF!>(Ps98j0hN+T-3cDvDc+^!%Rs&=5L@54pe7{XWh`L36mTF>zo&P= zXpWL}kEEPdD+H3Rr`>FFysCJ&xC`d5WPVS;&E0{_@0Bvw#lVy@LM8e80;7mlKu3wc z2p`*j1_emAG&3b*cz+o)A&;08Q(6H%+BTPmZL_iI9@-~pin^k}c}$3l}N8}F-=gfslPIc=5-;E(*eloS^fU=C0|o^_k}?A z%e3r^eN_*DmG~*TCW^itl6mp0GJg>=e<>g|1LW%%RboW}d=Yd|EFe=ra&OPt&so>TjQ-=SE?DONgtEvmzDs0IrJR>SQV-tZ=Vf7`l|RbBKc3CE>&f=;>tro=KN^@>Cb{> z6%8p~H~aY+Q#&daWtgv&jKVHo4ZEy=ZInUY5n?&)DZdC(`!8|i>-<*%gC*1ab$M!6 z2Kt+T!2w-+EcXrc^*(=Z3f&WhkwA~ zAA_9#4Dc6+Mz(7^M7DpkAgH_kZA#|{RsL_Dq<2ixd%AtoeLuw5I33e@(i2}r%jnf~ zAzlRJcA`-&9@y%3RT*GRTKtKnz{q;>XB<`O0HCUc=^esf!17-s%7Cfz(Dz13R9RQJ z3|N8%UI1dM;rN^QyKFrEfvXQkEPGG+vTv(s*?;O~|Al4$E&ij6ssSLt#r4Y5bQ(qe zeO*oU_n)EOHmBwa837fCgKR8L>(9nD*0-AdPz-d~5cP2~F_zW6&4|^swj{+Emul(4Y zSS^%q$L7XhCb7DpZs^o?!)o7Axllpju-4DG_1~Xv?y774aL6=PD>K)I98nL(VAiqu z0m-X$2ig)v@`un28tpsLX#W_T)IZdFihdTw!6!l-=&X7c4ioAz2%VJB|k8r;^wlv6!?C0vAd$qPYp12m|%{q}3`IB$_S z|CCtg#~Mz=DbYAxjr_wq5^?`h5y}Bmx(``z$ zfGh4OQ(}|!+t%r*C{E^wIMJ2{JXW89lakGuWt_w|1=bMh&Pp*`lrcoOvse^T&Ft>H z8476)+_^kdro=XB2bXZLJObg*un{Acg^11`Dld}h1PM@Xp7r32i@@c9=67h#r$n17 z^~=@1G6Z(05mEb2SPVbj(>rrdpjuf+^w-=cb`4aW#z#s#Iwf}Ven3j>#fHF?xSY*S z-swz<1M1E}|4y~K(~}a9QFj2noP(;ToUibcivDmY74HC7_>Gz7>Y2LF?5n;{9AU1G zfR`&%0-x8Y(kdk$Cy_tOX?#jpl0zPqNr~Z<81+Y8FGuD4QPX&;=N1QV=$|i|s8ejj z&;A-hYU-QD-vi(eKNi#~xdSt~8rjj*w7{Gc`S^9{unwlg4b79neHkTE;>2BATg_GR zNii`g9uG}5$1$v=F;#UaE?TKhTtrL77FsU0NiJ07u_JKW9kDHFw(#r=yQxcX6pL*I z2Y3?lFIt|!52SA6pXzG--V7h*-mvksxOHGkJn>-Fu9Z#qzn+#iH?3?vC2nh48NaUe z9`Teay7I|Yv9~<(P%~>PF82hMU@nOr*lyj4pZ>PdT6}N7OZ`9mQE zuN_^MksG`#E(5)sW;DkGK9!wF$gv{V;wR@jOtC#kS*bhdkOTOHTxhSR2v* E0h0!iGXMYp literal 0 HcmV?d00001 diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup/io/keploy/samples/javadedup/JavaDedupApplication.class.bin b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup/io/keploy/samples/javadedup/JavaDedupApplication.class.bin new file mode 100644 index 0000000000000000000000000000000000000000..337c0c2eee8b513423a99f176cc00eb9ac331eb8 GIT binary patch literal 698 zcmb7CO-~y!5PePp8$$R>3WdUjTgU-d6+%dk{{gUvtrlvi2WYf0gIR{6Y-rE@kXfk|OhTq97F^@bKX_m;G`?86g6d6CG+ljv&Wm%#k;gr!Wv@jo_ z-NphI8T!V?JkPAs@wFAHd@%NokBxDBSb~*GnraKpaL@~1nH?u0&j(i(Waxae!A=Ew zraxe49Gi(`SPGSv--~oC?WGtegmgj^iDV?K^6?AUa5pMv_!7SNkA6}r6mebaO`|rV zys$C#?oLTFw1%dzkvvtN+Qu{$cfA6XyC}3%sT`?X5$_|djVo2A)`#Jn(u%^F$mnYo z7goLq{f#uQh&{kI0@??EYG8gN$SzUD$usgZ{hwe@WqO&cPlHNU$hMy!po1<&bNGN& sLVWRl*$ilQg8nbe{dleS%Lm$x0P~d|O|0P~;Wez6b_1Wt`&pa7f3TLbLjV8( literal 0 HcmV?d00001 diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup/io/keploy/samples/javadedup/model/Item.class.bin b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup/io/keploy/samples/javadedup/model/Item.class.bin new file mode 100644 index 0000000000000000000000000000000000000000..0a64ec40fb4a46619fd8e27a5fabbf9545f00c5c GIT binary patch literal 1186 zcma))-EPxB5QWd?Kf!I%kcOsEpoLb6`SAtZ6?y>(sZvysigNEHt8|NA52ULp@wf9e*;^o^u*# zUm;_L3ia+v{XCeN@#t9eHZ4>|EH`$2EeBf`>bR+ppU%ucD>S1%-wiIDS7!7^4~*}6 z$Lx5`D-^0-GuCGdzprN>+`h+?dUrB#y`DQWvX2jQ!G)PCw7dTYP5qbc>%{9$l60LO zA28CvDIeZwcdobb^E~>zopOz|7ukNsnJ{Bzl$Ch)J{rr;U)AK*3MJOZRjpV&StJ)! z&L@l6K)*FIv$8n#R2;%SO~~;7$a1C6%rat880idS3B*T%YlFLIjEXUDe~0=SmThyD ze+v}Iv9-y}ii&UxJKTvti>nBDtjsYsM8JNuMA3e{#D@K3iP}o9$qrzXl~t5cW41+@ zdZ=wj{joNcLU4g=)goL7 zkyoJ6!&}7N4HcSE)jzmG45x566=Wj`vX=^SFCAnf39?U+gEb&xt4!cNO^>{|I+0ou ecr6A{~O*Ye2*r0eKjDaQFvWAGr+x literal 0 HcmV?d00001 diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup/io/keploy/samples/javadedup/model/Product.class.bin b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup/io/keploy/samples/javadedup/model/Product.class.bin new file mode 100644 index 0000000000000000000000000000000000000000..d94a11c064eaf166f2f721756cacac31f0836e49 GIT binary patch literal 1869 zcmb`HZ%@;35XY}ux4BWqWPom>C?ag-PZdxfKuC_UKr&e1)>x(w6>?Lg`w)nWr0+&rw06G13&OOUF17J9{+2?OcYWKJUD1@DJhH?K+je}DA4j;+XgOWuqv`p= z{oJ*H+&b|b+b}PJj5qiX@}6-0GpJ=wO{*^yDnvDWkSHR5AUn3{_g&mrh!Jd$7Ptc) z^(A_u&_qMpVz1vl67Hc1I&O&5GOecRO8y@?RsWL&?sDVWT}F!C+Gy(QdxkBnHx-(> zOm)^fCwu>Yk|cJC6KfseM}x!TGF7XOmVhYlQq*(;6pQ5`wuga&g$F{b2;ZyWOes{v ztxFDm>M)qD$@A>P<|wo{;!G?i%o*yGBfg;w(=Elli8>rvIB@!IOT3mmFj~}}HC`lo zfSpKTp;cV5n557x{2Rl08^4N2RouX8!I|QX3QlYgZr%y(St`KBCji&G@ND2zaAx!$ zq-@t_<}Qov&sVc@_=&s%7Di8)d7|CiaMY& ztEP!nt4K@0QmIUNnxZ0>y9~Grs^Bb}p;DBELXd=jX9ArKbL{A18kAo}%Wa$bJ?jLtxIT5532hkHjR+2$Vaga5D7*~MsR5Hja zt)W}-S5uCItS5kk@0Z``V|o&a?Bl$6MgPoPILXfd^!y5-a6N`olntind|}p+Wg{4u F&A(Z7TnPXG literal 0 HcmV?d00001 diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup/io/keploy/samples/javadedup/model/User.class.bin b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup/io/keploy/samples/javadedup/model/User.class.bin new file mode 100644 index 0000000000000000000000000000000000000000..4c294bf51baf9ef55bd86904e184944242987d85 GIT binary patch literal 889 zcma)(%TC)s6o&sv96L@O0)eEZEvRfD2^C(nJ1RC+yNc+7Hd2?9WRwp064?$CZv|2j z5(^%HhpKwc*lnuDm8y~EV*T^^J7*q$+&=(#i`@n)s5-Db)L;v&pUX?>2Qr%ZpT_5E zk_p)FbfmKrfoiWma#4rtpyA;OJb`^3`xj~+#8-YQ!+D@mp8-=fUCjM3o~pqAlBz_& zie#t++QTLPEK77WJ7(gVhZf^%DwH~)^n1E0&>HGUoi4($N1>RY}3YJ>vmuRx~+%7{84KIC=PDO_wDXYnGL zsE?Y Date: Fri, 20 Nov 2026 01:11:48 +0530 Subject: [PATCH 02/14] feat(dedup): send each class's probe-to-leaf map after the build table Keploy gets each test's coverage as JaCoCo probe hits per class, while the build table counts lines and branch edges (its leaves). To turn hits into leaves without the class files, Keploy needs, per class, which leaves a hit of each probe covers. keploy-sdk now computes that map in the app's JVM, with the same JaCoCo analysis steps as the table, and sends it right after the table, in the same answer to TABLE. - A probe covers more than its own line: JaCoCo marks every instruction on the path that ends at the probe, and the branch edges along it. So each probe maps to a list of leaves, and a test's leaves are the union of its hit probes' leaves. - The map is JaCoCo's own result, not a copy of its rules. JaCoCo's public API gives only per-line sums, which lose which branch edge of a line is covered, and two tests that cover different edges of one line must count two. A small class in JaCoCo's package (org.jacoco.core.internal.analysis.KeployBranchCoverage) runs JaCoCo's package-private analysis steps unchanged (the instruction graph, the filters, merges and branch replacements) and reads each branch through JaCoCo's own replaceBranches. It checks every line's sums against the coverage JaCoCo computes in the same pass and throws on a difference. keploy-sdk.jar relocates org.jacoco.core, and this class moves with it. - A method's probes never reach another method, so one analysis maps the k-th probe of every method: the map takes one analysis of a class file more than its largest method has probes, on top of the table's. Each one reads the whole class file, so a class with one very large method costs seconds of CPU on the analysis thread. - The map is a second stream of TABLE frames after the table's last frame: "part":"probes", seq from 0, the last frame with the record count and the sha256 of the stream. One record per class name of each measured class: its name, its JaCoCo class id, its unit, and for each probe the covered leaves as ranges of indices into the unit's leaves. Indices are relative to the unit, so a unit Keploy marks not measured only drops its records. The map is never uploaded. - A test's hits name a class by its name only, so a name has one record. The index can find one name under two paths: the same bytes count once, and two class files that differ share the record when their probes map alike (the record carries the first one's id). - The map is JaCoCo 0.8.15's probe layout, that of the agent keploy-sdk.jar starts. It is not for the hits of a JaCoCo agent the app's command attached (the index record names that case): another version can place the same number of probes differently. - A class whose probes cannot be mapped exactly is sent as not measured (analyzer_failed) in the table itself, so the table and the map always agree; that includes two class files of one name whose methods or probes differ. A class that is not measured has no map record, and the INFO summary counts the classes that failed this way. - Lines below 1 have no leaf (javac never writes line 0; bytecode tools can): a method keeps its other lines in the table and the map. Rows: for every class file of the java-dedup sample, of a Java 8 class of code shapes (&& on one line, loops with break and continue, an int switch with shared and fall-through cases, a String and an enum switch, try/finally, try-with-resources, synchronized, a lambda, an assert) and of a Java 21 one (records, a sealed type, pattern switches with guards and record patterns, an exhaustive enum switch, a String switch with yield, instanceof patterns), the leaves the map gives no probe, each probe alone, every pair of probes and all of them equal what JaCoCo's public Analyzer counts for that set, line by line (code leaf hit, branch leaves hit, branch leaves), and each record has as many probes as JaCoCo's instrumenter gives the class. The java-dedup map is a golden for the Go decoder, and the control-socket row checks it arrives after the table on a real socket. Rows cover one class file found twice, two class files of one name that map alike and that differ (moved line, missing method, probes in another order), and a method with line 0. Only BuildTable.java changes in the agent; KeployDedupAgent.java is untouched. Signed-off-by: slayerjain --- .../main/java/io/keploy/dedup/BuildTable.java | 333 +++++++++++- .../analysis/KeployBranchCoverage.java | 335 +++++++++++++ .../java/io/keploy/dedup/BuildTableTest.java | 18 +- .../io/keploy/dedup/ControlSocketTest.java | 14 + .../java/io/keploy/dedup/ProbeMapTest.java | 472 ++++++++++++++++++ .../dedup/buildtable/java-dedup.probes.ndjson | 5 + .../dedup/buildtable/modern/Modern.java.txt | 101 ++++ .../io/keploy/modern/Modern$Circle.class.bin | Bin 0 -> 1465 bytes .../io/keploy/modern/Modern$Color.class.bin | Bin 0 -> 1216 bytes .../io/keploy/modern/Modern$Pair.class.bin | Bin 0 -> 1588 bytes .../io/keploy/modern/Modern$Rect.class.bin | Bin 0 -> 1733 bytes .../io/keploy/modern/Modern$Shape.class.bin | Bin 0 -> 394 bytes .../io/keploy/modern/Modern$Square.class.bin | Bin 0 -> 1468 bytes .../modern/io/keploy/modern/Modern.class.bin | Bin 0 -> 5403 bytes .../dedup/buildtable/shapes/Shapes.java.txt | 113 +++++ .../io/keploy/shapes/Shapes$Color.class.bin | Bin 0 -> 1197 bytes .../io/keploy/shapes/Shapes$Res.class.bin | Bin 0 -> 489 bytes .../shapes/io/keploy/shapes/Shapes.class.bin | Bin 0 -> 3908 bytes 18 files changed, 1373 insertions(+), 18 deletions(-) create mode 100644 keploy-sdk/src/main/java/org/jacoco/core/internal/analysis/KeployBranchCoverage.java create mode 100644 keploy-sdk/src/test/java/io/keploy/dedup/ProbeMapTest.java create mode 100644 keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup.probes.ndjson create mode 100644 keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/Modern.java.txt create mode 100644 keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern$Circle.class.bin create mode 100644 keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern$Color.class.bin create mode 100644 keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern$Pair.class.bin create mode 100644 keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern$Rect.class.bin create mode 100644 keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern$Shape.class.bin create mode 100644 keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern$Square.class.bin create mode 100644 keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern.class.bin create mode 100644 keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/shapes/Shapes.java.txt create mode 100644 keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/shapes/io/keploy/shapes/Shapes$Color.class.bin create mode 100644 keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/shapes/io/keploy/shapes/Shapes$Res.class.bin create mode 100644 keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/shapes/io/keploy/shapes/Shapes.class.bin diff --git a/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java b/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java index d9ff5217..6bab08ae 100644 --- a/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java +++ b/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java @@ -7,6 +7,7 @@ import org.jacoco.core.analysis.IMethodCoverage; import org.jacoco.core.analysis.ISourceNode; import org.jacoco.core.data.ExecutionDataStore; +import org.jacoco.core.internal.analysis.KeployBranchCoverage; import org.jacoco.core.internal.data.CRC64; import org.newsclub.net.unix.AFUNIXSocket; import org.newsclub.net.unix.AFUNIXSocketAddress; @@ -26,6 +27,7 @@ import java.security.NoSuchAlgorithmException; import java.util.ArrayList; import java.util.Arrays; +import java.util.BitSet; import java.util.Collections; import java.util.Comparator; import java.util.HashMap; @@ -59,6 +61,13 @@ * class the stream is one frame: build "", last, 0 records. After each frame the SDK shuts its side's output * and waits for {@code OK} and a newline; a receiver that closes without it ends the stream, and the table * goes again only when Keploy sends TABLE again. + * + *

After the table's last frame the probe map follows as a second stream of the same frames, with + * {@code "part":"probes"} and seq from 0 again: one record per class file name of each measured class, giving + * for each JaCoCo probe the leaves (indices into its class's leaves, as ranges) that a hit of that probe + * covers. Keploy turns each test's probe hits, which name a class by its name, into its leaves with it; it is + * never uploaded. Two class files of one name whose bytes differ share the record when they map their probes + * alike, and fail their class closed (not measured) when not. */ final class BuildTable { @@ -72,6 +81,9 @@ final class BuildTable { static final int MAX_RECORD_BYTES = 8 << 20; /** The first 4 bytes of a TABLE frame. */ static final String MAGIC = "KTB1"; + /** The parts a TABLE answer streams, in this order: the table, then the probe map. */ + static final String PART_TABLE = "table"; + static final String PART_PROBES = "probes"; private static final int CONNECT_TIMEOUT_MILLIS = 5_000; private static final int REPLY_TIMEOUT_MILLIS = 30_000; @@ -201,12 +213,22 @@ static void send(Supplier> classes, FrameSender sender) { try { long cpu0 = threadCpuNanos(); long t0 = System.nanoTime(); - FrameWriter frames = new FrameWriter(sender); + FrameWriter frames = new FrameWriter(sender, PART_TABLE, MAX_FRAME_BYTES); Summary s = write(classes.get(), frames); frames.finish(s.build, s.records); + FrameWriter maps = new FrameWriter(sender, PART_PROBES, MAX_FRAME_BYTES); + for (byte[] p : s.probes) { + maps.record(s.build, p); + } + maps.finish(s.build, s.probes.size()); StringBuilder m = new StringBuilder("Keploy coverage: build table "); m.append(s.build.isEmpty() ? "(no app classes)" : s.build).append(" sent (").append(s.records) .append(" classes, ").append(s.notMeasured).append(" not measured"); + if (s.notMapped > 0) { + // Two copies of a class that differ, or JaCoCo's own steps disagreeing with its public result. + m.append(" (").append(s.notMapped).append(" of them as their probes could not be mapped: two" + + " different class files of one name, or a Keploy bug; the FINE log names each)"); + } if (s.noLines > 0) { m.append(", ").append(s.noLines) .append(" class files without line numbers, so without lines: compile them with -g"); @@ -214,7 +236,8 @@ static void send(Supplier> classes, FrameSender sender) { if (s.unnamed > 0) { m.append(", ").append(s.unnamed).append(" index entries without a class name left out"); } - m.append("; ").append(frames.seq).append(" frames, ").append((System.nanoTime() - t0) / 1_000_000) + m.append("; probe map of ").append(s.probes.size()).append(" class files; ").append(frames.seq + maps.seq) + .append(" frames, ").append((System.nanoTime() - t0) / 1_000_000) .append(" ms, cpu ").append((threadCpuNanos() - cpu0) / 1_000_000).append(" ms)"); LOGGER.log(Level.INFO, m.toString()); } catch (IOException e) { @@ -246,13 +269,20 @@ static final class Summary { final int notMeasured; final int noLines; final int unnamed; + /** How many of the not measured records JaCoCo analyzed but whose probes could not be mapped. */ + final int notMapped; + /** The probe map's records, in the table's order: one per class name of each measured unit. */ + final List probes; - Summary(String build, int records, int notMeasured, int noLines, int unnamed) { + Summary(String build, int records, int notMeasured, int noLines, int unnamed, int notMapped, + List probes) { this.build = build; this.records = records; this.notMeasured = notMeasured; this.noLines = noLines; this.unnamed = unnamed; + this.notMapped = notMapped; + this.probes = probes; } } @@ -271,6 +301,7 @@ static Summary write(List classes, RecordSink sink) throws IOExceptio static Summary write(List classes, RecordSink sink, int maxRecord) throws IOException { Map byId = new HashMap<>(); + Set seen = new HashSet<>(); long[] ids = new long[classes.size()]; int n = 0; int unnamed = 0; @@ -288,6 +319,11 @@ static Summary write(List classes, RecordSink sink, int maxRecord) th unnamed++; continue; } + if (!seen.add(f.name + "\n" + id)) { + // The same class file again (the index found it under two paths): it is one class file, and + // the build id and the unit's hash count its id once either way. + continue; + } Unit had = byId.get(u.id); if (had == null) { byId.put(u.id, u); @@ -296,13 +332,15 @@ static Summary write(List classes, RecordSink sink, int maxRecord) th had.files.add(new UnitFile(f, id)); } if (n == 0) { - return new Summary("", 0, 0, 0, unnamed); + return new Summary("", 0, 0, 0, unnamed, 0, Collections.emptyList()); } String build = "jv1-" + classIdsSum(Arrays.copyOf(ids, n)).substring(0, 32); List units = new ArrayList<>(byId.values()); Collections.sort(units, UNIT_ORDER); int notMeasured = 0; int noLines = 0; + int notMapped = 0; + List probes = new ArrayList<>(); for (Unit u : units) { byte[] rec = u.record(); if (rec.length + 1 > maxRecord) { @@ -311,12 +349,16 @@ static Summary write(List classes, RecordSink sink, int maxRecord) th } if (u.failed) { notMeasured++; + if (u.notMapped) { + notMapped++; + } } else { noLines += u.noLines; + probes.addAll(u.probes); } sink.record(build, rec); } - return new Summary(build, units.size(), notMeasured, noLines, unnamed); + return new Summary(build, units.size(), notMeasured, noLines, unnamed, notMapped, probes); } /** @@ -439,8 +481,12 @@ static final class Unit { final String id; final List files = new ArrayList<>(1); boolean failed; + // Set when JaCoCo analyzed the unit but its probes could not be mapped (so failed is set too). + boolean notMapped; // Class files with code but no line numbers: their methods have no leaf, so they are left out. int noLines; + // The probe map's records for the unit's class files, in file order; set when record() succeeds. + List probes = Collections.emptyList(); private Unit(String pkg, String name) { this.pkg = pkg; @@ -549,6 +595,41 @@ byte[] record() { int c = Integer.compare(a.line, b.line); return c != 0 ? c : compareCodePoints(a.id, b.id); }); + Map byId = new HashMap<>(); + int base = 0; + for (Method m : methods) { + m.base = base; + base += m.leaves.length / 3; + byId.put(m.id, m); + } + List maps = new ArrayList<>(); + try { + String lastName = null; + String lastProbes = null; + for (UnitFile f : sortedFiles()) { + String p = ProbeMap.probes(f, byId); + if (f.file.name.equals(lastName)) { + // Two class files of one name whose bytes differ (the index found both). Keploy finds a + // class's map by its name, as a test's hits name it, so they share one record: exact + // only when a hit of either one covers the same leaves. + if (!p.equals(lastProbes)) { + throw new IllegalStateException(f.file.name + ": two class files of this name map" + + " their probes differently"); + } + continue; + } + maps.add(ProbeMap.record(id, f, p)); + lastName = f.file.name; + lastProbes = p; + } + } catch (Exception | LinkageError e) { + // A class whose probes cannot be mapped exactly is not measured: its hits could not be read. + LOGGER.log(Level.FINE, "Keploy coverage: the probes of " + path() + " could not be mapped", e); + failed = true; + notMapped = true; + return notMeasured("analyzer_failed"); + } + probes = maps; StringBuilder sb = head(sourcePath(infos)); sb.append(",\"state\":\"ok\",\"methods\":["); for (int i = 0; i < methods.size(); i++) { @@ -610,24 +691,45 @@ static final class Method { final String display; final int line; final int[] leaves; // flat: line, count, kind per leaf + // The unit-relative index of the method's first leaf (the unit's leaves in record order). + int base; + // The lines of the method's code leaves, ascending, and each one's method-relative leaf index. + private final int[] codeLines; + private final int[] codeLeaves; private Method(String id, String display, int line, int[] leaves) { this.id = id; this.display = display; this.line = line; this.leaves = leaves; + int n = 0; + for (int i = 0; i < leaves.length; i += 3) { + if (leaves[i + 2] == 0) { + n++; + } + } + codeLines = new int[n]; + codeLeaves = new int[n]; + n = 0; + for (int i = 0; i < leaves.length; i += 3) { + if (leaves[i + 2] == 0) { + codeLines[n] = leaves[i]; + codeLeaves[n++] = i / 3; + } + } } /** * of makes the method's record from JaCoCo's per-method lines: one code leaf per line with - * instructions (n = their count), then one branch leaf per branch edge on that line. A method with no - * line (a class built without line numbers) has no leaf to show and is left out, as the contract's - * method count leaves out a method with no code leaf. + * instructions (n = their count), then one branch leaf per branch edge on that line. Lines below 1 + * have no leaf (javac never writes line 0; some bytecode tools do), and the method's line is its first + * line with a leaf. A method with no such line (a class built without line numbers) has no leaf to + * show and is left out, as the contract's method count leaves out a method with no code leaf. */ static Method of(String vmClass, String nested, String unitName, IMethodCoverage mc, ClassInfo info) { - int first = mc.getFirstLine(); + int first = Math.max(1, mc.getFirstLine()); int last = mc.getLastLine(); - if (first < 1 || last < first) { + if (last < first) { return null; } int[] buf = new int[16]; @@ -652,9 +754,12 @@ static Method of(String vmClass, String nested, String unitName, IMethodCoverage buf[k++] = 1; } } + if (k == 0) { + return null; + } boolean varargs = info != null && info.varargs.contains(mc.getName() + mc.getDesc()); return new Method("m:" + vmClass + "#" + mc.getName() + mc.getDesc(), - Names.method(unitName, nested, mc.getName(), mc.getDesc(), varargs), first, + Names.method(unitName, nested, mc.getName(), mc.getDesc(), varargs), buf[0], Arrays.copyOf(buf, k)); } @@ -674,6 +779,199 @@ void json(StringBuilder sb) { } sb.append("]}"); } + + /** codeLeaf is the method-relative index of line l's code leaf, or -1 when the method has none there. */ + int codeLeaf(int l) { + int at = Arrays.binarySearch(codeLines, l); + return at < 0 ? -1 : codeLeaves[at]; + } + + /** + * check throws unless the analysis has the method's leaves: on each line, the code leaf's instruction + * count and the line's branch edges (lines below 1 have no leaf). + */ + void check(KeployBranchCoverage.Method a) { + Map lines = new HashMap<>(); // line -> instructions, branches + for (int k = 0; k < a.lines.length; k++) { + if (a.lines[k] < 1) { + continue; + } + int[] s = lines.get(a.lines[k]); + if (s == null) { + s = new int[2]; + lines.put(a.lines[k], s); + } + s[0]++; + s[1] += a.branches[k].length; + } + Map mine = new HashMap<>(); + for (int i = 0; i < leaves.length; i += 3) { + int[] s = mine.get(leaves[i]); + if (s == null) { + s = new int[2]; + mine.put(leaves[i], s); + } + if (leaves[i + 2] == 0) { + s[0] += leaves[i + 1]; + } else { + s[1]++; + } + } + if (lines.size() != mine.size()) { + throw new IllegalStateException(id + ": " + lines.size() + " lines in the analysis, " + mine.size() + + " in the table"); + } + for (Map.Entry e : lines.entrySet()) { + int[] t = mine.get(e.getKey()); + if (t == null || !Arrays.equals(t, e.getValue())) { + throw new IllegalStateException(id + ": line " + e.getKey() + " differs from the table"); + } + } + } + + /** + * hits is the set of unit-relative leaf indices the analysis covers: a line's code leaf when one of its + * instructions is covered, and the leaf of each covered branch edge. A line's edges are its leaves after + * its code leaf, in the order of the instructions and of each instruction's branches. + */ + BitSet hits(KeployBranchCoverage.Method a) { + BitSet out = new BitSet(); + Map edges = new HashMap<>(); // line -> edges met so far + for (int k = 0; k < a.lines.length; k++) { + int l = a.lines[k]; + if (l < 1) { + continue; + } + int leaf = codeLeaf(l); + if (leaf < 0) { + throw new IllegalStateException(id + ": line " + l + " has no code leaf"); + } + int code = base + leaf; + if (a.covered[k]) { + out.set(code); + } + Integer met = edges.get(l); + int e = met == null ? 0 : met; + for (boolean covered : a.branches[k]) { + if (covered) { + out.set(code + 1 + e); + } + e++; + } + edges.put(l, e); + } + return out; + } + } + + /** + * ProbeMap maps each JaCoCo probe of a class file to the leaves a hit of that probe covers, from JaCoCo's + * own analysis (KeployBranchCoverage): a line's code leaf when an instruction of it is covered, and each + * covered branch edge's leaf. A probe covers more than its own line: JaCoCo marks every instruction on the + * path that ends at the probe, so a test's leaves are the union of its probes' leaves. + * + *

A method's probes never reach another method's instructions, so one analysis maps the k-th probe of + * every method at once: the map takes one analysis of the class file more than its largest method has + * probes, on top of the table's own analysis of it. + */ + static final class ProbeMap { + private ProbeMap() { + } + + /** + * record is the class file's probe-map record: {"class":vmName,"id":crc64,"unit":unitId,"probes":[...]}, + * with probes as {@link #probes} gives them. + */ + static byte[] record(String unitId, UnitFile f, String probes) { + StringBuilder sb = new StringBuilder(64 + probes.length()); + sb.append("{\"class\":"); + Json.string(sb, f.file.name); + sb.append(",\"id\":\"").append(String.format("%016x", f.classId)).append("\",\"unit\":"); + Json.string(sb, unitId); + sb.append(",\"probes\":").append(probes).append('}'); + return sb.toString().getBytes(StandardCharsets.UTF_8); + } + + /** + * probes is the class file's probe map as JSON: one entry per probe (the class's probe count), each the + * probe's leaves as half-open ranges of the unit's leaf indices, [start,end,start,end,...] ascending, no + * two ranges touching. It throws when the class's lines or branches differ from the unit's methods (the + * same name under two roots), when a method with lines is in one and not in the other, or when a round + * leaves a method out. + */ + static String probes(UnitFile f, Map byId) { + Map shape = new HashMap<>(); + int total = Math.max(0, KeployBranchCoverage.analyze(f.file.bytes, null, null, shape)); + Map table = new HashMap<>(); + int rounds = 0; + for (Map.Entry e : shape.entrySet()) { + Method m = byId.get("m:" + f.file.name + "#" + e.getKey()); + if (m == null) { + for (int l : e.getValue().lines) { + if (l >= 1) { + throw new IllegalStateException(f.file.name + "#" + e.getKey() + + " has lines but no method in the table"); + } + } + continue; // no line: no leaf for its probes to cover + } + m.check(e.getValue()); + table.put(e.getKey(), m); + rounds = Math.max(rounds, e.getValue().probes.length); + } + // And the other way: each of the table's methods of this class file has its probes mapped here. + String prefix = "m:" + f.file.name + "#"; + int inTable = 0; + for (String id : byId.keySet()) { + if (id.startsWith(prefix)) { + inTable++; + } + } + if (inTable != table.size()) { + throw new IllegalStateException(f.file.name + ": " + inTable + " methods in the table, " + + table.size() + " in the probe analysis"); + } + BitSet[] hits = new BitSet[total]; + for (int r = 0; r < rounds; r++) { + boolean[] probes = new boolean[total]; + Set only = new HashSet<>(); + for (Map.Entry e : table.entrySet()) { + int[] ids = shape.get(e.getKey()).probes; + if (ids.length > r) { + probes[ids[r]] = true; + only.add(e.getKey()); + } + } + Map run = new HashMap<>(); + // The first round analyzes every method, as the table's analysis does; see KeployBranchCoverage. + KeployBranchCoverage.analyze(f.file.bytes, probes, r == 0 ? null : only, run); + for (String key : only) { + KeployBranchCoverage.Method got = run.get(key); + if (got == null) { + throw new IllegalStateException(f.file.name + "#" + key + " has no coverage in round " + r); + } + Method m = table.get(key); + m.check(got); + hits[shape.get(key).probes[r]] = m.hits(got); + } + } + StringBuilder sb = new StringBuilder(2 + 8 * total); + sb.append('['); + for (int p = 0; p < total; p++) { + sb.append(p > 0 ? ",[" : "["); + BitSet b = hits[p]; + int at = b == null ? -1 : b.nextSetBit(0); + boolean first = true; + while (at >= 0) { + int end = b.nextClearBit(at); + sb.append(first ? "" : ",").append(at).append(',').append(end); + first = false; + at = b.nextSetBit(end); + } + sb.append(']'); + } + return sb.append(']').toString(); + } } /** Display names of methods (API-CONTRACT 2.3). */ @@ -875,6 +1173,7 @@ interface FrameSender { */ static final class FrameWriter implements RecordSink { private final FrameSender sender; + private final String part; private final int bodyCap; private final ByteArrayOutputStream body = new ByteArrayOutputStream(); private final MessageDigest digest; @@ -882,11 +1181,16 @@ static final class FrameWriter implements RecordSink { private String build; FrameWriter(FrameSender sender) { - this(sender, MAX_FRAME_BYTES); + this(sender, PART_TABLE, MAX_FRAME_BYTES); } FrameWriter(FrameSender sender, int maxFrame) { + this(sender, PART_TABLE, maxFrame); + } + + FrameWriter(FrameSender sender, String part, int maxFrame) { this.sender = sender; + this.part = part; this.bodyCap = maxFrame - HEADER_ROOM; if (bodyCap < 1) { throw new IllegalArgumentException("frame cap " + maxFrame + " leaves no room for a body"); @@ -929,7 +1233,7 @@ void finish(String build, int records) throws IOException { private void flush(boolean last, int records) throws IOException { StringBuilder h = new StringBuilder(160); - h.append("{\"v\":1,\"part\":\"table\",\"build\":"); + h.append("{\"v\":1,\"part\":\"").append(part).append("\",\"build\":"); Json.string(h, build); h.append(",\"seq\":").append(seq).append(",\"last\":").append(last); if (last) { @@ -951,7 +1255,8 @@ private void flush(boolean last, int records) throws IOException { try { sender.send(frame); } catch (IOException e) { - throw new IOException("TABLE frame " + seq + ": " + e.getMessage(), e); + throw new IOException("TABLE frame " + (PART_TABLE.equals(part) ? "" : part + " ") + seq + ": " + + e.getMessage(), e); } body.reset(); seq++; diff --git a/keploy-sdk/src/main/java/org/jacoco/core/internal/analysis/KeployBranchCoverage.java b/keploy-sdk/src/main/java/org/jacoco/core/internal/analysis/KeployBranchCoverage.java new file mode 100644 index 00000000..1d23d9c9 --- /dev/null +++ b/keploy-sdk/src/main/java/org/jacoco/core/internal/analysis/KeployBranchCoverage.java @@ -0,0 +1,335 @@ +package org.jacoco.core.internal.analysis; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.TreeSet; + +import org.jacoco.core.analysis.ICounter; +import org.jacoco.core.analysis.ILine; +import org.jacoco.core.internal.analysis.filter.Filters; +import org.jacoco.core.internal.analysis.filter.IFilter; +import org.jacoco.core.internal.analysis.filter.IFilterOutput; +import org.jacoco.core.internal.analysis.filter.Replacements; +import org.jacoco.core.internal.flow.ClassProbesAdapter; +import org.jacoco.core.internal.flow.MethodProbesVisitor; +import org.jacoco.core.internal.instr.InstrSupport; +import org.objectweb.asm.ClassReader; +import org.objectweb.asm.Label; +import org.objectweb.asm.MethodVisitor; +import org.objectweb.asm.Opcodes; +import org.objectweb.asm.tree.AbstractInsnNode; +import org.objectweb.asm.tree.MethodNode; + +/** + * KeployBranchCoverage runs JaCoCo's own analysis of one class (ClassAnalyzer's steps for each method: the + * instruction graph, the filters, the merges and the branch replacements) and keeps what JaCoCo's public + * coverage API sums away: which counted instructions are covered and which branch of each one is. Keploy's + * build table uses it to map each probe to the lines and branch edges that a hit of that probe covers. + * + *

It lives in JaCoCo's package so that it runs JaCoCo's package-private analysis classes as they are: the + * result is JaCoCo's own, not a copy of its rules. Each method's per-line sums are checked against the + * coverage JaCoCo computes for it in the same pass, so a mismatch throws instead of mapping a probe wrong. + */ +public final class KeployBranchCoverage { + + private KeployBranchCoverage() { + } + + /** One method JaCoCo counts (it has code), with its counted instructions in code order. */ + public static final class Method { + public final String name; + public final String desc; + /** The method's probe ids, in the order JaCoCo met them. */ + public int[] probes = new int[0]; + private int probeCount; + /** Per counted instruction: its line (-1 when it has none). */ + public int[] lines; + /** Per counted instruction: whether it is covered. */ + public boolean[] covered; + /** Per counted instruction: whether each of its branches is covered (empty below two branches). */ + public boolean[][] branches; + + Method(String name, String desc) { + this.name = name; + this.desc = desc; + } + + void addProbe(int id) { + if (probeCount == probes.length) { + int[] grown = new int[Math.max(4, probeCount * 2)]; + System.arraycopy(probes, 0, grown, 0, probeCount); + probes = grown; + } + probes[probeCount++] = id; + } + } + + /** + * analyze runs JaCoCo's analysis over the class bytes with the given probes (null: none executed) and puts + * each method that has code into out, keyed by name + descriptor. With only non-null, only the methods + * named there are analyzed (the others still take their probe ids, as JaCoCo's analyzer does). It returns + * the class's probe count, or -1 for a class JaCoCo's Analyzer does not analyze (module-info, + * package-info). It throws when the bytes are not a class JaCoCo reads, or when a method's per-line sums + * differ from JaCoCo's coverage of it. + */ + public static int analyze(byte[] bytes, boolean[] probes, Set only, Map out) { + ClassReader reader = InstrSupport.classReaderFor(bytes); + if ((reader.getAccess() & Opcodes.ACC_MODULE) != 0 || reader.getClassName().endsWith("/package-info")) { + return -1; + } + Visitor v = new Visitor(reader.getClassName(), probes, only, out); + reader.accept(new ClassProbesAdapter(v, false), 0); + return v.total; + } + + private static final class Visitor extends ClassAnalyzer { + private final boolean[] probes; + private final Set only; + private final Map out; + private final IFilter filter = Filters.all(); + int total; + + Visitor(String name, boolean[] probes, Set only, Map out) { + super(new ClassCoverageImpl(name, 0, false), probes, new StringPool()); + this.probes = probes; + this.only = only; + this.out = out; + } + + @Override + public void visitTotalProbeCount(int count) { + super.visitTotalProbeCount(count); + total = count; + } + + @Override + public MethodProbesVisitor visitMethod(int access, final String name, final String desc, + final String signature, String[] exceptions) { + // Leaving a method out keeps the others' results only while JaCoCo's filters carry nothing from one + // method to the next that changes them. One does: KotlinSerializableFilter keeps the line of a + // $Companion's ()V for serializer(). That method has one probe, so it is only ever mapped in + // the first round, which analyzes every method (only is null). Any other difference fails the + // class: the caller checks each method's lines and branches against the table in every round. + if (only != null && !only.contains(name + desc)) { + return null; + } + InstrSupport.assertNotInstrumented(name, getClassName()); + final Method m = new Method(name, desc); + final Builder builder = new Builder(probes, m); + return new MethodAnalyzer(builder) { + @Override + public void accept(MethodNode node, MethodVisitor visitor) { + super.accept(node, visitor); + collect(m, builder, node, signature); + } + }; + } + + private void collect(Method m, Builder builder, MethodNode node, String signature) { + Map instructions = builder.getInstructions(); + MethodCoverageCalculator calculator = new MethodCoverageCalculator(instructions); + Output output = new Output(calculator); + filter.filter(node, this, output); + MethodCoverageImpl coverage = new MethodCoverageImpl(m.name, m.desc, signature); + calculator.calculate(coverage); + if (!coverage.containsCode()) { + return; + } + int[] p = new int[m.probeCount]; + System.arraycopy(m.probes, 0, p, 0, m.probeCount); + m.probes = p; + List lines = new ArrayList<>(); + List covered = new ArrayList<>(); + List branches = new ArrayList<>(); + Map sums = new HashMap<>(); // line -> instructions, covered, branches, covered + for (AbstractInsnNode i = node.instructions.getFirst(); i != null; i = i.getNext()) { + if (output.ignored.contains(i)) { + continue; + } + final Instruction insn = instructions.get(i); + if (insn == null) { + continue; + } + boolean hit = insn.getInstructionCounter().getCoveredCount() > 0; + int n = insn.getBranchCounter().getTotalCount(); + // The instruction's branch numbers: those its builder gave it (a switch's may skip numbers), + // or 0..k-1 for the k new branches a filter replaced them with. + Integer replaced = output.replaced.get(i); + Set numbers = replaced != null ? range(replaced) : builder.branches.get(i); + if (n > 0 && (numbers == null || numbers.size() != n)) { + throw new IllegalStateException("JaCoCo counts " + n + " branches at " + getClassName() + "#" + + m.name + m.desc + " line " + insn.getLine() + ", its builder gave " + numbers); + } + boolean[] b = new boolean[n]; + int k = 0; + for (int number : n > 0 ? numbers : new TreeSet()) { + // replaceBranches with one new branch fed by this branch alone is covered exactly when the + // branch is: JaCoCo's own way to read one branch. + Replacements one = new Replacements(); + one.add(i, i, number); + b[k++] = insn.replaceBranches(one, x -> insn).getInstructionCounter().getCoveredCount() > 0; + } + lines.add(insn.getLine()); + covered.add(hit); + branches.add(b); + int[] s = sums.get(insn.getLine()); + if (s == null) { + s = new int[4]; + sums.put(insn.getLine(), s); + } + s[0]++; + s[1] += hit ? 1 : 0; + s[2] += n; + for (boolean x : b) { + s[3] += x ? 1 : 0; + } + } + for (Map.Entry e : sums.entrySet()) { + int l = e.getKey(); + if (l < 1) { + continue; + } + ILine line = coverage.getLine(l); + int[] s = e.getValue(); + if (!same(line.getInstructionCounter(), s[0], s[1]) || !same(line.getBranchCounter(), s[2], s[3])) { + throw new IllegalStateException("JaCoCo's coverage of " + getClassName() + "#" + m.name + m.desc + + " line " + l + " differs from its instructions: " + Arrays.toString(s) + " against " + + line.getInstructionCounter() + " " + line.getBranchCounter()); + } + } + m.lines = new int[lines.size()]; + m.covered = new boolean[lines.size()]; + m.branches = new boolean[lines.size()][]; + for (int k = 0; k < lines.size(); k++) { + m.lines[k] = lines.get(k); + m.covered[k] = covered.get(k); + m.branches[k] = branches.get(k); + } + out.put(m.name + m.desc, m); + } + + private static boolean same(ICounter c, int total, int covered) { + return c.getTotalCount() == total && c.getCoveredCount() == covered; + } + + private static Set range(int n) { + Set out = new TreeSet<>(); + for (int k = 0; k < n; k++) { + out.add(k); + } + return out; + } + } + + /** + * Builder is JaCoCo's InstructionsBuilder that also notes the method's probe ids and each instruction's + * branch numbers, as JaCoCo's builder gives them: 0 to the instruction an instruction falls through to, + * and the number MethodAnalyzer passes with each jump and probe. + */ + private static final class Builder extends InstructionsBuilder { + private final Method method; + final Map> branches = new HashMap<>(); + private AbstractInsnNode current; + + Builder(boolean[] probes, Method method) { + super(probes); + this.method = method; + } + + private void note(int branch) { + if (current != null) { + Set s = branches.get(current); + if (s == null) { + s = new TreeSet<>(); + branches.put(current, s); + } + s.add(branch); + } + } + + @Override + void addInstruction(AbstractInsnNode node) { + note(0); + super.addInstruction(node); + current = node; + } + + @Override + void noSuccessor() { + super.noSuccessor(); + current = null; + } + + @Override + void addJump(Label target, int branch) { + note(branch); + super.addJump(target, branch); + } + + @Override + void addProbe(int probeId, int branch) { + method.addProbe(probeId); + note(branch); + super.addProbe(probeId, branch); + } + } + + /** + * Output passes the filters' decisions on to JaCoCo's calculator and keeps the instructions it leaves out: + * the ignored ones and those merged into another (the calculator's own rule, findRepresentative). + */ + private static final class Output implements IFilterOutput { + private final MethodCoverageCalculator calculator; + final Set ignored = new HashSet<>(); + private final Map merged = new HashMap<>(); + // Instructions a filter replaced the branches of, with how many new branches each has. + final Map replaced = new HashMap<>(); + + Output(MethodCoverageCalculator calculator) { + this.calculator = calculator; + } + + @Override + public void ignore(AbstractInsnNode fromInclusive, AbstractInsnNode toInclusive) { + calculator.ignore(fromInclusive, toInclusive); + for (AbstractInsnNode i = fromInclusive; i != toInclusive; i = i.getNext()) { + ignored.add(i); + } + ignored.add(toInclusive); + } + + @Override + public void merge(AbstractInsnNode i1, AbstractInsnNode i2) { + calculator.merge(i1, i2); + i1 = representative(i1); + i2 = representative(i2); + if (i1 != i2) { + merged.put(i2, i1); + ignored.add(i2); + } + } + + private AbstractInsnNode representative(AbstractInsnNode i) { + AbstractInsnNode r; + while ((r = merged.get(i)) != null) { + i = r; + } + return i; + } + + @Override + public void replaceBranches(AbstractInsnNode source, Replacements replacements) { + calculator.replaceBranches(source, replacements); + int n = 0; + for (Object ignoredValue : replacements.values()) { + n++; + } + replaced.put(source, n); + } + } +} diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java index 317bbac1..580bcd1a 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java @@ -372,15 +372,17 @@ static String header(byte[] frame) { } @Test - public void noClassesIsOneEmptyLastFrame() throws Exception { + public void noClassesIsOneEmptyLastFrameOfEachPart() throws Exception { BuildTable.Summary s = BuildTable.write(new ArrayList(), new Collect()); assertEquals("", s.build); assertEquals(0, s.records); Frames f = new Frames(); BuildTable.send(ArrayList::new, f); - assertEquals(1, f.frames.size()); + assertEquals(2, f.frames.size()); assertEquals("{\"v\":1,\"part\":\"table\",\"build\":\"\",\"seq\":0,\"last\":true,\"records\":0,\"sha256\":" + "\"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\"}", header(f.frames.get(0))); + assertEquals("{\"v\":1,\"part\":\"probes\",\"build\":\"\",\"seq\":0,\"last\":true,\"records\":0,\"sha256\":" + + "\"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\"}", header(f.frames.get(1))); } // ---------------------------------------------------------------- an edge set, against Go @@ -700,12 +702,18 @@ public void aRequestedTableArrivesWithTheBuildId() throws Exception { assertTrue(sends.request(() -> BuildTable.send(BuildTableTest::javaDedupUnchecked, new BuildTable.SocketSender(sock)))); String last = null; - while (last == null) { + String lastProbes = null; + while (last == null || lastProbes == null) { synchronized (r.got) { for (byte[] b : r.got) { String h = new String(b, StandardCharsets.UTF_8); + h = h.substring(0, h.indexOf('\n')); if (h.contains("\"last\":true")) { - last = h.substring(0, h.indexOf('\n')); + if (h.contains("\"part\":\"table\"")) { + last = h; + } else { + lastProbes = h; + } } } } @@ -714,6 +722,8 @@ public void aRequestedTableArrivesWithTheBuildId() throws Exception { String build = new String(resource("java-dedup.build"), StandardCharsets.UTF_8).trim(); assertTrue(last, last.startsWith("{\"v\":1,\"part\":\"table\",\"build\":\"" + build + "\",")); assertTrue(last, last.contains("\"records\":5,")); + assertTrue(lastProbes, lastProbes.startsWith("{\"v\":1,\"part\":\"probes\",\"build\":\"" + build + "\",")); + assertTrue(lastProbes, lastProbes.contains("\"records\":5,")); } } diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/ControlSocketTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/ControlSocketTest.java index e4a8fd0e..6b90ee69 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/ControlSocketTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/ControlSocketTest.java @@ -214,6 +214,16 @@ public void theBuildTableGoesOnlyWhenKeployAsksForIt() throws Exception { assertEquals("{\"v\":1,\"part\":\"table\",\"build\":\"" + build + "\",\"seq\":0,\"last\":true," + "\"records\":5,\"sha256\":\"" + sha256(golden) + "\"}\n", frame.substring(header, body)); assertEquals(new String(golden, StandardCharsets.UTF_8), frame.substring(body)); + // The probe map follows the table, as its own stream with the same build. + String probes = data.next(); + assertNotNull("the probe map followed the table", probes); + byte[] map = BuildTableTest.resource("java-dedup.probes.ndjson"); + header = probes.indexOf("{\"v\":1,"); + body = probes.indexOf('\n', header) + 1; + assertEquals("{\"v\":1,\"part\":\"probes\",\"build\":\"" + build + "\",\"seq\":0,\"last\":true," + + "\"records\":5,\"sha256\":\"" + sha256(map) + "\"}\n", probes.substring(header, body)); + assertEquals(new String(map, StandardCharsets.UTF_8), probes.substring(body)); + assertNull("nothing more", data.payloads.poll(1, TimeUnit.SECONDS)); } @Test(timeout = 60000) @@ -672,6 +682,10 @@ private void serve() { try (Socket socket = server.accept()) { socket.setSoTimeout(30000); String payload = readAll(socket.getInputStream()); + if (payload.startsWith("KTB1")) { + // A TABLE frame waits for OK before the next one goes. + socket.getOutputStream().write("OK\n".getBytes(StandardCharsets.UTF_8)); + } all.add(payload); (payload.startsWith(INDEX_START) ? indexes : payloads).add(payload); } catch (IOException e) { diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/ProbeMapTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/ProbeMapTest.java new file mode 100644 index 00000000..0907eba3 --- /dev/null +++ b/keploy-sdk/src/test/java/io/keploy/dedup/ProbeMapTest.java @@ -0,0 +1,472 @@ +package io.keploy.dedup; + +import com.google.gson.JsonArray; +import com.google.gson.JsonElement; +import com.google.gson.JsonObject; +import com.google.gson.JsonParser; +import org.jacoco.core.analysis.Analyzer; +import org.jacoco.core.analysis.IClassCoverage; +import org.jacoco.core.analysis.ILine; +import org.jacoco.core.analysis.IMethodCoverage; +import org.jacoco.core.data.ExecutionData; +import org.jacoco.core.data.ExecutionDataStore; +import org.jacoco.core.instr.Instrumenter; +import org.jacoco.core.internal.data.CRC64; +import org.jacoco.core.runtime.IExecutionDataAccessorGenerator; +import org.junit.Test; +import org.objectweb.asm.ClassWriter; +import org.objectweb.asm.Label; +import org.objectweb.asm.MethodVisitor; +import org.objectweb.asm.Opcodes; + +import java.io.ByteArrayOutputStream; +import java.io.File; +import java.io.IOException; +import java.net.URL; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.BitSet; +import java.util.Collections; +import java.util.HashMap; +import java.util.List; +import java.util.Map; +import java.util.TreeMap; +import java.util.stream.Collectors; +import java.util.stream.Stream; + +import static org.junit.Assert.assertArrayEquals; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertTrue; + +/** + * Rows for the probe map: for every class file of the java-dedup sample and of a class of code shapes (if and + * &&, loops, an int, a String and an enum switch, try/finally, try-with-resources, synchronized, a lambda, + * an assert), the leaves the map gives a set of probes are what JaCoCo's own public Analyzer counts for that + * set, line by line: a line's code leaf is hit when JaCoCo covers an instruction of it, and as many of its + * branch leaves as JaCoCo covers branches. The sets are no probe, each probe alone, every pair of probes and + * all of them, so the branch leaves of a line keep their identity across probes (a union of two probes counts + * two edges only when JaCoCo does). The Shapes source is shapes/Shapes.java.txt (javac --release 8 -g); the + * Modern source (records, sealed types, pattern and enum switches, a String switch with yield) is + * modern/Modern.java.txt (javac --release 21 -g). + */ +public class ProbeMapTest { + + private static final String RES = "/io/keploy/dedup/buildtable/"; + + static List load(String dir) throws Exception { + URL root = ProbeMapTest.class.getResource(RES + dir); + assertNotNull(dir, root); + Path base = Paths.get(root.toURI()); + List out = new ArrayList<>(); + try (Stream walk = Files.walk(base)) { + for (Path p : walk.filter(p -> p.toString().endsWith(".class.bin")).sorted().collect(Collectors.toList())) { + String rel = base.relativize(p).toString().replace(File.separatorChar, '/'); + out.add(new BuildTable.ClassFile(rel.substring(0, rel.length() - ".class.bin".length()), + Files.readAllBytes(p))); + } + } + return out; + } + + /** probes writes the table and returns the probe map's NDJSON stream. */ + static String probes(List files, BuildTableTest.Collect table) throws IOException { + BuildTable.Summary s = BuildTable.write(files, table); + ByteArrayOutputStream b = new ByteArrayOutputStream(); + for (byte[] r : s.probes) { + b.write(r, 0, r.length); + b.write('\n'); + } + return new String(b.toByteArray(), StandardCharsets.UTF_8); + } + + @Test + public void javaDedupProbeMapIsTheGolden() throws Exception { + List files = BuildTableTest.javaDedup(); + String got = probes(files, new BuildTableTest.Collect()); + String dir = System.getProperty("buildtable.dump"); + if (dir != null) { + Files.write(Paths.get(dir, "java-dedup.probes.ndjson"), got.getBytes(StandardCharsets.UTF_8)); + } + assertEquals(new String(BuildTableTest.resource("java-dedup.probes.ndjson"), StandardCharsets.UTF_8), got); + // One record per class file, each with its class id and as many entries as JaCoCo has probes in it. + String[] lines = got.split("\n"); + assertEquals(files.size(), lines.length); + for (int i = 0; i < lines.length; i++) { + JsonObject r = JsonParser.parseString(lines[i]).getAsJsonObject(); + assertEquals(files.get(i).name, r.get("class").getAsString()); + assertEquals(String.format("%016x", CRC64.classId(files.get(i).bytes)), r.get("id").getAsString()); + assertEquals(files.get(i).name, instrumentedProbes(files.get(i)), r.getAsJsonArray("probes").size()); + } + } + + @Test + public void javaDedupProbesCoverWhatJacocoCounts() throws Exception { + agree(BuildTableTest.javaDedup()); + } + + @Test + public void everyCodeShapeCoversWhatJacocoCounts() throws Exception { + List files = load("shapes"); + assertEquals(3, files.size()); + int edges = agree(files); + assertTrue("the shapes have branch edges to tell apart: " + edges, edges >= 40); + } + + @Test + public void java21CodeShapesCoverWhatJacocoCounts() throws Exception { + // Filters Java 8 code never meets: records, pattern switches (typeSwitch, MatchException), record + // patterns, exhaustive switches. ASM reads the class version on every JDK the SDK runs on. + List files = load("modern"); + assertEquals(7, files.size()); + int edges = agree(files); + assertTrue("the shapes have branch edges to tell apart: " + edges, edges >= 40); + } + + /** + * instrumentedProbes is the length of the probe array JaCoCo's instrumenter gives the class: the array the + * agent fills, read without the analysis the map is built from. + */ + static int instrumentedProbes(BuildTable.ClassFile f) throws IOException { + int[] count = {0}; + new Instrumenter(new IExecutionDataAccessorGenerator() { + @Override + public int generateDataAccessor(long classid, String classname, int probecount, MethodVisitor mv) { + count[0] = probecount; + mv.visitInsn(Opcodes.ACONST_NULL); + mv.visitTypeInsn(Opcodes.CHECKCAST, "[Z"); + return 1; + } + }).instrument(f.bytes, f.name); + return count[0]; + } + + /** + * agree checks every class file's map against JaCoCo for no probe, each probe, each pair and all probes, + * and returns how many branch leaves the classes have. + */ + static int agree(List files) throws Exception { + BuildTableTest.Collect table = new BuildTableTest.Collect(); + String[] maps = probes(files, table).split("\n"); + Map units = new HashMap<>(); + for (String r : table.records) { + JsonObject o = JsonParser.parseString(r).getAsJsonObject(); + assertEquals(r, "ok", o.get("state").getAsString()); + units.put(o.get("id").getAsString(), o); + } + Map byName = new HashMap<>(); + for (BuildTable.ClassFile f : files) { + byName.put(f.name, f); + } + assertEquals(files.size(), maps.length); + int edges = 0; + for (String line : maps) { + JsonObject m = JsonParser.parseString(line).getAsJsonObject(); + BuildTable.ClassFile f = byName.get(m.get("class").getAsString()); + JsonObject unit = units.get(m.get("unit").getAsString()); + assertNotNull(line, f); + assertNotNull(line, unit); + List probes = new ArrayList<>(); + for (JsonElement p : m.getAsJsonArray("probes")) { + JsonArray a = p.getAsJsonArray(); + int[] r = new int[a.size()]; + for (int i = 0; i < r.length; i++) { + r[i] = a.get(i).getAsInt(); + } + assertEquals("ranges come in pairs", 0, r.length % 2); + for (int i = 0; i + 1 < r.length; i++) { + assertTrue(f.name + ": ranges ascend and never touch " + Arrays.toString(r), r[i] < r[i + 1]); + } + probes.add(r); + } + int n = probes.size(); + assertEquals(f.name, instrumentedProbes(f), n); + check(f, unit, probes, new boolean[n]); + boolean[] all = new boolean[n]; + Arrays.fill(all, true); + check(f, unit, probes, all); + for (int p = 0; p < n; p++) { + for (int q = p; q < n; q++) { + boolean[] set = new boolean[n]; + set[p] = true; + set[q] = true; + check(f, unit, probes, set); + } + } + for (List v : jacoco(f, new boolean[n]).values()) { + edges += v.get(2); + } + } + return edges; + } + + static void check(BuildTable.ClassFile f, JsonObject unit, List probes, boolean[] set) throws IOException { + Map> want = jacoco(f, set); + Map> got = mapped(f.name, unit, probes, set); + List on = new ArrayList<>(); + for (int i = 0; i < set.length; i++) { + if (set[i]) { + on.add(i); + } + } + assertEquals(f.name + " probes " + on, want, got); + } + + /** jacoco is what JaCoCo's Analyzer counts with these probes: per method and line, code hit, branches hit, branches. */ + static Map> jacoco(BuildTable.ClassFile f, boolean[] probes) throws IOException { + ExecutionDataStore store = new ExecutionDataStore(); + store.put(new ExecutionData(CRC64.classId(f.bytes), f.name, probes)); + List got = new ArrayList<>(); + new Analyzer(store, got::add).analyzeClass(f.bytes, f.name); + Map> out = new TreeMap<>(); + for (IClassCoverage cc : got) { + for (IMethodCoverage mc : cc.getMethods()) { + for (int l = Math.max(1, mc.getFirstLine()); l <= mc.getLastLine(); l++) { + ILine line = mc.getLine(l); + if (line.getInstructionCounter().getTotalCount() == 0) { + continue; + } + out.put("m:" + f.name + "#" + mc.getName() + mc.getDesc() + "@" + l, Arrays.asList( + line.getInstructionCounter().getCoveredCount() > 0 ? 1 : 0, + line.getBranchCounter().getCoveredCount(), line.getBranchCounter().getTotalCount())); + } + } + } + return out; + } + + /** mapped is the same count from the map: the union of the set's leaves, read back through the table. */ + static Map> mapped(String name, JsonObject unit, List probes, boolean[] set) { + List keys = new ArrayList<>(); + List kinds = new ArrayList<>(); + Map> out = new TreeMap<>(); + for (JsonElement me : unit.getAsJsonArray("methods")) { + JsonObject m = me.getAsJsonObject(); + String id = m.get("id").getAsString(); + for (JsonElement le : m.getAsJsonArray("leaves")) { + JsonArray leaf = le.getAsJsonArray(); + String key = id + "@" + leaf.get(0).getAsInt(); + keys.add(key); + kinds.add(leaf.get(5).getAsInt()); + if (id.startsWith("m:" + name + "#")) { + List v = out.get(key); + if (v == null) { + v = new ArrayList<>(Arrays.asList(0, 0, 0)); + out.put(key, v); + } + if (leaf.get(5).getAsInt() == 1) { + v.set(2, v.get(2) + 1); + } + } + } + } + BitSet hit = new BitSet(); + for (int p = 0; p < set.length; p++) { + if (set[p]) { + int[] r = probes.get(p); + for (int i = 0; i < r.length; i += 2) { + hit.set(r[i], r[i + 1]); + } + } + } + assertTrue(name + ": a leaf past the unit's", hit.length() <= keys.size()); + for (int i = hit.nextSetBit(0); i >= 0; i = hit.nextSetBit(i + 1)) { + List v = out.get(keys.get(i)); + assertNotNull(name + ": a probe of it covers another class's leaf " + keys.get(i), v); + if (kinds.get(i) == 0) { + v.set(0, 1); + } else { + v.set(1, v.get(1) + 1); + } + } + return out; + } + + @Test + public void aUnitThatIsNotMeasuredHasNoProbeRecord() throws Exception { + List files = new ArrayList<>(BuildTableTest.javaDedup()); + files.add(new BuildTable.ClassFile("io/keploy/samples/javadedup/model/Item$Later", + "not a class".getBytes(StandardCharsets.UTF_8))); + String got = probes(files, new BuildTableTest.Collect()); + assertFalse(got, got.contains("\"unit\":\"c:io/keploy/samples/javadedup/model/Item\"")); + assertEquals(4, got.split("\n").length); + } + + /** copy is a class d/C with one method per line given (m0 at the first, ...), and an int field when field. */ + static byte[] copy(boolean field, int... lines) { + int[][] methods = new int[lines.length][]; + for (int i = 0; i < lines.length; i++) { + methods[i] = new int[] {lines[i]}; + } + return cls(field, false, methods); + } + + /** + * cls is a class d/C with a method mi per entry of methods, one instruction on each of its lines in order + * (NOPs, then a RETURN), an int field when field, and the methods written last to first when reversed. + */ + static byte[] cls(boolean field, boolean reversed, int[]... methods) { + ClassWriter cw = new ClassWriter(ClassWriter.COMPUTE_MAXS); + cw.visit(Opcodes.V1_8, Opcodes.ACC_PUBLIC | Opcodes.ACC_SUPER, "d/C", null, "java/lang/Object", null); + cw.visitSource("C.java", null); + if (field) { + cw.visitField(Opcodes.ACC_STATIC, "x", "I", null, null).visitEnd(); + } + for (int j = 0; j < methods.length; j++) { + int i = reversed ? methods.length - 1 - j : j; + MethodVisitor mv = cw.visitMethod(Opcodes.ACC_STATIC, "m" + i, "()V", null, null); + mv.visitCode(); + for (int k = 0; k < methods[i].length; k++) { + Label l = new Label(); + mv.visitLabel(l); + mv.visitLineNumber(methods[i][k], l); + mv.visitInsn(k + 1 < methods[i].length ? Opcodes.NOP : Opcodes.RETURN); + } + mv.visitMaxs(0, 0); + mv.visitEnd(); + } + cw.visitEnd(); + return cw.toByteArray(); + } + + @Test + public void twoCopiesOfAClassThatDifferAreNotMeasured() throws Exception { + // The same class under two roots, as the index can find it. The table's methods come from one copy, + // so the other's probes cannot be mapped onto them: the unit fails closed in the table and the map. + byte[][][] cases = { + {copy(false, 10), copy(false, 20)}, // a line moved + {copy(false, 10, 11), copy(false, 10)}, // a method the other copy lacks + {copy(false, 10), copy(false, 10, 11)}, + // The same methods and lines, but their probes in another order: a hit of probe 0 of one copy is + // not a hit of probe 0 of the other, and a test's hits name only the class. + {cls(false, false, new int[] {10}, new int[] {12}), cls(false, true, new int[] {10}, new int[] {12})}, + }; + for (byte[][] c : cases) { + List files = new ArrayList<>(); + files.add(new BuildTable.ClassFile("d/C", c[0])); + files.add(new BuildTable.ClassFile("d/C", c[1])); + BuildTableTest.Collect table = new BuildTableTest.Collect(); + BuildTable.Summary s = BuildTable.write(files, table); + assertEquals(1, table.records.size()); + String rec = table.records.get(0); + assertTrue(rec, rec.contains("\"id\":\"c:d/C\"")); + assertTrue(rec, rec.contains("\"state\":\"not_measured\",\"reason\":{\"code\":\"analyzer_failed\"}")); + assertEquals(1, s.notMeasured); + assertEquals("counted as a probe map failure", 1, s.notMapped); + assertTrue("no probe record for a unit that is not measured", s.probes.isEmpty()); + } + } + + @Test + public void twoCopiesOfAClassThatMapAlikeShareOneRecord() throws Exception { + // Two class files of one name whose bytes differ but whose probes map alike: a test's hits name the + // class only, and a hit of either copy covers the same leaves, so the map has one record per name + // (the Go decoder refuses a name listed twice). Its id is the first copy the index listed. + byte[] a = copy(false, 10, 12); + byte[] b = copy(true, 10, 12); + assertNotEquals(CRC64.classId(a), CRC64.classId(b)); + for (byte[][] c : new byte[][][] {{a, b}, {b, a}}) { + List files = new ArrayList<>(); + files.add(new BuildTable.ClassFile("d/C", c[0])); + files.add(new BuildTable.ClassFile("d/C", c[1])); + BuildTableTest.Collect table = new BuildTableTest.Collect(); + String[] got = probes(files, table).split("\n"); + assertTrue(table.records.get(0), table.records.get(0).contains("\"state\":\"ok\"")); + assertEquals(1, got.length); + JsonObject r = JsonParser.parseString(got[0]).getAsJsonObject(); + assertEquals("d/C", r.get("class").getAsString()); + assertEquals(String.format("%016x", CRC64.classId(c[0])), r.get("id").getAsString()); + assertEquals("[[0,1],[1,2]]", r.getAsJsonArray("probes").toString()); + } + } + + @Test + public void theSameClassFileTwiceIsOneClassFile() throws Exception { + // The index can find one class file under two paths: the table, its build id and the map are those of + // one copy. + byte[] a = copy(false, 10, 12); + BuildTableTest.Collect once = new BuildTableTest.Collect(); + String onceMap = probes(Collections.singletonList(new BuildTable.ClassFile("d/C", a)), once); + BuildTableTest.Collect twice = new BuildTableTest.Collect(); + String twiceMap = probes(Arrays.asList(new BuildTable.ClassFile("d/C", a), + new BuildTable.ClassFile("x/d/C", a)), twice); + assertEquals(once.build, twice.build); + assertEquals(once.records, twice.records); + assertEquals(onceMap, twiceMap); + assertEquals(1, twiceMap.split("\n").length); + } + + @Test + public void aLineBelowOneHasNoLeafAndItsMethodStays() throws Exception { + // javac never writes line 0, but bytecode tools do. The table leaves such a line out and keeps the + // method's other lines: the map must agree, not fail the class as a method missing from the table. + int[][][] cases = { + {{0, 5}, {7}}, + {{5, 0}, {7}}, + {{0}, {7}}, // a method with no other line has no leaf: left out of both + }; + for (int[][] c : cases) { + List files = Collections.singletonList(new BuildTable.ClassFile("d/C", + cls(false, false, c))); + BuildTableTest.Collect table = new BuildTableTest.Collect(); + BuildTable.Summary s = BuildTable.write(files, table); + assertEquals(0, s.notMeasured); + assertEquals(0, s.notMapped); + JsonObject unit = JsonParser.parseString(table.records.get(0)).getAsJsonObject(); + JsonArray methods = unit.getAsJsonArray("methods"); + assertEquals(Arrays.deepToString(c), c[0].length == 1 ? 1 : 2, methods.size()); + if (c[0].length > 1) { + JsonObject m0 = methods.get(0).getAsJsonObject(); + assertEquals("m:d/C#m0()V", m0.get("id").getAsString()); + assertEquals(5, m0.get("line").getAsInt()); + assertEquals(1, m0.getAsJsonArray("leaves").size()); + } + agree(files); + } + } + + @Test + public void theProbeMapFollowsTheTableAsItsOwnStream() throws Exception { + BuildTableTest.Collect table = new BuildTableTest.Collect(); + String golden = probes(BuildTableTest.javaDedup(), table); + { + BuildTableTest.Frames sent = new BuildTableTest.Frames(); + List files = BuildTableTest.javaDedup(); + BuildTable.send(() -> files, frame -> { + assertTrue(frame.length <= BuildTable.MAX_FRAME_BYTES); + sent.send(frame); + }); + int tables = 0; + ByteArrayOutputStream stream = new ByteArrayOutputStream(); + int seq = 0; + for (byte[] f : sent.frames) { + String h = BuildTableTest.header(f); + JsonObject o = JsonParser.parseString(h).getAsJsonObject(); + if (o.get("part").getAsString().equals("table")) { + assertEquals("the table comes first", 0, stream.size()); + tables++; + continue; + } + assertEquals("probes", o.get("part").getAsString()); + assertEquals(table.build, o.get("build").getAsString()); + assertEquals(seq++, o.get("seq").getAsInt()); + stream.write(f, 8 + h.length() + 1, f.length - 8 - h.length() - 1); + if (o.get("last").getAsBoolean()) { + assertEquals(5, o.get("records").getAsInt()); + assertEquals(BuildTable.sha256Hex(golden.getBytes(StandardCharsets.UTF_8)), + o.get("sha256").getAsString()); + assertEquals("the last frame ends the answer", f, sent.frames.get(sent.frames.size() - 1)); + } + } + assertEquals(1, tables); + assertTrue(seq >= 1); + assertArrayEquals(golden.getBytes(StandardCharsets.UTF_8), stream.toByteArray()); + } + } +} diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup.probes.ndjson b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup.probes.ndjson new file mode 100644 index 00000000..c06bf3b5 --- /dev/null +++ b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup.probes.ndjson @@ -0,0 +1,5 @@ +{"class":"io/keploy/samples/javadedup/DedupController","id":"45a3f53a4c63794a","unit":"c:io/keploy/samples/javadedup/DedupController","probes":[[0,1],[1,2],[2,3],[3,4],[4,5],[5,6],[6,7],[7,8],[8,9],[9,10],[10,11],[11,12],[12,13],[13,14],[14,15],[15,16],[16,17],[17,18],[18,19],[19,20],[20,21],[21,22],[22,23],[23,24],[24,25],[25,26],[26,27],[27,28],[28,29],[29,30],[30,31],[31,32],[32,33],[33,34],[34,35],[35,36],[36,37],[37,38],[38,39],[39,40],[40,41],[41,42],[42,43],[43,44],[44,45],[45,46],[46,47],[47,48],[48,49],[49,50],[50,51],[51,52],[52,53],[53,54],[54,55],[55,56],[56,57],[57,58],[58,59],[59,60],[60,61],[61,62],[59,60],[62,63],[63,65,66,67],[63,64,65,66,67,68],[68,70],[69,71],[69,70,72,73],[69,70,71,72,73,74]]} +{"class":"io/keploy/samples/javadedup/JavaDedupApplication","id":"2fc08e98e5791eee","unit":"c:io/keploy/samples/javadedup/JavaDedupApplication","probes":[[0,1],[1,3]]} +{"class":"io/keploy/samples/javadedup/model/Item","id":"4b73722ef52560fa","unit":"c:io/keploy/samples/javadedup/model/Item","probes":[[0,2],[2,7],[7,8],[8,10],[10,11],[11,13],[13,14],[14,16]]} +{"class":"io/keploy/samples/javadedup/model/Product","id":"99d2d88ee67c142a","unit":"c:io/keploy/samples/javadedup/model/Product","probes":[[0,2],[2,8],[8,9],[9,11],[11,12],[12,14],[14,15],[15,17],[17,18],[18,20]]} +{"class":"io/keploy/samples/javadedup/model/User","id":"91a01d80e208b6dc","unit":"c:io/keploy/samples/javadedup/model/User","probes":[[0,2],[2,6],[6,7],[7,9],[9,10],[10,12]]} diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/Modern.java.txt b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/Modern.java.txt new file mode 100644 index 00000000..c2a60b66 --- /dev/null +++ b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/Modern.java.txt @@ -0,0 +1,101 @@ +package io.keploy.modern; + +import java.util.List; +import java.util.function.IntUnaryOperator; + +/** Java 17 and 21 code shapes for the probe map's rows (javac --release 21 -g). */ +public class Modern { + + sealed interface Shape permits Circle, Square, Rect { + } + + record Circle(int r) implements Shape { + } + + record Square(int side) implements Shape { + } + + record Rect(int w, int h) implements Shape { + Rect { + if (w < 0 || h < 0) { + throw new IllegalArgumentException("negative"); + } + } + } + + enum Color { RED, GREEN, BLUE } + + record Pair(Object left, Object right) { + } + + static int area(Shape s) { + return switch (s) { + case Circle c when c.r() > 10 -> 3 * c.r() * c.r() + 1; + case Circle c -> 3 * c.r() * c.r(); + case Square q -> q.side() * q.side(); + case Rect(int w, int h) when w == h -> w * w; + case Rect(int w, int h) -> w * h; + }; + } + + static int weight(Color c) { + return switch (c) { + case RED -> 1; + case GREEN -> 2; + case BLUE -> 3; + }; + } + + static int code(String s) { + return switch (s) { + case "a", "b" -> 1; + case "c" -> { + int n = s.length(); + yield n + 2; + } + default -> { + if (s.isEmpty()) { + yield 0; + } + yield -1; + } + }; + } + + static String describe(Object o) { + if (o instanceof Integer i && i > 0) { + return "positive " + i; + } + if (!(o instanceof String s)) { + return "other"; + } + return s.isBlank() ? "blank" : s; + } + + static int nested(Pair p) { + if (p instanceof Pair(Integer a, Integer b)) { + return a + b; + } + return switch (p.left()) { + case null -> -1; + case String s when s.length() > 3 -> s.length(); + case String s -> 0; + default -> 1; + }; + } + + static int guarded(List xs) { + int total = 0; + try { + for (String x : xs) { + total += code(x); + } + } catch (IllegalStateException e) { + total = -2; + } finally { + total++; + } + IntUnaryOperator twice = v -> v > 100 ? v : v * 2; + return twice.applyAsInt(total); + } +} diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern$Circle.class.bin b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern$Circle.class.bin new file mode 100644 index 0000000000000000000000000000000000000000..95ba1f4e8e270921c22d4486f1466d60db29dd69 GIT binary patch literal 1465 zcma)5T~8B16g^XDcPXod$~W?%fVPXWpx{>#qZmVzf<_1r`ZR4PbYZ(wXSXE$Cm$ry znDF2a@JAW%ew5Y##fRCub2I16Irq-}{pai#fY*4HMglz=dJQCzVi-8&$J}zbdtiMO z6{)HWsdd}6{SAhmTz_@ z*^;bqXP+xO*vH9Uzh;xH+3QL}z1kLjO;$f}#p}WsicmwjODYz2uE;Q)yUYpoM+Q>b zg^o=!q<3XgRm3|x5ZfPhb}2YVmERGbza>3i!y?1z6@{QOw_ee}8!3IySG*Ca_H^`T z7_>|ZD``B!V-2eY*6@U3rY%J^UEi(?E0Po=8T!gCGKSMmSGIdBkAxK++~RK45#D@B z%A;nZ(BUwCN*$RPxOdVJZ-rM;cEgvd6SC=W&+AOY9G@?ebl(2Y3qCKbQFL9QLP|kh zOSPsY89p2h1%`!o5xSVt@REkG`%&VLDu87g-UJxNX=Q@}rcp|UYm)M}>2HH#f@0eI z31&|IfObl|1g7YlA`}QSP2Z8|6lO3>J31e-G2Eg5B)t_1gEa1U2@=q*Pz=DDd4?On zH~PKEX+%e2&Ow^+xl7=Kz}NqRk77QCuhV6LDcxK=!|+x3$dZ0;Nyh^25kugnCBzE% zWYPQ%t(%NwOq0a@P;3tQFq(J}2nk7p>OZ6v_+Ca2WtU*#DP<(rg`zZfJ(O#lD@ literal 0 HcmV?d00001 diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern$Color.class.bin b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern$Color.class.bin new file mode 100644 index 0000000000000000000000000000000000000000..a7075f6292cc91489e66be04b70c7ab6b62289b5 GIT binary patch literal 1216 zcmah|?@!ZE6g_Wk*R2$p6B$epM3fDb@>9_*1jL~vV;H7O7C)8ogksm4t|RfAf0B+w zh(^O_|0v^qj~Pc4*QCAg-gnM<_r81kNBAPxrAgoY@CvQv1pG@z6f9E!Xz;t?k`IUJG*uQW(@RWFn0l zblM+4!E4s(*dyC*^6fVaOPLP#x@$KYxsE_N3>qF=M!iXAvYr7fj;cAXTHe_sYeVvC0d{@x2aAWX;uDnJbn95V5; zv+vnqGa#yx$yR6<@-?@8KXV;g;uiB=GF!MZpkMgM`iIwZy$AWK6lEsZLWNQ((2ghufUCJCP< zZ=9@nj23!7%!(?cQanQC(3;*kNtsU=Vgl(%QIZ9f zzCbGa4XmoPpp>g>3wpV#wWt9i-;4Fduz>kCd8199`AXd6m~w(!$55m*u1ZIj&TUOP l#wjL_(JL~)sHwJ7Tf`FWMeqO*rCr8jvf{5{SVnGj@E4>!`bz)+ literal 0 HcmV?d00001 diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern$Pair.class.bin b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern$Pair.class.bin new file mode 100644 index 0000000000000000000000000000000000000000..64f098b85828a3051a21030dbb4e392fa57a1024 GIT binary patch literal 1588 zcma)6+fvg|6kVsZOFkAk?01djRg?_-BmN?QIqQ)It-|)rsmSH*!af^TU7;@!v z*7uctUiC_nr(Z=@k|Z2+ukPh2o{5cjN=An&A+q^%Vl_Y2`u72%&qW|VK(2cZ`TxdsYXF`*|KT# z%~qqrogJfMQM|dbs2bL;;h6F}RK~r!NkQG}UKHx%2KVZswq-a*gL~W|%uv2Vt^BU8 zGYof#SLVUQ|G+RwhRn8TIaU77l)?4}eV><&rmAmp*IN^=r(lj@^nwObkp4a$CgU5KH;Hipdc+N0+K3q=A_RI#?gBU|9hIHkO zo#8l|sA(UH1Fi>#YldC3xI0}I;-J+mMYqZqJ z$ES9ZsP})y`ERAwb=&5SPsyo~ak*RyUNX#_r`{zM1+Qpgx|>ojg1oUnn;D};h|wz2 zj?*b8*OX+Eo&+Z7&rLdQ$cuwyR{H^_6;GgOb0_FKCRGf#=$!FgNNL4WT)zlE64-?9$r|^lW-{(x%!W+5|4lX2zC-DjMIxkOREkdEzN>%} x$orWVp!u?h`y~4*&Y-J7Pv?Cd-ExRi%sSP25idxWBfAueC{fBwSjKCN{Q+RMRowsp literal 0 HcmV?d00001 diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern$Rect.class.bin b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern$Rect.class.bin new file mode 100644 index 0000000000000000000000000000000000000000..443abf422c636979dc92f3398a0dfcae54fa381a GIT binary patch literal 1733 zcma)6TTc@~6#k~P-L@_iYQYQ0P0+R|^@g`1iYQVoXoQfcPeYl|h3!t=oua<^;J?tH z;DaU_5=`{nAElnXKr2ci&2)CpobNl|xy=6jwf!ByG)f7C(54}*BZ4SH=Q`iyd6Qdf z`KO{R?Nx^8q+uCqilHr&UDgm|7;ZQfO;fCKbJ||3SB0hSzb=cKGNctlf+1$nnKCv- z0vdETLOVlurqqBhD%-Hu3fbj2lIYaXr6Yx-41I>oZ-|;H-{h-uRoGU3-rGhwgP4jg za8Sn(hN)`eWu2Q2!{q~DP4OSEtW$c0>~jqx45#+fy;$Kj(i6!Pi`iv{urDuJGAyxB zudWDtiLaRCkSfVCHxbJ~6tSSbn>Tg{YhW16p%Nz5&7T`MPg{sKaC*0;$p@dBw z>CAtM=Z#5mPn&lWqee+rSYrZ$BZkDHtlMQVYq-*n`0XBd{iq2G!clY5Q5vo?9NVWZ zRP&vY?Yt|ca+J+$zG^3inv{8ZN{JkhmHP4#1%hlVlyd z@UuPNyz$J>@7FhgYn&+b2XR>iC_C<2v$UJ>DTULk&lePIUi8aMb^0-`c zRuTrOt9-^cxn;tz&A6&Df_~so-nvjK>=I5p>Il40O5*7B+`#Y|!vN1|iVW6b^lV!>hT literal 0 HcmV?d00001 diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern$Square.class.bin b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern$Square.class.bin new file mode 100644 index 0000000000000000000000000000000000000000..794bffb3e63d32fb756b1a64d697301876b19e1d GIT binary patch literal 1468 zcma)5T~pIQ6g^vL6H2tD@{N2bplwhB3Vs!F6vv@61s%)qpigPLLj!5HCMh%gCm&=+ z#~B{{0sbh*yGcsp0L6#oW;geoJ?GxNzyF;50`MBI(nz3BLBED1QViKc^Vl?O(>X9c z@|tk#45zGj=QNLgDivUi;C}B&H=+(zTBIlSSWWTg{Y$B%4xWTVd83-USo?6WthL7E08g& zVGQGBjFU*)V_3fy7}M~+ddO>jvG7U3B*XNT>KkTD`p=gN`xHWjnX={Z-FCCe-958v z)Bjvq)J%KdbS=4#Hv9dCMY86uD-E@4hx-jt|6saill$By)NuZiiUrLpF^uFdb3*+Q zQcAbbkx7PhMYP=-e`iUtgJE-*Wgb<2mwW!U@O%YJ3}aUmg38>vNZB_+_@3{Ytx&b6 zVlcy?WKvj7;}IS!SkthMCk(S)DQ?^GttK}@NfDA^pxPm0IE}lqoMUmsjWBWBbm}(u z7Ro{#wOhrQ!{{o-b|T>3NsGVbUd^>yzHsA^E!*_G_(sI>`65aD^nY&fd11AZ<8U{i zQd2tjJ85X-m=wV91OPa&psl*>v04p@V2{261lan({qm&GuDcZkHUz-#Y z6w~@oFn#I=lv7$IFir0ip+GY{L+{Zr3$vJ`73BwP40q@|Nq2?95T)KFh@@SmD8Y(; zh8w^)`n<@gg((r|5KZ{}CGa8O>wm$=un@sF=rCzY)tAmNauq(bq@G(+v50%bAl-C? z7{Qt>>ffRCl97yPlDHp;&7%-R9S@|CfK*ohAwAOf3i@bw83vxx&SbFC@eJeCD;>{M LDqdiNx-;x25&WaNDlITaBb85b82BrnO1HzqV=7pHgn0ep`r@a8fvgyK;&#( zsLYD<@b!;R=sEJ$VwHwE$rs3*LxzIY0=0{6wwvgys7FjggT2+Hibkwe(4?UmE%V^p zzUs+X#;`$Ml!JPD!q{~{qSoUD*`?@=fF6=!*K26A#ik+*fpP_x3RD)+TmVJICD@?h zeP|a5TUiG_feY4prMsZ2*oZ3>bZF?5qE{?tT6Z>`mC9ywLuN)#tB7Nhg3TJXNH$Ll z&hHjZ>G@PQoe;)0g04%BD>ZD#4#p?LO;0oMqFy&U(c9UyFN~e!7!2boHc82t(9kRQ z38jt9u$7WiyEI&bYpIc$-#$8KP1#P}C-LsrutySy&5X5IVjF=Udo^5-8w4&K)klo( zY$mB&*P9j<%;YUSV^NnzuP*c0p98Tl_6Y>DR?5gpJ#Nx)v#ZB8I(8(C4{&)IBO*l!^CU!=p6oLYS(5gkhDlr53@21f z;e!f3q~SJvn9wc~J2WlHI5@-GZv4rjaEHLEVi@Lu! zY4owN^wicoeI&i+y2!Tcz(gh~OGz8ouFL4Tsa<0Z05tR?xL3h_QpWuP7ZnX%l!tmV zA$!=!Fcw&nFf&Hq#OR=r+aqfa;YwtadU~&(Gv(Nw3|J|X;Z&Vi1ddJ$WNBe&tX{hq zO{_nqkFmIM^SVW8oYF>dV*Gq$P9>9MlqRUzszoI_y9$&M{NzZlK4zPw;IZ=*2AhYO zIcdnSJg-r94Y7?{Tm^@Qie@Cd#i($@g)u#3j58b#y_mbMna0u1M>ngF|}E1txYe0XWF~?$1u-lEw~GrdX+Sj<1)I zz)>H)(rd^RlmKkYX05!H)5m%ZE0rC}hw(kKk7e_wWgax@LVVMqkLO^z1mboCk=!X)?{- zTUkJs7!K`Ptl%$9{IkO1T;ZzVZ!BErC*`hAY@?oOg+dog9ES7;586P@|k=AOK%k7c~qWkIfbgebu(C2p+1FhAYPa|8Ir@4{R#{e@R}s) z2!yHwa@u(paE!CH9l`403|88&Q~{53uD&Bw9h$*K?W(`MG^!rQnrP`PT7hS|?;$?E zIsV)}8&pTtc15zRYuEhkWl>GCl+EH|*NV+9y~JZPpBg@wMYZ;DREdV4KuMsztT6W` z2MRT);X(=fCG3|6+Di*_cPR0~yZ<_xkQTSh;IbBJG(mn44~u$U^ZS{%=I-^G6UCWpt(NmEaWilm8L&(v{rfGt}X;e4fS9cjVXSqhd|T+lVNr zdJ9XUm2<4U8fWDKsTAr;Ms{O!^gX|9JS&_P`~eMfROXB|MxMPkM;T`<&9jxxQRj1R zHpkc#+>0`{CqhVlBt=CzD_e9~qD9i*XnzARH_p5IV|v z#~D@JTGH|359=91mtq|*Es-936b!~|T^TLpkw0mX zlp&uw+EGKRuimsG$lO`NZr?N%KP~eWEn7_;L3>iPC(TdIakTTx;0oN%z`Gk=xQA=f z$|}kpXDxV~!69?_FoWX@Y^A-+QSuIA3gV)yOs(~+s!S?6BrZcCj3ao$g`mJzs@T!| zG8(5*)qDz#2|}^^2-KEYTn8NEN_-lnT!|+N2t{r@#`XQv2uE(A;w{I>?RCjERHB=S zxE)Q{LC5U0m2D=_8sq$nc#=D=A%|2}hCv6ugs0fHa@CoW1}s6aru=RAsB$SIC{CrH z#+MzX#nluhy?=zN4NT+mW~uRDN1!@j8_`%DpcTEGJFqZUY0vew(lXYr-2+F7{h(VC z&H0kJ%~MU5Q~oMG8+`>dvvkE}9lSc^mccjVpn#=vp(8*CDs-*?>_Vb4=N@AGuVfm^ z)&Xh_(;7%{+{^zB?BYl2HT3qixC^@pRzHi$E>g=rvxJteApbT3=Oc$oK%bB*gYA^NAMLC*h&Z+P}2DhLJEAM-v2h162-R=Dw!iQBVnMJnqS4&D4IuxSjAQX zoA7rvpTMmGy{#v}8~8LXWM@LK7{&XRuRD%I16_r=e;yX1fM{>UyIpp#v?IW_I&g9p zck(!CU48eDruL16a20l;6q%F57ngi$t~oPkp_CRQs4N`&nIYBK-j4NGHe5 zOz>OCH^8*e$u&UebZo*Pu4JOlLsiRa^$a7+PijAF!w?dUyUMCtE>^?h?%WEoz zo`zDx9H|MWiuvrE)`PZ+&PL^Uo=P|y1>sZqUz`Ar+oDfkmTfuHdVtu}6Nf5R(HA(k zfpf2ru8M!PTzQ3?HuCM`yAeO)=tX-JXR`d1w!egve11dD4q{TpQ|7l=M$Rexj;*|L f{GRPbwtv{NjJr+Yj~qYB_D`WO{*1rk@2LJSN^m8A literal 0 HcmV?d00001 diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/shapes/Shapes.java.txt b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/shapes/Shapes.java.txt new file mode 100644 index 00000000..85057650 --- /dev/null +++ b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/shapes/Shapes.java.txt @@ -0,0 +1,113 @@ +package io.keploy.shapes; + +import java.io.Closeable; +import java.io.IOException; +import java.util.function.IntUnaryOperator; + +/** Code shapes for the probe map's rows: each one is a JaCoCo probe, branch or filter case. */ +public class Shapes { + enum Color { RED, GREEN, BLUE } + + static final class Res implements Closeable { + @Override + public void close() throws IOException { + } + } + + private final Object lock = new Object(); + private int total; + + int both(int a, int b) { + if (a > 0 && b > 0) { return 1; } + return a > b ? 2 : 3; + } + + int loop(int n) { + int s = 0; + for (int i = 0; i < n; i++) { + if (i % 3 == 0) { + continue; + } + s += i; + } + while (s > 100) { + s -= 7; + if (s == 50) { + break; + } + } + return s; + } + + String table(int k) { + switch (k) { + case 1: return "one"; + case 2: + case 3: return "few"; + case 7: total++; + case 8: return "many"; + default: return "other"; + } + } + + int strings(String s) { + switch (s) { + case "Aa": return 1; + case "BB": return 2; + case "c": return 3; + default: return 0; + } + } + + int colors(Color c) { + switch (c) { + case RED: return 1; + case BLUE: return 3; + default: return 0; + } + } + + int guarded(int x) { + try { + if (x < 0) { + throw new IllegalArgumentException("negative"); + } + return x * 2; + } finally { + if (x == 3) { total++; } + } + } + + int resources(int x) throws IOException { + try (Res r = new Res()) { + return x + 1; + } + } + + int locked(int x) { + synchronized (lock) { + total += x; + return total; + } + } + + int lambda(int x) { + IntUnaryOperator f = v -> v > 10 ? v - 10 : v + 10; + return f.applyAsInt(x); + } + + int checked(int x) { + assert x != 13 : "thirteen"; + int y = x; y++; if (y > 2) y--; return y; + } + + int nested(boolean a, boolean b, boolean c) { + if (a) { + if (b || c) { + return 1; + } + return 2; + } + return c ? 3 : 4; + } +} diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/shapes/io/keploy/shapes/Shapes$Color.class.bin b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/shapes/io/keploy/shapes/Shapes$Color.class.bin new file mode 100644 index 0000000000000000000000000000000000000000..c6a662110889179990cb351bc231dd1a59508f9a GIT binary patch literal 1197 zcmah|ZBNrs6n^g7u3ITGCo-5II8nBtl)>L_+C={gd>`IB@c zLNppa`=gBKUS=Fo+>-X(bMASb^U`NOet!85poo%+2*ZTq8}IBx*MD!c_N_y^WthTD zmwnd{;$V<5$6cy`F5qU$s+?gbRLnPR$o_m;EIkRBpB4` z+8TvShN(Awm^CDEO+`Y-Fs>7M-Ss`2VK|$s9#}_~;aXn9*xEg?>tRVl3Zp8Is-(;CPKvZv{>u_$PGS#3Vy>+3}okg<+(Z zs*Bvn)nZUIns_3Hcnmr}-qCRvGkkc_MvEbn&Gj3f3jNNO%l;lsK2&u)d$Zl#wS!mI zu1k@{e{_)RvsQ*=)vsG_%?cdO(xs9;t4WN>zn8&~cwvY8{@zO~pq$VSXaG6fcgVy| zr{P&)J0Pf(&DBT?%XPQApF$58-{Nwg&Q?kcn(4QLy1nl3@dnPNImf>ygTCQ;b|5~P z-BR(GA$5r~Ni0ImtMjz{6p3G69pjn?$dXcpC4B;kG?yi1C=}0PhVpadWytcM*1_Os zTs=ebBY6yYdPl`wobuC{MMUHm$#R62WcgE!oPqKD$c6lS6w%MKa36Ez8T2Rw`@mC4 zT95=;fviMVYJPV71mmCRco~z5c|L_;ibrT1I@41SkcEWCWk)6XNCsqj1F7gYupOxb z$=s1UP|Y2sLtE$lUYsw6MJ#l|>s|2NSHdQzq!Zjah9sPfEF4uhw-w=NrG33e{1s-kV{N1u#**N+^(}{LS0(alHW|wF zW)9T5Iuh|fq{`NdrkjnFA?$Sj8Ffl1Ms@(fzj%z~G*dd^1sa6L*8-j1H^QvGo~4nz zRCb^8eCH#}Bm|u#k!d>?#@Ii!2+cnxvk&}D9K&Ph2+M4v!-w7Ds?1r)k!R<^)tw<%(Tr)XBw%dCk4w!paXJErc~DyDYD>c#U%h>)7DvbI(H)n`nIk D?|ot+ literal 0 HcmV?d00001 diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/shapes/io/keploy/shapes/Shapes.class.bin b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/shapes/io/keploy/shapes/Shapes.class.bin new file mode 100644 index 0000000000000000000000000000000000000000..5ee86b300fe729219f614f3cfa6d181933596ed2 GIT binary patch literal 3908 zcma)9YjYFV8Gd%P(yoPM3sDg>*8R4l|x<{DC$JOv6l>X})FpPx`fA+6i@^v%6l&WT27f?Add9-}ia% z>rejs(cb_Z!P{}jh^SB!h$1E-bw$6b51G2PFm!(YicxeW#Ew<0iaRDD(${~f1sYmZ z#1q(pgoN!CduY+9n)dQgt*lp#+R(H&Nl;9?xF{h#`7py#4Xxy5TZV=dUoIQh1k-c^ z+wcVmQP*~LQ-YM&&_Om!y0xsKlLxk2Hk>%RkWrCMAcvis;7+?v#afV{mGxSA!Y&ye zU~=h}AfGxV2=^qg7d_O-c&)CRH3{u~pM}>yt6?8WPU;%^h@%+C01l`aOyHmpnR|rF zJrlNRI}%jeDOD`pR52uBPec2>X&MWO&(Q_(+2|Onuu2-w(^2u7>Vzlh<>-TM1 zcPr-5<+@cAN)GYtB};df&sPmccWq~M3(nwq6=wyHyo9|Tu*cId8v646c@Zt=5|~7R z`t;}o>If0F=PEU!4pB?9D$WypL)nuXw@y2b?P&NSP41Q}j%yfJ98;JUCcpTQ$^8VN zvTU=K)iO1d1PyZIcztn5r+$8_q?6 zRAh%;)Xht}QxWT6Qz0;t?wEXpk4MRHp2Z^0i(|Z(KTrB9-4Nj(QS9w(7 zusFyy>iUw~Y00M852A~An6qllOC{dklph-rNP*2+9g@b1A^}LXr|(hF{Tvw;n0Ni3 zJA|=N*PW8*&+8Ie7?O6~DH;?YG0k$1q-P4yMDZ}eYATjDBA+JBpAS!12}G3=B4Vb# zG+z>Syey&jb44%_7R!cDE>S|vGHR~Jt8aF8R_NH~GwlhcVRFC9z&lnngNhoT_Eb7k z5y7y8V8)`enVyyd%ynOUVD2Va-3Rh#iDRE8>+=Vo75AqD> z5260qqx>?TqA7!16>P;<_?GbR zi^wuu?ZzJd^|DbPKo6VYeh=>eWh&^zWnvTR4u{BxvEU(>dAA%QPa^sNiX=!M zgk)ccWFPt2mv@*WP&q=)a~@cm4*d^{6-7s-Z$vujfi z?Hs&MzF@TO<?A{;?2bbS^|>!(Od>%uB3 zH1-G;qYs!BoXFWFU)d`gw>b-c^xVS;E5_4>j4~(U;%IJdN=&QR@~m=jhjItUiBo1k zCx~X0rD!{|Z8X5^`}6`&r+I#nzMAH(3m#5U6*Stvj;o}nq!4jW6XHHZ)=9#D8Z<&3 z@$jw_trBbq+(HKx*T_SmUe`H)Ep)&efdg^_A3&Wc{M|EnMr0H>whn$wZ9euCPzk(~ zq@64bV)hO0hr!DfIK%2V%ad1{jGPN0=b9jI1MSS@vOfOR32zOdPVql># znE5MC%Mh6$H`UKqnc{Pyr9@fZD9EjWuo&(44SdtH-nT;TE_n;?y??rb6j=ff7^AXcF2R3hfPqb|Q&yhow)2-0`XS-wb#i z-=XzI%Xyu%2>Pzf4YXx%Le1R}ts=U{S!L~3)Q7Xa!uwYkA&b; Date: Fri, 20 Nov 2026 01:34:10 +0530 Subject: [PATCH 03/14] refactor(dedup): compute the INDEX and table build id in one place Keploy files the build table under the build id the INDEX record named, so the two ids must be one value over one class set. Until now they were two copies of the same algorithm: IndexReport.buildId for the INDEX record and BuildTable's own classIdsSum for the table. A change to one copy would have left Keploy looking for a table that never arrives. - IndexReport.classIdsSum is now the only implementation: the sha256, as 64 hex, over the class ids sorted as unsigned numbers, de-duplicated, each as 16 hex, joined by newlines. IndexReport.buildId is "jv1-" and its first 32 hex, or "" when there is no class. - BuildTable.write takes its class ids with IndexReport.classId and its build id with IndexReport.buildId, and a unit's hash is IndexReport.classIdsSum over that unit's ids. BuildTable's copy and its sha256Hex helper are removed (tests now take the sha256 digest themselves). - Both ids are taken over CoverageIndex.entries(), every class the index holds, whether a class reader can read it or not. Every entry has its bytes (the dir, jar and derived readers never store an entry without them), so the table's skip of an entry with no bytes never drops one the INDEX record counted. The table bytes are unchanged: the Go encoder's golden for the java-dedup sample still matches byte for byte. Rows: - ControlSocketTest.theIndexRecordAndTheTableNameOneBuild: the agent's index is the java-dedup sample plus one file no class reader reads; the INDEX record's build, and the build in every table and probe-map frame sent for TABLE, equal the id the test computes itself (sorted 16-hex ids, sha256), which differs from the sample's alone. - IndexReportTest.theUnitHashIsTheWholeSumAndTheBuildIdItsFirstHalf: the sum of the ids 1, 0xff and 0xffffffffffffffff, given out of order and with a repeat, is the full sha256 the Go side's build id is cut from. Signed-off-by: slayerjain --- .../main/java/io/keploy/dedup/BuildTable.java | 47 ++------------- .../java/io/keploy/dedup/IndexReport.java | 30 ++++++---- .../java/io/keploy/dedup/BuildTableTest.java | 19 +++++-- .../io/keploy/dedup/ControlSocketTest.java | 57 +++++++++++++++++++ .../java/io/keploy/dedup/IndexReportTest.java | 10 ++++ .../java/io/keploy/dedup/ProbeMapTest.java | 2 +- 6 files changed, 106 insertions(+), 59 deletions(-) diff --git a/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java b/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java index 6bab08ae..4076ac2d 100644 --- a/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java +++ b/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java @@ -8,7 +8,6 @@ import org.jacoco.core.analysis.ISourceNode; import org.jacoco.core.data.ExecutionDataStore; import org.jacoco.core.internal.analysis.KeployBranchCoverage; -import org.jacoco.core.internal.data.CRC64; import org.newsclub.net.unix.AFUNIXSocket; import org.newsclub.net.unix.AFUNIXSocketAddress; import org.objectweb.asm.ClassReader; @@ -309,8 +308,8 @@ static Summary write(List classes, RecordSink sink, int maxRecord) th if (in == null || in.bytes == null) { continue; } - // Every indexed class is in the build id, as in the index record's. - long id = CRC64.classId(in.bytes); + // Every indexed class is in the build id, named or not: the INDEX record's, by the same function. + long id = IndexReport.classId(in.bytes); ids[n++] = id; ClassFile f = new ClassFile(vmName(in), in.bytes); Unit u = f.name == null ? null : Unit.of(f.name); @@ -334,7 +333,7 @@ static Summary write(List classes, RecordSink sink, int maxRecord) th if (n == 0) { return new Summary("", 0, 0, 0, unnamed, 0, Collections.emptyList()); } - String build = "jv1-" + classIdsSum(Arrays.copyOf(ids, n)).substring(0, 32); + String build = IndexReport.buildId(Arrays.copyOf(ids, n)); List units = new ArrayList<>(byId.values()); Collections.sort(units, UNIT_ORDER); int notMeasured = 0; @@ -374,44 +373,6 @@ static String vmName(ClassFile f) { } } - /** classIdsSum is sha256, as hex, over the sorted, de-duplicated class ids, each %016x, joined by \n. */ - static String classIdsSum(long[] ids) { - long[] sorted = ids.clone(); - // Go sorts them as uint64. - Long[] boxed = new Long[sorted.length]; - for (int i = 0; i < sorted.length; i++) { - boxed[i] = sorted[i]; - } - Arrays.sort(boxed, Long::compareUnsigned); - StringBuilder sb = new StringBuilder(boxed.length * 17); - Long prev = null; - for (Long id : boxed) { - if (prev != null && prev.longValue() == id.longValue()) { - continue; - } - if (prev != null) { - sb.append('\n'); - } - String hex = Long.toHexString(id); - for (int i = hex.length(); i < 16; i++) { - sb.append('0'); - } - sb.append(hex); - prev = id; - } - return sha256Hex(sb.toString().getBytes(StandardCharsets.UTF_8)); - } - - static String sha256Hex(byte[] b) { - MessageDigest md; - try { - md = MessageDigest.getInstance("SHA-256"); - } catch (NoSuchAlgorithmException e) { - throw new IllegalStateException(e); - } - return hex(md.digest(b)); - } - static String hex(byte[] b) { StringBuilder sb = new StringBuilder(b.length * 2); for (byte x : b) { @@ -512,7 +473,7 @@ String hash() { for (int i = 0; i < ids.length; i++) { ids[i] = files.get(i).classId; } - return classIdsSum(ids); + return IndexReport.classIdsSum(ids); } byte[] notMeasured(String code) { diff --git a/keploy-sdk/src/main/java/io/keploy/dedup/IndexReport.java b/keploy-sdk/src/main/java/io/keploy/dedup/IndexReport.java index 84095e49..1f496134 100644 --- a/keploy-sdk/src/main/java/io/keploy/dedup/IndexReport.java +++ b/keploy-sdk/src/main/java/io/keploy/dedup/IndexReport.java @@ -144,15 +144,23 @@ static String hexId(long id) { /** * buildId is the build's id from its classes' JaCoCo ids: "jv1-" and the - * first 32 hex of the sha256 over the ids sorted as unsigned numbers, - * de-duplicated, each as 16 lowercase hex, joined by newlines. Keploy's - * codecoverage.JavaBuildID computes the same from the same ids. "" when - * there is no id. + * first 32 hex of classIdsSum, or "" when there is no id. Keploy's + * codecoverage.JavaBuildID computes the same from the same ids. It is the + * one build id: the INDEX record and the build table (BuildTable.write) + * both take it over the id of every class the index holds, read or not, + * so Keploy finds the table under the build the INDEX record named. */ static String buildId(long[] ids) { - if (ids.length == 0) { - return ""; - } + return ids.length == 0 ? "" : "jv1-" + classIdsSum(ids).substring(0, 32); + } + + /** + * classIdsSum is the sha256, as 64 lowercase hex, over the ids sorted as + * unsigned numbers, de-duplicated, each as 16 lowercase hex, joined by + * newlines. The build id is its first 32 hex; a build table unit's hash + * is all of it, over that unit's class ids. + */ + static String classIdsSum(long[] ids) { long[] sorted = Arrays.copyOf(ids, ids.length); // Unsigned order: flipping the sign bit maps it onto signed order. for (int i = 0; i < sorted.length; i++) { @@ -176,11 +184,11 @@ static String buildId(long[] ids) { // Every Java platform has SHA-256. throw new IllegalStateException(e); } - StringBuilder id = new StringBuilder("jv1-"); - for (int i = 0; i < 16; i++) { - id.append(Character.forDigit((sum[i] >> 4) & 0xf, 16)).append(Character.forDigit(sum[i] & 0xf, 16)); + StringBuilder hex = new StringBuilder(sum.length * 2); + for (byte b : sum) { + hex.append(Character.forDigit((b >> 4) & 0xf, 16)).append(Character.forDigit(b & 0xf, 16)); } - return id.toString(); + return hex.toString(); } /** diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java index 580bcd1a..92ad7be7 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java @@ -27,6 +27,8 @@ import java.nio.file.Files; import java.nio.file.Path; import java.nio.file.Paths; +import java.security.MessageDigest; +import java.security.NoSuchAlgorithmException; import java.util.ArrayList; import java.util.Arrays; import java.util.LinkedHashMap; @@ -98,6 +100,15 @@ static byte[] resource(String name) throws IOException { } } + /** sha256Hex is the sha256 of b as 64 lowercase hex, as a frame's "sha256" says it. */ + static String sha256Hex(byte[] b) { + try { + return BuildTable.hex(MessageDigest.getInstance("SHA-256").digest(b)); + } catch (NoSuchAlgorithmException e) { + throw new IllegalStateException(e); + } + } + /** Collects the records write hands out, as the NDJSON stream. */ static final class Collect implements BuildTable.RecordSink { final ByteArrayOutputStream ndjson = new ByteArrayOutputStream(); @@ -322,7 +333,7 @@ public void framesCarryTheStreamInOrderAndNeverPassTheCap() throws Exception { assertEquals(last, h.get("last").getAsBoolean()); if (last) { assertEquals(5, h.get("records").getAsInt()); - assertEquals(BuildTable.sha256Hex(c.ndjson.toByteArray()), h.get("sha256").getAsString()); + assertEquals(sha256Hex(c.ndjson.toByteArray()), h.get("sha256").getAsString()); } else { assertFalse(h.has("records")); } @@ -523,11 +534,11 @@ public void aClassIsNamedByItsBytesNotByWhereTheIndexFoundIt() throws Exception assertEquals("c:com/acme/Foo", u.get("id").getAsString()); assertEquals("com/acme", u.get("pkg").getAsString()); assertEquals("com/acme/Foo.java", u.get("path").getAsString()); - assertEquals(BuildTable.classIdsSum(new long[] {CRC64.classId(foo), CRC64.classId(in)}), + assertEquals(IndexReport.classIdsSum(new long[] {CRC64.classId(foo), CRC64.classId(in)}), u.get("hash").getAsString()); // Every indexed class is in the build id, as in the index record's, named or not. - assertEquals("jv1-" + BuildTable.classIdsSum(new long[] {CRC64.classId(foo), CRC64.classId(in), - CRC64.classId(junk)}).substring(0, 32), s.build); + assertEquals(IndexReport.buildId(new long[] {CRC64.classId(foo), CRC64.classId(in), CRC64.classId(junk)}), + s.build); assertTrue(c.records.get(0), c.records.get(0).contains("\"id\":\"m:com/acme/Foo$In#run(I[Ljava/lang/String;)I\"")); } diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/ControlSocketTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/ControlSocketTest.java index 6b90ee69..19e00745 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/ControlSocketTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/ControlSocketTest.java @@ -21,7 +21,9 @@ import java.nio.file.attribute.PosixFilePermission; import java.security.MessageDigest; import java.security.NoSuchAlgorithmException; +import java.util.ArrayList; import java.util.List; +import java.util.TreeSet; import java.util.concurrent.BlockingQueue; import java.util.concurrent.CopyOnWriteArrayList; import java.util.concurrent.LinkedBlockingQueue; @@ -30,6 +32,7 @@ import java.util.logging.Level; import java.util.logging.LogRecord; import java.util.logging.Logger; +import org.jacoco.core.internal.data.CRC64; import org.junit.After; import org.junit.Before; import org.junit.Test; @@ -226,6 +229,60 @@ public void theBuildTableGoesOnlyWhenKeployAsksForIt() throws Exception { assertNull("nothing more", data.payloads.poll(1, TimeUnit.SECONDS)); } + @Test(timeout = 60000) + public void theIndexRecordAndTheTableNameOneBuild() throws Exception { + // Keploy finds the table under the build the INDEX record named. The index is the java-dedup sample's + // classes and one file no class reader reads: both builds take every class the index holds. + File classes = new File(dir, "classes"); + List files = new ArrayList<>(BuildTableTest.javaDedup()); + files.add(new BuildTable.ClassFile("com/acme/Junk", "not a class".getBytes(StandardCharsets.UTF_8))); + TreeSet ids = new TreeSet<>(); + for (BuildTable.ClassFile f : files) { + File out = new File(classes, f.name + ".class"); + assertTrue(out.getParentFile().isDirectory() || out.getParentFile().mkdirs()); + Files.write(out.toPath(), f.bytes); + ids.add(String.format("%016x", CRC64.classId(f.bytes))); + } + // Keploy's codecoverage.JavaBuildID: the ids as 16 hex, sorted (as hex: unsigned), joined by newlines. + String build = "jv1-" + sha256(String.join("\n", ids).getBytes(StandardCharsets.UTF_8)).substring(0, 32); + assertNotEquals("the file no class reader reads is in the build", + new String(BuildTableTest.resource("java-dedup.build"), StandardCharsets.UTF_8).trim(), build); + System.setProperty(CLASS_DIRS_PROPERTY, classes.getPath()); + data = new DataReceiver(KeployDedupAgent.dataSocketPath); + startWithJacoco(); + String index = data.indexes.poll(30, TimeUnit.SECONDS); + assertNotNull("the INDEX record came once the data socket was bound", index); + assertTrue(index, index.contains("\"classes\":" + files.size() + ",")); + assertEquals(index, build, field(index, "build")); + + assertEquals(TAKEN, send("TABLE")); + boolean tableDone = false; + boolean probesDone = false; + while (!tableDone || !probesDone) { + String frame = data.next(); + assertNotNull("the table and the probe map each reached their last frame", frame); + int at = frame.indexOf("{\"v\":1,"); + assertTrue(frame, frame.startsWith("KTB1") && at > 0); + String header = frame.substring(at, frame.indexOf('\n', at)); + assertEquals(header, build, field(header, "build")); + boolean last = header.contains("\"last\":true"); + if (header.contains("\"part\":\"table\"")) { + tableDone |= last; + } else { + assertTrue(header, header.contains("\"part\":\"probes\"")); + probesDone |= last; + } + } + } + + // field is the string value of name in one JSON object's text. + private static String field(String json, String name) { + String key = "\"" + name + "\":\""; + int at = json.indexOf(key); + assertTrue(json, at >= 0); + return json.substring(at + key.length(), json.indexOf('"', at + key.length())); + } + @Test(timeout = 60000) public void aTableThatFailsToStartIsIgnoredAndTheServerGoesOn() throws Exception { startWithJacoco(); diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/IndexReportTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/IndexReportTest.java index 2a7464fb..bdc90720 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/IndexReportTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/IndexReportTest.java @@ -44,6 +44,16 @@ public void theBuildIdIsKeploysForTheSameIdsInAnyOrderAndRepeated() { assertEquals(GO_BUILD_ID, IndexReport.buildId(new long[] {0xff, 0xffffffffffffffffL, 0xff, 1})); } + // The whole sha256 over the same three ids: what a build table unit's + // hash is for a unit of those class ids. The build id is its first 32 hex. + private static final String IDS_SUM = "73ab094cbaa75db7d27e1cc885ca22604fb30a4555ebac8aae6e264612a027d0"; + + @Test + public void theUnitHashIsTheWholeSumAndTheBuildIdItsFirstHalf() { + assertEquals(IDS_SUM, IndexReport.classIdsSum(new long[] {0xffffffffffffffffL, 1, 0xff, 1})); + assertEquals(GO_BUILD_ID, "jv1-" + IDS_SUM.substring(0, 32)); + } + @Test public void noClassHasNoBuildId() { assertEquals("", IndexReport.buildId(new long[0])); diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/ProbeMapTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/ProbeMapTest.java index 0907eba3..06049ebf 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/ProbeMapTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/ProbeMapTest.java @@ -459,7 +459,7 @@ public void theProbeMapFollowsTheTableAsItsOwnStream() throws Exception { stream.write(f, 8 + h.length() + 1, f.length - 8 - h.length() - 1); if (o.get("last").getAsBoolean()) { assertEquals(5, o.get("records").getAsInt()); - assertEquals(BuildTable.sha256Hex(golden.getBytes(StandardCharsets.UTF_8)), + assertEquals(BuildTableTest.sha256Hex(golden.getBytes(StandardCharsets.UTF_8)), o.get("sha256").getAsString()); assertEquals("the last frame ends the answer", f, sent.frames.get(sent.frames.size() - 1)); } From ab786ac0f15725c2df78a47d57f560c802c26808 Mon Sep 17 00:00:00 2001 From: slayerjain Date: Fri, 20 Nov 2026 03:42:09 +0530 Subject: [PATCH 04/14] perf(dedup): map each probe by replaying its method, not the class The probe map analyzed each class once per probe round: a class with a method of K probes was parsed, sanitized, flow-analyzed and filtered K times, whole. KeployBranchCoverage now analyzes a class once. For each method it keeps the filters' output, then replays only that method's instructions through JaCoCo's own probe adapter, analyzer, instruction builder and coverage calculator, once per probe, with that probe alone executed. The result is still JaCoCo's analysis, not a copy of its rules, and a replay that meets other probes than the class's analysis did fails the class (not_measured, analyzer_failed). The filters now run once per class, in class order, exactly as for the table. A method with no line (a class built without -g) has no leaf to map, so it is not replayed. A class now costs, summed over its methods, each method's probes times that method's size. One harness, run before and after on the same machine (JDK 21, heap capped at 1 GiB), on a generated app of 50,000 classes (one 15-branch method each): the table and probe map took 51.4 s of wall time and 58.3 s of process CPU before, 29.1 s and 33.5 s after. BuildTableBudgetTest measures it in the test JVM on JDK 8, 17 and 21, heap capped at 1 GiB, table and probe map: - 50,000 generated classes: 28.6-31.2 s wall, 27.4-30.3 s on the table's thread, 35.7-38.1 s of process CPU (GC 0.6-1.0 s). At most 100-114 MB of heap stayed in use after any collection. Output: 55 MB of table and 21 MB of probe map. With the heap capped at 256 MB (JDK 21): 33-36 s wall, 41-64 s of process CPU (GC 1.9 s). - One class with a single 2000-branch method: 12.1-15.1 s; with 3500 branches (JDK 21), 57.4 s. No frame goes out while one class's map is computed, so such a class pauses the stream that long. - JaCoCo core's jar (210 classes): 0.5-0.6 s. The analysis runs on its own daemon thread after Keploy asks for it, and START and END are answered while it runs. Also: - A TABLE that arrives while a send runs now runs the newest request after it, not the first one again. Each request names the index and data socket it was made for, so after the agent's stop() and start() during a send, the rerun used the old index and the old socket. A send that throws no longer drops the request made while it ran. - BuildTable's logging can no longer throw. java.util.logging does not catch what a Handler throws (an app's logging bridge), so a log call that threw could fail a send, and one in the send loop left it marked running with no thread: every later TABLE was ACKed and never sent. - keploy-sdk's tests run with a 1 GiB heap (the argLine property; -DargLine overrides it). Left to the JVM, the heap was a quarter of the machine's RAM. In a 4 GiB container where JDK 8 sizes it from the host's RAM, the new rows let its collector grow the heap until the kernel killed the test JVM. - IndexReport.hex is the one lowercase-hex encoder (the build id and a frame's sha256); BuildTable's copy is removed. - BuildTable.write no longer skips an index entry without bytes: the agent never passes one, and the INDEX record has no such skip. - BuildTableTest's receiver keeps each frame before it answers OK, as Keploy does; it answered first, so a row could count one frame short. Rows: - ProbeMapTest.eachProbesReplayIsTheClassAnalyzedWithThatProbeExecuted: on JaCoCo core's and Gson's jars, each probe's replay equals the class analyzed whole with that round's probes executed, for every method and probe. - ProbeMapTest.aMethodTheCallerDeclinesIsNotReplayed: a method the caller declines gets no replay; the class pass still analyzes it. - BuildTableTest.theRunAfterASendIsTheNewestRequests: two requests during a send run the newest once (it ran the first one again). - BuildTableTest.aRequestMadeWhileASendFailsStillRuns: a request made during a send that throws still runs (it was dropped). - BuildTableTest.aLogHandlerThatThrowsNeitherFailsASendNorWedgesSends: with a log handler that throws, send does not throw, and a request after a failed send, and after a sent table, still runs. - ControlSocketTest.startAndEndAreAnsweredWhileTheTableIsAnalyzed: with a 5000-class index, START/END pairs are answered while the table is analyzed and sent; TABLE's own answer and the slowest pair each take under 5 s. - BuildTableBudgetTest: measures and prints the cost on a generated app, a one-method class and a real library; every class is measured. -Dkeploy.budget.classes and -Dkeploy.budget.branches set the sizes. - The java-dedup table and probe-map goldens are unchanged. Signed-off-by: slayerjain --- keploy-sdk/pom.xml | 4 + .../main/java/io/keploy/dedup/BuildTable.java | 157 ++++++----- .../java/io/keploy/dedup/IndexReport.java | 11 +- .../analysis/KeployBranchCoverage.java | 146 ++++++++-- .../io/keploy/dedup/BuildTableBudgetTest.java | 256 ++++++++++++++++++ .../java/io/keploy/dedup/BuildTableTest.java | 134 ++++++++- .../io/keploy/dedup/ControlSocketTest.java | 67 +++++ .../java/io/keploy/dedup/IndexReportTest.java | 2 +- .../java/io/keploy/dedup/ProbeMapTest.java | 115 ++++++++ 9 files changed, 781 insertions(+), 111 deletions(-) create mode 100644 keploy-sdk/src/test/java/io/keploy/dedup/BuildTableBudgetTest.java diff --git a/keploy-sdk/pom.xml b/keploy-sdk/pom.xml index 85783e36..40f08847 100644 --- a/keploy-sdk/pom.xml +++ b/keploy-sdk/pom.xml @@ -20,6 +20,10 @@ UTF-8 UTF-8 0.8.15 + + -Xmx1g diff --git a/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java b/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java index 4076ac2d..597016d1 100644 --- a/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java +++ b/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java @@ -72,6 +72,19 @@ final class BuildTable { private static final Logger LOGGER = Logger.getLogger(BuildTable.class.getName()); + /** + * log never throws. java.util.logging does not catch what a Handler throws (an app's logging bridge, an OOM + * while formatting), and a log that throws would end a send, or leave {@link Sends} running with no thread: + * every later TABLE would be ACKed and never sent. + */ + static void log(Level level, String message, Throwable thrown) { + try { + LOGGER.log(level, message, thrown); + } catch (Throwable ignored) { + // Nowhere left to say it: the log is what failed. + } + } + /** The control-socket command Keploy sends for the build table. */ static final String COMMAND = "TABLE"; /** A TABLE frame's size cap, magic and length included. */ @@ -118,7 +131,7 @@ static List files(List entries, Function name, Func /** * request answers Keploy's TABLE command: true (ACK) once the table's send is started on its own thread, - * or, when a send runs already, once that send is set to run once more after it. It never blocks. + * or, when a send runs already, once this request is set to run after it. It never blocks. * When no thread can start it answers false (IGNORED) and does not throw; the agent's command hook catches * anything else. The classes are read on that thread. */ @@ -140,14 +153,18 @@ static Thread thread(Runnable r) { } /** - * Sends runs one send at a time; a request while one runs has it run once more after it. It assumes one - * caller at a time (the control server handles one command at a time): a reset below clears {@code again}, - * which would drop a request that raced in from a second caller. + * Sends runs one send at a time. Requests made while one runs are run once after it, as the newest of them: + * each request names its own index and data socket, and only the newest is the agent's now (a stop() and + * start() during a send makes a new index and socket). A send that throws is logged, and the request made + * while it ran still runs. It assumes one caller at a time (the control server handles one command at a + * time): a thread that fails to start clears {@code pending}, which would drop a request that raced in from + * a second caller. */ static final class Sends { private final ThreadFactory threads; private boolean running; - private boolean again; + /** The newest request made while a send ran, to run after it; null for none. */ + private Runnable pending; private int requests; Sends(ThreadFactory threads) { @@ -162,7 +179,7 @@ boolean request(Runnable send) { synchronized (this) { requests++; if (running) { - again = true; + pending = send; return true; } running = true; @@ -175,31 +192,31 @@ boolean request(Runnable send) { // must not end the control server. Keploy reads IGNORED as no table from this SDK. synchronized (this) { running = false; - again = false; + pending = null; } - LOGGER.log(Level.WARNING, "Keploy coverage: the build table's thread did not start", t); + log(Level.WARNING, "Keploy coverage: the build table's thread did not start", t); return false; } } - private void loop(Runnable send) { - try { - while (true) { + private void loop(Runnable first) { + Runnable send = first; + while (true) { + try { send.run(); - synchronized (this) { - if (!again) { - running = false; - return; - } - again = false; - } + } catch (Throwable t) { + // A failed send does not drop the request made while it ran: that one names the newest + // index and socket. + log(Level.WARNING, "Keploy coverage: the build table was not sent", t); } - } catch (Throwable t) { synchronized (this) { - running = false; - again = false; + if (pending == null) { + running = false; + return; + } + send = pending; + pending = null; } - LOGGER.log(Level.WARNING, "Keploy coverage: the build table was not sent", t); } } } @@ -238,13 +255,13 @@ static void send(Supplier> classes, FrameSender sender) { m.append("; probe map of ").append(s.probes.size()).append(" class files; ").append(frames.seq + maps.seq) .append(" frames, ").append((System.nanoTime() - t0) / 1_000_000) .append(" ms, cpu ").append((threadCpuNanos() - cpu0) / 1_000_000).append(" ms)"); - LOGGER.log(Level.INFO, m.toString()); + log(Level.INFO, m.toString(), null); } catch (IOException e) { - LOGGER.log(Level.WARNING, "Keploy coverage: the build table was not sent: " + e.getMessage() - + "; it goes again when Keploy asks for it again"); + log(Level.WARNING, "Keploy coverage: the build table was not sent: " + e.getMessage() + + "; it goes again when Keploy asks for it again", null); } catch (Throwable t) { // Never let the table take the app down: Keploy names the coverage it did not get. - LOGGER.log(Level.WARNING, "Keploy coverage: the build table was not sent", t); + log(Level.WARNING, "Keploy coverage: the build table was not sent", t); } } @@ -305,7 +322,7 @@ static Summary write(List classes, RecordSink sink, int maxRecord) th int n = 0; int unnamed = 0; for (ClassFile in : classes) { - if (in == null || in.bytes == null) { + if (in == null) { continue; } // Every indexed class is in the build id, named or not: the INDEX record's, by the same function. @@ -373,14 +390,6 @@ static String vmName(ClassFile f) { } } - static String hex(byte[] b) { - StringBuilder sb = new StringBuilder(b.length * 2); - for (byte x : b) { - sb.append(Character.forDigit((x >> 4) & 0xf, 16)).append(Character.forDigit(x & 0xf, 16)); - } - return sb.toString(); - } - /** Units by package path, segment by segment ('/' sorts first), then by id: Go's comparePkg order. */ static final Comparator UNIT_ORDER = (a, b) -> { int c = comparePkg(a.pkg, b.pkg); @@ -534,7 +543,7 @@ byte[] record() { try { got = analyze(f.file); } catch (Exception | LinkageError e) { - LOGGER.log(Level.FINE, "Keploy coverage: JaCoCo could not analyze " + f.file.name, e); + log(Level.FINE, "Keploy coverage: JaCoCo could not analyze " + f.file.name, e); failed = true; return notMeasured("analyzer_failed"); } @@ -585,7 +594,7 @@ byte[] record() { } } catch (Exception | LinkageError e) { // A class whose probes cannot be mapped exactly is not measured: its hits could not be read. - LOGGER.log(Level.FINE, "Keploy coverage: the probes of " + path() + " could not be mapped", e); + log(Level.FINE, "Keploy coverage: the probes of " + path() + " could not be mapped", e); failed = true; notMapped = true; return notMeasured("analyzer_failed"); @@ -831,9 +840,9 @@ BitSet hits(KeployBranchCoverage.Method a) { * covered branch edge's leaf. A probe covers more than its own line: JaCoCo marks every instruction on the * path that ends at the probe, so a test's leaves are the union of its probes' leaves. * - *

A method's probes never reach another method's instructions, so one analysis maps the k-th probe of - * every method at once: the map takes one analysis of the class file more than its largest method has - * probes, on top of the table's own analysis of it. + *

A method's probes never reach another method's instructions, so the class file is analyzed once and + * each probe replays only its own method (KeployBranchCoverage): the map costs, summed over the methods + * with a line, each method's probes times that method's size, on top of the table's own analysis. */ static final class ProbeMap { private ProbeMap() { @@ -857,16 +866,32 @@ static byte[] record(String unitId, UnitFile f, String probes) { * probes is the class file's probe map as JSON: one entry per probe (the class's probe count), each the * probe's leaves as half-open ranges of the unit's leaf indices, [start,end,start,end,...] ascending, no * two ranges touching. It throws when the class's lines or branches differ from the unit's methods (the - * same name under two roots), when a method with lines is in one and not in the other, or when a round - * leaves a method out. + * same name under two roots), when a method with lines is in one and not in the other, or when a probe's + * replay meets other probes than the class's analysis did. */ static String probes(UnitFile f, Map byId) { Map shape = new HashMap<>(); - int total = Math.max(0, KeployBranchCoverage.analyze(f.file.bytes, null, null, shape)); - Map table = new HashMap<>(); - int rounds = 0; + Map hits = new HashMap<>(); + String prefix = "m:" + f.file.name + "#"; + // One analysis of the class; each probe's coverage replays only its own method (KeployBranchCoverage). + int total = Math.max(0, KeployBranchCoverage.analyze(f.file.bytes, shape, + new KeployBranchCoverage.EachProbe() { + @Override + public boolean wants(KeployBranchCoverage.Method s) { + // Checked below: a method with no line has no leaf for its probes to cover. + return byId.containsKey(prefix + s.name + s.desc); + } + + @Override + public void probe(KeployBranchCoverage.Method s, int r, KeployBranchCoverage.Method got) { + Method m = byId.get(prefix + s.name + s.desc); + m.check(got); + hits.put(s.probes[r], m.hits(got)); + } + })); + int inShape = 0; for (Map.Entry e : shape.entrySet()) { - Method m = byId.get("m:" + f.file.name + "#" + e.getKey()); + Method m = byId.get(prefix + e.getKey()); if (m == null) { for (int l : e.getValue().lines) { if (l >= 1) { @@ -874,53 +899,27 @@ static String probes(UnitFile f, Map byId) { + " has lines but no method in the table"); } } - continue; // no line: no leaf for its probes to cover + continue; } m.check(e.getValue()); - table.put(e.getKey(), m); - rounds = Math.max(rounds, e.getValue().probes.length); + inShape++; } // And the other way: each of the table's methods of this class file has its probes mapped here. - String prefix = "m:" + f.file.name + "#"; int inTable = 0; for (String id : byId.keySet()) { if (id.startsWith(prefix)) { inTable++; } } - if (inTable != table.size()) { + if (inTable != inShape) { throw new IllegalStateException(f.file.name + ": " + inTable + " methods in the table, " - + table.size() + " in the probe analysis"); - } - BitSet[] hits = new BitSet[total]; - for (int r = 0; r < rounds; r++) { - boolean[] probes = new boolean[total]; - Set only = new HashSet<>(); - for (Map.Entry e : table.entrySet()) { - int[] ids = shape.get(e.getKey()).probes; - if (ids.length > r) { - probes[ids[r]] = true; - only.add(e.getKey()); - } - } - Map run = new HashMap<>(); - // The first round analyzes every method, as the table's analysis does; see KeployBranchCoverage. - KeployBranchCoverage.analyze(f.file.bytes, probes, r == 0 ? null : only, run); - for (String key : only) { - KeployBranchCoverage.Method got = run.get(key); - if (got == null) { - throw new IllegalStateException(f.file.name + "#" + key + " has no coverage in round " + r); - } - Method m = table.get(key); - m.check(got); - hits[shape.get(key).probes[r]] = m.hits(got); - } + + inShape + " in the probe analysis"); } StringBuilder sb = new StringBuilder(2 + 8 * total); sb.append('['); for (int p = 0; p < total; p++) { sb.append(p > 0 ? ",[" : "["); - BitSet b = hits[p]; + BitSet b = hits.get(p); int at = b == null ? -1 : b.nextSetBit(0); boolean first = true; while (at >= 0) { @@ -1198,8 +1197,8 @@ private void flush(boolean last, int records) throws IOException { Json.string(h, build); h.append(",\"seq\":").append(seq).append(",\"last\":").append(last); if (last) { - h.append(",\"records\":").append(records).append(",\"sha256\":\"").append(hex(digest.digest())) - .append('"'); + h.append(",\"records\":").append(records) + .append(",\"sha256\":\"").append(IndexReport.hex(digest.digest())).append('"'); } h.append("}\n"); byte[] header = h.toString().getBytes(StandardCharsets.UTF_8); diff --git a/keploy-sdk/src/main/java/io/keploy/dedup/IndexReport.java b/keploy-sdk/src/main/java/io/keploy/dedup/IndexReport.java index 1f496134..b1151f3b 100644 --- a/keploy-sdk/src/main/java/io/keploy/dedup/IndexReport.java +++ b/keploy-sdk/src/main/java/io/keploy/dedup/IndexReport.java @@ -184,9 +184,14 @@ static String classIdsSum(long[] ids) { // Every Java platform has SHA-256. throw new IllegalStateException(e); } - StringBuilder hex = new StringBuilder(sum.length * 2); - for (byte b : sum) { - hex.append(Character.forDigit((b >> 4) & 0xf, 16)).append(Character.forDigit(b & 0xf, 16)); + return hex(sum); + } + + /** hex is b as lowercase hex, two digits a byte: the build id's and a TABLE frame's sha256. */ + static String hex(byte[] b) { + StringBuilder hex = new StringBuilder(b.length * 2); + for (byte x : b) { + hex.append(Character.forDigit((x >> 4) & 0xf, 16)).append(Character.forDigit(x & 0xf, 16)); } return hex.toString(); } diff --git a/keploy-sdk/src/main/java/org/jacoco/core/internal/analysis/KeployBranchCoverage.java b/keploy-sdk/src/main/java/org/jacoco/core/internal/analysis/KeployBranchCoverage.java index 1d23d9c9..33cdbf12 100644 --- a/keploy-sdk/src/main/java/org/jacoco/core/internal/analysis/KeployBranchCoverage.java +++ b/keploy-sdk/src/main/java/org/jacoco/core/internal/analysis/KeployBranchCoverage.java @@ -8,6 +8,7 @@ import java.util.Map; import java.util.Set; import java.util.TreeSet; +import java.util.function.Consumer; import org.jacoco.core.analysis.ICounter; import org.jacoco.core.analysis.ILine; @@ -16,6 +17,8 @@ import org.jacoco.core.internal.analysis.filter.IFilterOutput; import org.jacoco.core.internal.analysis.filter.Replacements; import org.jacoco.core.internal.flow.ClassProbesAdapter; +import org.jacoco.core.internal.flow.IProbeIdGenerator; +import org.jacoco.core.internal.flow.MethodProbesAdapter; import org.jacoco.core.internal.flow.MethodProbesVisitor; import org.jacoco.core.internal.instr.InstrSupport; import org.objectweb.asm.ClassReader; @@ -71,34 +74,66 @@ void addProbe(int id) { /** * analyze runs JaCoCo's analysis over the class bytes with the given probes (null: none executed) and puts - * each method that has code into out, keyed by name + descriptor. With only non-null, only the methods - * named there are analyzed (the others still take their probe ids, as JaCoCo's analyzer does). It returns + * each method that has code into out, keyed by name + descriptor. It returns * the class's probe count, or -1 for a class JaCoCo's Analyzer does not analyze (module-info, * package-info). It throws when the bytes are not a class JaCoCo reads, or when a method's per-line sums * differ from JaCoCo's coverage of it. */ - public static int analyze(byte[] bytes, boolean[] probes, Set only, Map out) { + public static int analyze(byte[] bytes, boolean[] probes, Map out) { + return analyze(bytes, probes, out, null); + } + + /** What {@link #analyze(byte[], Map, EachProbe)} hands over for each probe of each method with code. */ + public interface EachProbe { + /** + * probe gets method (as analyze with no probe executed puts it in out), the index r of one of its + * probes in method.probes, and the method's coverage when that probe alone was executed. + */ + void probe(Method method, int r, Method got); + + /** + * wants says whether method's probes are replayed at all: a method it declines costs no replay, and + * probe gets none of its probes. Every method with code is wanted unless this is overridden. + */ + default boolean wants(Method method) { + return true; + } + } + + /** + * analyze is {@link #analyze(byte[], boolean[], Map)} with no probe executed, and it also hands each + * probe's own coverage to each, method by method, as a class analysis with only that probe executed gives + * it. It parses the class and runs the filters once: each probe replays only its method's instructions + * through JaCoCo's probe adapter, instruction builder and coverage calculator, with the filters' output of + * that one pass. So a class costs, summed over its methods, each method's probes times that method's size, + * not the whole class once per probe. A replay that meets other probes than the pass did throws. + */ + public static int analyze(byte[] bytes, Map out, EachProbe each) { + return analyze(bytes, null, out, each); + } + + private static int analyze(byte[] bytes, boolean[] probes, Map out, EachProbe each) { ClassReader reader = InstrSupport.classReaderFor(bytes); if ((reader.getAccess() & Opcodes.ACC_MODULE) != 0 || reader.getClassName().endsWith("/package-info")) { return -1; } - Visitor v = new Visitor(reader.getClassName(), probes, only, out); + Visitor v = new Visitor(reader.getClassName(), probes, out, each); reader.accept(new ClassProbesAdapter(v, false), 0); return v.total; } private static final class Visitor extends ClassAnalyzer { private final boolean[] probes; - private final Set only; private final Map out; + private final EachProbe each; private final IFilter filter = Filters.all(); int total; - Visitor(String name, boolean[] probes, Set only, Map out) { + Visitor(String name, boolean[] probes, Map out, EachProbe each) { super(new ClassCoverageImpl(name, 0, false), probes, new StringPool()); this.probes = probes; - this.only = only; this.out = out; + this.each = each; } @Override @@ -110,14 +145,6 @@ public void visitTotalProbeCount(int count) { @Override public MethodProbesVisitor visitMethod(int access, final String name, final String desc, final String signature, String[] exceptions) { - // Leaving a method out keeps the others' results only while JaCoCo's filters carry nothing from one - // method to the next that changes them. One does: KotlinSerializableFilter keeps the line of a - // $Companion's ()V for serializer(). That method has one probe, so it is only ever mapped in - // the first round, which analyzes every method (only is null). Any other difference fails the - // class: the caller checks each method's lines and branches against the table in every round. - if (only != null && !only.contains(name + desc)) { - return null; - } InstrSupport.assertNotInstrumented(name, getClassName()); final Method m = new Method(name, desc); final Builder builder = new Builder(probes, m); @@ -131,14 +158,61 @@ public void accept(MethodNode node, MethodVisitor visitor) { } private void collect(Method m, Builder builder, MethodNode node, String signature) { + Output output = new Output(); + filter.filter(node, this, output); + if (!fill(m, builder, node, signature, output)) { + return; + } + out.put(m.name + m.desc, m); + if (each != null && each.wants(m)) { + replay(m, node, signature, output); + } + } + + /** + * replay hands each of m's probes to each with m's coverage when that probe alone is executed: the + * method's node again through a new probe adapter (its probe ids from the method's first) and analyzer, + * and the filters' output of the pass that made m. + */ + private void replay(Method m, MethodNode node, String signature, Output output) { + if (m.probes.length == 0) { + return; + } + int first = Integer.MAX_VALUE; + int last = -1; + for (int id : m.probes) { + first = Math.min(first, id); + last = Math.max(last, id); + } + boolean[] one = new boolean[last + 1]; + for (int r = 0; r < m.probes.length; r++) { + one[m.probes[r]] = true; + Method got = new Method(m.name, m.desc); + Builder builder = new Builder(one, got); + MethodAnalyzer analyzer = new MethodAnalyzer(builder); + analyzer.accept(node, new MethodProbesAdapter(analyzer, new Ids(first))); + one[m.probes[r]] = false; + if (!fill(got, builder, node, signature, output) || !Arrays.equals(got.probes, m.probes)) { + throw new IllegalStateException("JaCoCo's replay of " + getClassName() + "#" + m.name + m.desc + + " met probes " + Arrays.toString(got.probes) + ", its analysis " + + Arrays.toString(m.probes)); + } + each.probe(m, r, got); + } + } + + /** + * fill sets m's counted instructions from builder's graph and the filters' output, as ClassAnalyzer + * computes a method's coverage. It returns false for a method without code. + */ + private boolean fill(Method m, Builder builder, MethodNode node, String signature, Output output) { Map instructions = builder.getInstructions(); MethodCoverageCalculator calculator = new MethodCoverageCalculator(instructions); - Output output = new Output(calculator); - filter.filter(node, this, output); + output.applyTo(calculator); MethodCoverageImpl coverage = new MethodCoverageImpl(m.name, m.desc, signature); calculator.calculate(coverage); if (!coverage.containsCode()) { - return; + return false; } int[] p = new int[m.probeCount]; System.arraycopy(m.probes, 0, p, 0, m.probeCount); @@ -210,7 +284,7 @@ private void collect(Method m, Builder builder, MethodNode node, String signatur m.covered[k] = covered.get(k); m.branches[k] = branches.get(k); } - out.put(m.name + m.desc, m); + return true; } private static boolean same(ICounter c, int total, int covered) { @@ -279,24 +353,40 @@ void addProbe(int probeId, int branch) { } } + /** Ids counts probe ids up from a method's first, as ClassProbesAdapter counted them for the class. */ + private static final class Ids implements IProbeIdGenerator { + private int next; + + Ids(int first) { + next = first; + } + + @Override + public int nextId() { + return next++; + } + } + /** - * Output passes the filters' decisions on to JaCoCo's calculator and keeps the instructions it leaves out: - * the ignored ones and those merged into another (the calculator's own rule, findRepresentative). + * Output keeps what the filters said of one method, in their order, to be given to the coverage calculator + * of each analysis of that method: the filters read only the method's code and the class, never coverage. */ private static final class Output implements IFilterOutput { - private final MethodCoverageCalculator calculator; + private final List> calls = new ArrayList<>(); final Set ignored = new HashSet<>(); private final Map merged = new HashMap<>(); // Instructions a filter replaced the branches of, with how many new branches each has. final Map replaced = new HashMap<>(); - Output(MethodCoverageCalculator calculator) { - this.calculator = calculator; + void applyTo(IFilterOutput calculator) { + for (Consumer call : calls) { + call.accept(calculator); + } } @Override public void ignore(AbstractInsnNode fromInclusive, AbstractInsnNode toInclusive) { - calculator.ignore(fromInclusive, toInclusive); + calls.add(c -> c.ignore(fromInclusive, toInclusive)); for (AbstractInsnNode i = fromInclusive; i != toInclusive; i = i.getNext()) { ignored.add(i); } @@ -305,7 +395,9 @@ public void ignore(AbstractInsnNode fromInclusive, AbstractInsnNode toInclusive) @Override public void merge(AbstractInsnNode i1, AbstractInsnNode i2) { - calculator.merge(i1, i2); + final AbstractInsnNode a = i1; + final AbstractInsnNode b = i2; + calls.add(c -> c.merge(a, b)); i1 = representative(i1); i2 = representative(i2); if (i1 != i2) { @@ -324,7 +416,7 @@ private AbstractInsnNode representative(AbstractInsnNode i) { @Override public void replaceBranches(AbstractInsnNode source, Replacements replacements) { - calculator.replaceBranches(source, replacements); + calls.add(c -> c.replaceBranches(source, replacements)); int n = 0; for (Object ignoredValue : replacements.values()) { n++; diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableBudgetTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableBudgetTest.java new file mode 100644 index 00000000..1324d915 --- /dev/null +++ b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableBudgetTest.java @@ -0,0 +1,256 @@ +package io.keploy.dedup; + +import com.sun.management.GarbageCollectionNotificationInfo; +import org.jacoco.core.analysis.Analyzer; +import org.junit.Test; +import org.objectweb.asm.ClassWriter; +import org.objectweb.asm.Label; +import org.objectweb.asm.MethodVisitor; +import org.objectweb.asm.Opcodes; + +import java.io.File; +import java.lang.management.GarbageCollectorMXBean; +import java.lang.management.ManagementFactory; +import java.lang.management.MemoryPoolMXBean; +import java.lang.management.MemoryType; +import java.lang.management.MemoryUsage; +import java.lang.management.OperatingSystemMXBean; +import java.lang.management.ThreadMXBean; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.HashSet; +import java.util.List; +import java.util.Locale; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicLong; +import javax.management.NotificationEmitter; +import javax.management.NotificationListener; +import javax.management.openmbean.CompositeData; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +/** + * The build table's cost inside the app's JVM, measured and reported (a BUDGET line on stdout): on a generated + * app (classes of one if/else chain method each, as the large-app profile generates them), on one class with a + * single long method, and on a real library. -Dkeploy.budget.classes=50000 and -Dkeploy.budget.branches=2000 + * measure the large app and a 2000-branch method; the defaults keep the rows quick. It checks that every class + * is measured; the numbers are for reading, not asserted. + * + *

heap_after_gc_max_mb is the most heap any collection during the send left in use: an upper bound of what + * the analysis keeps live (a young collection also counts old garbage not collected yet). It is -1 when no + * collection ran. heap_max_mb is the heap cap the JVM ran with (-Xmx, or its default from the machine's RAM). + */ +public class BuildTableBudgetTest { + + static final int BRANCHES = 15; + + @Test(timeout = 1_800_000) + public void aGeneratedAppsTableIsMeasured() throws Exception { + int n = Integer.getInteger("keploy.budget.classes", 2000); + List app = new ArrayList<>(n); + for (int i = 0; i < n; i++) { + app.add(new BuildTable.ClassFile(name(i), gen(name(i), i, BRANCHES))); + } + Budget b = measure("app-" + n, app); + assertEquals(n, b.tableRecords); + assertEquals(n, b.probeRecords); + assertEquals("every class is measured", 0, b.notMeasured); + } + + @Test(timeout = 600_000) + public void aClassWithOneLongMethodIsMeasured() throws Exception { + int branches = Integer.getInteger("keploy.budget.branches", 500); + List one = new ArrayList<>(); + one.add(new BuildTable.ClassFile("gen/big/Long", gen("gen/big/Long", 0, branches))); + Budget b = measure("one-method-" + branches + "-branches", one); + assertEquals(1, b.probeRecords); + assertEquals(0, b.notMeasured); + } + + @Test(timeout = 600_000) + public void aRealLibraryIsMeasured() throws Exception { + File jar = new File(Analyzer.class.getProtectionDomain().getCodeSource().getLocation().toURI()); + List lib = ProbeMapTest.jarClasses(jar); + Budget b = measure("jacoco-core-jar", lib); + assertTrue(b.tableRecords > 100); + assertEquals(0, b.notMeasured); + } + + static String name(int i) { + return String.format(Locale.ROOT, "gen/app/p%03d/C%05d", i / 100, i); + } + + /** gen is a class with a static run(int) of branches if/else pairs, a line each (about 2*branches+1 probes). */ + static byte[] gen(String name, int i, int branches) { + ClassWriter cw = new ClassWriter(ClassWriter.COMPUTE_FRAMES | ClassWriter.COMPUTE_MAXS); + cw.visit(Opcodes.V1_8, Opcodes.ACC_PUBLIC | Opcodes.ACC_SUPER, name, null, "java/lang/Object", null); + cw.visitSource(name.substring(name.lastIndexOf('/') + 1) + ".java", null); + MethodVisitor init = cw.visitMethod(Opcodes.ACC_PUBLIC, "", "()V", null, null); + init.visitCode(); + Label first = new Label(); + init.visitLabel(first); + init.visitLineNumber(5, first); + init.visitVarInsn(Opcodes.ALOAD, 0); + init.visitMethodInsn(Opcodes.INVOKESPECIAL, "java/lang/Object", "", "()V", false); + init.visitInsn(Opcodes.RETURN); + init.visitMaxs(0, 0); + init.visitEnd(); + MethodVisitor mv = cw.visitMethod(Opcodes.ACC_PUBLIC | Opcodes.ACC_STATIC, "run", "(I)I", null, null); + mv.visitCode(); + mv.visitInsn(Opcodes.ICONST_0); + mv.visitVarInsn(Opcodes.ISTORE, 1); + for (int b = 0; b < branches; b++) { + Label line = new Label(); + mv.visitLabel(line); + mv.visitLineNumber(10 + b, line); + Label orElse = new Label(); + Label next = new Label(); + mv.visitVarInsn(Opcodes.ILOAD, 0); + mv.visitIntInsn(Opcodes.SIPUSH, (b * 7 + i) % 64); + mv.visitJumpInsn(Opcodes.IF_ICMPLE, orElse); + mv.visitIincInsn(1, b % 100 + 1); + mv.visitJumpInsn(Opcodes.GOTO, next); + mv.visitLabel(orElse); + mv.visitIincInsn(1, -1); + mv.visitLabel(next); + } + mv.visitVarInsn(Opcodes.ILOAD, 1); + mv.visitInsn(Opcodes.IRETURN); + mv.visitMaxs(0, 0); + mv.visitEnd(); + cw.visitEnd(); + return cw.toByteArray(); + } + + static final class Budget { + int tableRecords; + int probeRecords; + int notMeasured; + } + + /** measure sends the table of classes to a sink that counts, as the table's thread does, and reports. */ + static Budget measure(String label, List classes) throws Exception { + Set heapPools = new HashSet<>(); + for (MemoryPoolMXBean p : ManagementFactory.getMemoryPoolMXBeans()) { + if (p.getType() == MemoryType.HEAP) { + heapPools.add(p.getName()); + } + } + AtomicLong afterGcMax = new AtomicLong(-1); + AtomicInteger collections = new AtomicInteger(); + NotificationListener listener = (n, handback) -> { + if (!GarbageCollectionNotificationInfo.GARBAGE_COLLECTION_NOTIFICATION.equals(n.getType())) { + return; + } + GarbageCollectionNotificationInfo info = GarbageCollectionNotificationInfo.from( + (CompositeData) n.getUserData()); + if ("System.gc()".equals(info.getGcCause())) { + // This method's own collection before the send, whose notification can come late. + return; + } + long used = 0; + for (Map.Entry e : info.getGcInfo().getMemoryUsageAfterGc().entrySet()) { + if (heapPools.contains(e.getKey())) { + used += e.getValue().getUsed(); + } + } + collections.incrementAndGet(); + afterGcMax.accumulateAndGet(used, Math::max); + }; + List emitters = new ArrayList<>(); + for (GarbageCollectorMXBean g : ManagementFactory.getGarbageCollectorMXBeans()) { + if (g instanceof NotificationEmitter) { + ((NotificationEmitter) g).addNotificationListener(listener, null, null); + emitters.add((NotificationEmitter) g); + } + } + try { + return measure(label, classes, heapPools, afterGcMax, collections); + } finally { + for (NotificationEmitter e : emitters) { + e.removeNotificationListener(listener); + } + } + } + + private static Budget measure(String label, List classes, Set heapPools, + AtomicLong afterGcMax, AtomicInteger collections) throws Exception { + System.gc(); + long base = 0; + for (MemoryPoolMXBean p : ManagementFactory.getMemoryPoolMXBeans()) { + if (heapPools.contains(p.getName())) { + base += p.getUsage().getUsed(); + } + } + long classBytes = 0; + for (BuildTable.ClassFile f : classes) { + classBytes += f.bytes.length; + } + ThreadMXBean threads = ManagementFactory.getThreadMXBean(); + long[] bytes = new long[2]; + // A record may run on into the next frame: not_measured is counted over the table's stream, keeping + // the end of each frame that could begin a split match. + String notMeasured = "\"state\":\"not_measured\""; + String[] carry = {""}; + Budget b = new Budget(); + long gc0 = gcMillis(); + long process0 = processCpuNanos(); + long cpu0 = threads.getCurrentThreadCpuTime(); + long t0 = System.nanoTime(); + BuildTable.send(() -> classes, frame -> { + String f = new String(frame, StandardCharsets.UTF_8); + int at = f.indexOf("{\"v\":1,"); + String header = f.substring(at, f.indexOf('\n', at)); + boolean probes = header.contains("\"part\":\"probes\""); + bytes[probes ? 1 : 0] += frame.length; + if (header.contains("\"last\":true")) { + int records = Integer.parseInt(header.replaceAll(".*\"records\":(\\d+).*", "$1")); + if (probes) { + b.probeRecords = records; + } else { + b.tableRecords = records; + } + } + if (!probes) { + String body = carry[0] + f.substring(f.indexOf('\n', at) + 1); + for (int k = body.indexOf(notMeasured); k >= 0; k = body.indexOf(notMeasured, k + 1)) { + b.notMeasured++; + } + carry[0] = body.substring(Math.max(0, body.length() - (notMeasured.length() - 1))); + } + }); + long wall = System.nanoTime() - t0; + long cpu = threads.getCurrentThreadCpuTime() - cpu0; + long process = processCpuNanos() - process0; + long gc = gcMillis() - gc0; + // The JVM sends a collection's notification after it ends, from a thread of its own. + Thread.sleep(200); + long afterGc = afterGcMax.get(); + System.out.printf(Locale.ROOT, "BUDGET %s java=%s heap_max_mb=%d classes=%d class_bytes=%d wall_ms=%d" + + " thread_cpu_ms=%d process_cpu_ms=%d gc_ms=%d collections=%d heap_before_mb=%d" + + " heap_after_gc_max_mb=%d table_bytes=%d probes_bytes=%d not_measured=%d%n", + label, System.getProperty("java.version"), Runtime.getRuntime().maxMemory() >> 20, classes.size(), + classBytes, wall / 1_000_000, cpu / 1_000_000, process / 1_000_000, gc, collections.get(), + base >> 20, afterGc < 0 ? -1 : afterGc >> 20, bytes[0], bytes[1], b.notMeasured); + return b; + } + + private static long gcMillis() { + long t = 0; + for (GarbageCollectorMXBean g : ManagementFactory.getGarbageCollectorMXBeans()) { + t += Math.max(0, g.getCollectionTime()); + } + return t; + } + + private static long processCpuNanos() { + OperatingSystemMXBean os = ManagementFactory.getOperatingSystemMXBean(); + if (os instanceof com.sun.management.OperatingSystemMXBean) { + return ((com.sun.management.OperatingSystemMXBean) os).getProcessCpuTime(); + } + return -1; + } +} diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java index 92ad7be7..34efe025 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java @@ -31,6 +31,7 @@ import java.security.NoSuchAlgorithmException; import java.util.ArrayList; import java.util.Arrays; +import java.util.Collections; import java.util.LinkedHashMap; import java.util.List; import java.util.Map; @@ -103,7 +104,11 @@ static byte[] resource(String name) throws IOException { /** sha256Hex is the sha256 of b as 64 lowercase hex, as a frame's "sha256" says it. */ static String sha256Hex(byte[] b) { try { - return BuildTable.hex(MessageDigest.getInstance("SHA-256").digest(b)); + StringBuilder hex = new StringBuilder(); + for (byte x : MessageDigest.getInstance("SHA-256").digest(b)) { + hex.append(String.format("%02x", x)); + } + return hex.toString(); } catch (NoSuchAlgorithmException e) { throw new IllegalStateException(e); } @@ -574,8 +579,14 @@ static final class Receiver implements AutoCloseable { byte[] body = new byte[d.readInt()]; d.readFully(body); read = body; + // Kept before the OK, as Keploy keeps a frame before it answers: once the sender has + // its OK, count() holds the frame. + synchronized (got) { + got.add(read); + } s.getOutputStream().write("OK\n".getBytes(StandardCharsets.UTF_8)); s.getOutputStream().flush(); + continue; } else if (mode == Mode.READ_512_AND_CLOSE) { byte[] b = new byte[512]; int n = in.read(b); @@ -738,6 +749,127 @@ public void aRequestedTableArrivesWithTheBuildId() throws Exception { } } + @Test(timeout = 30_000) + public void theRunAfterASendIsTheNewestRequests() throws Exception { + // A request names its index and data socket: after KeployDedupAgent.stop() and start() during a send, the + // new server's TABLE must be served from the new index to the new socket, not the first request's again. + CountDownLatch inFirst = new CountDownLatch(1); + CountDownLatch release = new CountDownLatch(1); + List ran = Collections.synchronizedList(new ArrayList<>()); + BuildTable.Sends sends = new BuildTable.Sends(BuildTable::thread); + assertTrue(sends.request(() -> { + ran.add("old"); + inFirst.countDown(); + try { + release.await(10, TimeUnit.SECONDS); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + })); + assertTrue(inFirst.await(10, TimeUnit.SECONDS)); + assertTrue(sends.request(() -> ran.add("middle"))); + assertTrue(sends.request(() -> ran.add("new"))); + release.countDown(); + while (ran.size() < 2) { + Thread.sleep(10); + } + Thread.sleep(300); + assertEquals(Arrays.asList("old", "new"), new ArrayList<>(ran)); + } + + @Test(timeout = 30_000) + public void aRequestMadeWhileASendFailsStillRuns() throws Exception { + // A send that throws is logged; the request made while it ran names the agent's newest index and data + // socket, so it must still run, and Sends must take requests again after it. + CountDownLatch inFirst = new CountDownLatch(1); + CountDownLatch release = new CountDownLatch(1); + CountDownLatch next = new CountDownLatch(1); + CountDownLatch after = new CountDownLatch(1); + List ran = Collections.synchronizedList(new ArrayList<>()); + BuildTable.Sends sends = new BuildTable.Sends(BuildTable::thread); + assertTrue(sends.request(() -> { + ran.add("failing"); + inFirst.countDown(); + try { + release.await(10, TimeUnit.SECONDS); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + throw new IllegalStateException("a send that fails (this row's own)"); + })); + assertTrue(inFirst.await(10, TimeUnit.SECONDS)); + assertTrue(sends.request(() -> { + ran.add("next"); + next.countDown(); + })); + release.countDown(); + assertTrue("the request made while the send failed ran", next.await(10, TimeUnit.SECONDS)); + assertTrue(sends.request(() -> { + ran.add("after"); + after.countDown(); + })); + assertTrue("a request after them runs", after.await(10, TimeUnit.SECONDS)); + assertEquals(Arrays.asList("failing", "next", "after"), new ArrayList<>(ran)); + } + + /** Throwing is a JUL handler on BuildTable's logger that throws on every record, as an app's bridge can. */ + static final class Throwing extends Handler implements AutoCloseable { + final AtomicInteger published = new AtomicInteger(); + private final Logger logger = Logger.getLogger(BuildTable.class.getName()); + + Throwing() { + logger.addHandler(this); + } + + @Override + public void publish(LogRecord record) { + published.incrementAndGet(); + throw new IllegalStateException("a log handler that throws (this row's own)"); + } + + @Override + public void flush() { + } + + @Override + public void close() { + logger.removeHandler(this); + } + } + + @Test(timeout = 60_000) + public void aLogHandlerThatThrowsNeitherFailsASendNorWedgesSends() throws Exception { + List classes = javaDedup(); + try (Throwing h = new Throwing()) { + // send logs what it sent (INFO) or why it stopped (WARNING), and never throws. + List frames = Collections.synchronizedList(new ArrayList<>()); + BuildTable.send(() -> classes, frames::add); + assertFalse("the table was sent", frames.isEmpty()); + BuildTable.send(() -> { + throw new IllegalStateException("classes that cannot be read (this row's own)"); + }, f -> fail("nothing to send")); + assertEquals("both sends logged through the handler", 2, h.published.get()); + // Sends logs a send that throws through the same handler: the request made after it still runs. + BuildTable.Sends sends = new BuildTable.Sends(BuildTable::thread); + CountDownLatch failing = new CountDownLatch(1); + assertTrue(sends.request(() -> { + failing.countDown(); + throw new IllegalStateException("a send that fails (this row's own)"); + })); + assertTrue(failing.await(10, TimeUnit.SECONDS)); + CountDownLatch next = new CountDownLatch(1); + assertTrue(sends.request(next::countDown)); + assertTrue("a request after a send that failed runs", next.await(10, TimeUnit.SECONDS)); + // The production send through Sends, then one more request. + CountDownLatch sent = new CountDownLatch(1); + assertTrue(sends.request(() -> BuildTable.send(() -> classes, frame -> sent.countDown()))); + assertTrue(sent.await(20, TimeUnit.SECONDS)); + CountDownLatch last = new CountDownLatch(1); + assertTrue(sends.request(last::countDown)); + assertTrue("a request after the production send runs", last.await(20, TimeUnit.SECONDS)); + } + } + @Test(timeout = 30_000) public void aRequestDuringASendRunsItOnceMore() throws Exception { CountDownLatch inFirst = new CountDownLatch(1); diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/ControlSocketTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/ControlSocketTest.java index 19e00745..541964be 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/ControlSocketTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/ControlSocketTest.java @@ -273,6 +273,73 @@ public void theIndexRecordAndTheTableNameOneBuild() throws Exception { probesDone |= last; } } + assertNull("nothing more", data.payloads.poll(1, TimeUnit.SECONDS)); + } + + @Test(timeout = 300000) + public void startAndEndAreAnsweredWhileTheTableIsAnalyzed() throws Exception { + // Keploy sends TABLE after the first ACK and goes on with its tests: the table's analysis runs on its own + // thread, so no START or END waits for it. The index is a generated app large enough that its analysis + // outlasts many START/END pairs. + File classes = new File(dir, "classes"); + int n = 5000; + for (int i = 0; i < n; i++) { + String name = BuildTableBudgetTest.name(i); + File out = new File(classes, name + ".class"); + assertTrue(out.getParentFile().isDirectory() || out.getParentFile().mkdirs()); + Files.write(out.toPath(), BuildTableBudgetTest.gen(name, i, BuildTableBudgetTest.BRANCHES)); + } + System.setProperty(CLASS_DIRS_PROPERTY, classes.getPath()); + data = new DataReceiver(KeployDedupAgent.dataSocketPath); + startWithJacoco(); + assertNotNull("the INDEX record (the index is read)", data.indexes.poll(60, TimeUnit.SECONDS)); + assertEquals(TAKEN, send("START test-set-0/test-0")); + assertEquals(TAKEN, send("END test-set-0/test-0")); + assertNotNull(data.next()); + + long t0 = System.nanoTime(); + assertEquals(TAKEN, send("TABLE")); + long tableAck = System.nanoTime() - t0; + boolean tableDone = false; + boolean probesDone = false; + int during = 0; + long worst = 0; + for (int k = 1; !tableDone || !probesDone; k++) { + long s = System.nanoTime(); + assertEquals(TAKEN, send("START test-set-0/test-" + k)); + assertEquals(TAKEN, send("END test-set-0/test-" + k)); + worst = Math.max(worst, System.nanoTime() - s); + boolean ended = false; + for (String p; (p = data.payloads.poll(ended ? 0 : 30, TimeUnit.SECONDS)) != null; ) { + if (!p.startsWith("KTB1")) { + assertTrue(p, p.contains("\"id\":\"test-set-0/test-" + k + "\"")); + ended = true; + continue; + } + int at = p.indexOf("{\"v\":1,"); + String header = p.substring(at, p.indexOf('\n', at)); + boolean last = header.contains("\"last\":true"); + tableDone |= last && header.contains("\"part\":\"table\""); + probesDone |= last && header.contains("\"part\":\"probes\""); + } + assertTrue("the END's coverage reached the data socket", ended); + if (!tableDone || !probesDone) { + during++; + } + } + long table = System.nanoTime() - t0; + System.out.printf("START/END pairs answered while the table of %d classes was analyzed and sent: %d in %d ms;" + + " slowest pair %d ms; TABLE answered in %d ms%n", n, during, table / 1_000_000, worst / 1_000_000, + tableAck / 1_000_000); + assertTrue("START and END were answered while the table was analyzed: " + during + " pairs in " + + table / 1_000_000 + " ms", during >= 3); + // The bounds are generous (a pair takes about 0.1 s): they fail when a START/END or TABLE waits on a + // slow analysis. A fast runner can finish the analysis inside them; during >= 3 above is what fails + // when the control thread does the analysis itself. + assertTrue("TABLE itself was answered at once: " + tableAck / 1_000_000 + " ms", + tableAck < TimeUnit.SECONDS.toNanos(5)); + assertTrue("no START/END pair waited on the analysis: the slowest took " + worst / 1_000_000 + " ms", + worst < TimeUnit.SECONDS.toNanos(5)); } // field is the string value of name in one JSON object's text. diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/IndexReportTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/IndexReportTest.java index bdc90720..0388cea7 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/IndexReportTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/IndexReportTest.java @@ -51,7 +51,7 @@ public void theBuildIdIsKeploysForTheSameIdsInAnyOrderAndRepeated() { @Test public void theUnitHashIsTheWholeSumAndTheBuildIdItsFirstHalf() { assertEquals(IDS_SUM, IndexReport.classIdsSum(new long[] {0xffffffffffffffffL, 1, 0xff, 1})); - assertEquals(GO_BUILD_ID, "jv1-" + IDS_SUM.substring(0, 32)); + assertEquals(GO_BUILD_ID, IndexReport.buildId(new long[] {0xffffffffffffffffL, 1, 0xff, 1})); } @Test diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/ProbeMapTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/ProbeMapTest.java index 06049ebf..50b91671 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/ProbeMapTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/ProbeMapTest.java @@ -11,6 +11,7 @@ import org.jacoco.core.data.ExecutionData; import org.jacoco.core.data.ExecutionDataStore; import org.jacoco.core.instr.Instrumenter; +import org.jacoco.core.internal.analysis.KeployBranchCoverage; import org.jacoco.core.internal.data.CRC64; import org.jacoco.core.runtime.IExecutionDataAccessorGenerator; import org.junit.Test; @@ -22,6 +23,7 @@ import java.io.ByteArrayOutputStream; import java.io.File; import java.io.IOException; +import java.io.InputStream; import java.net.URL; import java.nio.charset.StandardCharsets; import java.nio.file.Files; @@ -37,6 +39,8 @@ import java.util.TreeMap; import java.util.stream.Collectors; import java.util.stream.Stream; +import java.util.zip.ZipEntry; +import java.util.zip.ZipFile; import static org.junit.Assert.assertArrayEquals; import static org.junit.Assert.assertEquals; @@ -106,6 +110,117 @@ public void javaDedupProbeMapIsTheGolden() throws Exception { } } + @Test + public void aMethodTheCallerDeclinesIsNotReplayed() { + // The map declines the methods the table has no line of (a class built without -g): their probes cover + // no leaf, so their replays would be thrown away. The class pass still analyzes them. + byte[] bytes = BuildTableBudgetTest.gen("gen/app/Declined", 0, 40); + Map shape = new HashMap<>(); + List replayed = new ArrayList<>(); + int total = KeployBranchCoverage.analyze(bytes, shape, new KeployBranchCoverage.EachProbe() { + @Override + public boolean wants(KeployBranchCoverage.Method m) { + return !m.name.equals("run"); + } + + @Override + public void probe(KeployBranchCoverage.Method m, int r, KeployBranchCoverage.Method got) { + replayed.add(m.name); + } + }); + assertTrue(shape.containsKey("run(I)I")); + assertTrue("run has probes to replay", shape.get("run(I)I").probes.length > 40); + assertEquals(total - shape.get("run(I)I").probes.length, replayed.size()); + assertFalse("a declined method is not replayed", replayed.contains("run")); + assertTrue(replayed.contains("")); + } + + @Test(timeout = 300000) + public void eachProbesReplayIsTheClassAnalyzedWithThatProbeExecuted() throws Exception { + // The oracle is the class analyzed whole, once per round r, with the r-th probe of every method executed + // (a method's coverage reads only its own probes). Against it, each probe's one-pass replay, on real + // libraries' classes: JaCoCo core's and Gson's, the jars on this test's classpath. + int classes = 0; + int probes = 0; + for (Class c : new Class[] {Analyzer.class, JsonParser.class}) { + File jar = new File(c.getProtectionDomain().getCodeSource().getLocation().toURI()); + assertTrue(jar + " is a jar", jar.isFile()); + for (BuildTable.ClassFile f : jarClasses(jar)) { + Map shape = new HashMap<>(); + Map> replayed = new HashMap<>(); + int total = KeployBranchCoverage.analyze(f.bytes, shape, (m, r, got) -> { + List l = replayed.computeIfAbsent(m.name + m.desc, + k -> new ArrayList<>()); + assertEquals(f.name + "#" + m.name + m.desc, r, l.size()); + l.add(got); + }); + if (total < 0) { + continue; + } + classes++; + int rounds = 0; + int replays = 0; + for (Map.Entry e : shape.entrySet()) { + int n = e.getValue().probes.length; + rounds = Math.max(rounds, n); + replays += n; + List l = replayed.get(e.getKey()); + assertEquals(f.name + "#" + e.getKey(), n, l == null ? 0 : l.size()); + } + // A method whose code the filters leave out entirely (an enum's values()) keeps its probes: no leaf. + assertTrue(f.name + ": " + replays + " probes replayed of " + total, replays <= total); + for (int r = 0; r < rounds; r++) { + boolean[] set = new boolean[total]; + for (KeployBranchCoverage.Method m : shape.values()) { + if (m.probes.length > r) { + set[m.probes[r]] = true; + } + } + Map run = new HashMap<>(); + KeployBranchCoverage.analyze(f.bytes, set, run); + for (Map.Entry e : shape.entrySet()) { + if (e.getValue().probes.length <= r) { + continue; + } + KeployBranchCoverage.Method want = run.get(e.getKey()); + KeployBranchCoverage.Method got = replayed.get(e.getKey()).get(r); + String at = f.name + "#" + e.getKey() + " probe " + r; + assertArrayEquals(at, want.lines, got.lines); + assertArrayEquals(at, want.covered, got.covered); + assertEquals(at, want.branches.length, got.branches.length); + for (int k = 0; k < want.branches.length; k++) { + assertArrayEquals(at, want.branches[k], got.branches[k]); + } + probes++; + } + } + } + } + assertTrue("classes: " + classes, classes > 300); + assertTrue("probes: " + probes, probes > 5000); + } + + static List jarClasses(File jar) throws IOException { + List out = new ArrayList<>(); + try (ZipFile z = new ZipFile(jar)) { + for (ZipEntry e : Collections.list(z.entries())) { + String name = e.getName(); + if (!name.endsWith(".class") || name.startsWith("META-INF/")) { + continue; + } + ByteArrayOutputStream b = new ByteArrayOutputStream(); + try (InputStream in = z.getInputStream(e)) { + byte[] buf = new byte[8192]; + for (int k; (k = in.read(buf)) > 0; ) { + b.write(buf, 0, k); + } + } + out.add(new BuildTable.ClassFile(name.substring(0, name.length() - 6), b.toByteArray())); + } + } + return out; + } + @Test public void javaDedupProbesCoverWhatJacocoCounts() throws Exception { agree(BuildTableTest.javaDedup()); From 46ce07cc675fc1172f20e8714ec99010e995d3c9 Mon Sep 17 00:00:00 2001 From: slayerjain Date: Fri, 20 Nov 2026 05:31:05 +0530 Subject: [PATCH 05/14] feat(dedup): stream the probe map and decline TABLE on a short heap The app's JVM builds the coverage build table and probe map when Keploy asks (TABLE). The table already went out record by record, but every probe-map record stayed in the heap until the table's last frame, as the map streams after the table: about 25 MB more live heap for 50,000 classes, and more for classes with more probes. An app with a tight -Xmx (a k8s pod) could run out of heap while the table was built. Each measured class's probe records now go, as they are made, to a temp file in java.io.tmpdir (owner-only, opened without following a link, unlinked once open on Unix, so a JVM that dies mid-send leaves none and one that dies before the open leaves an empty file), which is streamed back as the probe map's frames after the table's last frame. The wire is unchanged. Each class is dropped once written and the grouping's lookups end before the analysis, so a send holds the class list, one frame and one class's analysis. A tmpdir the app cannot write ends the send before any frame, with a WARNING naming the dir: Keploy gets no table for that build and asks again on its next run. When TABLE arrives the SDK reads the free heap: the heap cap the JVM reports (a little under -Xmx with the Serial and Parallel collectors) less the heap in use. That is what the JVM's newest collection left in the heap pools (the collectors' last GcInfo), or the heap in use now when that is smaller, when no collection ran yet or when the runtime does not report one. The SDK never runs a collection. With less free than 256 MiB it answers IGNORED proto=2 reason=low_heap heap_free_mb=F heap_needed_mb=N logs a WARNING and sends nothing, so Keploy can mark the run not measured with that reason. KEPLOY_COVERAGE_TABLE_MIN_HEAP_MB, else -Dkeploy.coverage.table.min.heap.mb, sets the bound in MiB; 0 turns the check off. A Keploy that does not read the reason sees IGNORED, as from an SDK that sends no table. Measured on 50,000 generated classes, against the parent commit in the same run: the live heap at each frame (after a full collection) is 18-22 MB above the heap in use before the send (the classes' bytes, the list of them the send is handed and the harness), from 44-47 MB (JDK 8 and 21); what is left is the send's own entry for each class, about 0.4 KB per class. Time is unchanged: 25.3-26.2 s wall, 24.7-25.4 s on the analysis thread at -Xmx1g (JDK 8/17/21). With -Xmx96m it takes 44 s, from 71 s, and with -Xmx80m it completes in 48 s where the parent ran out of heap (JDK 21). Rows: a measured class's probe records reach their sink just before its own table record; the map waits in a temp file that is gone during and after the send, and both parts still match the java-dedup goldens; a tmpdir that cannot be written sends no frame and names it; the temp file is never opened through a link; the low_heap answer at and around the bound, with 0 and no cap; the env, property and default of the bound; the heap in use is the newest collection's, heap pools only, or the heap in use now when smaller; and through the control socket, a TABLE with too little heap is answered low_heap, starts no send and sends no frame (it failed first: ACK). The other TABLE rows turn the bound off, so the test JVM's -Xmx does not decide them. The budget row also reports the heap delta, the pools' peak and, with -Dkeploy.budget.live=true, the live set at each frame. Signed-off-by: slayerjain --- .../main/java/io/keploy/dedup/BuildTable.java | 428 ++++++++++++++++-- .../io/keploy/dedup/KeployDedupAgent.java | 8 +- .../io/keploy/dedup/BuildTableBudgetTest.java | 40 +- .../java/io/keploy/dedup/BuildTableTest.java | 202 +++++++++ .../io/keploy/dedup/ControlSocketTest.java | 42 ++ .../java/io/keploy/dedup/ProbeMapTest.java | 13 +- 6 files changed, 679 insertions(+), 54 deletions(-) diff --git a/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java b/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java index 597016d1..51eb3915 100644 --- a/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java +++ b/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java @@ -15,13 +15,26 @@ import org.objectweb.asm.MethodVisitor; import org.objectweb.asm.Opcodes; +import java.io.BufferedOutputStream; import java.io.BufferedReader; import java.io.ByteArrayOutputStream; import java.io.File; import java.io.IOException; +import java.io.InputStream; import java.io.InputStreamReader; import java.io.OutputStream; +import java.lang.management.GarbageCollectorMXBean; +import java.lang.management.ManagementFactory; +import java.lang.management.MemoryPoolMXBean; +import java.lang.management.MemoryType; +import java.lang.management.MemoryUsage; +import java.nio.channels.Channels; +import java.nio.channels.FileChannel; import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.LinkOption; +import java.nio.file.Path; +import java.nio.file.StandardOpenOption; import java.security.MessageDigest; import java.security.NoSuchAlgorithmException; import java.util.ArrayList; @@ -67,6 +80,12 @@ * covers. Keploy turns each test's probe hits, which name a class by its name, into its leaves with it; it is * never uploaded. Two class files of one name whose bytes differ share the record when they map their probes * alike, and fail their class closed (not measured) when not. + * + *

The send holds no whole table in the heap: each class's table record goes into the frame being filled + * as it is made, and its probe-map records into a temp file (owner-only, unlinked once open where the OS + * allows), which is streamed back as the probe map's frames once the table's last frame is sent. So what a + * send keeps is the class list (names and ids, no bytes of its own), one frame and one class's analysis. + * A TABLE asked when the heap is short is not taken: see {@link #answer}. */ final class BuildTable { @@ -129,9 +148,187 @@ static List files(List entries, Function name, Func return out; } + /** The env var, then the system property, that set the free heap a table needs, in MiB; 0 turns it off. */ + static final String MIN_HEAP_ENV = "KEPLOY_COVERAGE_TABLE_MIN_HEAP_MB"; + static final String MIN_HEAP_PROPERTY = "keploy.coverage.table.min.heap.mb"; + /** The free heap a table needs when neither is set, in MiB. */ + static final long DEFAULT_MIN_HEAP_MB = 256; + /** The reason a TABLE ask is declined when the heap is short: Keploy marks the run not_measured with it. */ + static final String LOW_HEAP = "low_heap"; + + /** + * answer is the control socket's reply to Keploy's TABLE command. When the heap has less free than the + * table needs ({@link #lowHeap}) the analysis is skipped and the reply is {@code IGNORED proto=2 + * reason=low_heap heap_free_mb=F heap_needed_mb=N}, with a WARNING in the app's log; else it is + * {@link #request}'s ACK or IGNORED. The heap is read once, here, when Keploy asks: a send that runs after + * another one was checked when it was asked. It never blocks. + */ + static String answer(Supplier> classes, File dataSocket) { + String low = lowHeap(Heap.read(), minHeapMb(System.getenv(MIN_HEAP_ENV), + System.getProperty(MIN_HEAP_PROPERTY))); + if (low != null) { + return KeployDedupAgent.UNIX_REPLY_IGNORED + " " + low; + } + return request(classes, dataSocket) ? KeployDedupAgent.UNIX_REPLY_TAKEN + : KeployDedupAgent.UNIX_REPLY_IGNORED; + } + + /** + * lowHeap is null when heap has at least minMb MiB free, else what the reply names after IGNORED proto=2 + * ({@code reason=low_heap heap_free_mb=F heap_needed_mb=N}), logged at WARNING. + */ + static String lowHeap(Heap heap, long minMb) { + long free = heap.freeMb(); + if (free >= minMb) { + return null; + } + log(Level.WARNING, "Keploy coverage: the build table is not built, as the heap is short: " + free + + " MiB free of the " + (heap.max >> 20) + " MiB heap cap the JVM reports (in use " + (heap.used >> 20) + + " MiB" + (heap.afterGc ? " after the last collection" : " now") + "), and it needs " + + minMb + " MiB. Keploy marks this run's coverage not measured (" + LOW_HEAP + "). Give the JVM" + + " more heap (-Xmx), or set " + MIN_HEAP_ENV + " (0 turns the check off)", null); + return "reason=" + LOW_HEAP + " heap_free_mb=" + free + " heap_needed_mb=" + minMb; + } + + /** + * minHeapMb is the free heap a table needs, in MiB: env, else property (as the agent reads its settings), + * else {@link #DEFAULT_MIN_HEAP_MB}. A value that is not a whole number of 0 or more is logged and the + * default holds. + */ + static long minHeapMb(String env, String property) { + String v = env == null || env.trim().isEmpty() ? property : env; + if (v == null || v.trim().isEmpty()) { + return DEFAULT_MIN_HEAP_MB; + } + try { + long mb = Long.parseLong(v.trim()); + if (mb >= 0) { + return mb; + } + } catch (NumberFormatException ignored) { + // logged below + } + log(Level.WARNING, "Keploy coverage: " + MIN_HEAP_ENV + " (or -D" + MIN_HEAP_PROPERTY + ") is '" + v.trim() + + "', not a whole number of MiB (0 or more); the build table needs " + DEFAULT_MIN_HEAP_MB + + " MiB of free heap", null); + return DEFAULT_MIN_HEAP_MB; + } + + /** + * Heap is what the low-heap check reads: the heap cap and the heap in use. In use is what the JVM's newest + * collection left (the live set and any garbage it did not reach); with no collection seen, or a JVM that + * does not report one, it is the heap in use now, garbage included. The check never runs a collection: a + * System.gc() would pause the app. + */ + static final class Heap { + /** + * The heap cap in bytes, as Runtime.maxMemory() reports it (a little under -Xmx with the Serial and + * Parallel collectors, which leave out a survivor space); Long.MAX_VALUE when the JVM has none. + */ + final long max; + final long used; + /** afterGc is true when used is what the newest collection left, false when it is the heap in use now. */ + final boolean afterGc; + + Heap(long max, long used, boolean afterGc) { + this.max = max; + this.used = used; + this.afterGc = afterGc; + } + + long freeMb() { + return max == Long.MAX_VALUE ? Long.MAX_VALUE : Math.max(0, max - used) >> 20; + } + + static Heap read() { + Runtime rt = Runtime.getRuntime(); + long now = rt.totalMemory() - rt.freeMemory(); + long after; + try { + after = LastCollection.heapUsedAfter(); + } catch (Throwable t) { + // No com.sun.management (a trimmed runtime) or a JMX error: the heap as it is now. + after = -1; + } + return of(rt.maxMemory(), now, after); + } + + /** + * of is the heap with cap max, now bytes in use now and after bytes in use when the newest collection + * ended (-1 with none seen): the smaller of the two. More in use now is garbage and objects made since + * that collection; less means a collection the JVM did not report ran since, so now is the newer figure. + */ + static Heap of(long max, long now, long after) { + return after < 0 || after > now ? new Heap(max, now, false) : new Heap(max, after, true); + } + + /** + * usedAfterNewest is the heap in use when the newest of ends ended: its use after it, summed over + * heapPools only (the JVM also reports Metaspace and its code cache, which are not the heap); -1 when + * ends is empty. + */ + static long usedAfterNewest(List ends, Set heapPools) { + GcEnd newest = null; + for (GcEnd e : ends) { + if (newest == null || e.endTime > newest.endTime) { + newest = e; + } + } + if (newest == null) { + return -1; + } + long used = 0; + for (Map.Entry p : newest.usedAfter.entrySet()) { + if (heapPools.contains(p.getKey())) { + used += p.getValue(); + } + } + return used; + } + } + + /** GcEnd is one collector's last collection: when it ended (ms since JVM start) and each pool's use after it. */ + static final class GcEnd { + final long endTime; + final Map usedAfter; + + GcEnd(long endTime, Map usedAfter) { + this.endTime = endTime; + this.usedAfter = usedAfter; + } + } + + /** LastCollection reads com.sun.management, in a class of its own so a runtime without it fails only here. */ + static final class LastCollection { + /** heapUsedAfter is the heap in use when the JVM's newest collection ended, or -1 with none seen. */ + static long heapUsedAfter() { + Set heapPools = new HashSet<>(); + for (MemoryPoolMXBean p : ManagementFactory.getMemoryPoolMXBeans()) { + if (p.getType() == MemoryType.HEAP) { + heapPools.add(p.getName()); + } + } + List ends = new ArrayList<>(); + for (GarbageCollectorMXBean g : ManagementFactory.getGarbageCollectorMXBeans()) { + if (g instanceof com.sun.management.GarbageCollectorMXBean) { + com.sun.management.GcInfo info = + ((com.sun.management.GarbageCollectorMXBean) g).getLastGcInfo(); + if (info != null) { + Map usedAfter = new HashMap<>(); + for (Map.Entry e : info.getMemoryUsageAfterGc().entrySet()) { + usedAfter.put(e.getKey(), e.getValue().getUsed()); + } + ends.add(new GcEnd(info.getEndTime(), usedAfter)); + } + } + } + return Heap.usedAfterNewest(ends, heapPools); + } + } + /** - * request answers Keploy's TABLE command: true (ACK) once the table's send is started on its own thread, - * or, when a send runs already, once this request is set to run after it. It never blocks. + * request starts the table's send: true (ACK) once it is started on its own thread, or, when a send runs + * already, once this request is set to run after it. It never blocks. * When no thread can start it answers false (IGNORED) and does not throw; the agent's command hook catches * anything else. The classes are read on that thread. */ @@ -139,7 +336,10 @@ static boolean request(Supplier> classes, File dataSocket) { return JVM.request(() -> send(classes, new SocketSender(dataSocket))); } - /** requests is how many times the JVM's table was asked for, answered or not: only TABLE asks. */ + /** + * requests is how many times the JVM's table send was asked for, started or not: only TABLE asks the heap + * check let through. + */ static int requests() { return JVM.requests(); } @@ -222,21 +422,28 @@ private void loop(Runnable first) { } /** - * send analyzes the classes and streams the table through sender. It logs what it sent, or why it stopped, - * and never throws. + * send analyzes the classes and streams the table through sender, then the probe map, which waits in a + * temp file in java.io.tmpdir while the table streams ({@link Spill}). It logs what it sent, or why it + * stopped, and never throws. */ static void send(Supplier> classes, FrameSender sender) { + send(classes, sender, new File(System.getProperty("java.io.tmpdir", "/tmp"))); + } + + /** send with the probe map's temp file made in spillDir. */ + static void send(Supplier> classes, FrameSender sender, File spillDir) { try { long cpu0 = threadCpuNanos(); long t0 = System.nanoTime(); FrameWriter frames = new FrameWriter(sender, PART_TABLE, MAX_FRAME_BYTES); - Summary s = write(classes.get(), frames); - frames.finish(s.build, s.records); FrameWriter maps = new FrameWriter(sender, PART_PROBES, MAX_FRAME_BYTES); - for (byte[] p : s.probes) { - maps.record(s.build, p); + Summary s; + try (Spill spill = Spill.open(spillDir)) { + s = write(classes.get(), frames, spill); + frames.finish(s.build, s.records); + spill.sendTo(maps, s.build); } - maps.finish(s.build, s.probes.size()); + maps.finish(s.build, s.probeRecords); StringBuilder m = new StringBuilder("Keploy coverage: build table "); m.append(s.build.isEmpty() ? "(no app classes)" : s.build).append(" sent (").append(s.records) .append(" classes, ").append(s.notMeasured).append(" not measured"); @@ -252,7 +459,7 @@ static void send(Supplier> classes, FrameSender sender) { if (s.unnamed > 0) { m.append(", ").append(s.unnamed).append(" index entries without a class name left out"); } - m.append("; probe map of ").append(s.probes.size()).append(" class files; ").append(frames.seq + maps.seq) + m.append("; probe map of ").append(s.probeRecords).append(" class files; ").append(frames.seq + maps.seq) .append(" frames, ").append((System.nanoTime() - t0) / 1_000_000) .append(" ms, cpu ").append((threadCpuNanos() - cpu0) / 1_000_000).append(" ms)"); log(Level.INFO, m.toString(), null); @@ -287,18 +494,18 @@ static final class Summary { final int unnamed; /** How many of the not measured records JaCoCo analyzed but whose probes could not be mapped. */ final int notMapped; - /** The probe map's records, in the table's order: one per class name of each measured unit. */ - final List probes; + /** How many probe-map records went to the probes sink: one per class name of each measured unit. */ + final int probeRecords; Summary(String build, int records, int notMeasured, int noLines, int unnamed, int notMapped, - List probes) { + int probeRecords) { this.build = build; this.records = records; this.notMeasured = notMeasured; this.noLines = noLines; this.unnamed = unnamed; this.notMapped = notMapped; - this.probes = probes; + this.probeRecords = probeRecords; } } @@ -307,15 +514,79 @@ interface RecordSink { void record(String build, byte[] json) throws IOException; } + /** A sink that drops what it is given: write's probe map when only the table is wanted. */ + static final RecordSink DROP = (build, json) -> { }; + /** - * write analyzes the classes and hands each table record to sink, in the contract's order. With no class - * bytes the build id is "" and there is no record. + * write analyzes the classes and hands each table record to sink, in the contract's order, and each + * measured class's probe-map records to probes as they are made, in the same order: it keeps neither. With + * no class bytes the build id is "" and there is no record. */ + static Summary write(List classes, RecordSink sink, RecordSink probes) throws IOException { + return write(classes, sink, probes, MAX_RECORD_BYTES); + } + + /** write with the probe map dropped. */ static Summary write(List classes, RecordSink sink) throws IOException { - return write(classes, sink, MAX_RECORD_BYTES); + return write(classes, sink, DROP, MAX_RECORD_BYTES); } static Summary write(List classes, RecordSink sink, int maxRecord) throws IOException { + return write(classes, sink, DROP, maxRecord); + } + + static Summary write(List classes, RecordSink sink, RecordSink probes, int maxRecord) + throws IOException { + Grouped g = group(classes); + if (g.build.isEmpty()) { + return new Summary("", 0, 0, 0, g.unnamed, 0, 0); + } + String build = g.build; + List units = g.units; + int notMeasured = 0; + int noLines = 0; + int notMapped = 0; + int probeRecords = 0; + for (int i = 0; i < units.size(); i++) { + // A unit written is dropped: what stays is the class list (names, ids), not the table. + Unit u = units.set(i, null); + byte[] rec = u.record(); + if (rec.length + 1 > maxRecord) { + rec = u.notMeasured("record_too_large"); + u.failed = true; + } + if (u.failed) { + notMeasured++; + if (u.notMapped) { + notMapped++; + } + } else { + noLines += u.noLines; + for (byte[] p : u.probes) { + probes.record(build, p); + probeRecords++; + } + } + sink.record(build, rec); + } + return new Summary(build, units.size(), notMeasured, noLines, g.unnamed, notMapped, probeRecords); + } + + /** What group makes of the index's classes: the build id ("" with none), the units in table order. */ + private static final class Grouped { + final String build; + final List units; + final int unnamed; + + Grouped(String build, List units, int unnamed) { + this.build = build; + this.units = units; + this.unnamed = unnamed; + } + } + + /** group folds the classes into units by name, in table order; its lookups end with it. */ + private static Grouped group(List classes) { Map byId = new HashMap<>(); Set seen = new HashSet<>(); long[] ids = new long[classes.size()]; @@ -348,33 +619,11 @@ static Summary write(List classes, RecordSink sink, int maxRecord) th had.files.add(new UnitFile(f, id)); } if (n == 0) { - return new Summary("", 0, 0, 0, unnamed, 0, Collections.emptyList()); + return new Grouped("", Collections.emptyList(), unnamed); } - String build = IndexReport.buildId(Arrays.copyOf(ids, n)); List units = new ArrayList<>(byId.values()); Collections.sort(units, UNIT_ORDER); - int notMeasured = 0; - int noLines = 0; - int notMapped = 0; - List probes = new ArrayList<>(); - for (Unit u : units) { - byte[] rec = u.record(); - if (rec.length + 1 > maxRecord) { - rec = u.notMeasured("record_too_large"); - u.failed = true; - } - if (u.failed) { - notMeasured++; - if (u.notMapped) { - notMapped++; - } - } else { - noLines += u.noLines; - probes.addAll(u.probes); - } - sink.record(build, rec); - } - return new Summary(build, units.size(), notMeasured, noLines, unnamed, notMapped, probes); + return new Grouped(IndexReport.buildId(Arrays.copyOf(ids, n)), units, unnamed); } /** @@ -1185,6 +1434,13 @@ private void write(byte[] b, int off, int len) throws IOException { } } + /** bytes adds whole records, each ended by its newline, to the stream as they are. */ + void bytes(String build, byte[] b, int off, int len) throws IOException { + this.build = build; + digest.update(b, off, len); + write(b, off, len); + } + /** finish sends the last frame, with the record count and the stream's sha256. */ void finish(String build, int records) throws IOException { this.build = build; @@ -1223,6 +1479,94 @@ private void flush(boolean last, int records) throws IOException { } } + /** + * Spill keeps the probe map's records while the table streams, as the wire sends the map after the table's + * last frame: in a temp file made owner-only, which is unlinked as soon as it is open where the OS allows + * (so a JVM that dies mid-send leaves no file; one that dies between the file's making and its opening + * leaves it empty), and is removed on close otherwise. It is opened without following a link, so a link + * put in its place is never written through. A dir where no such file + * can be made, or a write that fails (a full disk), ends the send before the table's last frame: Keploy + * gets no table for that build and names it (table_incomplete), and the WARNING names the dir. + */ + static final class Spill implements RecordSink, AutoCloseable { + private final File dir; + private final Path path; + private final FileChannel channel; + private final OutputStream out; + + private Spill(File dir, Path path, FileChannel channel) { + this.dir = dir; + this.path = path; + this.channel = channel; + this.out = new BufferedOutputStream(Channels.newOutputStream(channel), 1 << 16); + } + + static Spill open(File dir) throws IOException { + Path path; + try { + path = Files.createTempFile(dir.toPath(), "keploy-probe-map-", ".ndjson"); + } catch (IOException | RuntimeException e) { + throw new IOException("no temp file for the probe map in " + dir + " (" + e + + "); set java.io.tmpdir to a dir the app can write", e); + } + try { + return new Spill(dir, path, openOwned(path)); + } catch (IOException | RuntimeException e) { + deleteQuietly(path); + throw new IOException("could not open the probe map's temp file " + path + " (" + e + ")", e); + } + } + + /** + * openOwned opens the file just made at path to write and read back, never through a link (a link put + * in its place fails the open), and deleted on close. + */ + static FileChannel openOwned(Path path) throws IOException { + return FileChannel.open(path, StandardOpenOption.READ, StandardOpenOption.WRITE, + StandardOpenOption.DELETE_ON_CLOSE, LinkOption.NOFOLLOW_LINKS); + } + + @Override + public void record(String build, byte[] json) throws IOException { + try { + out.write(json); + out.write('\n'); + } catch (IOException e) { + throw new IOException("could not keep the probe map in a temp file in " + dir + ": " + + e.getMessage(), e); + } + } + + /** sendTo streams what was kept into w, from its first byte. */ + void sendTo(FrameWriter w, String build) throws IOException { + out.flush(); + channel.position(0); + InputStream in = Channels.newInputStream(channel); + byte[] buf = new byte[1 << 16]; + for (int n = in.read(buf); n >= 0; n = in.read(buf)) { + w.bytes(build, buf, 0, n); + } + } + + @Override + public void close() { + try { + channel.close(); + } catch (IOException ignored) { + // the file is removed below either way + } + deleteQuietly(path); + } + + private static void deleteQuietly(Path path) { + try { + Files.deleteIfExists(path); + } catch (IOException | RuntimeException ignored) { + // best effort: the file is owner-only, and gone already where it was unlinked once open + } + } + } + /** * SocketSender sends each frame on its own connection to the data socket, shuts its output so a receiver * reading to the end of the connection is never left waiting, and waits for OK. diff --git a/keploy-sdk/src/main/java/io/keploy/dedup/KeployDedupAgent.java b/keploy-sdk/src/main/java/io/keploy/dedup/KeployDedupAgent.java index b39d02a0..1af21954 100644 --- a/keploy-sdk/src/main/java/io/keploy/dedup/KeployDedupAgent.java +++ b/keploy-sdk/src/main/java/io/keploy/dedup/KeployDedupAgent.java @@ -808,15 +808,15 @@ private void handle(Socket socket) { } if (BuildTable.COMMAND.equals(line.trim())) { // the build table: ACK, then sent on its own thread - boolean asked; + String reply; // or IGNORED, with reason=low_heap when the heap is short try { - asked = BuildTable.request(() -> BuildTable.files(collector.coverageIndex.entries(), + reply = BuildTable.answer(() -> BuildTable.files(collector.coverageIndex.entries(), e -> e.className, e -> e.bytes), new File(dataSocketPath)); } catch (Throwable t) { // thrown past here, it would end the accept loop: IGNORED instead log(Level.WARNING, "Keploy coverage: the build table was not started", t); - asked = false; + reply = UNIX_REPLY_IGNORED; } - writeReply(commandSocket.getOutputStream(), asked ? UNIX_REPLY_TAKEN : UNIX_REPLY_IGNORED); + writeReply(commandSocket.getOutputStream(), reply); return; } boolean taken; diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableBudgetTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableBudgetTest.java index 1324d915..a2979861 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableBudgetTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableBudgetTest.java @@ -41,7 +41,15 @@ * *

heap_after_gc_max_mb is the most heap any collection during the send left in use: an upper bound of what * the analysis keeps live (a young collection also counts old garbage not collected yet). It is -1 when no - * collection ran. heap_max_mb is the heap cap the JVM ran with (-Xmx, or its default from the machine's RAM). + * collection ran. heap_delta_mb is that less the heap in use before the send (heap_before_mb, after a + * collection: the classes' bytes and the test's own): what the send kept live at most. heap_peak_mb sums each + * heap pool's peak during the send (garbage included, so it follows the collector and the cap, not the live + * set). heap_max_mb is the heap cap the JVM ran with (-Xmx, or its default from the machine's RAM). + * + *

With -Dkeploy.budget.live=true each frame, as it is sent, first runs a full collection and reads the heap + * in use: live_max_mb is the most of those (what the send holds live at a frame boundary, the table's last + * frame included), live_delta_mb that less heap_before_mb. Those collections slow the send: read its times + * from a run without it. Without it both are -1. */ public class BuildTableBudgetTest { @@ -190,7 +198,14 @@ private static Budget measure(String label, List classes, classBytes += f.bytes.length; } ThreadMXBean threads = ManagementFactory.getThreadMXBean(); + for (MemoryPoolMXBean p : ManagementFactory.getMemoryPoolMXBeans()) { + if (heapPools.contains(p.getName())) { + p.resetPeakUsage(); + } + } long[] bytes = new long[2]; + boolean live = Boolean.getBoolean("keploy.budget.live"); + long[] liveMax = {-1}; // A record may run on into the next frame: not_measured is counted over the table's stream, keeping // the end of each frame that could begin a split match. String notMeasured = "\"state\":\"not_measured\""; @@ -201,6 +216,16 @@ private static Budget measure(String label, List classes, long cpu0 = threads.getCurrentThreadCpuTime(); long t0 = System.nanoTime(); BuildTable.send(() -> classes, frame -> { + if (live) { + System.gc(); + long used = 0; + for (MemoryPoolMXBean p : ManagementFactory.getMemoryPoolMXBeans()) { + if (heapPools.contains(p.getName())) { + used += p.getUsage().getUsed(); + } + } + liveMax[0] = Math.max(liveMax[0], used); + } String f = new String(frame, StandardCharsets.UTF_8); int at = f.indexOf("{\"v\":1,"); String header = f.substring(at, f.indexOf('\n', at)); @@ -226,15 +251,24 @@ private static Budget measure(String label, List classes, long cpu = threads.getCurrentThreadCpuTime() - cpu0; long process = processCpuNanos() - process0; long gc = gcMillis() - gc0; + long peak = 0; + for (MemoryPoolMXBean p : ManagementFactory.getMemoryPoolMXBeans()) { + if (heapPools.contains(p.getName())) { + peak += p.getPeakUsage().getUsed(); + } + } // The JVM sends a collection's notification after it ends, from a thread of its own. Thread.sleep(200); long afterGc = afterGcMax.get(); System.out.printf(Locale.ROOT, "BUDGET %s java=%s heap_max_mb=%d classes=%d class_bytes=%d wall_ms=%d" + " thread_cpu_ms=%d process_cpu_ms=%d gc_ms=%d collections=%d heap_before_mb=%d" - + " heap_after_gc_max_mb=%d table_bytes=%d probes_bytes=%d not_measured=%d%n", + + " heap_after_gc_max_mb=%d heap_delta_mb=%d heap_peak_mb=%d live_max_mb=%d live_delta_mb=%d" + + " table_bytes=%d probes_bytes=%d not_measured=%d%n", label, System.getProperty("java.version"), Runtime.getRuntime().maxMemory() >> 20, classes.size(), classBytes, wall / 1_000_000, cpu / 1_000_000, process / 1_000_000, gc, collections.get(), - base >> 20, afterGc < 0 ? -1 : afterGc >> 20, bytes[0], bytes[1], b.notMeasured); + base >> 20, afterGc < 0 ? -1 : afterGc >> 20, afterGc < 0 ? -1 : (afterGc - base) >> 20, peak >> 20, + liveMax[0] < 0 ? -1 : liveMax[0] >> 20, liveMax[0] < 0 ? -1 : (liveMax[0] - base) >> 20, + bytes[0], bytes[1], b.notMeasured); return b; } diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java index 34efe025..e3bd621c 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java @@ -32,6 +32,8 @@ import java.util.ArrayList; import java.util.Arrays; import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; import java.util.LinkedHashMap; import java.util.List; import java.util.Map; @@ -919,4 +921,204 @@ public void aThreadThatCannotStartIsIgnoredNeverThrown() throws Exception { assertTrue("nothing is left running: the next request starts", sends.request(ran::countDown)); assertTrue(ran.await(10, TimeUnit.SECONDS)); } + + // ---------------------------------------------------------------- what a send holds, and the heap it needs + + /** body is a frame's bytes after its header line. */ + static String body(byte[] frame) { + int nl = 8; + while (frame[nl] != '\n') { + nl++; + } + return new String(frame, nl + 1, frame.length - nl - 1, StandardCharsets.UTF_8); + } + + @Test + public void eachClassesProbeRecordsGoOutAsItIsWrittenNotKeptForTheEnd() throws Exception { + // Nothing of the probe map waits for the table's end in the heap: a measured unit's probe records reach + // their sink just before its own table record, so before the next unit is analyzed. + List events = new ArrayList<>(); + BuildTable.Summary s = BuildTable.write(javaDedup(), + (build, json) -> events.add("T" + new String(json, StandardCharsets.UTF_8)), + (build, json) -> events.add("P" + new String(json, StandardCharsets.UTF_8))); + assertEquals(5, s.records); + assertEquals(5, s.probeRecords); + List since = new ArrayList<>(); + int units = 0; + for (String e : events) { + JsonObject o = JsonParser.parseString(e.substring(1)).getAsJsonObject(); + if (e.charAt(0) == 'P') { + since.add(o.get("unit").getAsString()); + continue; + } + units++; + boolean measured = "ok".equals(o.get("state").getAsString()); + assertEquals(e, measured, !since.isEmpty()); + for (String unit : since) { + assertEquals("a probe record goes just before its own unit's table record", o.get("id").getAsString(), + unit); + } + since.clear(); + } + assertEquals(5, units); + assertTrue("no probe record after the last table record", since.isEmpty()); + } + + @Test + public void theProbeMapWaitsInATempFileThatIsGoneOnceTheSendEnds() throws Exception { + File dir = tmp.newFolder("spill"); + List frames = new ArrayList<>(); + List during = new ArrayList<>(); + BuildTable.send(BuildTableTest::javaDedupUnchecked, frame -> { + frames.add(frame); + during.add(Arrays.toString(dir.list())); + }, dir); + assertEquals(2, frames.size()); + String build = new String(resource("java-dedup.build"), StandardCharsets.UTF_8).trim(); + byte[] map = resource("java-dedup.probes.ndjson"); + assertEquals("{\"v\":1,\"part\":\"probes\",\"build\":\"" + build + "\",\"seq\":0,\"last\":true,\"records\":5," + + "\"sha256\":\"" + sha256Hex(map) + "\"}", header(frames.get(1))); + assertEquals(new String(map, StandardCharsets.UTF_8), body(frames.get(1))); + assertEquals(new String(resource("java-dedup.table.ndjson"), StandardCharsets.UTF_8), body(frames.get(0))); + assertArrayEquals("the temp file is removed", new String[0], dir.list()); + if (File.separatorChar == '/') { + // Unlinked as soon as it is open: a JVM that dies mid-send leaves no file behind. + assertEquals(Arrays.asList("[]", "[]"), during); + } + } + + @Test + public void aTempDirTheAppCannotWriteSendsNoFrameAndSaysWhich() throws Exception { + File notADir = tmp.newFile("not-a-dir"); + List frames = new ArrayList<>(); + AtomicInteger read = new AtomicInteger(); + try (Logs logs = new Logs()) { + BuildTable.send(() -> { + read.incrementAndGet(); + return javaDedupUnchecked(); + }, frames::add, notADir); + assertTrue("no frame: Keploy gets no table for the build, and asks again next run", frames.isEmpty()); + assertEquals("the classes are not read", 0, read.get()); + LogRecord last = logs.records.get(logs.records.size() - 1); + assertEquals(Level.WARNING, last.getLevel()); + assertTrue(last.getMessage(), last.getMessage().contains(notADir.getPath()) + && last.getMessage().contains("java.io.tmpdir")); + } + } + + @Test + public void aShortHeapDeclinesTheTableWithLowHeap() { + long mb = 1 << 20; + try (Logs logs = new Logs()) { + assertEquals("reason=low_heap heap_free_mb=112 heap_needed_mb=256", + BuildTable.lowHeap(new BuildTable.Heap(512 * mb, 400 * mb, true), 256)); + assertEquals(1, logs.records.size()); + LogRecord r = logs.records.get(0); + assertEquals(Level.WARNING, r.getLevel()); + assertTrue(r.getMessage(), r.getMessage().contains("112 MiB free of the 512 MiB heap cap the JVM reports" + + " (in use 400 MiB after the last collection), and it needs 256 MiB") + && r.getMessage().contains("low_heap") + && r.getMessage().contains("KEPLOY_COVERAGE_TABLE_MIN_HEAP_MB")); + } + assertNull("exactly what it needs is enough", BuildTable.lowHeap(new BuildTable.Heap(512 * mb, 256 * mb, + true), 256)); + assertEquals("reason=low_heap heap_free_mb=255 heap_needed_mb=256", + BuildTable.lowHeap(new BuildTable.Heap(512 * mb, 256 * mb + 1, true), 256)); + assertNull("0 turns the check off", BuildTable.lowHeap(new BuildTable.Heap(512 * mb, 512 * mb, false), 0)); + assertEquals("in use past the cap is no free heap", "reason=low_heap heap_free_mb=0 heap_needed_mb=1", + BuildTable.lowHeap(new BuildTable.Heap(512 * mb, 600 * mb, false), 1)); + assertNull("a JVM with no heap cap", BuildTable.lowHeap(new BuildTable.Heap(Long.MAX_VALUE, 400 * mb, + true), 256)); + } + + @Test + public void theFreeHeapATableNeedsIsTheEnvElseThePropertyElse256() { + assertEquals(256, BuildTable.minHeapMb(null, null)); + assertEquals(256, BuildTable.minHeapMb(" ", "")); + assertEquals(100, BuildTable.minHeapMb("100", "50")); + assertEquals(50, BuildTable.minHeapMb(" ", "50")); + assertEquals(7, BuildTable.minHeapMb(" 7 ", null)); + assertEquals(0, BuildTable.minHeapMb(null, "0")); + try (Logs logs = new Logs()) { + assertEquals(256, BuildTable.minHeapMb("-1", "50")); + assertEquals(256, BuildTable.minHeapMb(null, "1g")); + assertEquals(2, logs.records.size()); + assertEquals(Level.WARNING, logs.records.get(1).getLevel()); + assertTrue(logs.records.get(1).getMessage(), logs.records.get(1).getMessage().contains("'1g'")); + } + } + + @Test + public void theHeapInUseIsWhatTheNewestCollectionLeftInTheHeapPools() { + long mb = 1 << 20; + java.util.Set heap = new HashSet<>(Arrays.asList("Eden", "Old")); + // The newest is neither the first nor the last; the older ones left more in use; Metaspace and the code + // cache are not the heap. + BuildTable.GcEnd older = gcEnd(100, "Eden", 0, "Old", 300 * mb, "Metaspace", 40 * mb); + BuildTable.GcEnd newest = gcEnd(300, "Eden", 2 * mb, "Old", 100 * mb, "Metaspace", 50 * mb, + "CodeHeap", 20 * mb); + BuildTable.GcEnd middle = gcEnd(200, "Eden", 0, "Old", 250 * mb); + assertEquals(102 * mb, BuildTable.Heap.usedAfterNewest(Arrays.asList(older, newest, middle), heap)); + assertEquals(102 * mb, BuildTable.Heap.usedAfterNewest(Arrays.asList(middle, older, newest), heap)); + assertEquals("no collection seen", -1, + BuildTable.Heap.usedAfterNewest(Collections.emptyList(), heap)); + } + + @Test + public void theHeapInUseIsTheSmallerOfNowAndTheNewestCollection() { + BuildTable.Heap h = BuildTable.Heap.of(512, 400, 300); + assertEquals(512, h.max); + assertEquals("garbage made since the collection does not count", 300, h.used); + assertTrue(h.afterGc); + h = BuildTable.Heap.of(512, 400, 400); + assertEquals(400, h.used); + assertTrue(h.afterGc); + h = BuildTable.Heap.of(512, 400, 500); + assertEquals("less in use now than the collection left: now is newer", 400, h.used); + assertFalse("the figure is the heap in use now", h.afterGc); + h = BuildTable.Heap.of(512, 400, -1); + assertEquals("no collection seen", 400, h.used); + assertFalse(h.afterGc); + } + + @Test + public void thisJvmsCollectionsAreRead() { + // This JVM reports its collections; the check itself never runs one. + System.gc(); + long after = BuildTable.LastCollection.heapUsedAfter(); + assertTrue("a collection ran: " + after, after > 0); + BuildTable.Heap h = BuildTable.Heap.read(); + assertEquals(Runtime.getRuntime().maxMemory(), h.max); + assertTrue(h.used + " of " + h.max, h.used > 0 && h.used <= h.max); + } + + @Test + public void theProbeMapsTempFileIsNeverOpenedThroughALink() throws IOException { + // Another user who swaps the just-made temp file for a link to a file the app can write gets nothing + // written through it. + Path target = tmp.newFile("target").toPath(); + Files.write(target, "keep".getBytes(StandardCharsets.UTF_8)); + Path link = tmp.getRoot().toPath().resolve("keploy-probe-map-1.ndjson"); + try { + Files.createSymbolicLink(link, target); + } catch (IOException | UnsupportedOperationException e) { + org.junit.Assume.assumeNoException(e); + } + try { + BuildTable.Spill.openOwned(link).close(); + fail("opened through a link"); + } catch (IOException expected) { + // ELOOP + } + assertEquals("keep", new String(Files.readAllBytes(target), StandardCharsets.UTF_8)); + assertTrue("the target is not deleted", Files.exists(target)); + } + + private static BuildTable.GcEnd gcEnd(long endTime, Object... poolUsed) { + Map used = new HashMap<>(); + for (int i = 0; i < poolUsed.length; i += 2) { + used.put((String) poolUsed[i], ((Number) poolUsed[i + 1]).longValue()); + } + return new BuildTable.GcEnd(endTime, used); + } } diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/ControlSocketTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/ControlSocketTest.java index 541964be..2e6fcb6b 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/ControlSocketTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/ControlSocketTest.java @@ -98,6 +98,15 @@ public void setUp() throws IOException { KeployDedupAgent.dataSocketPath = new File(dir, "data.sock").getAbsolutePath(); // The warmup loads every class on the test classpath; nothing here needs it. System.setProperty(WARMUP_DISABLED_PROPERTY, "true"); + // A TABLE row answers ACK whatever heap this JVM was given (-DargLine=-Xmx...): the heap check is off + // unless a row sets its own bound. + System.setProperty(BuildTable.MIN_HEAP_PROPERTY, "0"); + } + + /** A TABLE row needs the heap check's env var unset: it would win over the property setUp sets. */ + private static void assumeNoMinHeapEnv() { + String v = System.getenv(BuildTable.MIN_HEAP_ENV); + org.junit.Assume.assumeTrue(BuildTable.MIN_HEAP_ENV + " is set", v == null || v.trim().isEmpty()); } @After @@ -114,6 +123,7 @@ public void tearDown() throws Exception { System.clearProperty(JACOCO_PORT_PROPERTY); System.clearProperty(WARMUP_DISABLED_PROPERTY); System.clearProperty(CLASS_DIRS_PROPERTY); + System.clearProperty(BuildTable.MIN_HEAP_PROPERTY); if (jacoco != null) { jacoco.close(); } @@ -183,6 +193,7 @@ public void aTestStartedAndEndedIsAnsweredTakenAndItsCoverageIsSent() throws Exc @Test(timeout = 60000) public void theBuildTableGoesOnlyWhenKeployAsksForIt() throws Exception { + assumeNoMinHeapEnv(); // The index is the java-dedup sample's classes, so the table is the Go encoder's golden for them. File classes = new File(dir, "classes"); for (BuildTable.ClassFile f : BuildTableTest.javaDedup()) { @@ -229,8 +240,37 @@ public void theBuildTableGoesOnlyWhenKeployAsksForIt() throws Exception { assertNull("nothing more", data.payloads.poll(1, TimeUnit.SECONDS)); } + @Test(timeout = 60000) + public void aTableAskedWhenTheHeapIsShortIsDeclinedWithLowHeapAndSendsNothing() throws Exception { + assumeNoMinHeapEnv(); + File classes = new File(dir, "classes"); + for (BuildTable.ClassFile f : BuildTableTest.javaDedup()) { + File out = new File(classes, f.name + ".class"); + assertTrue(out.getParentFile().isDirectory() || out.getParentFile().mkdirs()); + Files.write(out.toPath(), f.bytes); + } + System.setProperty(CLASS_DIRS_PROPERTY, classes.getPath()); + long needed = Long.MAX_VALUE >> 20; + System.setProperty(BuildTable.MIN_HEAP_PROPERTY, String.valueOf(needed)); + try { + startWithJacoco(); + data = new DataReceiver(KeployDedupAgent.dataSocketPath); + int asked = BuildTable.requests(); + assertEquals(TAKEN, send("START test-set-0/test-1")); + String reply = send("TABLE"); + assertTrue(reply, reply.matches("IGNORED proto=2 reason=low_heap heap_free_mb=\\d+ heap_needed_mb=" + + needed + "\n")); + assertEquals("no send started", asked, BuildTable.requests()); + assertNull("no table frame", data.payloads.poll(2, TimeUnit.SECONDS)); + assertEquals("the control server goes on", TAKEN, send("END test-set-0/test-1")); + } finally { + System.clearProperty(BuildTable.MIN_HEAP_PROPERTY); + } + } + @Test(timeout = 60000) public void theIndexRecordAndTheTableNameOneBuild() throws Exception { + assumeNoMinHeapEnv(); // Keploy finds the table under the build the INDEX record named. The index is the java-dedup sample's // classes and one file no class reader reads: both builds take every class the index holds. File classes = new File(dir, "classes"); @@ -278,6 +318,7 @@ public void theIndexRecordAndTheTableNameOneBuild() throws Exception { @Test(timeout = 300000) public void startAndEndAreAnsweredWhileTheTableIsAnalyzed() throws Exception { + assumeNoMinHeapEnv(); // Keploy sends TABLE after the first ACK and goes on with its tests: the table's analysis runs on its own // thread, so no START or END waits for it. The index is a generated app large enough that its analysis // outlasts many START/END pairs. @@ -352,6 +393,7 @@ private static String field(String json, String name) { @Test(timeout = 60000) public void aTableThatFailsToStartIsIgnoredAndTheServerGoesOn() throws Exception { + assumeNoMinHeapEnv(); startWithJacoco(); String dataPath = KeployDedupAgent.dataSocketPath; data = new DataReceiver(dataPath); diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/ProbeMapTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/ProbeMapTest.java index 50b91671..f2e7421b 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/ProbeMapTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/ProbeMapTest.java @@ -81,12 +81,13 @@ static List load(String dir) throws Exception { /** probes writes the table and returns the probe map's NDJSON stream. */ static String probes(List files, BuildTableTest.Collect table) throws IOException { - BuildTable.Summary s = BuildTable.write(files, table); ByteArrayOutputStream b = new ByteArrayOutputStream(); - for (byte[] r : s.probes) { + BuildTable.Summary s = BuildTable.write(files, table, (build, r) -> { b.write(r, 0, r.length); b.write('\n'); - } + }); + assertEquals("the summary counts the probe records", s.probeRecords, + new String(b.toByteArray(), StandardCharsets.UTF_8).split("\n", -1).length - 1); return new String(b.toByteArray(), StandardCharsets.UTF_8); } @@ -467,14 +468,16 @@ public void twoCopiesOfAClassThatDifferAreNotMeasured() throws Exception { files.add(new BuildTable.ClassFile("d/C", c[0])); files.add(new BuildTable.ClassFile("d/C", c[1])); BuildTableTest.Collect table = new BuildTableTest.Collect(); - BuildTable.Summary s = BuildTable.write(files, table); + BuildTableTest.Collect map = new BuildTableTest.Collect(); + BuildTable.Summary s = BuildTable.write(files, table, map); assertEquals(1, table.records.size()); String rec = table.records.get(0); assertTrue(rec, rec.contains("\"id\":\"c:d/C\"")); assertTrue(rec, rec.contains("\"state\":\"not_measured\",\"reason\":{\"code\":\"analyzer_failed\"}")); assertEquals(1, s.notMeasured); assertEquals("counted as a probe map failure", 1, s.notMapped); - assertTrue("no probe record for a unit that is not measured", s.probes.isEmpty()); + assertTrue("no probe record for a unit that is not measured", map.records.isEmpty()); + assertEquals(0, s.probeRecords); } } From 0c65ab3411846dd93b18a3f3e2104b29407f791f Mon Sep 17 00:00:00 2001 From: slayerjain Date: Fri, 20 Nov 2026 07:27:57 +0530 Subject: [PATCH 06/14] perf(dedup): map each probe from one analysis pass and bound its cost Each probe's leaves are now read off the class's one JaCoCo pass. JaCoCo marks an executed probe's instruction and walks back through each instruction's predecessor (the one edge into it that carries no probe; JaCoCo puts a probe on every edge into an instruction that more than one edge reaches), stopping after an instruction it marked already; its calculator then merges and replaces branches as the filters said. KeployBranchCoverage takes that walk over the edges the pass's builder wired and applies the pass's filter output to what it marked. A method costs its size plus its probes' walks, with no object per probe. Before, each probe ran JaCoCo's builder and calculator over its whole method again. The walks are short in an if/else chain, whose statements join at labels two edges reach. In a chain of early returns ("if (x == k) return k;" lines, or a run of guard throws) each probe walks back over every check before it and covers each one, so the map itself is quadratic: about 4.4 MB at 1,000 lines and 43 MB at 3,000, and at 4,500 it ran a 256 MiB heap out of memory. A unit's probe map is now bounded at 8 MiB (its records with their newlines, the table record's bound) and 64 Mi walk steps. Past either, the unit is sent not_measured record_too_large with no probe record and is counted in the send's log line, and the map is given up as soon as it is past, so 3,000 and 4,500 lines now take 0.07-0.2 s of CPU and 40-51 MiB of allocation. The map's text is ASCII in one buffer and copied once into an exact-size record. Two class files of one name in the index are both charged against the one bound, so such a unit is declined once its two maps together pass 8 MiB, though its one record would fit (counted, never unbounded). A method whose flow has another shape (an instruction that two edges without a probe reach, or a replaced branch that reads another replaced instruction) throws, so its class is not measured (analyzer_failed), as a mismatch with JaCoCo's coverage did before. The pass reads an instruction's branches only when it is covered (an uncovered instruction has none; the table's pass executes no probe), and keeps each instruction's branch numbers in one sorted array, not a set per instruction. The low_heap bound's default is 64 MiB plus 1 KiB per class the index holds (113 MiB at 50,000 classes). 256 MiB declined every 1 GiB pod run with no -Xmx. The count is the index's only when it is read already, so the answer reads no class file and waits on no lock; an index not read yet is bounded by the 64 MiB base, and read on the send's thread. Measured on 50,000 generated classes (JDK 21 and 8, G1 and Parallel) before the map's bound was added, which only takes copies out of the map's path, so these are upper bounds: the send allocates 6,142-6,915 MiB, 125-141 KiB a class, against 90,611-98,076 MiB (about 1.9 MiB a class) before, and takes 4.3-5.8 s of CPU against 29-42 s. A method of 3,000 if/else branches maps in about 0.2 s of CPU. The heap the JVM commits while the table is built still follows the analysis's allocation rate, as it did before (up to 657 MiB more with G1 on JDK 8 in a 4 GiB pod): the analysis is not paced, and this commit does not bound that growth. Rows: the probe oracle (each probe's map against JaCoCo's analysis of the class with that probe executed) now covers JaCoCo core, Gson, the java-dedup, code-shape and Java 21 fixtures, a 300-branch generated class and an 80-return chain; the budget rows bound the allocation per class (256 KiB), a 3,000-branch method's CPU (1 s), and a chain of 1,000, 3,000 and 4,500 early returns (mapped, then record_too_large, each under 1 s of CPU and 64 MiB of allocation); the map's bytes and steps bounds have rows at both edges; the heap bound has rows at both edges of its default and through the TABLE answer with an index count given and with none (-1). Signed-off-by: slayerjain --- .../main/java/io/keploy/dedup/BuildTable.java | 447 +++++++++++++--- .../io/keploy/dedup/KeployDedupAgent.java | 9 +- .../analysis/KeployBranchCoverage.java | 482 ++++++++++++++---- .../io/keploy/dedup/BuildTableBudgetTest.java | 177 ++++++- .../java/io/keploy/dedup/BuildTableTest.java | 111 +++- .../java/io/keploy/dedup/ProbeMapTest.java | 145 +++--- 6 files changed, 1129 insertions(+), 242 deletions(-) diff --git a/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java b/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java index 51eb3915..9f60e9ce 100644 --- a/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java +++ b/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java @@ -49,6 +49,7 @@ import java.util.Set; import java.util.concurrent.ThreadFactory; import java.util.function.Function; +import java.util.function.IntSupplier; import java.util.function.Supplier; import java.util.logging.Level; import java.util.logging.Logger; @@ -110,6 +111,18 @@ static void log(Level level, String message, Throwable thrown) { static final int MAX_FRAME_BYTES = 1 << 20; /** A record (with its newline) larger than a whole inflated chunk can never be sent: record_too_large. */ static final int MAX_RECORD_BYTES = 8 << 20; + /** The reason a unit's record is sent without its methods when it, or its probe map, is past a bound. */ + static final String RECORD_TOO_LARGE = "record_too_large"; + /** + * The most a unit's probe map may take: its records' bytes, newlines included (a table record's bound), + * and the steps its probes' walks take (KeployBranchCoverage.Hits#steps, about a second of CPU). A unit + * past either is sent not_measured record_too_large with no probe record, and its map is given up as + * soon as it is past one, so neither the heap nor the time it takes grows with it: a method of n early + * returns maps to about n * n ranges (its probe k covers the k checks before it), 4.4 MB at 1,000 and + * 43 MB at 3,000. + */ + static final int MAX_MAP_BYTES = MAX_RECORD_BYTES; + static final long MAX_MAP_STEPS = 64L << 20; /** The first 4 bytes of a TABLE frame. */ static final String MAGIC = "KTB1"; /** The parts a TABLE answer streams, in this order: the table, then the probe map. */ @@ -151,8 +164,10 @@ static List files(List entries, Function name, Func /** The env var, then the system property, that set the free heap a table needs, in MiB; 0 turns it off. */ static final String MIN_HEAP_ENV = "KEPLOY_COVERAGE_TABLE_MIN_HEAP_MB"; static final String MIN_HEAP_PROPERTY = "keploy.coverage.table.min.heap.mb"; - /** The free heap a table needs when neither is set, in MiB. */ - static final long DEFAULT_MIN_HEAP_MB = 256; + /** The free heap a table needs when neither is set: this many MiB, plus {@link #MIN_HEAP_KB_PER_CLASS}. */ + static final long MIN_HEAP_BASE_MB = 64; + /** ...and this many KiB per class the index holds (113 MiB at 50,000 classes). */ + static final long MIN_HEAP_KB_PER_CLASS = 1; /** The reason a TABLE ask is declined when the heap is short: Keploy marks the run not_measured with it. */ static final String LOW_HEAP = "low_heap"; @@ -161,11 +176,21 @@ static List files(List entries, Function name, Func * table needs ({@link #lowHeap}) the analysis is skipped and the reply is {@code IGNORED proto=2 * reason=low_heap heap_free_mb=F heap_needed_mb=N}, with a WARNING in the app's log; else it is * {@link #request}'s ACK or IGNORED. The heap is read once, here, when Keploy asks: a send that runs after - * another one was checked when it was asked. It never blocks. + * another one was checked when it was asked. The bound takes loaded, the class count of the index when it + * is read already (the app's start or its first START read it, before Keploy asks for a table); when it is + * not read yet (loaded is -1: no START read it, the index was not warmed up and the JVM started the SDK + * from agentmain or start(), or with a JaCoCo of its own), the bound is the base alone, as the count is + * not known without reading the class files. The answer reads no class file and waits on no lock: classes + * is read on the send's thread. It never blocks. */ - static String answer(Supplier> classes, File dataSocket) { - String low = lowHeap(Heap.read(), minHeapMb(System.getenv(MIN_HEAP_ENV), - System.getProperty(MIN_HEAP_PROPERTY))); + static String answer(IntSupplier loaded, Supplier> classes, File dataSocket) { + return answer(Heap.read(), loaded.getAsInt(), classes, dataSocket); + } + + /** answer with the heap read and the index's class count (-1: not read yet) given. */ + static String answer(Heap heap, int loaded, Supplier> classes, File dataSocket) { + String low = lowHeap(heap, minHeapMb(System.getenv(MIN_HEAP_ENV), System.getProperty(MIN_HEAP_PROPERTY), + loaded)); if (low != null) { return KeployDedupAgent.UNIX_REPLY_IGNORED + " " + low; } @@ -191,14 +216,14 @@ static String lowHeap(Heap heap, long minMb) { } /** - * minHeapMb is the free heap a table needs, in MiB: env, else property (as the agent reads its settings), - * else {@link #DEFAULT_MIN_HEAP_MB}. A value that is not a whole number of 0 or more is logged and the - * default holds. + * minHeapMb is the free heap a table of the given number of classes needs, in MiB: env, else property (as + * the agent reads its settings), else {@link #defaultMinHeapMb}. A value that is not a whole number of 0 or + * more is logged and the default holds. */ - static long minHeapMb(String env, String property) { + static long minHeapMb(String env, String property, int classes) { String v = env == null || env.trim().isEmpty() ? property : env; if (v == null || v.trim().isEmpty()) { - return DEFAULT_MIN_HEAP_MB; + return defaultMinHeapMb(classes); } try { long mb = Long.parseLong(v.trim()); @@ -209,9 +234,14 @@ static long minHeapMb(String env, String property) { // logged below } log(Level.WARNING, "Keploy coverage: " + MIN_HEAP_ENV + " (or -D" + MIN_HEAP_PROPERTY + ") is '" + v.trim() - + "', not a whole number of MiB (0 or more); the build table needs " + DEFAULT_MIN_HEAP_MB + + "', not a whole number of MiB (0 or more); the build table needs " + defaultMinHeapMb(classes) + " MiB of free heap", null); - return DEFAULT_MIN_HEAP_MB; + return defaultMinHeapMb(classes); + } + + /** defaultMinHeapMb is {@link #MIN_HEAP_BASE_MB} MiB plus 1 KiB per class, rounded up to a whole MiB. */ + static long defaultMinHeapMb(int classes) { + return MIN_HEAP_BASE_MB + (Math.max(0, classes) * MIN_HEAP_KB_PER_CLASS + 1023) / 1024; } /** @@ -452,6 +482,12 @@ static void send(Supplier> classes, FrameSender sender, File spi m.append(" (").append(s.notMapped).append(" of them as their probes could not be mapped: two" + " different class files of one name, or a Keploy bug; the FINE log names each)"); } + if (s.tooLarge > 0) { + m.append(", ").append(s.tooLarge).append(" not measured as too large (").append(RECORD_TOO_LARGE) + .append(": a record over ").append(MAX_RECORD_BYTES >> 20).append(" MiB, or a probe map over ") + .append(MAX_MAP_BYTES >> 20).append(" MiB or ").append(MAX_MAP_STEPS >> 20) + .append(" Mi steps; the FINE log names each)"); + } if (s.noLines > 0) { m.append(", ").append(s.noLines) .append(" class files without line numbers, so without lines: compile them with -g"); @@ -494,10 +530,12 @@ static final class Summary { final int unnamed; /** How many of the not measured records JaCoCo analyzed but whose probes could not be mapped. */ final int notMapped; + /** How many of the not measured records are record_too_large: the record or its probe map too large. */ + final int tooLarge; /** How many probe-map records went to the probes sink: one per class name of each measured unit. */ final int probeRecords; - Summary(String build, int records, int notMeasured, int noLines, int unnamed, int notMapped, + Summary(String build, int records, int notMeasured, int noLines, int unnamed, int notMapped, int tooLarge, int probeRecords) { this.build = build; this.records = records; @@ -505,6 +543,7 @@ static final class Summary { this.noLines = noLines; this.unnamed = unnamed; this.notMapped = notMapped; + this.tooLarge = tooLarge; this.probeRecords = probeRecords; } } @@ -537,29 +576,40 @@ static Summary write(List classes, RecordSink sink, int maxRecord) th static Summary write(List classes, RecordSink sink, RecordSink probes, int maxRecord) throws IOException { + return write(classes, sink, probes, maxRecord, MAX_MAP_BYTES, MAX_MAP_STEPS); + } + + /** write with the bounds of a unit's table record (maxRecord) and probe map (maxMap bytes, maxSteps). */ + static Summary write(List classes, RecordSink sink, RecordSink probes, int maxRecord, int maxMap, + long maxSteps) throws IOException { Grouped g = group(classes); if (g.build.isEmpty()) { - return new Summary("", 0, 0, 0, g.unnamed, 0, 0); + return new Summary("", 0, 0, 0, g.unnamed, 0, 0, 0); } String build = g.build; List units = g.units; int notMeasured = 0; int noLines = 0; int notMapped = 0; + int tooLarge = 0; int probeRecords = 0; for (int i = 0; i < units.size(); i++) { // A unit written is dropped: what stays is the class list (names, ids), not the table. Unit u = units.set(i, null); - byte[] rec = u.record(); + byte[] rec = u.record(new MapBudget(maxMap, maxSteps)); if (rec.length + 1 > maxRecord) { - rec = u.notMeasured("record_too_large"); + rec = u.notMeasured(RECORD_TOO_LARGE); u.failed = true; + u.tooLarge = true; } if (u.failed) { notMeasured++; if (u.notMapped) { notMapped++; } + if (u.tooLarge) { + tooLarge++; + } } else { noLines += u.noLines; for (byte[] p : u.probes) { @@ -569,7 +619,8 @@ static Summary write(List classes, RecordSink sink, RecordSink probes } sink.record(build, rec); } - return new Summary(build, units.size(), notMeasured, noLines, g.unnamed, notMapped, probeRecords); + return new Summary(build, units.size(), notMeasured, noLines, g.unnamed, notMapped, tooLarge, + probeRecords); } /** What group makes of the index's classes: the build id ("" with none), the units in table order. */ @@ -702,6 +753,8 @@ static final class Unit { boolean failed; // Set when JaCoCo analyzed the unit but its probes could not be mapped (so failed is set too). boolean notMapped; + // Set when the unit's record or its probe map is past its bound: record_too_large (failed is set too). + boolean tooLarge; // Class files with code but no line numbers: their methods have no leaf, so they are left out. int noLines; // The probe map's records for the unit's class files, in file order; set when record() succeeds. @@ -780,8 +833,11 @@ private List sortedFiles() { return sorted; } - /** record analyzes the unit's class files and writes its record; any class JaCoCo refuses fails it. */ - byte[] record() { + /** + * record analyzes the unit's class files and writes its record; any class JaCoCo refuses fails it, and + * a probe map past the budget makes it record_too_large. + */ + byte[] record(MapBudget budget) { Map infos = new HashMap<>(); List methods = new ArrayList<>(); Set methodIds = new HashSet<>(); @@ -824,23 +880,32 @@ byte[] record() { List maps = new ArrayList<>(); try { String lastName = null; - String lastProbes = null; + byte[] last = null; for (UnitFile f : sortedFiles()) { - String p = ProbeMap.probes(f, byId); + byte[] p = ProbeMap.record(id, f, ProbeMap.probes(f, byId, budget)); if (f.file.name.equals(lastName)) { // Two class files of one name whose bytes differ (the index found both). Keploy finds a // class's map by its name, as a test's hits name it, so they share one record: exact // only when a hit of either one covers the same leaves. - if (!p.equals(lastProbes)) { + if (!ProbeMap.sameProbes(p, last)) { throw new IllegalStateException(f.file.name + ": two class files of this name map" + " their probes differently"); } + budget.drop(); continue; } - maps.add(ProbeMap.record(id, f, p)); + // The unit's records, newlines included, within the budget (its text alone was, as it grew). + budget.take(p.length + 1); + maps.add(p); lastName = f.file.name; - lastProbes = p; + last = p; } + } catch (TooLarge e) { + log(Level.FINE, "Keploy coverage: the probe map of " + path() + " is past its bound: " + e.getMessage(), + null); + failed = true; + tooLarge = true; + return notMeasured(RECORD_TOO_LARGE); } catch (Exception | LinkageError e) { // A class whose probes cannot be mapped exactly is not measured: its hits could not be read. log(Level.FINE, "Keploy coverage: the probes of " + path() + " could not be mapped", e); @@ -1048,38 +1113,63 @@ void check(KeployBranchCoverage.Method a) { } } + // Per counted instruction of the class file's analysis of this method (set by prepare): the unit + // leaf index of its line's code leaf, and of its first branch edge (-1: a line below 1, no leaf). + private int[] codeAt; + private int[] edgeAt; + private KeployBranchCoverage.Method prepared; + /** - * hits is the set of unit-relative leaf indices the analysis covers: a line's code leaf when one of its - * instructions is covered, and the leaf of each covered branch edge. A line's edges are its leaves after - * its code leaf, in the order of the instructions and of each instruction's branches. + * prepare checks the analysis has the method's leaves ({@link #check}) and notes, per counted + * instruction, its leaves: its line's code leaf, and its line's edges after the code leaf in the + * order of the instructions and of each instruction's branches. */ - BitSet hits(KeployBranchCoverage.Method a) { - BitSet out = new BitSet(); + void prepare(KeployBranchCoverage.Method a) { + check(a); + codeAt = new int[a.lines.length]; + edgeAt = new int[a.lines.length]; Map edges = new HashMap<>(); // line -> edges met so far for (int k = 0; k < a.lines.length; k++) { int l = a.lines[k]; if (l < 1) { + codeAt[k] = -1; + edgeAt[k] = -1; continue; } int leaf = codeLeaf(l); if (leaf < 0) { throw new IllegalStateException(id + ": line " + l + " has no code leaf"); } - int code = base + leaf; - if (a.covered[k]) { - out.set(code); - } Integer met = edges.get(l); int e = met == null ? 0 : met; - for (boolean covered : a.branches[k]) { - if (covered) { - out.set(code + 1 + e); + codeAt[k] = base + leaf; + edgeAt[k] = base + leaf + 1 + e; + edges.put(l, e + a.branches[k].length); + } + prepared = a; + } + + /** + * hits sets in out the unit-relative leaf indices a probe covers: a line's code leaf when one of its + * instructions is covered, and the leaf of each covered branch edge. + */ + void hits(KeployBranchCoverage.Method a, KeployBranchCoverage.Hits h, BitSet out) { + if (prepared != a) { + throw new IllegalStateException(id + ": probes of an analysis not prepared"); + } + for (int j = 0; j < h.count; j++) { + int k = h.at[j]; + if (codeAt[k] < 0) { + continue; + } + out.set(codeAt[k]); + boolean[] b = h.branches[k]; + for (int t = 0; t < b.length; t++) { + if (b[t]) { + out.set(edgeAt[k] + t); } - e++; } - edges.put(l, e); } - return out; } } @@ -1089,9 +1179,11 @@ BitSet hits(KeployBranchCoverage.Method a) { * covered branch edge's leaf. A probe covers more than its own line: JaCoCo marks every instruction on the * path that ends at the probe, so a test's leaves are the union of its probes' leaves. * - *

A method's probes never reach another method's instructions, so the class file is analyzed once and - * each probe replays only its own method (KeployBranchCoverage): the map costs, summed over the methods - * with a line, each method's probes times that method's size, on top of the table's own analysis. + *

The class file is analyzed once, and each probe's leaves are read off that analysis (KeployBranchCoverage + * walks back from the probe over the edges JaCoCo's builder wired): the map costs the class's size plus its + * probes' walks and their ranges' text, on top of the table's own analysis, with no object per probe. The + * walks and the text are quadratic in a chain of early returns, so a unit's map has a budget + * ({@link MapBudget}): past it the unit is record_too_large. */ static final class ProbeMap { private ProbeMap() { @@ -1099,43 +1191,88 @@ private ProbeMap() { /** * record is the class file's probe-map record: {"class":vmName,"id":crc64,"unit":unitId,"probes":[...]}, - * with probes as {@link #probes} gives them. + * with probes as {@link #probes} made them, written once into a buffer of its size. */ - static byte[] record(String unitId, UnitFile f, String probes) { - StringBuilder sb = new StringBuilder(64 + probes.length()); + static byte[] record(String unitId, UnitFile f, Ranges probes) { + StringBuilder sb = new StringBuilder(64 + f.file.name.length() + unitId.length()); sb.append("{\"class\":"); Json.string(sb, f.file.name); sb.append(",\"id\":\"").append(String.format("%016x", f.classId)).append("\",\"unit\":"); Json.string(sb, unitId); - sb.append(",\"probes\":").append(probes).append('}'); - return sb.toString().getBytes(StandardCharsets.UTF_8); + sb.append(",\"probes\":"); + byte[] head = sb.toString().getBytes(StandardCharsets.UTF_8); + byte[] rec = new byte[head.length + probes.jsonLength() + 1]; + System.arraycopy(head, 0, rec, 0, head.length); + int at = probes.json(rec, head.length); + rec[at] = '}'; + return rec; } /** - * probes is the class file's probe map as JSON: one entry per probe (the class's probe count), each the + * sameProbes says whether two records of one class name and unit map their probes alike: they are + * alike but for the id (16 hex digits), which ends at the same place in both. + */ + static boolean sameProbes(byte[] a, byte[] b) { + if (a.length != b.length) { + return false; + } + int id = indexOf(a, ID_KEY) + ID_KEY.length + 16; + for (int i = id; i < a.length; i++) { + if (a[i] != b[i]) { + return false; + } + } + return true; + } + + private static final byte[] ID_KEY = ",\"id\":\"".getBytes(StandardCharsets.US_ASCII); + + private static int indexOf(byte[] a, byte[] key) { + outer: + for (int i = 0; i + key.length <= a.length; i++) { + for (int j = 0; j < key.length; j++) { + if (a[i + j] != key[j]) { + continue outer; + } + } + return i; + } + throw new IllegalStateException("a probe record with no id"); + } + + /** + * probes is the class file's probe map: one entry per probe (the class's probe count), each the * probe's leaves as half-open ranges of the unit's leaf indices, [start,end,start,end,...] ascending, no * two ranges touching. It throws when the class's lines or branches differ from the unit's methods (the - * same name under two roots), when a method with lines is in one and not in the other, or when a probe's - * replay meets other probes than the class's analysis did. + * same name under two roots), when a method with lines is in one and not in the other, or when a + * method's flow is not one KeployBranchCoverage maps; and TooLarge as soon as the unit's map is past + * its budget's bytes or steps. */ - static String probes(UnitFile f, Map byId) { + static Ranges probes(UnitFile f, Map byId, MapBudget budget) { Map shape = new HashMap<>(); - Map hits = new HashMap<>(); String prefix = "m:" + f.file.name + "#"; - // One analysis of the class; each probe's coverage replays only its own method (KeployBranchCoverage). + Ranges ranges = new Ranges(budget); + // One analysis of the class; each probe's leaves come from it (KeployBranchCoverage). int total = Math.max(0, KeployBranchCoverage.analyze(f.file.bytes, shape, new KeployBranchCoverage.EachProbe() { + private Method method; + @Override public boolean wants(KeployBranchCoverage.Method s) { // Checked below: a method with no line has no leaf for its probes to cover. - return byId.containsKey(prefix + s.name + s.desc); + method = byId.get(prefix + s.name + s.desc); + if (method == null) { + return false; + } + method.prepare(s); + return true; } @Override - public void probe(KeployBranchCoverage.Method s, int r, KeployBranchCoverage.Method got) { - Method m = byId.get(prefix + s.name + s.desc); - m.check(got); - hits.put(s.probes[r], m.hits(got)); + public void probe(KeployBranchCoverage.Method s, int r, KeployBranchCoverage.Hits h) { + budget.step(h.steps); + method.hits(s, h, ranges.bits); + ranges.put(s.probes[r]); } })); int inShape = 0; @@ -1164,22 +1301,188 @@ public void probe(KeployBranchCoverage.Method s, int r, KeployBranchCoverage.Met throw new IllegalStateException(f.file.name + ": " + inTable + " methods in the table, " + inShape + " in the probe analysis"); } - StringBuilder sb = new StringBuilder(2 + 8 * total); - sb.append('['); + ranges.close(f.file.name, total); + return ranges; + } + } + + /** + * TooLarge ends a unit's probe map that is past its budget (MapBudget): the unit is sent not_measured + * record_too_large. It carries no stack trace: it is an answer, not a bug. + */ + static final class TooLarge extends RuntimeException { + private static final long serialVersionUID = 1L; + + TooLarge(String message) { + super(message, null, false, false); + } + } + + /** + * MapBudget is what one unit's probe map may still take: bytes (its records', newlines included; its + * ranges' text counts as it is written, since a record holds it) and walk steps. Past either it throws + * TooLarge, so a map is given up as soon as it is past, not once it is made. + */ + static final class MapBudget { + private final long maxBytes; + private final long maxSteps; + private long text; + private long bytes; + private long steps; + + MapBudget(long maxBytes, long maxSteps) { + this.maxBytes = maxBytes; + this.maxSteps = maxSteps; + } + + /** step counts a probe's walk. */ + void step(int n) { + steps += n; + if (steps > maxSteps) { + throw new TooLarge("its probes' walks take more than " + maxSteps + " steps"); + } + } + + /** grew counts ranges' text written for a class file's record not taken yet. */ + void grew(int n) { + text += n; + if (bytes + text > maxBytes) { + throw new TooLarge("its probe map is more than " + maxBytes + " bytes"); + } + } + + /** drop forgets the text counted so far: its record shares the record before it. */ + void drop() { + text = 0; + } + + /** take counts a whole record (with its newline) in place of its text counted so far. */ + void take(int n) { + text = 0; + bytes += n; + if (bytes > maxBytes) { + throw new TooLarge("its probe records are more than " + maxBytes + " bytes"); + } + } + } + + /** + * Ranges keeps a class file's probe map as it is made: each probe's leaf ranges written once, as ASCII + * JSON, into one buffer, with where each probe's text starts and ends, so a probe costs its text and no + * object, and the record copies the text once ({@link ProbeMap#record}). + */ + static final class Ranges { + /** The probe being written: its leaves, set by the caller and cleared by {@link #put}. */ + final BitSet bits = new BitSet(); + private final MapBudget budget; + private byte[] text = new byte[256]; + private int length; + private int[] from = new int[16]; + private int[] to = new int[16]; + private int total = -1; + + Ranges(MapBudget budget) { + this.budget = budget; + } + + /** put writes bits as the given probe's ranges ("start,end,..." ascending, none touching), then clears it. */ + void put(int probe) { + if (probe >= from.length) { + int size = Math.max(probe + 1, from.length * 2); + from = Arrays.copyOf(from, size); + to = Arrays.copyOf(to, size); + } + if (to[probe] != 0) { + throw new IllegalStateException("probe " + probe + " mapped twice"); + } + from[probe] = length; + int at = bits.nextSetBit(0); + while (at >= 0) { + int end = bits.nextClearBit(at); + int was = length; + if (length > from[probe]) { + append(','); + } + appendInt(at); + append(','); + appendInt(end); + budget.grew(length - was); + at = bits.nextSetBit(end); + } + // to is one past the text's end, so 0 is "not mapped" and an empty range list is still mapped. + to[probe] = length + 1; + bits.clear(); + } + + private void append(char c) { + if (length == text.length) { + text = Arrays.copyOf(text, text.length * 2); + } + text[length++] = (byte) c; + } + + private void appendInt(int v) { + if (v < 0) { + throw new IllegalStateException("leaf " + v); + } + int digits = 1; + for (int x = v; x >= 10; x /= 10) { + digits++; + } + if (length + digits > text.length) { + text = Arrays.copyOf(text, Math.max(text.length * 2, length + digits)); + } + for (int i = length + digits - 1; i >= length; i--) { + text[i] = (byte) ('0' + v % 10); + v /= 10; + } + length += digits; + } + + /** close ends the map at the class's probe count (total): a probe at or past it is a bug. */ + void close(String className, int total) { + for (int p = total; p < from.length; p++) { + if (to[p] != 0) { + throw new IllegalStateException(className + ": probe " + p + " of " + total + " probes"); + } + } + this.total = total; + } + + /** jsonLength is the length of {@link #json}'s text: [ ], one [ ] a probe, the commas, the ranges. */ + int jsonLength() { + long n = 2 + 3L * total - (total > 0 ? 1 : 0); + for (int p = 0; p < total && p < from.length; p++) { + if (to[p] != 0) { + n += to[p] - 1 - from[p]; + } + } + if (total < 0 || n > Integer.MAX_VALUE - 1024) { + throw new IllegalStateException("a probe map of " + n + " bytes"); + } + return (int) n; + } + + /** + * json writes the map into out at at and returns where it ends: one entry per probe of the class, + * [] for a probe no method mapped. + */ + int json(byte[] out, int at) { + out[at++] = '['; for (int p = 0; p < total; p++) { - sb.append(p > 0 ? ",[" : "["); - BitSet b = hits.get(p); - int at = b == null ? -1 : b.nextSetBit(0); - boolean first = true; - while (at >= 0) { - int end = b.nextClearBit(at); - sb.append(first ? "" : ",").append(at).append(',').append(end); - first = false; - at = b.nextSetBit(end); + if (p > 0) { + out[at++] = ','; + } + out[at++] = '['; + if (p < from.length && to[p] != 0) { + int n = to[p] - 1 - from[p]; + System.arraycopy(text, from[p], out, at, n); + at += n; } - sb.append(']'); + out[at++] = ']'; } - return sb.append(']').toString(); + out[at++] = ']'; + return at; } } diff --git a/keploy-sdk/src/main/java/io/keploy/dedup/KeployDedupAgent.java b/keploy-sdk/src/main/java/io/keploy/dedup/KeployDedupAgent.java index 1af21954..078d99ba 100644 --- a/keploy-sdk/src/main/java/io/keploy/dedup/KeployDedupAgent.java +++ b/keploy-sdk/src/main/java/io/keploy/dedup/KeployDedupAgent.java @@ -810,8 +810,13 @@ private void handle(Socket socket) { if (BuildTable.COMMAND.equals(line.trim())) { // the build table: ACK, then sent on its own thread String reply; // or IGNORED, with reason=low_heap when the heap is short try { - reply = BuildTable.answer(() -> BuildTable.files(collector.coverageIndex.entries(), - e -> e.className, e -> e.bytes), new File(dataSocketPath)); + CoverageIndex index = collector.coverageIndex; // its count only if read: no load here + reply = BuildTable.answer(() -> { + List read = index.entries; + return read == null ? -1 : read.size(); + }, + () -> BuildTable.files(index.entries(), e -> e.className, e -> e.bytes), + new File(dataSocketPath)); } catch (Throwable t) { // thrown past here, it would end the accept loop: IGNORED instead log(Level.WARNING, "Keploy coverage: the build table was not started", t); reply = UNIX_REPLY_IGNORED; diff --git a/keploy-sdk/src/main/java/org/jacoco/core/internal/analysis/KeployBranchCoverage.java b/keploy-sdk/src/main/java/org/jacoco/core/internal/analysis/KeployBranchCoverage.java index 33cdbf12..0ef41d80 100644 --- a/keploy-sdk/src/main/java/org/jacoco/core/internal/analysis/KeployBranchCoverage.java +++ b/keploy-sdk/src/main/java/org/jacoco/core/internal/analysis/KeployBranchCoverage.java @@ -2,12 +2,13 @@ import java.util.ArrayList; import java.util.Arrays; +import java.util.Collection; import java.util.HashMap; import java.util.HashSet; +import java.util.IdentityHashMap; import java.util.List; import java.util.Map; import java.util.Set; -import java.util.TreeSet; import java.util.function.Consumer; import org.jacoco.core.analysis.ICounter; @@ -17,8 +18,6 @@ import org.jacoco.core.internal.analysis.filter.IFilterOutput; import org.jacoco.core.internal.analysis.filter.Replacements; import org.jacoco.core.internal.flow.ClassProbesAdapter; -import org.jacoco.core.internal.flow.IProbeIdGenerator; -import org.jacoco.core.internal.flow.MethodProbesAdapter; import org.jacoco.core.internal.flow.MethodProbesVisitor; import org.jacoco.core.internal.instr.InstrSupport; import org.objectweb.asm.ClassReader; @@ -26,6 +25,7 @@ import org.objectweb.asm.MethodVisitor; import org.objectweb.asm.Opcodes; import org.objectweb.asm.tree.AbstractInsnNode; +import org.objectweb.asm.tree.InsnList; import org.objectweb.asm.tree.MethodNode; /** @@ -34,9 +34,10 @@ * coverage API sums away: which counted instructions are covered and which branch of each one is. Keploy's * build table uses it to map each probe to the lines and branch edges that a hit of that probe covers. * - *

It lives in JaCoCo's package so that it runs JaCoCo's package-private analysis classes as they are: the - * result is JaCoCo's own, not a copy of its rules. Each method's per-line sums are checked against the - * coverage JaCoCo computes for it in the same pass, so a mismatch throws instead of mapping a probe wrong. + *

It lives in JaCoCo's package so that it runs JaCoCo's package-private analysis classes as they are. Each + * method's per-line sums are checked against the coverage JaCoCo computes for it in the same pass, so a + * mismatch throws instead of mapping a probe wrong. What one probe covers is read off that same pass (the + * edges JaCoCo's builder wired and the filters' output), as {@link #analyze(byte[], Map, EachProbe)} says. */ public final class KeployBranchCoverage { @@ -83,17 +84,53 @@ public static int analyze(byte[] bytes, boolean[] probes, Map ou return analyze(bytes, probes, out, null); } + /** + * Hits is what one probe covers in its method: the method's counted instructions that a class analysis + * with only that probe executed marks covered, and which of their branches. It is valid during + * {@link EachProbe#probe} only: the arrays are the method's, reused for each of its probes. + */ + public static final class Hits { + /** How many of the method's counted instructions the probe covers. */ + public int count; + /** The first count entries: those instructions' indexes into the method's lines, in no set order. */ + public final int[] at; + /** Per counted instruction, which of its branches the probe covers; all false for one not in at. */ + public final boolean[][] branches; + /** + * The steps this probe's walk took: the instructions it walked back over, plus each branch the + * filters' replacements fed from them. It is the probe's cost, which a caller can bound: in a method + * where each probe walks back over all the probes before it (a chain of early returns), the walks + * sum to the square of its length. + */ + public int steps; + + Hits(Method m) { + at = new int[m.lines.length]; + branches = new boolean[m.lines.length][]; + for (int k = 0; k < branches.length; k++) { + branches[k] = new boolean[m.branches[k].length]; + } + } + + void clear() { + for (int j = 0; j < count; j++) { + Arrays.fill(branches[at[j]], false); + } + count = 0; + } + } + /** What {@link #analyze(byte[], Map, EachProbe)} hands over for each probe of each method with code. */ public interface EachProbe { /** * probe gets method (as analyze with no probe executed puts it in out), the index r of one of its - * probes in method.probes, and the method's coverage when that probe alone was executed. + * probes in method.probes, and what that probe alone covers (valid during this call only). */ - void probe(Method method, int r, Method got); + void probe(Method method, int r, Hits hits); /** - * wants says whether method's probes are replayed at all: a method it declines costs no replay, and - * probe gets none of its probes. Every method with code is wanted unless this is overridden. + * wants says whether method's probes are walked at all: a method it declines costs no walk, and probe + * gets none of its probes. Every method with code is wanted unless this is overridden. */ default boolean wants(Method method) { return true; @@ -103,10 +140,18 @@ default boolean wants(Method method) { /** * analyze is {@link #analyze(byte[], boolean[], Map)} with no probe executed, and it also hands each * probe's own coverage to each, method by method, as a class analysis with only that probe executed gives - * it. It parses the class and runs the filters once: each probe replays only its method's instructions - * through JaCoCo's probe adapter, instruction builder and coverage calculator, with the filters' output of - * that one pass. So a class costs, summed over its methods, each method's probes times that method's size, - * not the whole class once per probe. A replay that meets other probes than the pass did throws. + * it, from that one pass: no method is analyzed again. JaCoCo's builder marks an executed probe's + * instruction and walks back through each instruction's predecessor, the one edge into it that carries no + * probe (JaCoCo puts a probe on every edge into an instruction that more than one edge reaches), stopping + * after an instruction it marked already (a loop); its coverage calculator then merges and replaces + * branches as the filters said, reading only which instructions and branches are marked. Each probe here + * takes that walk over the edges the pass's builder wired and applies the pass's filter output to what it + * marked, so a method costs its size plus its probes' walks ({@link Hits#steps}), with no allocation per + * probe. The walks are short when the method's statements join at labels many edges reach (JaCoCo puts a + * probe on each of those edges), and quadratic in a chain of early returns, where each probe walks back + * over every check before it. A method whose + * flow does not have that shape (an instruction two edges without a probe reach, a replacement that reads + * another replaced instruction) throws. */ public static int analyze(byte[] bytes, Map out, EachProbe each) { return analyze(bytes, null, out, each); @@ -147,10 +192,11 @@ public MethodProbesVisitor visitMethod(int access, final String name, final Stri final String signature, String[] exceptions) { InstrSupport.assertNotInstrumented(name, getClassName()); final Method m = new Method(name, desc); - final Builder builder = new Builder(probes, m); + final Builder builder = new Builder(probes, m, each != null); return new MethodAnalyzer(builder) { @Override public void accept(MethodNode node, MethodVisitor visitor) { + builder.start(node.instructions); super.accept(node, visitor); collect(m, builder, node, signature); } @@ -165,42 +211,215 @@ private void collect(Method m, Builder builder, MethodNode node, String signatur } out.put(m.name + m.desc, m); if (each != null && each.wants(m)) { - replay(m, node, signature, output); + map(m, builder, node, output); } } /** - * replay hands each of m's probes to each with m's coverage when that probe alone is executed: the - * method's node again through a new probe adapter (its probe ids from the method's first) and analyzer, - * and the filters' output of the pass that made m. + * map hands each of m's probes to each with what that probe alone covers, from this pass: the edges + * builder wired (each instruction's predecessor) and the filters' output. */ - private void replay(Method m, MethodNode node, String signature, Output output) { + private void map(Method m, Builder builder, MethodNode node, Output output) { if (m.probes.length == 0) { return; } - int first = Integer.MAX_VALUE; - int last = -1; - for (int id : m.probes) { - first = Math.min(first, id); - last = Math.max(last, id); + InsnList insns = node.instructions; + int n = insns.size(); + Map instructions = builder.getInstructions(); + // Each instruction's predecessor and the branch of it that leads here: its fall-through, or the + // jump into it. JaCoCo's builder wires both and keeps the last; here none may have two. + int[] pred = Arrays.copyOf(builder.fall, n); + int[] predBranch = new int[n]; + for (int j = 0; j < builder.jumps; j++) { + Integer t = builder.labelAt.get(builder.jumpTarget.get(j)); + if (t == null) { + throw shape(m, "a jump to a label with no instruction"); + } + if (pred[t] >= 0) { + throw shape(m, "two edges without a probe into instruction " + t); + } + if (builder.jumpFrom[j] < 0) { + throw shape(m, "a jump from no instruction"); + } + pred[t] = builder.jumpFrom[j]; + predBranch[t] = builder.jumpBranch[j]; } - boolean[] one = new boolean[last + 1]; + // The counted instructions, in fill's order, and each instruction's merge representative. + int[] counted = new int[n]; + int[] root = new int[n]; + Arrays.fill(counted, -1); + int c = 0; + for (AbstractInsnNode i = insns.getFirst(); i != null; i = i.getNext()) { + if (instructions.get(i) == null) { + continue; + } + root[insns.indexOf(i)] = insns.indexOf(output.representative(i)); + if (!output.ignored.contains(i)) { + counted[insns.indexOf(i)] = c++; + } + } + if (c != m.lines.length) { + throw shape(m, c + " counted instructions, " + m.lines.length + " in its coverage"); + } + // A counted instruction's branch numbers in fill's order (its row of branches), and, for each new + // branch of a replaced one, the instruction branches it reads: key root << 32 | branch, sorted, the + // new branch (counted instruction << 32 | its index) beside it. + int[][] numbers = new int[c][]; + long[] reads = new long[8]; + long[] feeds = new long[8]; + int nReads = 0; + for (AbstractInsnNode i = insns.getFirst(); i != null; i = i.getNext()) { + int at = insns.indexOf(i); + int k = counted[at]; + if (k < 0) { + continue; + } + Replacements r = output.replacements.get(i); + if (r == null) { + // fill checked that a counted instruction's branch numbers are as many as its branches. + numbers[k] = new int[m.branches[k].length]; + for (int x = 0; x < numbers[k].length; x++) { + numbers[k][x] = builder.number(at, x); + } + continue; + } + int t = 0; + for (Collection branch : r.values()) { + for (Replacements.InstructionBranch from : branch) { + if (from.instruction != i && output.replacements.containsKey(from.instruction)) { + throw shape(m, "a replaced branch reads another replaced instruction"); + } + if (instructions.get(from.instruction) == null) { + throw shape(m, "a replaced branch reads no instruction"); + } + if (nReads == reads.length) { + reads = Arrays.copyOf(reads, nReads * 2); + feeds = Arrays.copyOf(feeds, nReads * 2); + } + reads[nReads] = key(root[insns.indexOf(from.instruction)], from.branch); + feeds[nReads++] = (long) k << 32 | t; + } + t++; + } + } + sortTogether(reads, feeds, nReads); + // Per probe: the walk, by stamp (the probe's round, so nothing is cleared between probes). + int[] stamp = new int[n]; + int[] walked = new int[16]; + int[] walkedBranch = new int[16]; + int[] newStamp = new int[c]; + Hits hits = new Hits(m); for (int r = 0; r < m.probes.length; r++) { - one[m.probes[r]] = true; - Method got = new Method(m.name, m.desc); - Builder builder = new Builder(one, got); - MethodAnalyzer analyzer = new MethodAnalyzer(builder); - analyzer.accept(node, new MethodProbesAdapter(analyzer, new Ids(first))); - one[m.probes[r]] = false; - if (!fill(got, builder, node, signature, output) || !Arrays.equals(got.probes, m.probes)) { - throw new IllegalStateException("JaCoCo's replay of " + getClassName() + "#" + m.name + m.desc - + " met probes " + Arrays.toString(got.probes) + ", its analysis " - + Arrays.toString(m.probes)); + int round = r + 1; + int w = 0; + int i = builder.probeFrom[r]; + int b = builder.probeBranch[r]; + if (i < 0) { + throw shape(m, "a probe on no instruction"); + } + while (i >= 0) { + if (w == walked.length) { + walked = Arrays.copyOf(walked, w * 2); + walkedBranch = Arrays.copyOf(walkedBranch, w * 2); + } + walked[w] = i; + walkedBranch[w++] = b; + if (stamp[i] == round) { + break; + } + stamp[i] = round; + b = predBranch[i]; + i = pred[i]; } - each.probe(m, r, got); + int fed = 0; + for (int x = 0; x < w; x++) { + int rt = root[walked[x]]; + int br = walkedBranch[x]; + int k = counted[rt]; + if (k >= 0 && numbers[k] != null) { + // Merged: the representative carries its group's marks. + mark(hits, newStamp, round, k); + if (numbers[k].length > 0) { + int pos = Arrays.binarySearch(numbers[k], br); + if (pos < 0) { + throw shape(m, "branch " + br + " marked, not one of its instruction's"); + } + hits.branches[k][pos] = true; + } + } + // Replaced: each new branch that reads this instruction branch. + long key = key(rt, br); + for (int y = lowerBound(reads, nReads, key); y < nReads && reads[y] == key; y++) { + int kk = (int) (feeds[y] >>> 32); + int t = (int) feeds[y]; + fed++; + mark(hits, newStamp, round, kk); + if (t < hits.branches[kk].length) { + hits.branches[kk][t] = true; + } + } + } + hits.steps = w + fed; + each.probe(m, r, hits); + hits.clear(); } } + private static long key(int instruction, int branch) { + return (long) instruction << 32 | (branch & 0xffffffffL); + } + + /** lowerBound is the first index of the n sorted keys whose key is at least key. */ + private static int lowerBound(long[] keys, int n, long key) { + int lo = 0; + int hi = n; + while (lo < hi) { + int mid = (lo + hi) >>> 1; + if (keys[mid] < key) { + lo = mid + 1; + } else { + hi = mid; + } + } + return lo; + } + + /** sortTogether sorts the first n keys ascending, moving each value with its key (a merge sort). */ + private static void sortTogether(long[] keys, long[] values, int n) { + long[] k2 = new long[n]; + long[] v2 = new long[n]; + for (int width = 1; width < n; width *= 2) { + for (int lo = 0; lo < n; lo += 2 * width) { + int mid = Math.min(lo + width, n); + int hi = Math.min(lo + 2 * width, n); + int a = lo; + int b = mid; + for (int o = lo; o < hi; o++) { + if (a < mid && (b >= hi || keys[a] <= keys[b])) { + k2[o] = keys[a]; + v2[o] = values[a++]; + } else { + k2[o] = keys[b]; + v2[o] = values[b++]; + } + } + } + System.arraycopy(k2, 0, keys, 0, n); + System.arraycopy(v2, 0, values, 0, n); + } + } + + private static void mark(Hits hits, int[] newStamp, int round, int k) { + if (newStamp[k] != round) { + newStamp[k] = round; + hits.at[hits.count++] = k; + } + } + + private IllegalStateException shape(Method m, String what) { + return new IllegalStateException(getClassName() + "#" + m.name + m.desc + ": " + what); + } + /** * fill sets m's counted instructions from builder's graph and the filters' output, as ClassAnalyzer * computes a method's coverage. It returns false for a method without code. @@ -217,11 +436,13 @@ private boolean fill(Method m, Builder builder, MethodNode node, String signatur int[] p = new int[m.probeCount]; System.arraycopy(m.probes, 0, p, 0, m.probeCount); m.probes = p; - List lines = new ArrayList<>(); - List covered = new ArrayList<>(); - List branches = new ArrayList<>(); + InsnList insns = node.instructions; + int[] lines = new int[insns.size()]; + boolean[] covered = new boolean[insns.size()]; + boolean[][] branches = new boolean[insns.size()][]; + int c = 0; Map sums = new HashMap<>(); // line -> instructions, covered, branches, covered - for (AbstractInsnNode i = node.instructions.getFirst(); i != null; i = i.getNext()) { + for (AbstractInsnNode i = insns.getFirst(); i != null; i = i.getNext()) { if (output.ignored.contains(i)) { continue; } @@ -234,23 +455,25 @@ private boolean fill(Method m, Builder builder, MethodNode node, String signatur // The instruction's branch numbers: those its builder gave it (a switch's may skip numbers), // or 0..k-1 for the k new branches a filter replaced them with. Integer replaced = output.replaced.get(i); - Set numbers = replaced != null ? range(replaced) : builder.branches.get(i); - if (n > 0 && (numbers == null || numbers.size() != n)) { + int at = insns.indexOf(i); + int count = replaced != null ? replaced : builder.numbers(at); + if (n > 0 && count != n) { throw new IllegalStateException("JaCoCo counts " + n + " branches at " + getClassName() + "#" - + m.name + m.desc + " line " + insn.getLine() + ", its builder gave " + numbers); + + m.name + m.desc + " line " + insn.getLine() + ", its builder gave " + count); } boolean[] b = new boolean[n]; - int k = 0; - for (int number : n > 0 ? numbers : new TreeSet()) { + // An instruction not covered has no covered branch (JaCoCo covers an instruction when one of + // its branches is), so only a covered one's branches are read. + for (int k = 0; hit && k < n; k++) { // replaceBranches with one new branch fed by this branch alone is covered exactly when the // branch is: JaCoCo's own way to read one branch. Replacements one = new Replacements(); - one.add(i, i, number); - b[k++] = insn.replaceBranches(one, x -> insn).getInstructionCounter().getCoveredCount() > 0; + one.add(i, i, replaced != null ? k : builder.number(at, k)); + b[k] = insn.replaceBranches(one, x -> insn).getInstructionCounter().getCoveredCount() > 0; } - lines.add(insn.getLine()); - covered.add(hit); - branches.add(b); + lines[c] = insn.getLine(); + covered[c] = hit; + branches[c++] = b; int[] s = sums.get(insn.getLine()); if (s == null) { s = new int[4]; @@ -276,28 +499,15 @@ private boolean fill(Method m, Builder builder, MethodNode node, String signatur + line.getInstructionCounter() + " " + line.getBranchCounter()); } } - m.lines = new int[lines.size()]; - m.covered = new boolean[lines.size()]; - m.branches = new boolean[lines.size()][]; - for (int k = 0; k < lines.size(); k++) { - m.lines[k] = lines.get(k); - m.covered[k] = covered.get(k); - m.branches[k] = branches.get(k); - } + m.lines = Arrays.copyOf(lines, c); + m.covered = Arrays.copyOf(covered, c); + m.branches = Arrays.copyOf(branches, c); return true; } private static boolean same(ICounter c, int total, int covered) { return c.getTotalCount() == total && c.getCoveredCount() == covered; } - - private static Set range(int n) { - Set out = new TreeSet<>(); - for (int k = 0; k < n; k++) { - out.add(k); - } - return out; - } } /** @@ -307,29 +517,114 @@ private static Set range(int n) { */ private static final class Builder extends InstructionsBuilder { private final Method method; - final Map> branches = new HashMap<>(); private AbstractInsnNode current; - - Builder(boolean[] probes, Method method) { + private InsnList insns; + // Each branch number the builder gave an instruction, as index << 32 | number; sorted, de-duplicated + // and indexed (noteStart: where each instruction's numbers start; one more for the end) on first read. + private long[] notes = new long[16]; + private int noteCount; + private int[] noteStart; + private Map wired; + // With flow kept (for map), by index in the method's instruction list: each instruction's fall-through + // predecessor (-1: none), each jump (from, branch, target label, in the order the builder wires them), + // the instruction each label is at, and each probe's instruction and branch (in method.probes order). + private final boolean flow; + int[] fall; + int jumps; + int[] jumpFrom = new int[0]; + int[] jumpBranch = new int[0]; + final List

heap_after_gc_max_mb is the most heap any collection during the send left in use: an upper bound of what * the analysis keeps live (a young collection also counts old garbage not collected yet). It is -1 when no @@ -66,8 +67,15 @@ public void aGeneratedAppsTableIsMeasured() throws Exception { assertEquals(n, b.tableRecords); assertEquals(n, b.probeRecords); assertEquals("every class is measured", 0, b.notMeasured); + // What the send allocates per class: about 1.9 MiB when each probe re-analyzed its method, 125-141 KiB + // (JaCoCo's analysis of the class, twice, and the records' text) since each probe is read off one pass. + long perClass = b.allocated / n; + assertTrue("allocated " + perClass + " bytes a class", perClass < ALLOCATED_PER_CLASS); } + /** The most a generated class's table and probe map may allocate on the send's thread. */ + static final long ALLOCATED_PER_CLASS = 256 << 10; + @Test(timeout = 600_000) public void aClassWithOneLongMethodIsMeasured() throws Exception { int branches = Integer.getInteger("keploy.budget.branches", 500); @@ -78,6 +86,45 @@ public void aClassWithOneLongMethodIsMeasured() throws Exception { assertEquals(0, b.notMeasured); } + @Test(timeout = 600_000) + public void aMethodOf3000BranchesMapsInUnderASecond() throws Exception { + // An if/else chain's statements join at labels two edges reach, so its probes' walks are short. When + // each probe re-analyzed its method, 3,500 took about a minute, past the CLI's 30 s wait for a frame, + // so one such class left every run's table incomplete. + List one = new ArrayList<>(); + one.add(new BuildTable.ClassFile("gen/big/Long3000", gen("gen/big/Long3000", 0, 3000))); + Budget b = measure("one-method-3000-branches", one); + assertEquals(1, b.probeRecords); + assertEquals(0, b.notMeasured); + assertTrue("the send took " + (b.cpuNanos / 1_000_000) + " ms of CPU", b.cpuNanos < 1_000_000_000L); + } + + @Test(timeout = 600_000) + public void aChainOfEarlyReturnsIsMappedOrTooLargeAndNeverUnbounded() throws Exception { + // Probe k of a chain of n early returns walks back over the k checks before it and covers each one's + // line and branch, so the map is about n * n ranges: 4.4 MB at 1,000 and 43 MB at 3,000; 4,500 ran a + // 256 MiB heap out of memory (and 3,000 a 128 MiB one) before the map had a bound. Past it the class + // is record_too_large, given up as its map grows past the bound. + for (int n : new int[] {1000, 3000, 4500}) { + String name = "gen/big/Returns" + n; + List one = new ArrayList<>(); + one.add(new BuildTable.ClassFile(name, returns(name, n))); + Budget b = measure("early-returns-" + n, one); + boolean mapped = n == 1000; + assertEquals(1, b.tableRecords); + assertEquals(name, mapped ? 1 : 0, b.probeRecords); + assertEquals(name, mapped ? 0 : 1, b.notMeasured); + assertEquals(name, mapped ? 0 : 1, b.tooLarge); + assertTrue(name + ": the send took " + (b.cpuNanos / 1_000_000) + " ms of CPU", + b.cpuNanos < 1_000_000_000L); + assertTrue(name + ": the send allocated " + (b.allocated >> 20) + " MiB", + b.allocated < MAP_ALLOCATED); + } + } + + /** The most one class's table and probe map may allocate, its map at the bound: 8 MiB of text. */ + static final long MAP_ALLOCATED = 64 << 20; + @Test(timeout = 600_000) public void aRealLibraryIsMeasured() throws Exception { File jar = new File(Analyzer.class.getProtectionDomain().getCodeSource().getLocation().toURI()); @@ -133,10 +180,57 @@ static byte[] gen(String name, int i, int branches) { return cw.toByteArray(); } + /** count is how many times s holds what, ending past its first carried chars (counted with the frame before). */ + static int count(String s, String what, int carried) { + int n = 0; + for (int k = s.indexOf(what); k >= 0; k = s.indexOf(what, k + 1)) { + if (k + what.length() > carried) { + n++; + } + } + return n; + } + + /** returns is a class with a static run(int) of n lines "if (x == k) return k;", then "return -1;". */ + static byte[] returns(String name, int n) { + ClassWriter cw = new ClassWriter(ClassWriter.COMPUTE_FRAMES | ClassWriter.COMPUTE_MAXS); + cw.visit(Opcodes.V1_8, Opcodes.ACC_PUBLIC | Opcodes.ACC_SUPER, name, null, "java/lang/Object", null); + cw.visitSource(name.substring(name.lastIndexOf('/') + 1) + ".java", null); + MethodVisitor mv = cw.visitMethod(Opcodes.ACC_PUBLIC | Opcodes.ACC_STATIC, "run", "(I)I", null, null); + mv.visitCode(); + for (int k = 0; k < n; k++) { + Label line = new Label(); + mv.visitLabel(line); + mv.visitLineNumber(10 + k, line); + Label next = new Label(); + mv.visitVarInsn(Opcodes.ILOAD, 0); + mv.visitIntInsn(Opcodes.SIPUSH, k); + mv.visitJumpInsn(Opcodes.IF_ICMPNE, next); + mv.visitIntInsn(Opcodes.SIPUSH, k); + mv.visitInsn(Opcodes.IRETURN); + mv.visitLabel(next); + } + Label end = new Label(); + mv.visitLabel(end); + mv.visitLineNumber(10 + n, end); + mv.visitInsn(Opcodes.ICONST_M1); + mv.visitInsn(Opcodes.IRETURN); + mv.visitMaxs(0, 0); + mv.visitEnd(); + cw.visitEnd(); + return cw.toByteArray(); + } + static final class Budget { int tableRecords; int probeRecords; int notMeasured; + /** How many of the table's records are record_too_large. */ + int tooLarge; + /** Bytes the send allocated on its thread (the sink's own allocation left out). */ + long allocated; + /** The send thread's CPU time, in nanoseconds. */ + long cpuNanos; } /** measure sends the table of classes to a sink that counts, as the table's thread does, and reports. */ @@ -148,6 +242,7 @@ static Budget measure(String label, List classes) throws E } } AtomicLong afterGcMax = new AtomicLong(-1); + AtomicLong committedGcMax = new AtomicLong(-1); AtomicInteger collections = new AtomicInteger(); NotificationListener listener = (n, handback) -> { if (!GarbageCollectionNotificationInfo.GARBAGE_COLLECTION_NOTIFICATION.equals(n.getType())) { @@ -165,6 +260,19 @@ static Budget measure(String label, List classes) throws E used += e.getValue().getUsed(); } } + long before = 0; + long after = 0; + for (Map.Entry e : info.getGcInfo().getMemoryUsageBeforeGc().entrySet()) { + if (heapPools.contains(e.getKey())) { + before += e.getValue().getCommitted(); + } + } + for (Map.Entry e : info.getGcInfo().getMemoryUsageAfterGc().entrySet()) { + if (heapPools.contains(e.getKey())) { + after += e.getValue().getCommitted(); + } + } + committedGcMax.accumulateAndGet(Math.max(before, after), Math::max); collections.incrementAndGet(); afterGcMax.accumulateAndGet(used, Math::max); }; @@ -176,7 +284,7 @@ static Budget measure(String label, List classes) throws E } } try { - return measure(label, classes, heapPools, afterGcMax, collections); + return measure(label, classes, heapPools, afterGcMax, committedGcMax, collections); } finally { for (NotificationEmitter e : emitters) { e.removeNotificationListener(listener); @@ -185,8 +293,23 @@ static Budget measure(String label, List classes) throws E } private static Budget measure(String label, List classes, Set heapPools, - AtomicLong afterGcMax, AtomicInteger collections) throws Exception { + AtomicLong afterGcMax, AtomicLong committedGcMax, AtomicInteger collections) throws Exception { + long committedPre = heapCommitted(); System.gc(); + long committedBase = heapCommitted(); + AtomicLong committedSampled = new AtomicLong(committedBase); + java.util.concurrent.atomic.AtomicBoolean sending = new java.util.concurrent.atomic.AtomicBoolean(true); + Thread sampler = new Thread(() -> { + while (sending.get()) { + committedSampled.accumulateAndGet(heapCommitted(), Math::max); + try { + Thread.sleep(2); + } catch (InterruptedException e) { + return; + } + } + }, "budget-committed"); + sampler.setDaemon(true); long base = 0; for (MemoryPoolMXBean p : ManagementFactory.getMemoryPoolMXBeans()) { if (heapPools.contains(p.getName())) { @@ -206,16 +329,23 @@ private static Budget measure(String label, List classes, long[] bytes = new long[2]; boolean live = Boolean.getBoolean("keploy.budget.live"); long[] liveMax = {-1}; - // A record may run on into the next frame: not_measured is counted over the table's stream, keeping - // the end of each frame that could begin a split match. + // A record may run on into the next frame: not_measured and record_too_large are counted over the + // table's stream, keeping the end of each frame that could begin a split match. String notMeasured = "\"state\":\"not_measured\""; + String tooLarge = "\"reason\":{\"code\":\"record_too_large\"}"; String[] carry = {""}; Budget b = new Budget(); long gc0 = gcMillis(); long process0 = processCpuNanos(); + com.sun.management.ThreadMXBean allocation = (com.sun.management.ThreadMXBean) threads; + long self = Thread.currentThread().getId(); + long[] sinkAllocated = {0}; + long allocated0 = allocation.getThreadAllocatedBytes(self); long cpu0 = threads.getCurrentThreadCpuTime(); long t0 = System.nanoTime(); + sampler.start(); BuildTable.send(() -> classes, frame -> { + long sink0 = allocation.getThreadAllocatedBytes(self); if (live) { System.gc(); long used = 0; @@ -241,13 +371,17 @@ private static Budget measure(String label, List classes, } if (!probes) { String body = carry[0] + f.substring(f.indexOf('\n', at) + 1); - for (int k = body.indexOf(notMeasured); k >= 0; k = body.indexOf(notMeasured, k + 1)) { - b.notMeasured++; - } - carry[0] = body.substring(Math.max(0, body.length() - (notMeasured.length() - 1))); + b.notMeasured += count(body, notMeasured, carry[0].length()); + b.tooLarge += count(body, tooLarge, carry[0].length()); + carry[0] = body.substring(Math.max(0, body.length() - (tooLarge.length() - 1))); } + sinkAllocated[0] += allocation.getThreadAllocatedBytes(self) - sink0; }); long wall = System.nanoTime() - t0; + long allocated = allocation.getThreadAllocatedBytes(self) - allocated0 - sinkAllocated[0]; + sending.set(false); + sampler.join(); + long committedEnd = heapCommitted(); long cpu = threads.getCurrentThreadCpuTime() - cpu0; long process = processCpuNanos() - process0; long gc = gcMillis() - gc0; @@ -263,15 +397,36 @@ private static Budget measure(String label, List classes, System.out.printf(Locale.ROOT, "BUDGET %s java=%s heap_max_mb=%d classes=%d class_bytes=%d wall_ms=%d" + " thread_cpu_ms=%d process_cpu_ms=%d gc_ms=%d collections=%d heap_before_mb=%d" + " heap_after_gc_max_mb=%d heap_delta_mb=%d heap_peak_mb=%d live_max_mb=%d live_delta_mb=%d" - + " table_bytes=%d probes_bytes=%d not_measured=%d%n", + + " table_bytes=%d probes_bytes=%d not_measured=%d allocated_mb=%d allocated_kb_per_class=%d%n", label, System.getProperty("java.version"), Runtime.getRuntime().maxMemory() >> 20, classes.size(), classBytes, wall / 1_000_000, cpu / 1_000_000, process / 1_000_000, gc, collections.get(), base >> 20, afterGc < 0 ? -1 : afterGc >> 20, afterGc < 0 ? -1 : (afterGc - base) >> 20, peak >> 20, liveMax[0] < 0 ? -1 : liveMax[0] >> 20, liveMax[0] < 0 ? -1 : (liveMax[0] - base) >> 20, - bytes[0], bytes[1], b.notMeasured); + bytes[0], bytes[1], b.notMeasured, allocated >> 20, (allocated / Math.max(1, classes.size())) >> 10); + b.allocated = allocated; + b.cpuNanos = cpu; + long committedPeak = Math.max(committedSampled.get(), committedGcMax.get()); + System.out.printf(Locale.ROOT, "BUDGET-COMMITTED %s java=%s gc=%s heap_max_mb=%d committed_pre_gc_mb=%d" + + " committed_before_mb=%d committed_peak_mb=%d committed_end_mb=%d committed_growth_mb=%d" + + " committed_growth_from_pre_gc_mb=%d%n", + label, System.getProperty("java.version"), gcNames(), Runtime.getRuntime().maxMemory() >> 20, + committedPre >> 20, committedBase >> 20, committedPeak >> 20, committedEnd >> 20, + (committedPeak - committedBase) >> 20, (committedPeak - committedPre) >> 20); return b; } + private static long heapCommitted() { + return ManagementFactory.getMemoryMXBean().getHeapMemoryUsage().getCommitted(); + } + + private static String gcNames() { + StringBuilder names = new StringBuilder(); + for (GarbageCollectorMXBean g : ManagementFactory.getGarbageCollectorMXBeans()) { + names.append(names.length() == 0 ? "" : "+").append(g.getName().replace(' ', '_')); + } + return names.toString(); + } + private static long gcMillis() { long t = 0; for (GarbageCollectorMXBean g : ManagementFactory.getGarbageCollectorMXBeans()) { diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java index e3bd621c..f3e29a9e 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java @@ -7,6 +7,7 @@ import org.junit.Rule; import org.junit.Test; import org.junit.rules.TemporaryFolder; +import org.jacoco.core.internal.analysis.KeployBranchCoverage; import org.jacoco.core.internal.data.CRC64; import org.objectweb.asm.ClassWriter; import org.objectweb.asm.Label; @@ -37,6 +38,7 @@ import java.util.LinkedHashMap; import java.util.List; import java.util.Map; +import java.util.function.Supplier; import java.util.stream.Collectors; import java.util.concurrent.CountDownLatch; import java.util.concurrent.TimeUnit; @@ -301,6 +303,75 @@ public void aRecordLargerThanAChunkInflatesToIsRecordTooLarge() throws Exception assertEquals(2, s.notMeasured); } + @Test + public void aProbeMapPastItsBytesOrStepsIsRecordTooLargeWithNoProbeRecord() throws Exception { + String name = "gen/big/Returns"; + byte[] bytes = BuildTableBudgetTest.returns(name, 200); + List one = Collections.singletonList(new BuildTable.ClassFile(name, bytes)); + Collect probes = new Collect(); + BuildTable.write(one, new Collect(), probes, BuildTable.MAX_RECORD_BYTES); + assertEquals(1, probes.records.size()); + // What the map takes: its record and newline (ASCII), and its probes' walks (the method has a line, so + // the table's walks are all of them). + int mapBytes = probes.records.get(0).length() + 1; + long[] steps = {0}; + KeployBranchCoverage.analyze(bytes, new HashMap<>(), (m, r, h) -> steps[0] += h.steps); + // A chain of 200 early returns: each probe walks back over the checks before it. + assertTrue(steps[0] + " steps", steps[0] > 200L * 200); + assertTrue(mapBytes + " bytes", mapBytes > 100_000); + + long[][] bounds = {{mapBytes, steps[0]}, {mapBytes - 1, steps[0]}, {mapBytes, steps[0] - 1}}; + for (int i = 0; i < bounds.length; i++) { + boolean at = i == 0; + Collect table = new Collect(); + Collect map = new Collect(); + BuildTable.Summary s = BuildTable.write(one, table, map, BuildTable.MAX_RECORD_BYTES, + (int) bounds[i][0], bounds[i][1]); + String why = "bytes " + bounds[i][0] + ", steps " + bounds[i][1]; + assertEquals(why, 1, s.records); + assertEquals(why, at ? 0 : 1, s.notMeasured); + assertEquals(why, at ? 0 : 1, s.tooLarge); + assertEquals(why, 0, s.notMapped); + assertEquals(why, at ? 1 : 0, s.probeRecords); + assertEquals(why, at ? 1 : 0, map.records.size()); + String rec = table.records.get(0); + if (at) { + assertTrue(rec, rec.contains("\"state\":\"ok\"")); + assertEquals(probes.records.get(0), map.records.get(0)); + } else { + assertTrue(rec, rec.contains("\"state\":\"not_measured\",\"reason\":{\"code\":\"record_too_large\"}") + && rec.endsWith("\"methods\":[]}")); + } + } + } + + @Test + public void theTableAnswerBoundsTheHeapByTheIndexCountHeldAndReadsNoClassFile() { + long mb = 1 << 20; + String property = System.getProperty(BuildTable.MIN_HEAP_PROPERTY); + System.clearProperty(BuildTable.MIN_HEAP_PROPERTY); + try (Logs logs = new Logs()) { + // 63 MiB free. The classes are never read here: the count is the index's when it is read already. + BuildTable.Heap heap = new BuildTable.Heap(512 * mb, 449 * mb, true); + Supplier> never = () -> { + throw new AssertionError("the answer read the classes"); + }; + File none = new File("/nonexistent/keploy-data.sock"); + assertEquals(KeployDedupAgent.UNIX_REPLY_IGNORED + " reason=low_heap heap_free_mb=63 heap_needed_mb=66", + BuildTable.answer(heap, 1025, never, none)); + assertEquals("50,000 classes", KeployDedupAgent.UNIX_REPLY_IGNORED + + " reason=low_heap heap_free_mb=63 heap_needed_mb=113", BuildTable.answer(heap, 50_000, never, + none)); + assertEquals("an index not read yet: the base alone", KeployDedupAgent.UNIX_REPLY_IGNORED + + " reason=low_heap heap_free_mb=63 heap_needed_mb=64", BuildTable.answer(heap, -1, never, none)); + assertEquals(3, logs.records.size()); + } finally { + if (property != null) { + System.setProperty(BuildTable.MIN_HEAP_PROPERTY, property); + } + } + } + /** Reads frames back: each is KTB1, a length, a header line and the next bytes of the stream. */ static final class Frames implements BuildTable.FrameSender { final List frames = new ArrayList<>(); @@ -1032,20 +1103,40 @@ public void aShortHeapDeclinesTheTableWithLowHeap() { } @Test - public void theFreeHeapATableNeedsIsTheEnvElseThePropertyElse256() { - assertEquals(256, BuildTable.minHeapMb(null, null)); - assertEquals(256, BuildTable.minHeapMb(" ", "")); - assertEquals(100, BuildTable.minHeapMb("100", "50")); - assertEquals(50, BuildTable.minHeapMb(" ", "50")); - assertEquals(7, BuildTable.minHeapMb(" 7 ", null)); - assertEquals(0, BuildTable.minHeapMb(null, "0")); + public void theFreeHeapATableNeedsIsTheEnvElseThePropertyElse64MiBAnd1KiBAClass() { + assertEquals(64, BuildTable.minHeapMb(null, null, 0)); + assertEquals(65, BuildTable.minHeapMb(" ", "", 1)); + assertEquals(65, BuildTable.minHeapMb(null, null, 1024)); + assertEquals(66, BuildTable.minHeapMb(null, null, 1025)); + assertEquals("50,000 classes", 113, BuildTable.minHeapMb(null, null, 50_000)); + assertEquals(100, BuildTable.minHeapMb("100", "50", 50_000)); + assertEquals(50, BuildTable.minHeapMb(" ", "50", 50_000)); + assertEquals(7, BuildTable.minHeapMb(" 7 ", null, 50_000)); + assertEquals(0, BuildTable.minHeapMb(null, "0", 50_000)); try (Logs logs = new Logs()) { - assertEquals(256, BuildTable.minHeapMb("-1", "50")); - assertEquals(256, BuildTable.minHeapMb(null, "1g")); + assertEquals(113, BuildTable.minHeapMb("-1", "50", 50_000)); + assertEquals(64, BuildTable.minHeapMb(null, "1g", 0)); assertEquals(2, logs.records.size()); assertEquals(Level.WARNING, logs.records.get(1).getLevel()); - assertTrue(logs.records.get(1).getMessage(), logs.records.get(1).getMessage().contains("'1g'")); + assertTrue(logs.records.get(1).getMessage(), logs.records.get(1).getMessage().contains("'1g'") + && logs.records.get(1).getMessage().contains("needs 64 MiB")); + } + } + + @Test + public void aTableOf50000ClassesIsBuiltWith113MiBFreeAndDeclinedWith112() { + // Both edges of the default bound, as the answer reads it: the count of the index's class list. + long mb = 1 << 20; + long need = BuildTable.minHeapMb(null, null, 50_000); + assertNull("113 MiB free is enough", BuildTable.lowHeap(new BuildTable.Heap(1024 * mb, (1024 - 113) * mb, + true), need)); + try (Logs logs = new Logs()) { + assertEquals("reason=low_heap heap_free_mb=112 heap_needed_mb=113", BuildTable.lowHeap( + new BuildTable.Heap(1024 * mb, (1024 - 113) * mb + 1, true), need)); } + // A 1 GiB pod with no -Xmx caps the heap at about 256 MiB: with its app's live set under 143 MiB, a + // 50,000-class app's table is built (the 256 MiB bound before declined every such pod). + assertNull(BuildTable.lowHeap(new BuildTable.Heap(256 * mb, 143 * mb, true), need)); } @Test diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/ProbeMapTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/ProbeMapTest.java index f2e7421b..7767581e 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/ProbeMapTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/ProbeMapTest.java @@ -112,12 +112,12 @@ public void javaDedupProbeMapIsTheGolden() throws Exception { } @Test - public void aMethodTheCallerDeclinesIsNotReplayed() { + public void aMethodTheCallerDeclinesIsNotWalked() { // The map declines the methods the table has no line of (a class built without -g): their probes cover - // no leaf, so their replays would be thrown away. The class pass still analyzes them. + // no leaf, so their walks would be thrown away. The class pass still analyzes them. byte[] bytes = BuildTableBudgetTest.gen("gen/app/Declined", 0, 40); Map shape = new HashMap<>(); - List replayed = new ArrayList<>(); + List walked = new ArrayList<>(); int total = KeployBranchCoverage.analyze(bytes, shape, new KeployBranchCoverage.EachProbe() { @Override public boolean wants(KeployBranchCoverage.Method m) { @@ -125,75 +125,106 @@ public boolean wants(KeployBranchCoverage.Method m) { } @Override - public void probe(KeployBranchCoverage.Method m, int r, KeployBranchCoverage.Method got) { - replayed.add(m.name); + public void probe(KeployBranchCoverage.Method m, int r, KeployBranchCoverage.Hits h) { + walked.add(m.name); } }); assertTrue(shape.containsKey("run(I)I")); - assertTrue("run has probes to replay", shape.get("run(I)I").probes.length > 40); - assertEquals(total - shape.get("run(I)I").probes.length, replayed.size()); - assertFalse("a declined method is not replayed", replayed.contains("run")); - assertTrue(replayed.contains("")); + assertTrue("run has probes to walk", shape.get("run(I)I").probes.length > 40); + assertEquals(total - shape.get("run(I)I").probes.length, walked.size()); + assertFalse("a declined method is not walked", walked.contains("run")); + assertTrue(walked.contains("")); + } + + /** What one probe covers, copied out of the analysis's reused Hits: per counted instruction. */ + static final class Got { + final boolean[] covered; + final boolean[][] branches; + + Got(KeployBranchCoverage.Method m, KeployBranchCoverage.Hits h) { + covered = new boolean[m.lines.length]; + branches = new boolean[m.lines.length][]; + for (int k = 0; k < branches.length; k++) { + branches[k] = h.branches[k].clone(); + } + for (int j = 0; j < h.count; j++) { + assertFalse("an instruction once in hits", covered[h.at[j]]); + covered[h.at[j]] = true; + } + for (int k = 0; k < branches.length; k++) { + for (boolean b : branches[k]) { + assertTrue("a covered branch of a covered instruction", !b || covered[k]); + } + } + } } @Test(timeout = 300000) - public void eachProbesReplayIsTheClassAnalyzedWithThatProbeExecuted() throws Exception { - // The oracle is the class analyzed whole, once per round r, with the r-th probe of every method executed - // (a method's coverage reads only its own probes). Against it, each probe's one-pass replay, on real - // libraries' classes: JaCoCo core's and Gson's, the jars on this test's classpath. + public void eachProbesMapIsTheClassAnalyzedWithThatProbeExecuted() throws Exception { + // The oracle is the class analyzed whole by JaCoCo's builder and calculator, once per round r, with the + // r-th probe of every method executed (a method's coverage reads only its own probes). Against it, what + // the one pass maps each probe to, on real libraries' classes (JaCoCo core's and Gson's, the jars on + // this test's classpath) and on every fixture (java-dedup, the code shapes, the Java 21 shapes). int classes = 0; int probes = 0; + List all = new ArrayList<>(); for (Class c : new Class[] {Analyzer.class, JsonParser.class}) { File jar = new File(c.getProtectionDomain().getCodeSource().getLocation().toURI()); assertTrue(jar + " is a jar", jar.isFile()); - for (BuildTable.ClassFile f : jarClasses(jar)) { - Map shape = new HashMap<>(); - Map> replayed = new HashMap<>(); - int total = KeployBranchCoverage.analyze(f.bytes, shape, (m, r, got) -> { - List l = replayed.computeIfAbsent(m.name + m.desc, - k -> new ArrayList<>()); - assertEquals(f.name + "#" + m.name + m.desc, r, l.size()); - l.add(got); - }); - if (total < 0) { - continue; + all.addAll(jarClasses(jar)); + } + all.addAll(BuildTableTest.javaDedup()); + all.addAll(load("shapes")); + all.addAll(load("modern")); + all.add(new BuildTable.ClassFile("gen/app/Long", BuildTableBudgetTest.gen("gen/app/Long", 3, 300))); + // Each probe of a chain of early returns walks back over every check before it. + all.add(new BuildTable.ClassFile("gen/app/Returns", BuildTableBudgetTest.returns("gen/app/Returns", 80))); + for (BuildTable.ClassFile f : all) { + Map shape = new HashMap<>(); + Map> replayed = new HashMap<>(); + int total = KeployBranchCoverage.analyze(f.bytes, shape, (m, r, h) -> { + List l = replayed.computeIfAbsent(m.name + m.desc, k -> new ArrayList<>()); + assertEquals(f.name + "#" + m.name + m.desc, r, l.size()); + l.add(new Got(m, h)); + }); + if (total < 0) { + continue; + } + classes++; + int rounds = 0; + int replays = 0; + for (Map.Entry e : shape.entrySet()) { + int n = e.getValue().probes.length; + rounds = Math.max(rounds, n); + replays += n; + List l = replayed.get(e.getKey()); + assertEquals(f.name + "#" + e.getKey(), n, l == null ? 0 : l.size()); + } + // A method whose code the filters leave out entirely (an enum's values()) keeps its probes: no leaf. + assertTrue(f.name + ": " + replays + " probes replayed of " + total, replays <= total); + for (int r = 0; r < rounds; r++) { + boolean[] set = new boolean[total]; + for (KeployBranchCoverage.Method m : shape.values()) { + if (m.probes.length > r) { + set[m.probes[r]] = true; + } } - classes++; - int rounds = 0; - int replays = 0; + Map run = new HashMap<>(); + KeployBranchCoverage.analyze(f.bytes, set, run); for (Map.Entry e : shape.entrySet()) { - int n = e.getValue().probes.length; - rounds = Math.max(rounds, n); - replays += n; - List l = replayed.get(e.getKey()); - assertEquals(f.name + "#" + e.getKey(), n, l == null ? 0 : l.size()); - } - // A method whose code the filters leave out entirely (an enum's values()) keeps its probes: no leaf. - assertTrue(f.name + ": " + replays + " probes replayed of " + total, replays <= total); - for (int r = 0; r < rounds; r++) { - boolean[] set = new boolean[total]; - for (KeployBranchCoverage.Method m : shape.values()) { - if (m.probes.length > r) { - set[m.probes[r]] = true; - } + if (e.getValue().probes.length <= r) { + continue; } - Map run = new HashMap<>(); - KeployBranchCoverage.analyze(f.bytes, set, run); - for (Map.Entry e : shape.entrySet()) { - if (e.getValue().probes.length <= r) { - continue; - } - KeployBranchCoverage.Method want = run.get(e.getKey()); - KeployBranchCoverage.Method got = replayed.get(e.getKey()).get(r); - String at = f.name + "#" + e.getKey() + " probe " + r; - assertArrayEquals(at, want.lines, got.lines); - assertArrayEquals(at, want.covered, got.covered); - assertEquals(at, want.branches.length, got.branches.length); - for (int k = 0; k < want.branches.length; k++) { - assertArrayEquals(at, want.branches[k], got.branches[k]); - } - probes++; + KeployBranchCoverage.Method want = run.get(e.getKey()); + Got got = replayed.get(e.getKey()).get(r); + String at = f.name + "#" + e.getKey() + " probe " + r; + assertArrayEquals(at, want.lines, e.getValue().lines); + assertArrayEquals(at, want.covered, got.covered); + assertEquals(at, want.branches.length, got.branches.length); + for (int k = 0; k < want.branches.length; k++) { + assertArrayEquals(at, want.branches[k], got.branches[k]); } + probes++; } } } From 7082533118f04dbd1cf7139325ad87da59820381 Mon Sep 17 00:00:00 2001 From: slayerjain Date: Fri, 20 Nov 2026 08:43:05 +0530 Subject: [PATCH 07/14] fix(dedup): name a build by its table format and analyzer's JaCoCo too The build id hashed the class ids alone, so an SDK release that writes a different table or probe map for unchanged classes named the same build as the older SDK did. Keploy keeps one table per build, so every later save of that build was refused (409 build_table_mismatch) until the app itself was rebuilt. The id is now "jv1-" and the first 32 hex of the sha256 over the ASCII bytes "keploy-java-table " F "\n" "jacoco " V "\n" ID_1 "\n" ... ID_n with no newline after the last id, where F is the table's format (BuildTable.TABLE_FORMAT, 1) and V the release of the JaCoCo core whose analyzer builds the table: the one keploy-sdk.jar carries renamed, read from that core's own JaCoCo.VERSION less its build qualifier ("0.8.15"), not the measuring agent's, since an attached agent of another version only records the probes. The ids are as before: CRC64, sorted unsigned, de-duplicated, 16 lowercase hex. These are the bytes the contract gives Keploy's codecoverage.JavaBuildID. F covers the bounds that decide a class's records (the record, probe-map bytes and probe-map steps bounds) and the reason a unit past one is not measured, so a release that changes one raises it. The frames that carry the records are not in F: their header names its own version. The INDEX record and the table still take the id from one function, and a unit's hash is unchanged: the sum over that unit's class ids alone. V is read once, at the first build id, from a holder of its own, so the class-id helpers a testcase's END runs before its ACK never load JaCoCo's version. If the renamed core cannot read its version resource, V is "" (logged once) and the INDEX record and the table still name one build. The smoke test, which CI runs on JDK 8, 17 and 21 against the built jar, now fails a jar whose analyzer names a JaCoCo release other than its org.jacoco.core's Maven version, read both as the SDK loads from the class path and from the jar alone in a loader of its own. The README says that an SDK upgrade that changes the table format or the analyzer's JaCoCo gives the same classes a new build id. Rows: the build id's three golden vectors (ids 1, 0xff and 0xffffffffffffffff with F=1 and V="0.8.15", with F=2, and with V="0.8.14"), which differ from each other and from the id over the ids alone; the SDK's F is 1 and its V the core jar's pom.properties version; a JaCoCo version less its qualifier; the class-id helpers never load the analyzer's JaCoCo version; and a tripwire that pins F=1 together with the bounds, the step count of a 200-return chain, the sha256 of the record each cap writes (a unit past the probe-map steps, and records past a 1600-byte chunk) and of the golden table and probe-map records, so a change to any of them, its reason code included, fails until F is decided. The independent build id checks of the INDEX record and of the table now hash the same bytes, and the java-dedup and edge goldens are regenerated from an oracle that also reproduces their old ids-only values. Their table and probe-map records are unchanged. Signed-off-by: slayerjain --- README.md | 2 +- .../main/java/io/keploy/dedup/BuildTable.java | 9 ++ .../java/io/keploy/dedup/IndexReport.java | 99 ++++++++++++-- .../java/io/keploy/dedup/AppIndexTest.java | 4 +- .../java/io/keploy/dedup/BuildTableTest.java | 83 ++++++++++-- .../io/keploy/dedup/ControlSocketTest.java | 6 +- .../java/io/keploy/dedup/IndexReportTest.java | 125 ++++++++++++++++-- .../io/keploy/dedup/buildtable/edge.build | 2 +- .../keploy/dedup/buildtable/java-dedup.build | 2 +- scripts/smoke-javaagent.sh | 42 ++++++ 10 files changed, 333 insertions(+), 41 deletions(-) diff --git a/README.md b/README.md index b53a1a5c..e061d232 100644 --- a/README.md +++ b/README.md @@ -31,7 +31,7 @@ Since 2.2.0 a testcase's record also names, in `"classMismatch":[className, ...] With the JaCoCo agent in the application's JVM (the usual case), steps 2 and 5 are one pass over the probe arrays of the application's classes alone, split across up to four threads (half the cores): a probe is cleared only once it was read as hit, so a hit made while the pass runs is reported by that pass or the next one, never lost. Library classes are neither read nor cleared. If JaCoCo's runtime data cannot be read that way, or the pass fails once, the agent uses JaCoCo's own reset and dump for the rest of the run, as it does when JaCoCo is reached over TCP. -Before the first testcase's coverage, the agent sends Enterprise one INDEX record on `/tmp/coverage_data.sock`: a JSON object starting with `{"kind":"index",` that says how it found the application's own classes (`source`: `override` when the dirs and jars in `KEPLOY_JAVA_CLASS_DIRS` held them, `derived` from the launch or from `target/classes` or `build/classes/java/main` under the working directory, `whole-jar` or `classpath` when it could not tell them from the libraries, `none`), what it found them from in words (`launch`), the dirs and jars it read them from (`roots`, at most 64, and `rootCount`), the package roots it kept (`packages`), how many classes it indexed, the build id of those classes (`jv1-` and 32 hex), which JaCoCo agent measures them (`jacoco`: `embedded`, the one keploy-sdk.jar starts, `attached` by the command, `none`, or `unchecked` when the agent was not started as a `-javaagent`), that embedded agent's `includes=`, and two versions as their jars' manifests name them, or `""` when not known: keploy-sdk.jar's (`sdkVersion`) and that of the JaCoCo agent that measures the classes, embedded or attached (`jacocoVersion`). It carries no class list. It is sent as the agent starts when that path's socket file is there (Enterprise's receiver makes it), and otherwise, or when that send fails, right before the first testcase's coverage; a receiver Enterprise binds anew on that path gets it again before its first record. On the TCP transport used in Kubernetes no INDEX record is sent. +Before the first testcase's coverage, the agent sends Enterprise one INDEX record on `/tmp/coverage_data.sock`: a JSON object starting with `{"kind":"index",` that says how it found the application's own classes (`source`: `override` when the dirs and jars in `KEPLOY_JAVA_CLASS_DIRS` held them, `derived` from the launch or from `target/classes` or `build/classes/java/main` under the working directory, `whole-jar` or `classpath` when it could not tell them from the libraries, `none`), what it found them from in words (`launch`), the dirs and jars it read them from (`roots`, at most 64, and `rootCount`), the package roots it kept (`packages`), how many classes it indexed, the build id of those classes (`jv1-` and 32 hex; it also names the format of keploy-sdk's build table and the JaCoCo release keploy-sdk analyzes the classes with, so an SDK upgrade that changes either gives the same classes a new build id), which JaCoCo agent measures them (`jacoco`: `embedded`, the one keploy-sdk.jar starts, `attached` by the command, `none`, or `unchecked` when the agent was not started as a `-javaagent`), that embedded agent's `includes=`, and two versions as their jars' manifests name them, or `""` when not known: keploy-sdk.jar's (`sdkVersion`) and that of the JaCoCo agent that measures the classes, embedded or attached (`jacocoVersion`). It carries no class list. It is sent as the agent starts when that path's socket file is there (Enterprise's receiver makes it), and otherwise, or when that send fails, right before the first testcase's coverage; a receiver Enterprise binds anew on that path gets it again before its first record. On the TCP transport used in Kubernetes no INDEX record is sent. Since 2.2.0 Keploy can also ask for that background, to keep it out of what it compares: it sends the line `BACKGROUND` on `/tmp/coverage_control.sock`. The agent answers `ACK` when it reads JaCoCo's probes in place and a receiver is on `/tmp/coverage_data.sock`, and `IGNORED` otherwise (`KEPLOY_JAVA_SCAN=off`, JaCoCo reached over TCP, the in-place read failed, or no receiver), as an older agent answers a line it does not know. The ask holds for this run of the application and that receiver only: an application started again (native `keploy test` starts it again for each test set) and a receiver Enterprise binds anew on that path (the next docker-compose test set) get no record until Keploy asks again, so Keploy asks after each INDEX record, which comes once per start of the application and once per new receiver. From then on, each `START` that cleared hits made since the previous testcase's `END` sends one BACKGROUND record on `/tmp/coverage_data.sock`, right after that `START` is answered (after the INDEX record, when that receiver does not have it yet): `{"kind":"background","v":1,"build":"","after":"","before":"","executedLinesByFile":{className:[probeIdx]}}`, under the key and in the shape of a testcase's record (with its `classMismatch`, left out when empty), plus `"lost":N` when the records of the N windows before it could not be made or sent. A window with no hits sends nothing. What ran before the first testcase (the application's start, the warmup) is not background, and neither is what ran after an `END` that was not taken or a `START` whose `END` never came (a testcase's own hits): none of it is sent. A test's own work that goes on after its `END` (an async task it started) lands in the next window, and so is reported as background. A Keploy that never asks gets no record, short of a new receiver that replaces the one that asked within the same instant (the agent tells receivers apart by the socket file's inode and modification time). On the TCP transport used in Kubernetes there is no BACKGROUND. diff --git a/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java b/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java index 9f60e9ce..76d72f0f 100644 --- a/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java +++ b/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java @@ -107,6 +107,15 @@ static void log(Level level, String message, Throwable thrown) { /** The control-socket command Keploy sends for the build table. */ static final String COMMAND = "TABLE"; + /** + * The table's format, F of the build id (IndexReport.buildId): the contract's section 3.3 records as this + * class writes them and the probe map of leaf ranges, with every bound that decides what a class's records + * are ({@link #MAX_RECORD_BYTES}, {@link #MAX_MAP_BYTES}, {@link #MAX_MAP_STEPS}, how + * KeployBranchCoverage.Hits counts its steps, and the reason a unit past one is not measured). Raise it in + * any release where the table or probe-map records written for the same class bytes can change, so they + * name another build. The KTB1 frames that carry the records are not in it: their header's "v" names them. + */ + static final int TABLE_FORMAT = 1; /** A TABLE frame's size cap, magic and length included. */ static final int MAX_FRAME_BYTES = 1 << 20; /** A record (with its newline) larger than a whole inflated chunk can never be sent: record_too_large. */ diff --git a/keploy-sdk/src/main/java/io/keploy/dedup/IndexReport.java b/keploy-sdk/src/main/java/io/keploy/dedup/IndexReport.java index b1151f3b..8ae26b00 100644 --- a/keploy-sdk/src/main/java/io/keploy/dedup/IndexReport.java +++ b/keploy-sdk/src/main/java/io/keploy/dedup/IndexReport.java @@ -11,6 +11,8 @@ import java.util.Collection; import java.util.List; import java.util.TreeSet; +import java.util.logging.Level; +import java.util.logging.Logger; /** * IndexReport is the INDEX record: what the SDK took as the app's classes, @@ -53,7 +55,8 @@ final class IndexReport { // Serialized in this order, by name. final String kind = KIND; final int v = VERSION; - // "jv1-" + 32 hex (buildId), or "" when the index holds no class. + // "jv1-" + 32 hex (buildId: the class ids, the table format and the analyzer's JaCoCo), or "" when the + // index holds no class. final String build; final String source; // How the JVM was launched, or why the app's classes could not be told @@ -143,24 +146,87 @@ static String hexId(long id) { } /** - * buildId is the build's id from its classes' JaCoCo ids: "jv1-" and the - * first 32 hex of classIdsSum, or "" when there is no id. Keploy's - * codecoverage.JavaBuildID computes the same from the same ids. It is the - * one build id: the INDEX record and the build table (BuildTable.write) - * both take it over the id of every class the index holds, read or not, - * so Keploy finds the table under the build the INDEX record named. + * buildId is the build's id: "jv1-" and the first 32 lowercase hex of the sha256 over the ASCII bytes + * {@code "keploy-java-table " F "\n" "jacoco " V "\n"} and the ids as {@link #classIdsSum} joins them, where + * F is the build table's format ({@link BuildTable#TABLE_FORMAT}) and V the JaCoCo release the SDK's analyzer + * runs ({@link Analyzer#JACOCO}); "" when there is no id. Keploy's codecoverage.JavaBuildID computes the same + * from the same bytes. It is the one build id: the INDEX record and the build table (BuildTable.write) both + * take it over the id of every class the index holds, read or not, so Keploy finds the table under the build + * the INDEX record named. An SDK whose table or probe map can differ for the same class bytes (another F or + * V) names another build, so Keploy never holds its table against one an older SDK saved for that build. */ static String buildId(long[] ids) { - return ids.length == 0 ? "" : "jv1-" + classIdsSum(ids).substring(0, 32); + return buildId(ids, BuildTable.TABLE_FORMAT, Analyzer.JACOCO); + } + + /** buildId is the build id of the ids for a table of format F and an analyzer of JaCoCo release V. */ + static String buildId(long[] ids, int format, String jacoco) { + if (ids.length == 0) { + return ""; + } + return "jv1-" + sha256("keploy-java-table " + format + "\njacoco " + jacoco + "\n", joinedIds(ids)) + .substring(0, 32); + } + + /** + * Analyzer holds V of the build id apart from IndexReport, so reading it (JaCoCo's class init and its + * version resource) is paid by the first build id alone, never by the class-id helpers a testcase's END + * runs before its ACK, and a JaCoCo core that cannot load leaves those helpers working. + */ + static final class Analyzer { + /** + * JACOCO is the release of the JaCoCo core whose analyzer builds the table (the one keploy-sdk.jar + * carries renamed, read from its own JaCoCo.VERSION less the build qualifier, as "0.8.15"), not the + * measuring agent's: an attached JaCoCo agent of another version only records the probes. "" if that + * core cannot say (its version resource is missing), so the INDEX record and the table still name one + * build; the release smoke test fails a jar whose analyzer cannot say. + */ + static final String JACOCO = analyzerJacoco(); + + private Analyzer() { + } + + private static String analyzerJacoco() { + try { + return release(org.jacoco.core.JaCoCo.VERSION); + } catch (Throwable t) { + // JaCoCo reads its version from a resource bundle when the class loads (MissingResourceException, + // ExceptionInInitializerError): a jar without it still names one build in INDEX and TABLE. + // Logged here, not through BuildTable, whose link needs the JaCoCo core that may be missing. + try { + Logger.getLogger(IndexReport.class.getName()).log(Level.INFO, "Keploy coverage: keploy-sdk" + + " cannot read its JaCoCo core's version, so the build id names none", t); + } catch (Throwable ignored) { + // Nowhere left to say it: the log is what failed. + } + return ""; + } + } + } + + /** release is a JaCoCo version less its build qualifier: its first three dot-separated parts. */ + static String release(String version) { + int dots = 0; + for (int i = 0; i < version.length(); i++) { + if (version.charAt(i) == '.' && ++dots == 3) { + return version.substring(0, i); + } + } + return version; } /** * classIdsSum is the sha256, as 64 lowercase hex, over the ids sorted as * unsigned numbers, de-duplicated, each as 16 lowercase hex, joined by - * newlines. The build id is its first 32 hex; a build table unit's hash - * is all of it, over that unit's class ids. + * newlines: a build table unit's hash, over that unit's class ids. */ static String classIdsSum(long[] ids) { + return sha256("", joinedIds(ids)); + } + + // joinedIds is the ids sorted as unsigned numbers, de-duplicated, each as 16 lowercase hex, joined by + // newlines, with none after the last. + private static String joinedIds(long[] ids) { long[] sorted = Arrays.copyOf(ids, ids.length); // Unsigned order: flipping the sign bit maps it onto signed order. for (int i = 0; i < sorted.length; i++) { @@ -177,14 +243,21 @@ static String classIdsSum(long[] ids) { } joined.append(hexId(sorted[i] ^ Long.MIN_VALUE)); } - byte[] sum; + return joined.toString(); + } + + // sha256 is the sha256, as 64 lowercase hex, over the ASCII bytes of prefix and then of ids, taken in + // two parts so the ids (about 17 bytes a class) are not copied once more behind the prefix. + private static String sha256(String prefix, String ids) { + MessageDigest digest; try { - sum = MessageDigest.getInstance("SHA-256").digest(joined.toString().getBytes(StandardCharsets.US_ASCII)); + digest = MessageDigest.getInstance("SHA-256"); } catch (NoSuchAlgorithmException e) { // Every Java platform has SHA-256. throw new IllegalStateException(e); } - return hex(sum); + digest.update(prefix.getBytes(StandardCharsets.US_ASCII)); + return hex(digest.digest(ids.getBytes(StandardCharsets.US_ASCII))); } /** hex is b as lowercase hex, two digits a byte: the build id's and a TABLE frame's sha256. */ diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/AppIndexTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/AppIndexTest.java index e79042ba..3caee801 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/AppIndexTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/AppIndexTest.java @@ -437,7 +437,9 @@ private static void assertIndex(String source, String launch, String packages, i String build = ""; if (classes > 0) { String hex = String.format("%016x", CRC64.classId(new byte[] {1})); - build = "jv1-" + sha256(hex).substring(0, 32); + // Keploy's codecoverage.JavaBuildID bytes: the table format, the analyzer's JaCoCo release, then the ids. + build = "jv1-" + sha256("keploy-java-table 1\njacoco " + IndexReportTest.jacocoCore() + "\n" + hex) + .substring(0, 32); } assertEquals(index.toString(), build, index.get("build").getAsString()); // start() is not premain: it starts no JaCoCo and looks for none. diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java index f3e29a9e..ac056714 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java @@ -60,15 +60,23 @@ /** * Rows for the build table: the java-dedup sample's classes (compiled once with javac --release 8 -g from * keploy/samples-java java-dedup, kept as .class.bin) encode to the golden table byte for byte. The golden - * (java-dedup.table.ndjson, java-dedup.build) is the Go encoder's output (enterprise pkg/service/codecoverage - * NewTable + EncodeTable) over these records, with the hashes and the build id computed in Go from the class - * bytes, so the order, the JSON form, the fold and the ids are Go's, not this class's. So is edge.table.ndjson, - * over a set of classes generated here ({@link #edge}) for the names, packages and escapes the sample lacks. + * table (java-dedup.table.ndjson) is the Go encoder's output (enterprise pkg/service/codecoverage NewTable + + * EncodeTable) over these records, with the unit hashes computed in Go from the class bytes, so the order, the + * JSON form, the fold and the unit hashes are Go's, not this class's. So is edge.table.ndjson, over a set of + * classes generated here ({@link #edge}) for the names, packages and escapes the sample lacks. * - *

To regenerate a golden: run these rows with -Dbuildtable.dump=DIR (each golden row writes DIR/NAME.ndjson - * and the class files as DIR/NAME/VM-NAME.class.bin), then in an enterprise checkout read each record with - * codecoverage's Unit JSON, recompute its hash and the build id from the class files with JaCoCo's CRC64, - * NewTable(LanguageJava, build, units) and EncodeTable, and write the result back over the golden here. + *

The .build goldens (java-dedup.build, edge.build) are the build id of IndexReport.buildId's bytes: the + * table format F ({@link BuildTable#TABLE_FORMAT}), the JaCoCo release V of the analyzer keploy-sdk.jar + * carries, then the class ids (JaCoCo's CRC64 of each class file), computed by an independent CRC64 and + * sha256 oracle until codecoverage.JavaBuildID takes F and V. A JaCoCo upgrade or a raised TABLE_FORMAT + * changes both .build files on purpose. + * + *

To regenerate a golden table: run these rows with -Dbuildtable.dump=DIR (each golden row writes + * DIR/NAME.ndjson and the class files as DIR/NAME/VM-NAME.class.bin), then in an enterprise checkout read + * each record with codecoverage's Unit JSON, recompute its hash from the class files with JaCoCo's CRC64, + * NewTable(LanguageJava, build, units) and EncodeTable, and write the result back over the golden here. Then + * update {@link #aTableThatChangesForTheSameClassesNamesAnotherBuild}. To regenerate a .build golden, take + * the sha256 of IndexReport.buildId's bytes over the dumped class files' ids. */ public class BuildTableTest { @@ -154,12 +162,67 @@ static void dump(String name, List files, Collect c) throw } } + private static final String BUILD_GOLDEN = "the build id of IndexReport.buildId's bytes (F, the analyzer's JaCoCo" + + " release, the class ids): a JaCoCo upgrade or a raised BuildTable.TABLE_FORMAT changes it on purpose"; + + /** + * A tripwire for the build id's F: TABLE_FORMAT 1 is these bounds, Hits' step count, the record each cap + * writes (with its reason) and the records these golden tables hold. A change to any of them changes the + * table for the same class bytes, and an SDK that kept F would name the build an older SDK saved its table + * under, so Keploy would refuse each later save of it. In a release, raise BuildTable.TABLE_FORMAT, then + * update this row; before the first release that writes a table, update the pinned values alone. + */ + @Test + public void aTableThatChangesForTheSameClassesNamesAnotherBuild() throws Exception { + String why = "the table for the same class bytes changed: in a release, raise BuildTable.TABLE_FORMAT," + + " then update this row"; + assertEquals(why, 1, BuildTable.TABLE_FORMAT); + assertEquals(why, 8 << 20, BuildTable.MAX_RECORD_BYTES); + assertEquals(why, 8 << 20, BuildTable.MAX_MAP_BYTES); + assertEquals(why, 64L << 20, BuildTable.MAX_MAP_STEPS); + // The record each cap writes, not the constants: a unit past the probe map's steps (a chain of 200 + // early returns, whose walks Hits counts) and a record past a 1600-byte chunk (java-dedup's two). + String name = "gen/big/Returns"; + byte[] bytes = BuildTableBudgetTest.returns(name, 200); + List one = Collections.singletonList(new BuildTable.ClassFile(name, bytes)); + long[] steps = {0}; + KeployBranchCoverage.analyze(bytes, new HashMap<>(), (m, r, h) -> steps[0] += h.steps); + assertEquals(why, 61302L, steps[0]); + Collect probes = new Collect(); + BuildTable.write(one, new Collect(), probes, BuildTable.MAX_RECORD_BYTES); + assertEquals(why, 1, probes.records.size()); + assertEquals(why, "c4271098e6e5103773be528dee9ca19edd1fded1409e2cba5a4aad84232e61c9", + sha256Hex(probes.ndjson.toByteArray())); + int mapBytes = probes.records.get(0).length() + 1; + assertEquals(why, 149124, mapBytes); + Collect mapped = new Collect(); + Collect map = new Collect(); + BuildTable.write(one, mapped, map, BuildTable.MAX_RECORD_BYTES, mapBytes, steps[0]); + assertEquals(why, probes.records, map.records); + assertTrue(why, mapped.records.get(0).contains("\"state\":\"ok\"")); + Collect capped = new Collect(); + BuildTable.write(one, capped, new Collect(), BuildTable.MAX_RECORD_BYTES, mapBytes, steps[0] - 1); + assertTrue(why, capped.records.get(0).contains("\"state\":\"not_measured\"")); + assertEquals(why, "3ee7977c74cf5939ab7af3957bfc1bbd4e495614d4dcb7bd066c707618f7ec1c", + sha256Hex(capped.ndjson.toByteArray())); + Collect chunked = new Collect(); + BuildTable.write(javaDedup(), chunked, 1600); + assertEquals(why, "4c5ebe7cc05a4f0e4c34faa45c7521216df376409cac5cc4f8e6447b4359d584", + sha256Hex(chunked.ndjson.toByteArray())); + assertEquals(why, "d9d2f53a4e36fb2a63b63a307187373a1ee87ec4e44bcbcdec1a2c4fb47caf1c", + sha256Hex(resource("java-dedup.table.ndjson"))); + assertEquals(why, "ddb2d5ac7990ed620e370825fc84aec6782ea89ccda029716cdfb1b27519a7dc", + sha256Hex(resource("java-dedup.probes.ndjson"))); + assertEquals(why, "e0e30865ebc060b2a576f961df79bd002191791daa6336f47f210e666feadd20", + sha256Hex(resource("edge.table.ndjson"))); + } + @Test public void javaDedupTableIsTheGoEncodersByteForByte() throws Exception { List files = javaDedup(); Collect c = table(files); dump("java-dedup", files, c); - assertEquals(new String(resource("java-dedup.build"), StandardCharsets.UTF_8).trim(), c.build); + assertEquals(BUILD_GOLDEN, new String(resource("java-dedup.build"), StandardCharsets.UTF_8).trim(), c.build); assertEquals(new String(resource("java-dedup.table.ndjson"), StandardCharsets.UTF_8), new String(c.ndjson.toByteArray(), StandardCharsets.UTF_8)); // The same classes in another order give the same bytes. @@ -589,7 +652,7 @@ public void anEdgeSetIsTheGoEncodersByteForByte() throws Exception { List files = edge(); Collect c = table(files); dump("edge", files, c); - assertEquals(new String(resource("edge.build"), StandardCharsets.UTF_8).trim(), c.build); + assertEquals(BUILD_GOLDEN, new String(resource("edge.build"), StandardCharsets.UTF_8).trim(), c.build); assertEquals(new String(resource("edge.table.ndjson"), StandardCharsets.UTF_8), new String(c.ndjson.toByteArray(), StandardCharsets.UTF_8)); BuildTable.Summary s = BuildTable.write(files, new Collect()); diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/ControlSocketTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/ControlSocketTest.java index 2e6fcb6b..4524f0a6 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/ControlSocketTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/ControlSocketTest.java @@ -283,8 +283,10 @@ public void theIndexRecordAndTheTableNameOneBuild() throws Exception { Files.write(out.toPath(), f.bytes); ids.add(String.format("%016x", CRC64.classId(f.bytes))); } - // Keploy's codecoverage.JavaBuildID: the ids as 16 hex, sorted (as hex: unsigned), joined by newlines. - String build = "jv1-" + sha256(String.join("\n", ids).getBytes(StandardCharsets.UTF_8)).substring(0, 32); + // Keploy's codecoverage.JavaBuildID: the table format, the analyzer's JaCoCo release, then + // the ids as 16 hex, sorted (as hex: unsigned), joined by newlines. + String build = "jv1-" + sha256(("keploy-java-table 1\njacoco " + IndexReportTest.jacocoCore() + "\n" + + String.join("\n", ids)).getBytes(StandardCharsets.UTF_8)).substring(0, 32); assertNotEquals("the file no class reader reads is in the build", new String(BuildTableTest.resource("java-dedup.build"), StandardCharsets.UTF_8).trim(), build); System.setProperty(CLASS_DIRS_PROPERTY, classes.getPath()); diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/IndexReportTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/IndexReportTest.java index 0388cea7..b3cabc30 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/IndexReportTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/IndexReportTest.java @@ -8,6 +8,7 @@ import java.io.File; import java.io.FileOutputStream; +import java.io.IOException; import java.io.InputStream; import java.net.URL; import java.net.URLClassLoader; @@ -16,6 +17,7 @@ import java.util.Arrays; import java.util.Collections; import java.util.List; +import java.util.Properties; import java.util.jar.Attributes; import java.util.jar.JarOutputStream; import java.util.jar.Manifest; @@ -23,6 +25,7 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertNull; import static org.junit.Assert.assertTrue; @@ -31,27 +34,125 @@ public class IndexReportTest { @Rule public final TemporaryFolder tmp = new TemporaryFolder(); - // Keploy's codecoverage.JavaBuildID of the ids 1, 0xff and - // 0xffffffffffffffff (its TestBuildIDs): sha256 over - // "0000000000000001\n00000000000000ff\nffffffffffffffff". - private static final String GO_BUILD_ID = "jv1-73ab094cbaa75db7d27e1cc885ca2260"; + // The build id's golden vectors (shared with Keploy's codecoverage.JavaBuildID): the sha256 over + // "keploy-java-table 1\njacoco 0.8.15\n0000000000000001\n00000000000000ff\nffffffffffffffff". + private static final long[] GOLDEN_IDS = {1, 0xff, 0xffffffffffffffffL}; + private static final String GOLDEN_BUILD_ID = "jv1-ac2499a6a1d337817534c44eb82f6ab3"; @Test public void theBuildIdIsKeploysForTheSameIdsInAnyOrderAndRepeated() { - assertEquals(GO_BUILD_ID, IndexReport.buildId(new long[] {1, 0xff, 0xffffffffffffffffL})); + assertEquals(GOLDEN_BUILD_ID, IndexReport.buildId(GOLDEN_IDS, 1, "0.8.15")); // A signed sort puts 0xffffffffffffffff (-1) first, a different id. - assertEquals(GO_BUILD_ID, IndexReport.buildId(new long[] {0xffffffffffffffffL, 1, 0xff, 1})); - assertEquals(GO_BUILD_ID, IndexReport.buildId(new long[] {0xff, 0xffffffffffffffffL, 0xff, 1})); + assertEquals(GOLDEN_BUILD_ID, + IndexReport.buildId(new long[] {0xffffffffffffffffL, 1, 0xff, 1}, 1, "0.8.15")); + assertEquals(GOLDEN_BUILD_ID, + IndexReport.buildId(new long[] {0xff, 0xffffffffffffffffL, 0xff, 1}, 1, "0.8.15")); + } + + @Test + public void anotherTableFormatOrAnalyzerNamesAnotherBuildOfTheSameClasses() { + // An SDK whose table can differ for the same class bytes must not name the build an older SDK saved a + // table under: Keploy would refuse every later save of it (409 build_table_mismatch). + assertEquals("jv1-32f6f460d59e38d89bac87928b9421ae", IndexReport.buildId(GOLDEN_IDS, 2, "0.8.15")); + assertEquals("jv1-776b07757cb4a25493b652ff31a8ca86", IndexReport.buildId(GOLDEN_IDS, 1, "0.8.14")); + // Over the ids alone (the build id before the format and the analyzer were in it): none of the above. + assertEquals("jv1-73ab094cbaa75db7d27e1cc885ca2260", + "jv1-" + IndexReport.classIdsSum(GOLDEN_IDS).substring(0, 32)); + } + + @Test + public void theSdksBuildIdIsOfThisTablesFormatAndItsAnalyzersJacoco() { + assertEquals(1, BuildTable.TABLE_FORMAT); + assertEquals(jacocoCore(), IndexReport.Analyzer.JACOCO); + assertEquals(IndexReport.buildId(GOLDEN_IDS, 1, jacocoCore()), IndexReport.buildId(GOLDEN_IDS)); + } + + /** + * An END reads class ids (hexId, classId) before its ACK; only the build id needs V, so those helpers must + * not run JaCoCo's class init and its version lookup. Run in a loader of its own over the SDK's classes, + * JaCoCo's core and Gson, so the classes it loads are its own to count. + */ + @Test + public void theClassIdHelpersNeverLoadTheAnalyzersJacoco() throws Exception { + URL[] urls = { + IndexReport.class.getProtectionDomain().getCodeSource().getLocation(), + org.jacoco.core.JaCoCo.class.getProtectionDomain().getCodeSource().getLocation(), + com.google.gson.Gson.class.getProtectionDomain().getCodeSource().getLocation(), + }; + byte[] classBytes = Files.readAllBytes(new File(urls[0].toURI()).toPath() + .resolve(IndexReport.class.getName().replace('.', '/') + ".class")); + try (Counting loader = new Counting(urls)) { + Class report = Class.forName(IndexReport.class.getName(), true, loader); + assertTrue(report != IndexReport.class); + assertEquals("0000000000000001", call(report, "hexId", long.class, 1L)); + assertEquals(IndexReport.classId(classBytes), call(report, "classId", byte[].class, classBytes)); + assertFalse("V was read by a class-id helper", loader.loaded(IndexReport.Analyzer.class.getName())); + assertFalse("JaCoCo's version was loaded by a class-id helper", + loader.loaded(org.jacoco.core.JaCoCo.class.getName())); + assertEquals(IndexReport.buildId(new long[] {1}), call(report, "buildId", long[].class, new long[] {1})); + assertTrue(loader.loaded(IndexReport.Analyzer.class.getName())); + assertTrue(loader.loaded(org.jacoco.core.JaCoCo.class.getName())); + } + } + + private static final class Counting extends URLClassLoader { + Counting(URL[] urls) { + super(urls, null); + } + + boolean loaded(String name) { + return findLoadedClass(name) != null; + } + } + + private static Object call(Class c, String name, Class parameter, Object argument) throws Exception { + java.lang.reflect.Method m = c.getDeclaredMethod(name, parameter); + m.setAccessible(true); + return m.invoke(null, argument); + } + + @Test + public void aJacocoReleaseIsItsVersionLessTheBuildQualifier() { + assertEquals("0.8.15", IndexReport.release("0.8.15.202606040825")); + assertEquals("0.8.15", IndexReport.release("0.8.15")); + assertEquals("1.0.0", IndexReport.release("1.0.0.SNAPSHOT.x")); + assertEquals("0.8", IndexReport.release("0.8")); + assertEquals("", IndexReport.release("")); + } + + /** + * jacocoCore is the release of the org.jacoco.core jar the analyzer runs, as its Maven build named it (its + * pom.properties, read from the jar JaCoCo's class was loaded from, so another jar's pom.properties on the + * class path cannot answer), not as JaCoCo.VERSION says: what IndexReport.Analyzer.JACOCO must be. + */ + static String jacocoCore() { + try { + File jar = new File(org.jacoco.core.JaCoCo.class.getProtectionDomain().getCodeSource().getLocation() + .toURI()); + assertTrue(jar + " is org.jacoco.core's jar", jar.isFile()); + try (java.util.jar.JarFile file = new java.util.jar.JarFile(jar)) { + ZipEntry entry = file.getEntry("META-INF/maven/org.jacoco/org.jacoco.core/pom.properties"); + assertNotNull("org.jacoco.core's pom.properties in " + jar, entry); + Properties p = new Properties(); + try (InputStream in = file.getInputStream(entry)) { + p.load(in); + } + String version = p.getProperty("version"); + assertTrue(version, version.matches("\\d+\\.\\d+\\.\\d+")); + return version; + } + } catch (IOException | java.net.URISyntaxException e) { + throw new AssertionError(e); + } } // The whole sha256 over the same three ids: what a build table unit's - // hash is for a unit of those class ids. The build id is its first 32 hex. + // hash is for a unit of those class ids. private static final String IDS_SUM = "73ab094cbaa75db7d27e1cc885ca22604fb30a4555ebac8aae6e264612a027d0"; @Test - public void theUnitHashIsTheWholeSumAndTheBuildIdItsFirstHalf() { + public void theUnitHashIsTheSumOverTheUnitsIdsAlone() { assertEquals(IDS_SUM, IndexReport.classIdsSum(new long[] {0xffffffffffffffffL, 1, 0xff, 1})); - assertEquals(GO_BUILD_ID, IndexReport.buildId(new long[] {0xffffffffffffffffL, 1, 0xff, 1})); } @Test @@ -92,13 +193,13 @@ public void aRootIsTheLocationWithoutTheClassPath() { @Test public void theReportIsOneSmallJsonObjectWithNoClassList() { - IndexReport report = new IndexReport(GO_BUILD_ID, IndexReport.SOURCE_DERIVED, "java -jar /app/app.jar", + IndexReport report = new IndexReport(GOLDEN_BUILD_ID, IndexReport.SOURCE_DERIVED, "java -jar /app/app.jar", Arrays.asList("/app/libs/b.jar", "/app/classes", "/app/libs/b.jar"), Collections.singletonList("org.acme"), 3, "org.acme.*", IndexReport.JACOCO_EMBEDDED, "2.2.0", "0.8.15"); JsonObject json = JsonParser.parseString(report.toJson()).getAsJsonObject(); assertEquals("index", json.get("kind").getAsString()); assertEquals(1, json.get("v").getAsInt()); - assertEquals(GO_BUILD_ID, json.get("build").getAsString()); + assertEquals(GOLDEN_BUILD_ID, json.get("build").getAsString()); assertEquals("derived", json.get("source").getAsString()); assertEquals("java -jar /app/app.jar", json.get("launch").getAsString()); assertEquals("[\"/app/classes\",\"/app/libs/b.jar\"]", json.get("roots").toString()); diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/edge.build b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/edge.build index 350796a9..876c4c16 100644 --- a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/edge.build +++ b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/edge.build @@ -1 +1 @@ -jv1-19348879a23c5d3452cc7da8f15505f9 +jv1-91aa3fda68544563b2a8cca5ad1dde54 diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup.build b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup.build index a32ff64c..1179d0ee 100644 --- a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup.build +++ b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup.build @@ -1 +1 @@ -jv1-41cdf8ddf2cd4e64008b050d972818b2 +jv1-d2b0c051a964d63b882ad20b37ebda64 diff --git a/scripts/smoke-javaagent.sh b/scripts/smoke-javaagent.sh index 08d499f0..974df3a0 100755 --- a/scripts/smoke-javaagent.sh +++ b/scripts/smoke-javaagent.sh @@ -259,6 +259,26 @@ public final class SmokeHarness { throw new IllegalStateException("the INDEX report did not name keploy-sdk " + wantSdkVersion + " and JaCoCo " + wantJacocoVersion + " in " + mode + ": " + report); } + // The build id's V: the release of the JaCoCo core keploy-sdk.jar carries renamed, as the SDK reads it + // from that core's own version resource, must be the one the core's Maven build named. A jar whose + // renamed core cannot say gives V "", and a JaCoCo upgrade would then keep the build id. It is read as + // the SDK loaded here (from the class path, with this mode's agents and libraries) and from the jar alone + // in a loader of its own with no parent, so V is the jar's own core's with nothing else visible. + File sdkJar = new File(System.getProperty("smoke.sdkJar")); + String core = corePomVersion(sdkJar); + String analyzer = analyzerJacoco(ClassLoader.getSystemClassLoader()); + URLClassLoader alone = new URLClassLoader(new URL[] {sdkJar.toURI().toURL()}, null); + String analyzerAlone; + try { + analyzerAlone = analyzerJacoco(alone); + } finally { + alone.close(); + } + if (core.isEmpty() || !core.equals(analyzer) || !core.equals(analyzerAlone)) { + throw new IllegalStateException("keploy-sdk's analyzer names JaCoCo '" + analyzer + "' ('" + + analyzerAlone + "' in a loader of its own) and its org.jacoco.core is '" + core + "' in " + mode); + } + System.out.println(mode + ": analyzer JaCoCo " + analyzer); String ownLibraries = System.getProperty("smoke.ownLibraries", ""); if (!ownLibraries.isEmpty()) { loadsItsOwn(ownLibraries); @@ -385,6 +405,28 @@ public final class SmokeHarness { } } + // analyzerJacoco is IndexReport.Analyzer.JACOCO, V of the build id, of keploy-sdk as loader loads it. + private static String analyzerJacoco(ClassLoader loader) throws Exception { + java.lang.reflect.Field field = Class.forName("io.keploy.dedup.IndexReport$Analyzer", true, loader) + .getDeclaredField("JACOCO"); + field.setAccessible(true); + return (String) field.get(null); + } + + private static String corePomVersion(File jar) throws Exception { + try (JarFile file = new JarFile(jar)) { + java.util.zip.ZipEntry entry = file.getEntry("META-INF/maven/org.jacoco/org.jacoco.core/pom.properties"); + if (entry == null) { + return ""; + } + java.util.Properties p = new java.util.Properties(); + try (InputStream in = file.getInputStream(entry)) { + p.load(in); + } + return p.getProperty("version", ""); + } + } + private static void waitFor(File file) throws InterruptedException { long deadline = System.currentTimeMillis() + 10000L; while (System.currentTimeMillis() < deadline) { From 39e72a3544878fd070dd22d990ecfdfc1276d4a3 Mon Sep 17 00:00:00 2001 From: slayerjain Date: Fri, 20 Nov 2026 09:42:18 +0530 Subject: [PATCH 08/14] perf(dedup): analyze each class once for its table and its probe map JaCoCo analyzed every class file twice while the build table was made: once through its public Analyzer, for the table record's methods, and once through KeployBranchCoverage, for the probe map. The second pass already runs the same steps (ClassAnalyzer's instruction graph, filters and MethodCoverageCalculator), so the first was pure cost: about 30% of what the send allocates and of its CPU. Now one pass gives both. KeployBranchCoverage hands each method's JaCoCo coverage (Method.coverage, computed as ClassAnalyzer computes the coverage it adds to its class) to the caller before the method's probes are walked, and BuildTable makes the table methods from it. A probe's ranges are kept relative to its method and get the method's base when the unit's records are written, since the base is known only once all of the unit's methods are ordered. The map budget counts the text before the bases are added (never more than the records hold), then each whole record in its place, so the bound holds as before. The table and probe-map records are byte for byte what they were: the java-dedup and edge goldens and the records the build-id tripwire pins pass unchanged. What changes: - At 50,000 generated classes (G1 and Parallel; 1 and 4 GiB pods) the send's thread allocates 84-96 KiB a class (4,145-4,697 MiB in all; JDK 8, 17 and 21), from 123-134 KiB (6,021-6,577 MiB; JDK 8 and 21), in 2.5-3.4 s of CPU, from 3.5-4.8 s. The committed heap does not drop (G1 in 4 GiB pods grows it by 222-518 MiB, from 254-420): the collectors size it to the allocation rate, not to the garbage's total. - A check of the map that fails (KeployBranchCoverage.NotMapped, or one of BuildTable's) counts as not mapped, as before; anything else the pass throws counts as JaCoCo failing to read the class (before, an error in the map's own pass counted as not mapped). Both are analyzer_failed. An error while a table method is made from its coverage now fails only its unit, as not mapped; before, it ended the whole send. - A unit's class files are now all analyzed first, in name order, and the first analysis that fails decides; only then are its probe-map records made and charged to the budget. Before, every file's JaCoCo read came first, then each file's map and record in turn. So a unit with two failures of different kinds can now be given the other one's reason: analyzer_failed instead of record_too_large, or the reverse. Either way it is not measured and has no probe record. - A second class file of one name is charged its text against the budget while it is analyzed, not its record, so the smallest bound that maps a unit of two such copies is a few hundred bytes lower. - While a unit's records are made, only the class file being written keeps its ranges; a record whose text is not the length counted for it fails its unit as not mapped instead of being sent. Rows: - aGeneratedAppsTableIsMeasured bounds the allocation at 112 KiB a class, below what two analyses took (123-141 KiB in every run so far); on JDK 21 it fails on the two-pass code (135,287 bytes a class at its default 2,000 classes) and passes on this one on JDK 8, 17 and 21. - theTableNeedsNoSecondAnalysisOfAClass: no class of BuildTable's names JaCoCo's public Analyzer (the two-pass BuildTable does). - eachTableMethodIsWhatJacocosAnalyzerCounts: on JaCoCo core's and Gson's classes and every fixture, with no probe executed and with all of them, each method's coverage from the one pass equals what the Analyzer counts (the same methods in the same order, every counter, line by line). It fails when the pass skips the filters. - aClassJacocoCannotAnalyzeIsNotMeasuredAndSoIsItsUnit now also adds a class JaCoCo's instrumenter wrote, and counts none of the three refusals as a map that failed (it fails when they are). Signed-off-by: slayerjain --- .../main/java/io/keploy/dedup/BuildTable.java | 364 +++++++++++------- .../analysis/KeployBranchCoverage.java | 44 ++- .../io/keploy/dedup/BuildTableBudgetTest.java | 9 +- .../java/io/keploy/dedup/BuildTableTest.java | 21 +- .../java/io/keploy/dedup/ProbeMapTest.java | 107 +++++ 5 files changed, 389 insertions(+), 156 deletions(-) diff --git a/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java b/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java index 76d72f0f..4686ba12 100644 --- a/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java +++ b/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java @@ -1,12 +1,8 @@ package io.keploy.dedup; -import org.jacoco.core.analysis.Analyzer; -import org.jacoco.core.analysis.IClassCoverage; -import org.jacoco.core.analysis.ICoverageVisitor; import org.jacoco.core.analysis.ILine; import org.jacoco.core.analysis.IMethodCoverage; import org.jacoco.core.analysis.ISourceNode; -import org.jacoco.core.data.ExecutionDataStore; import org.jacoco.core.internal.analysis.KeployBranchCoverage; import org.newsclub.net.unix.AFUNIXSocket; import org.newsclub.net.unix.AFUNIXSocketAddress; @@ -128,7 +124,9 @@ static void log(Level level, String message, Throwable thrown) { * past either is sent not_measured record_too_large with no probe record, and its map is given up as * soon as it is past one, so neither the heap nor the time it takes grows with it: a method of n early * returns maps to about n * n ranges (its probe k covers the k checks before it), 4.4 MB at 1,000 and - * 43 MB at 3,000. + * 43 MB at 3,000. The bound is on the text; what a unit's map holds live until its records are made is + * its ranges' int bounds (4 bytes each, a bound's text at least about 1.5), so up to about 2.7 times + * the bound. */ static final int MAX_MAP_BYTES = MAX_RECORD_BYTES; static final long MAX_MAP_STEPS = 64L << 20; @@ -843,55 +841,72 @@ private List sortedFiles() { } /** - * record analyzes the unit's class files and writes its record; any class JaCoCo refuses fails it, and - * a probe map past the budget makes it record_too_large. + * record analyzes each of the unit's class files once, all of them first, in name order, then makes + * the unit's records. The first analysis that fails decides: analyzer_failed when JaCoCo refuses the + * class file or its probes cannot be mapped, record_too_large when its probe map is past the budget. + * Only then are the probe-map records made and each charged in place of its text: a record past the + * budget makes the unit record_too_large, and a second class file of one name that maps its probes + * otherwise makes it analyzer_failed. A unit with two failures of different kinds gets the first one's + * reason in that order; either way it is not measured and has no probe record. */ byte[] record(MapBudget budget) { Map infos = new HashMap<>(); List methods = new ArrayList<>(); - Set methodIds = new HashSet<>(); - for (UnitFile f : sortedFiles()) { - ClassInfo info = ClassInfo.read(f.file.bytes); - infos.put(f.file.name, info); - List got; - try { - got = analyze(f.file); - } catch (Exception | LinkageError e) { - log(Level.FINE, "Keploy coverage: JaCoCo could not analyze " + f.file.name, e); - failed = true; - return notMeasured("analyzer_failed"); - } - for (IClassCoverage cc : got) { - if (cc.getFirstLine() == ISourceNode.UNKNOWN_LINE - && cc.getInstructionCounter().getTotalCount() > 0) { - noLines++; - } - String nested = f.file.name.substring(path().length()); - for (IMethodCoverage mc : cc.getMethods()) { - Method m = Method.of(f.file.name, nested, name, mc, info); - if (m != null && methodIds.add(m.id)) { - methods.add(m); - } + Map byId = new HashMap<>(); + List sorted = sortedFiles(); + List ranges = new ArrayList<>(sorted.size()); + UnitFile at = null; + try { + String lastName = null; + for (UnitFile f : sorted) { + at = f; + ClassInfo info = ClassInfo.read(f.file.bytes); + infos.put(f.file.name, info); + Ranges r = ProbeMap.analyze(this, f, info, methods, byId, budget); + if (f.file.name.equals(lastName)) { + // A second class file of this name: its record is the one before it or none (below), + // so its text is not one the records take. + budget.forget(r.counted()); } + ranges.add(r); + lastName = f.file.name; } + } catch (TooLarge e) { + log(Level.FINE, "Keploy coverage: the probe map of " + path() + " is past its bound: " + e.getMessage(), + null); + failed = true; + tooLarge = true; + return notMeasured(RECORD_TOO_LARGE); + } catch (KeployBranchCoverage.NotMapped e) { + // A class whose probes cannot be mapped exactly is not measured: its hits could not be read. + log(Level.FINE, "Keploy coverage: the probes of " + path() + " could not be mapped", e); + failed = true; + notMapped = true; + return notMeasured("analyzer_failed"); + } catch (Exception | LinkageError e) { + log(Level.FINE, "Keploy coverage: JaCoCo could not analyze " + at.file.name, e); + failed = true; + return notMeasured("analyzer_failed"); } Collections.sort(methods, (a, b) -> { int c = Integer.compare(a.line, b.line); return c != 0 ? c : compareCodePoints(a.id, b.id); }); - Map byId = new HashMap<>(); int base = 0; for (Method m : methods) { m.base = base; base += m.leaves.length / 3; - byId.put(m.id, m); } List maps = new ArrayList<>(); try { String lastName = null; byte[] last = null; - for (UnitFile f : sortedFiles()) { - byte[] p = ProbeMap.record(id, f, ProbeMap.probes(f, byId, budget)); + for (int i = 0; i < sorted.size(); i++) { + UnitFile f = sorted.get(i); + Ranges r = ranges.get(i); + // Only one class file's ranges stay alive while the records are made. + ranges.set(i, null); + byte[] p = ProbeMap.record(id, f, r); if (f.file.name.equals(lastName)) { // Two class files of one name whose bytes differ (the index found both). Keploy finds a // class's map by its name, as a test's hits name it, so they share one record: exact @@ -900,11 +915,11 @@ byte[] record(MapBudget budget) { throw new IllegalStateException(f.file.name + ": two class files of this name map" + " their probes differently"); } - budget.drop(); continue; } - // The unit's records, newlines included, within the budget (its text alone was, as it grew). - budget.take(p.length + 1); + // The unit's records, newlines included, within the budget, in place of their ranges' text + // counted as the analysis wrote it. + budget.take(p.length + 1, r.counted()); maps.add(p); lastName = f.file.name; last = p; @@ -916,7 +931,6 @@ byte[] record(MapBudget budget) { tooLarge = true; return notMeasured(RECORD_TOO_LARGE); } catch (Exception | LinkageError e) { - // A class whose probes cannot be mapped exactly is not measured: its hits could not be read. log(Level.FINE, "Keploy coverage: the probes of " + path() + " could not be mapped", e); failed = true; notMapped = true; @@ -936,19 +950,6 @@ byte[] record(MapBudget budget) { } } - /** analyze runs JaCoCo's analyzer over one class file with no execution data. */ - static List analyze(ClassFile f) throws IOException { - final List out = new ArrayList<>(1); - Analyzer analyzer = new Analyzer(new ExecutionDataStore(), new ICoverageVisitor() { - @Override - public void visitCoverage(IClassCoverage coverage) { - out.add(coverage); - } - }); - analyzer.analyzeClass(f.bytes, f.name); - return out; - } - /** What a cheap ASM pass (no code) gives: the SourceFile and which methods take varargs. */ static final class ClassInfo { String sourceFile; @@ -1122,8 +1123,9 @@ void check(KeployBranchCoverage.Method a) { } } - // Per counted instruction of the class file's analysis of this method (set by prepare): the unit - // leaf index of its line's code leaf, and of its first branch edge (-1: a line below 1, no leaf). + // Per counted instruction of the class file's analysis of this method (set by prepare, until release): + // the method-relative index of its line's code leaf, and of its first branch edge (-1: a line below 1, + // no leaf). A probe's ranges add the method's base once the unit's methods are ordered. private int[] codeAt; private int[] edgeAt; private KeployBranchCoverage.Method prepared; @@ -1151,15 +1153,22 @@ void prepare(KeployBranchCoverage.Method a) { } Integer met = edges.get(l); int e = met == null ? 0 : met; - codeAt[k] = base + leaf; - edgeAt[k] = base + leaf + 1 + e; + codeAt[k] = leaf; + edgeAt[k] = leaf + 1 + e; edges.put(l, e + a.branches[k].length); } prepared = a; } + /** release forgets the analysis prepare noted, once its probes are mapped. */ + void release() { + codeAt = null; + edgeAt = null; + prepared = null; + } + /** - * hits sets in out the unit-relative leaf indices a probe covers: a line's code leaf when one of its + * hits sets in out the method-relative leaf indices a probe covers: a line's code leaf when one of its * instructions is covered, and the leaf of each covered branch edge. */ void hits(KeployBranchCoverage.Method a, KeployBranchCoverage.Hits h, BitSet out) { @@ -1188,9 +1197,9 @@ void hits(KeployBranchCoverage.Method a, KeployBranchCoverage.Hits h, BitSet out * covered branch edge's leaf. A probe covers more than its own line: JaCoCo marks every instruction on the * path that ends at the probe, so a test's leaves are the union of its probes' leaves. * - *

The class file is analyzed once, and each probe's leaves are read off that analysis (KeployBranchCoverage - * walks back from the probe over the edges JaCoCo's builder wired): the map costs the class's size plus its - * probes' walks and their ranges' text, on top of the table's own analysis, with no object per probe. The + *

The class file is analyzed once, for both its table methods and its map: each probe's leaves are read + * off that analysis (KeployBranchCoverage walks back from the probe over the edges JaCoCo's builder wired), + * so the map costs the class's probes' walks and their ranges, with no object per probe. The * walks and the text are quadratic in a chain of early returns, so a unit's map has a budget * ({@link MapBudget}): past it the unit is record_too_large. */ @@ -1200,7 +1209,7 @@ private ProbeMap() { /** * record is the class file's probe-map record: {"class":vmName,"id":crc64,"unit":unitId,"probes":[...]}, - * with probes as {@link #probes} made them, written once into a buffer of its size. + * with probes as {@link #analyze} made them, written once into a buffer of its size. */ static byte[] record(String unitId, UnitFile f, Ranges probes) { StringBuilder sb = new StringBuilder(64 + f.file.name.length() + unitId.length()); @@ -1213,6 +1222,10 @@ static byte[] record(String unitId, UnitFile f, Ranges probes) { byte[] rec = new byte[head.length + probes.jsonLength() + 1]; System.arraycopy(head, 0, rec, 0, head.length); int at = probes.json(rec, head.length); + if (at != rec.length - 1) { + throw new IllegalStateException(f.file.name + ": its probes wrote " + (at - head.length) + + " bytes, not the " + probes.jsonLength() + " counted"); + } rec[at] = '}'; return rec; } @@ -1250,69 +1263,111 @@ private static int indexOf(byte[] a, byte[] key) { } /** - * probes is the class file's probe map: one entry per probe (the class's probe count), each the - * probe's leaves as half-open ranges of the unit's leaf indices, [start,end,start,end,...] ascending, no - * two ranges touching. It throws when the class's lines or branches differ from the unit's methods (the - * same name under two roots), when a method with lines is in one and not in the other, or when a - * method's flow is not one KeployBranchCoverage maps; and TooLarge as soon as the unit's map is past - * its budget's bytes or steps. + * analyze is the class file's one analysis (KeployBranchCoverage): it adds the file's methods to the + * unit's (methods and byId; the first class file of a name makes a method, a second one of that name + * is checked against it), counts a file with code but no line in u.noLines, and returns the file's + * probe map: one entry per probe (the class's probe count), each the probe's leaves as half-open ranges + * of its method's leaf indices, ascending, no two ranges touching; its record adds the method's base + * once the unit's methods are ordered ({@link Ranges#json}). It throws NotMapped when the class's lines + * or branches differ from the unit's methods (the same name under two roots), when a method with lines + * is in one and not in the other, or when a method's flow is not one KeployBranchCoverage maps; TooLarge + * as soon as the unit's map is past its budget's bytes or steps; and what JaCoCo throws for a class it + * does not read. */ - static Ranges probes(UnitFile f, Map byId, MapBudget budget) { + static Ranges analyze(Unit u, UnitFile f, ClassInfo info, List methods, Map byId, + MapBudget budget) { Map shape = new HashMap<>(); String prefix = "m:" + f.file.name + "#"; + String nested = f.file.name.substring(u.path().length()); Ranges ranges = new Ranges(budget); - // One analysis of the class; each probe's leaves come from it (KeployBranchCoverage). + boolean[] seen = new boolean[2]; // a method with code, and one with a line int total = Math.max(0, KeployBranchCoverage.analyze(f.file.bytes, shape, new KeployBranchCoverage.EachProbe() { private Method method; @Override public boolean wants(KeployBranchCoverage.Method s) { - // Checked below: a method with no line has no leaf for its probes to cover. - method = byId.get(prefix + s.name + s.desc); - if (method == null) { - return false; + try { + // As JaCoCo sums a class: code with no line when no method of it has one. + seen[0] = true; + if (s.coverage.getFirstLine() != ISourceNode.UNKNOWN_LINE) { + seen[1] = true; + } + method = byId.get(prefix + s.name + s.desc); + if (method == null) { + method = Method.of(f.file.name, nested, u.name, s.coverage, info); + if (method == null) { + // Checked below: a method with no line has no leaf for its probes. + return false; + } + byId.put(method.id, method); + methods.add(method); + } + method.prepare(s); + return true; + } catch (RuntimeException e) { + throw notMapped(e); } - method.prepare(s); - return true; } @Override public void probe(KeployBranchCoverage.Method s, int r, KeployBranchCoverage.Hits h) { - budget.step(h.steps); - method.hits(s, h, ranges.bits); - ranges.put(s.probes[r]); + try { + budget.step(h.steps); + method.hits(s, h, ranges.bits); + ranges.put(s.probes[r], method); + } catch (RuntimeException e) { + throw notMapped(e); + } } })); - int inShape = 0; - for (Map.Entry e : shape.entrySet()) { - Method m = byId.get(prefix + e.getKey()); - if (m == null) { - for (int l : e.getValue().lines) { - if (l >= 1) { - throw new IllegalStateException(f.file.name + "#" + e.getKey() - + " has lines but no method in the table"); + if (seen[0] && !seen[1]) { + u.noLines++; + } + try { + int inShape = 0; + for (Map.Entry e : shape.entrySet()) { + Method m = byId.get(prefix + e.getKey()); + if (m == null) { + for (int l : e.getValue().lines) { + if (l >= 1) { + throw new IllegalStateException(f.file.name + "#" + e.getKey() + + " has lines but no method in the table"); + } } + continue; + } + m.release(); + inShape++; + } + // And the other way: each of the table's methods of this class file has its probes mapped here. + int inTable = 0; + for (String id : byId.keySet()) { + if (id.startsWith(prefix)) { + inTable++; } - continue; } - m.check(e.getValue()); - inShape++; - } - // And the other way: each of the table's methods of this class file has its probes mapped here. - int inTable = 0; - for (String id : byId.keySet()) { - if (id.startsWith(prefix)) { - inTable++; + if (inTable != inShape) { + throw new IllegalStateException(f.file.name + ": " + inTable + " methods in the table, " + + inShape + " in the probe analysis"); } + ranges.close(f.file.name, total); + } catch (RuntimeException e) { + throw notMapped(e); } - if (inTable != inShape) { - throw new IllegalStateException(f.file.name + ": " + inTable + " methods in the table, " - + inShape + " in the probe analysis"); - } - ranges.close(f.file.name, total); return ranges; } + + /** + * notMapped is e as a map that failed: TooLarge and NotMapped as they are, a check that failed as + * NotMapped. + */ + private static RuntimeException notMapped(RuntimeException e) { + if (e instanceof TooLarge || e instanceof KeployBranchCoverage.NotMapped) { + return e; + } + return new KeployBranchCoverage.NotMapped(String.valueOf(e.getMessage()), e); + } } /** @@ -1352,7 +1407,10 @@ void step(int n) { } } - /** grew counts ranges' text written for a class file's record not taken yet. */ + /** + * grew counts ranges' text an analysis wrote for a record not taken yet: at most what the record writes + * for them, so the budget is past only when the records would be. + */ void grew(int n) { text += n; if (bytes + text > maxBytes) { @@ -1360,77 +1418,85 @@ void grew(int n) { } } - /** drop forgets the text counted so far: its record shares the record before it. */ - void drop() { - text = 0; + /** forget takes back the text counted for a record not taken: it shares the record before it. */ + void forget(long counted) { + text -= counted; } - /** take counts a whole record (with its newline) in place of its text counted so far. */ - void take(int n) { - text = 0; + /** take counts a whole record (with its newline) in place of the text counted for it. */ + void take(int n, long counted) { + text -= counted; bytes += n; - if (bytes > maxBytes) { + if (bytes + text > maxBytes) { throw new TooLarge("its probe records are more than " + maxBytes + " bytes"); } } } /** - * Ranges keeps a class file's probe map as it is made: each probe's leaf ranges written once, as ASCII - * JSON, into one buffer, with where each probe's text starts and ends, so a probe costs its text and no - * object, and the record copies the text once ({@link ProbeMap#record}). + * Ranges keeps a class file's probe map as its analysis makes it: each probe's leaf ranges relative to its + * method, as ints in one buffer, with where each probe's ranges start and end and the probe's method, so a + * probe costs its ranges and no object. Its record writes them once as ASCII JSON, each plus its method's + * base, after the unit's methods are ordered ({@link ProbeMap#record}). */ static final class Ranges { - /** The probe being written: its leaves, set by the caller and cleared by {@link #put}. */ + /** The probe being written: its method-relative leaves, set by the caller and cleared by {@link #put}. */ final BitSet bits = new BitSet(); private final MapBudget budget; - private byte[] text = new byte[256]; + private int[] bounds = new int[64]; private int length; private int[] from = new int[16]; private int[] to = new int[16]; + private Method[] owner = new Method[16]; private int total = -1; + // The text put counted in the budget: the ranges' text before the bases are added, so at most what + // the record writes for them. + private long counted; Ranges(MapBudget budget) { this.budget = budget; } - /** put writes bits as the given probe's ranges ("start,end,..." ascending, none touching), then clears it. */ - void put(int probe) { + /** + * put keeps bits as the given probe's ranges of m's leaves (start, end, ... ascending, none touching), + * then clears it. + */ + void put(int probe, Method m) { if (probe >= from.length) { int size = Math.max(probe + 1, from.length * 2); from = Arrays.copyOf(from, size); to = Arrays.copyOf(to, size); + owner = Arrays.copyOf(owner, size); } if (to[probe] != 0) { throw new IllegalStateException("probe " + probe + " mapped twice"); } from[probe] = length; + owner[probe] = m; int at = bits.nextSetBit(0); while (at >= 0) { int end = bits.nextClearBit(at); - int was = length; - if (length > from[probe]) { - append(','); + int n = (length > from[probe] ? 1 : 0) + digits(at) + 1 + digits(end); + if (length + 2 > bounds.length) { + bounds = Arrays.copyOf(bounds, bounds.length * 2); } - appendInt(at); - append(','); - appendInt(end); - budget.grew(length - was); + bounds[length++] = at; + bounds[length++] = end; + counted += n; + budget.grew(n); at = bits.nextSetBit(end); } - // to is one past the text's end, so 0 is "not mapped" and an empty range list is still mapped. + // to is one past the ranges' end, so 0 is "not mapped" and an empty range list is still mapped. to[probe] = length + 1; bits.clear(); } - private void append(char c) { - if (length == text.length) { - text = Arrays.copyOf(text, text.length * 2); - } - text[length++] = (byte) c; + /** counted is the text put counted in the budget for this map. */ + long counted() { + return counted; } - private void appendInt(int v) { + private static int digits(int v) { if (v < 0) { throw new IllegalStateException("leaf " + v); } @@ -1438,14 +1504,7 @@ private void appendInt(int v) { for (int x = v; x >= 10; x /= 10) { digits++; } - if (length + digits > text.length) { - text = Arrays.copyOf(text, Math.max(text.length * 2, length + digits)); - } - for (int i = length + digits - 1; i >= length; i--) { - text[i] = (byte) ('0' + v % 10); - v /= 10; - } - length += digits; + return digits; } /** close ends the map at the class's probe count (total): a probe at or past it is a bug. */ @@ -1458,12 +1517,21 @@ void close(String className, int total) { this.total = total; } - /** jsonLength is the length of {@link #json}'s text: [ ], one [ ] a probe, the commas, the ranges. */ + /** + * jsonLength is the length of {@link #json}'s text: [ ], one [ ] a probe, the commas, and each probe's + * ranges plus its method's base. + */ int jsonLength() { long n = 2 + 3L * total - (total > 0 ? 1 : 0); for (int p = 0; p < total && p < from.length; p++) { - if (to[p] != 0) { - n += to[p] - 1 - from[p]; + if (to[p] == 0) { + continue; + } + int end = to[p] - 1; + n += Math.max(0, end - from[p] - 1); + int base = owner[p].base; + for (int i = from[p]; i < end; i++) { + n += digits(base + bounds[i]); } } if (total < 0 || n > Integer.MAX_VALUE - 1024) { @@ -1474,7 +1542,7 @@ int jsonLength() { /** * json writes the map into out at at and returns where it ends: one entry per probe of the class, - * [] for a probe no method mapped. + * [] for a probe no method mapped, each range's bounds plus its method's base (unit leaf indices). */ int json(byte[] out, int at) { out[at++] = '['; @@ -1484,9 +1552,19 @@ int json(byte[] out, int at) { } out[at++] = '['; if (p < from.length && to[p] != 0) { - int n = to[p] - 1 - from[p]; - System.arraycopy(text, from[p], out, at, n); - at += n; + int base = owner[p].base; + for (int i = from[p]; i < to[p] - 1; i++) { + if (i > from[p]) { + out[at++] = ','; + } + int v = base + bounds[i]; + int d = digits(v); + for (int j = at + d - 1; j >= at; j--) { + out[j] = (byte) ('0' + v % 10); + v /= 10; + } + at += d; + } } out[at++] = ']'; } diff --git a/keploy-sdk/src/main/java/org/jacoco/core/internal/analysis/KeployBranchCoverage.java b/keploy-sdk/src/main/java/org/jacoco/core/internal/analysis/KeployBranchCoverage.java index 0ef41d80..67a39552 100644 --- a/keploy-sdk/src/main/java/org/jacoco/core/internal/analysis/KeployBranchCoverage.java +++ b/keploy-sdk/src/main/java/org/jacoco/core/internal/analysis/KeployBranchCoverage.java @@ -13,6 +13,7 @@ import org.jacoco.core.analysis.ICounter; import org.jacoco.core.analysis.ILine; +import org.jacoco.core.analysis.IMethodCoverage; import org.jacoco.core.internal.analysis.filter.Filters; import org.jacoco.core.internal.analysis.filter.IFilter; import org.jacoco.core.internal.analysis.filter.IFilterOutput; @@ -32,7 +33,9 @@ * KeployBranchCoverage runs JaCoCo's own analysis of one class (ClassAnalyzer's steps for each method: the * instruction graph, the filters, the merges and the branch replacements) and keeps what JaCoCo's public * coverage API sums away: which counted instructions are covered and which branch of each one is. Keploy's - * build table uses it to map each probe to the lines and branch edges that a hit of that probe covers. + * build table makes each class's methods from the coverage this pass computes ({@link Method#coverage}) and maps + * each probe to the lines and branch edges that a hit of that probe covers, from the same pass: a class is + * analyzed once. * *

It lives in JaCoCo's package so that it runs JaCoCo's package-private analysis classes as they are. Each * method's per-line sums are checked against the coverage JaCoCo computes for it in the same pass, so a @@ -57,6 +60,12 @@ public static final class Method { public boolean[] covered; /** Per counted instruction: whether each of its branches is covered (empty below two branches). */ public boolean[][] branches; + /** + * JaCoCo's coverage of the method from this pass, computed as ClassAnalyzer computes the coverage it + * adds to its class (a MethodCoverageCalculator over the same instructions and filter output), with + * the probes analyze was given: with none executed, what JaCoCo's Analyzer gives with no execution data. + */ + public IMethodCoverage coverage; Method(String name, String desc) { this.name = name; @@ -84,6 +93,24 @@ public static int analyze(byte[] bytes, boolean[] probes, Map ou return analyze(bytes, probes, out, null); } + /** + * NotMapped is a class file whose probes cannot be mapped exactly from this pass: a method's flow has a + * shape the walk does not map, its per-instruction coverage differs from JaCoCo's per-line sums, or a + * check of the caller's fails. The caller marks the class not measured; it is not JaCoCo failing to read + * the class. + */ + public static final class NotMapped extends IllegalStateException { + private static final long serialVersionUID = 1L; + + public NotMapped(String message) { + super(message); + } + + public NotMapped(String message, Throwable cause) { + super(message, cause); + } + } + /** * Hits is what one probe covers in its method: the method's counted instructions that a class analysis * with only that probe executed marks covered, and which of their branches. It is valid during @@ -129,8 +156,10 @@ public interface EachProbe { void probe(Method method, int r, Hits hits); /** - * wants says whether method's probes are walked at all: a method it declines costs no walk, and probe - * gets none of its probes. Every method with code is wanted unless this is overridden. + * wants gets each method with code once, in the class's order, as soon as its coverage + * ({@link Method#coverage}) is computed and before any of its probes, and says whether its probes are + * walked at all: a method it declines costs no walk, and probe gets none of its probes. Every method + * with code is wanted unless this is overridden. */ default boolean wants(Method method) { return true; @@ -416,8 +445,8 @@ private static void mark(Hits hits, int[] newStamp, int round, int k) { } } - private IllegalStateException shape(Method m, String what) { - return new IllegalStateException(getClassName() + "#" + m.name + m.desc + ": " + what); + private NotMapped shape(Method m, String what) { + return new NotMapped(getClassName() + "#" + m.name + m.desc + ": " + what); } /** @@ -458,7 +487,7 @@ private boolean fill(Method m, Builder builder, MethodNode node, String signatur int at = insns.indexOf(i); int count = replaced != null ? replaced : builder.numbers(at); if (n > 0 && count != n) { - throw new IllegalStateException("JaCoCo counts " + n + " branches at " + getClassName() + "#" + throw new NotMapped("JaCoCo counts " + n + " branches at " + getClassName() + "#" + m.name + m.desc + " line " + insn.getLine() + ", its builder gave " + count); } boolean[] b = new boolean[n]; @@ -494,7 +523,7 @@ private boolean fill(Method m, Builder builder, MethodNode node, String signatur ILine line = coverage.getLine(l); int[] s = e.getValue(); if (!same(line.getInstructionCounter(), s[0], s[1]) || !same(line.getBranchCounter(), s[2], s[3])) { - throw new IllegalStateException("JaCoCo's coverage of " + getClassName() + "#" + m.name + m.desc + throw new NotMapped("JaCoCo's coverage of " + getClassName() + "#" + m.name + m.desc + " line " + l + " differs from its instructions: " + Arrays.toString(s) + " against " + line.getInstructionCounter() + " " + line.getBranchCounter()); } @@ -502,6 +531,7 @@ private boolean fill(Method m, Builder builder, MethodNode node, String signatur m.lines = Arrays.copyOf(lines, c); m.covered = Arrays.copyOf(covered, c); m.branches = Arrays.copyOf(branches, c); + m.coverage = coverage; return true; } diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableBudgetTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableBudgetTest.java index bb676575..afc3ce7f 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableBudgetTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableBudgetTest.java @@ -67,14 +67,15 @@ public void aGeneratedAppsTableIsMeasured() throws Exception { assertEquals(n, b.tableRecords); assertEquals(n, b.probeRecords); assertEquals("every class is measured", 0, b.notMeasured); - // What the send allocates per class: about 1.9 MiB when each probe re-analyzed its method, 125-141 KiB - // (JaCoCo's analysis of the class, twice, and the records' text) since each probe is read off one pass. + // What the send allocates per class: about 1.9 MiB when each probe re-analyzed its method, 123-141 KiB + // when JaCoCo analyzed each class twice (once for its table, once for its probe map), and 84-96 KiB + // since one analysis gives both (and the records' text). long perClass = b.allocated / n; assertTrue("allocated " + perClass + " bytes a class", perClass < ALLOCATED_PER_CLASS); } - /** The most a generated class's table and probe map may allocate on the send's thread. */ - static final long ALLOCATED_PER_CLASS = 256 << 10; + /** The most a generated class's table and probe map may allocate on the send's thread: below two analyses'. */ + static final long ALLOCATED_PER_CLASS = 112 << 10; @Test(timeout = 600_000) public void aClassWithOneLongMethodIsMeasured() throws Exception { diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java index ac056714..b62eb3b5 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java @@ -321,8 +321,15 @@ public void aClassJacocoCannotAnalyzeIsNotMeasuredAndSoIsItsUnit() throws Except // A nested class of Item no ASM in this SDK reads (class file version 99), and a class that is // not a class file at all. files.add(new BuildTable.ClassFile(item.name + "$Later", patchedMajor(item.bytes, 99))); - files.add(new BuildTable.ClassFile("io/keploy/samples/javadedup/Junk", "not a class".getBytes(StandardCharsets.UTF_8))); - Map ours = byId(table(files).records); + files.add(new BuildTable.ClassFile("io/keploy/samples/javadedup/Junk", + "not a class".getBytes(StandardCharsets.UTF_8))); + // A class JaCoCo's offline instrumenter wrote: JaCoCo refuses to analyze it. + String instrumented = "io/keploy/samples/javadedup/Instrumented"; + files.add(new BuildTable.ClassFile(instrumented, ProbeMapTest.instrumented(instrumented, + edgeClass(instrumented, "Instrumented.java", true, 10)))); + Collect c = new Collect(); + BuildTable.Summary s = BuildTable.write(files, c); + Map ours = byId(c.records); Map golden = byId(Arrays.asList( new String(resource("java-dedup.table.ndjson"), StandardCharsets.UTF_8).split("\n"))); @@ -340,6 +347,16 @@ public void aClassJacocoCannotAnalyzeIsNotMeasuredAndSoIsItsUnit() throws Except assertEquals("", junk.get("path").getAsString()); assertEquals("[]", junk.get("methods").toString()); + JsonObject in = ours.get("c:" + instrumented); + assertEquals("not_measured", in.get("state").getAsString()); + assertEquals("{\"code\":\"analyzer_failed\"}", in.get("reason").toString()); + assertEquals("[]", in.get("methods").toString()); + + // Each of the three is JaCoCo refusing a class file, none a probe map that failed. + assertEquals(3, s.notMeasured); + assertEquals("JaCoCo's refusals are not counted as maps that failed", 0, s.notMapped); + assertEquals(0, s.tooLarge); + // The other classes are as they were. for (String id : golden.keySet()) { if (!id.endsWith("/model/Item")) { diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/ProbeMapTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/ProbeMapTest.java index 7767581e..9ce1c665 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/ProbeMapTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/ProbeMapTest.java @@ -6,8 +6,11 @@ import com.google.gson.JsonParser; import org.jacoco.core.analysis.Analyzer; import org.jacoco.core.analysis.IClassCoverage; +import org.jacoco.core.analysis.ICounter; +import org.jacoco.core.analysis.ICoverageNode; import org.jacoco.core.analysis.ILine; import org.jacoco.core.analysis.IMethodCoverage; +import org.jacoco.core.analysis.ISourceNode; import org.jacoco.core.data.ExecutionData; import org.jacoco.core.data.ExecutionDataStore; import org.jacoco.core.instr.Instrumenter; @@ -34,9 +37,11 @@ import java.util.BitSet; import java.util.Collections; import java.util.HashMap; +import java.util.LinkedHashMap; import java.util.List; import java.util.Map; import java.util.TreeMap; +import java.util.regex.Pattern; import java.util.stream.Collectors; import java.util.stream.Stream; import java.util.zip.ZipEntry; @@ -232,6 +237,108 @@ public void eachProbesMapIsTheClassAnalyzedWithThatProbeExecuted() throws Except assertTrue("probes: " + probes, probes > 5000); } + @Test(timeout = 300000) + public void eachTableMethodIsWhatJacocosAnalyzerCounts() throws Exception { + // The table's methods come from the probe map's pass (KeployBranchCoverage.Method#coverage), not from + // JaCoCo's public Analyzer. Against the Analyzer, on real libraries' classes (JaCoCo core's and Gson's) + // and every fixture, with no probe executed and with every probe: the same methods in the same order, + // and each one's counters, whole and line by line. + List all = new ArrayList<>(); + for (Class c : new Class[] {Analyzer.class, JsonParser.class}) { + all.addAll(jarClasses(new File(c.getProtectionDomain().getCodeSource().getLocation().toURI()))); + } + all.addAll(BuildTableTest.javaDedup()); + all.addAll(load("shapes")); + all.addAll(load("modern")); + int classes = 0; + int methods = 0; + for (BuildTable.ClassFile f : all) { + int total = KeployBranchCoverage.analyze(f.bytes, null, new HashMap<>()); + for (boolean hit : new boolean[] {false, true}) { + boolean[] probes = null; + ExecutionDataStore store = new ExecutionDataStore(); + if (hit && total > 0) { + probes = new boolean[total]; + Arrays.fill(probes, true); + store.put(new ExecutionData(CRC64.classId(f.bytes), f.name, probes)); + } + Map ours = new LinkedHashMap<>(); + assertEquals(f.name, total, KeployBranchCoverage.analyze(f.bytes, probes, ours)); + List got = new ArrayList<>(); + for (KeployBranchCoverage.Method m : ours.values()) { + got.add(counts(m.coverage)); + } + List classCoverage = new ArrayList<>(); + new Analyzer(store, classCoverage::add).analyzeClass(f.bytes, f.name); + List want = new ArrayList<>(); + for (IClassCoverage cc : classCoverage) { + for (IMethodCoverage mc : cc.getMethods()) { + want.add(counts(mc)); + } + } + assertEquals(f.name + (hit ? ", every probe executed" : ", none executed"), want, got); + if (!hit && total >= 0) { + classes++; + methods += got.size(); + } + } + } + assertTrue("classes: " + classes, classes > 300); + assertTrue("methods: " + methods, methods > 1500); + } + + /** counts is a method's coverage as JaCoCo's API gives it: its counters, then each line's, with its lines. */ + static String counts(IMethodCoverage m) { + StringBuilder sb = new StringBuilder(); + sb.append(m.getName()).append(m.getDesc()).append(' ').append(m.getSignature()).append(" lines ") + .append(m.getFirstLine()).append('-').append(m.getLastLine()); + for (ICoverageNode.CounterEntity e : ICoverageNode.CounterEntity.values()) { + ICounter c = m.getCounter(e); + sb.append(' ').append(e).append(' ').append(c.getMissedCount()).append('/').append(c.getCoveredCount()); + } + if (m.getFirstLine() != ISourceNode.UNKNOWN_LINE) { + for (int l = m.getFirstLine(); l <= m.getLastLine(); l++) { + ILine line = m.getLine(l); + sb.append(" @").append(l).append(' ').append(line.getStatus()) + .append(" i ").append(line.getInstructionCounter().getMissedCount()).append('/') + .append(line.getInstructionCounter().getCoveredCount()) + .append(" b ").append(line.getBranchCounter().getMissedCount()).append('/') + .append(line.getBranchCounter().getCoveredCount()); + } + } + return sb.toString(); + } + + @Test + public void theTableNeedsNoSecondAnalysisOfAClass() throws Exception { + // One JaCoCo analysis of each class file gives both its table methods and its probe map. JaCoCo's public + // Analyzer would be a second one: no class of BuildTable's names it (the per-test CoverageReporter + // still does, for the coverage of a test's hits). + File dir = new File(BuildTable.class.getProtectionDomain().getCodeSource().getLocation().toURI()); + assertTrue(dir + " is the SDK's classes directory", dir.isDirectory()); + File[] own = new File(dir, "io/keploy/dedup").listFiles( + (d, n) -> n.equals("BuildTable.class") || n.startsWith("BuildTable$")); + assertNotNull(own); + assertTrue(Arrays.toString(own), own.length > 10); + Pattern analyzer = Pattern.compile("org/jacoco/core/analysis/Analyzer(?![\\w$])"); + for (File f : own) { + String text = new String(Files.readAllBytes(f.toPath()), StandardCharsets.ISO_8859_1); + assertFalse(f.getName() + " names JaCoCo's Analyzer", analyzer.matcher(text).find()); + } + } + + /** instrumented is the class file as JaCoCo's offline instrumenter writes it, its probes read from null. */ + static byte[] instrumented(String name, byte[] bytes) throws IOException { + return new Instrumenter(new IExecutionDataAccessorGenerator() { + @Override + public int generateDataAccessor(long classid, String classname, int probecount, MethodVisitor mv) { + mv.visitInsn(Opcodes.ACONST_NULL); + mv.visitTypeInsn(Opcodes.CHECKCAST, "[Z"); + return 1; + } + }).instrument(bytes, name); + } + static List jarClasses(File jar) throws IOException { List out = new ArrayList<>(); try (ZipFile z = new ZipFile(jar)) { From 836b0166a942de93f9c0f42b55029a02eba367d5 Mon Sep 17 00:00:00 2001 From: slayerjain Date: Fri, 20 Nov 2026 12:30:25 +0530 Subject: [PATCH 09/14] perf(dedup): pace the build table's send by the JVM's collections At 50,000 classes the build table's send allocates 4.2-4.8 GB in 2.5-6.3 s, nearly all of it short-lived. G1 and Parallel answer that rate by committing more heap, and the app keeps that heap after the send: unpaced, G1 on JDK 21 grew a 4 GiB pod's heap by 175-343 MiB. Two changes: - The send's thread is paced by the collections (BuildTable.Pacer). After each unit it owes 200 times the collection time since the send began (the app's collections and its own), less the time since, and sleeps that off, so collecting takes about 0.5% of the send's time. A JVM that does not collect is never paced. The sleeps come from an allowance of 0.8 ms per analyzed class file, of which at most 1 s is saved up: no one sleep is over a second, frames keep coming (so Keploy's wait for the next frame never runs out), and the sleeps add up to at most 0.8 ms a class file (40 s at 50,000). No sleep is taken that would end the table past 50 s at 50,000 class files (1 ms a class file past that), reckoned as the time so far plus the rest of the class files at the rate the work went so far, so on a slow or busy CPU the pacing gives way to the table's 60 s bound. A clock or collector bean that fails ends the pacing, and the rest is sent unpaced. An interrupt ends the send, with a WARNING that names it, and the interrupt is kept; before, the next write to the probe map's temp file failed and the WARNING blamed java.io.tmpdir. The send's INFO line now says how long it was paced. - FrameWriter fills one frame buffer, reused for every frame, and the socket writes it in place. Before, each 1 MiB frame was allocated twice (the body's copy and the frame), as G1 humongous objects. FrameSender gains send(b, off, len); its default copies the bytes, for a sender that keeps them. At 50,000 generated classes (G1 and Parallel; 1 and 4 GiB pods with no -Xmx; JDK 8, 17 and 21), three runs paced against two of the same code with the pacer off, the committed heap grows over what it was just after a full collection by: - JDK 21: 19-46 MiB (unpaced: G1 in a 4 GiB pod 175-343 MiB). - JDK 17 and 8, on Parallel and on G1 in a 1 GiB pod: 11-92 MiB. - G1 in a 4 GiB pod: 13-166 MiB on JDK 17, and 420-421 MiB on JDK 8 in every run (unpaced 422-707). The collection shrank G1's heap (JDK 8: 632 to 120 MiB) and its pause raised G1's pause-time ratio, so G1 grows the heap back while the table is sent, paced or not. Never past -Xmx. The table takes 10.3-46.1 s, from 2.5-6.3 s, in 2.4-6.7 s of thread CPU; the largest gap between two frames is 2.25 s. Where the pacing does not cut the growth it still costs that time: Parallel on JDK 8 takes 28-46 s, from 3.6-4.1 s, for about the same growth. Rows: - BuildTablePacerTest (16 rows, on a clock the rows drive): no sleep without a collection; a collection is slept off to 200 times its time, less the work since; the sleeps over any run of units stay within a second plus the allowance of their class files; a quiet stretch saves up at most a second; an overrun sleep is charged, and a sleep the clock did not see is charged what it asked; the sleeps give way to the deadline, told the class files or not; a table whose work alone fills its deadline is not paced; an interrupt ends the send and is kept; a failing collector bean ends the pacing; write paces after each unit by its class files and tells the pacer how many come. - anInterruptEndsTheSendAndIsSaidSo: an interrupt before the send, or after the table's last frame (when it closes the probe map's real temp file), ends the send with a WARNING that names it, the interrupt kept and no temp file left. - aGeneratedAppsTableIsMeasured asserts the growth over the heap just after its full collection that the paced send reaches: under 128 MiB, but under 192 MiB on G1 with a heap cap of 512 MiB or more on JDK 9-20 and under 448 MiB on JDK 8; the table within 60 s (scaled past 50,000 classes); and no frame more than 10 s after the one before. It fails with the pacer off (G1, JDK 21, 4 GiB pod: 175 and 343 MiB). Its sink now reads each frame in place, counting a match split over two frames: a String of a 1 MiB frame is itself a humongous object, and grew the heap the row measures. Signed-off-by: slayerjain --- .../main/java/io/keploy/dedup/BuildTable.java | 293 ++++++++++++- .../io/keploy/dedup/BuildTableBudgetTest.java | 199 +++++++-- .../io/keploy/dedup/BuildTablePacerTest.java | 391 ++++++++++++++++++ .../java/io/keploy/dedup/BuildTableTest.java | 49 +++ 4 files changed, 869 insertions(+), 63 deletions(-) create mode 100644 keploy-sdk/src/test/java/io/keploy/dedup/BuildTablePacerTest.java diff --git a/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java b/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java index 4686ba12..d4cb7289 100644 --- a/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java +++ b/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java @@ -13,9 +13,9 @@ import java.io.BufferedOutputStream; import java.io.BufferedReader; -import java.io.ByteArrayOutputStream; import java.io.File; import java.io.IOException; +import java.io.InterruptedIOException; import java.io.InputStream; import java.io.InputStreamReader; import java.io.OutputStream; @@ -475,8 +475,9 @@ static void send(Supplier> classes, FrameSender sender, File spi FrameWriter frames = new FrameWriter(sender, PART_TABLE, MAX_FRAME_BYTES); FrameWriter maps = new FrameWriter(sender, PART_PROBES, MAX_FRAME_BYTES); Summary s; + Pacer pacer = Pacer.jvm(); try (Spill spill = Spill.open(spillDir)) { - s = write(classes.get(), frames, spill); + s = write(classes.get(), frames, spill, MAX_RECORD_BYTES, MAX_MAP_BYTES, MAX_MAP_STEPS, pacer); frames.finish(s.build, s.records); spill.sendTo(maps, s.build); } @@ -504,10 +505,14 @@ static void send(Supplier> classes, FrameSender sender, File spi } m.append("; probe map of ").append(s.probeRecords).append(" class files; ").append(frames.seq + maps.seq) .append(" frames, ").append((System.nanoTime() - t0) / 1_000_000) - .append(" ms, cpu ").append((threadCpuNanos() - cpu0) / 1_000_000).append(" ms)"); + .append(" ms, cpu ").append((threadCpuNanos() - cpu0) / 1_000_000).append(" ms, paced ") + .append(pacer.waitedNanos() / 1_000_000).append(" ms)"); log(Level.INFO, m.toString(), null); } catch (IOException e) { - log(Level.WARNING, "Keploy coverage: the build table was not sent: " + e.getMessage() + // An interrupt ends the send: the Pacer's, or a temp file's channel that it closed. Name it, and not + // the channel's error, which would point at java.io.tmpdir. + String why = Thread.currentThread().isInterrupted() ? "its thread was interrupted" : e.getMessage(); + log(Level.WARNING, "Keploy coverage: the build table was not sent: " + why + "; it goes again when Keploy asks for it again", null); } catch (Throwable t) { // Never let the table take the app down: Keploy names the coverage it did not get. @@ -523,6 +528,213 @@ private static long threadCpuNanos() { } } + /** + * Pacer paces the table's thread by the JVM's collections, so its garbage does not make the collector grow + * the app's heap. The analysis allocates fast (about 80-90 KiB a class, over 1 GB/s at full speed, nearly + * all of it short-lived), and G1 and Parallel commit more heap when collecting takes more than their goal + * of the time (Parallel 1%; G1 a few percent, as low as 1% while it has committed a small part of its cap). + * So after each unit the thread owes {@link #WAIT_PER_GC} times the time the collections took since the + * send began (the app's and its own) less the time since, and waits it out: collections then take at most + * 1/200 of the send's time, and a JVM that does not collect is not paced at all. The time read is every + * collector bean's: on a collector whose beans count concurrent cycles (ZGC, Shenandoah) the send is paced + * at its full allowance. + * + *

The waits are bounded. They are drawn from an allowance of {@link #ALLOWANCE_NANOS_PER_CLASS} per + * class file analyzed, of which at most {@link #MAX_BANK_NANOS} is saved up: no one wait is over a second, + * the waits over any k class files add up to at most a second plus 0.8 k ms, so frames keep coming (Keploy's + * wait for the table is a bound on the time since its last frame), and all of them to at most 0.8 ms a + * class file (40 s at 50,000). And no wait is taken that would end the table past {@link #deadlineNanos} + * (50 s at 50,000 class files), reckoned as the time so far plus the rest of its class files at the rate + * the work went so far: on a slow or busy CPU the pacing gives way to the table's 60 s bound, with 10 s + * left for the probe map's frames and the estimate's error. A clock or collector bean that fails ends the + * pacing, and the rest is sent unpaced. An interrupt ends the send: unitDone throws, and the interrupt is + * kept for the thread. + */ + static final class Pacer { + /** Each millisecond of collection asks for this many of the send's: at most 0.5% of it collecting. */ + static final long WAIT_PER_GC = 200; + /** The wait each analyzed class file adds to what the thread may wait: the table's time bound. */ + static final long ALLOWANCE_NANOS_PER_CLASS = 800_000L; + /** The most of the allowance kept for a later wait: the longest wait after a quiet stretch. */ + static final long MAX_BANK_NANOS = 1_000_000_000L; + /** Waits shorter than this are put off to the next unit, so the thread does not sleep per class. */ + static final long MIN_WAIT_NANOS = 10_000_000L; + /** The pacing's time bound up to {@link #DEADLINE_CLASS_FILES} class files: see deadlineNanos. */ + static final long DEADLINE_NANOS = 50_000_000_000L; + static final long DEADLINE_CLASS_FILES = 50_000; + // The most the thread may owe (an hour), so the sum cannot overflow; the allowance bounds it anyway. + private static final long MAX_DEBT_NANOS = 3_600_000_000_000L; + + /** The clock, the collections' time and the sleep a Pacer reads; the JVM's, or a row's. */ + interface Clock { + long nanoTime(); + + /** The collectors' total time in ms since the JVM started. */ + long gcMillis(); + + void sleep(long nanos) throws InterruptedException; + } + + private final Clock clock; + private boolean off; + private long start; + private long total; + private long done; + private long last; + private long gc; + private long debt; + private long bank; + private long waited; + + Pacer(Clock clock) { + this.clock = clock; + try { + last = clock.nanoTime(); + start = last; + gc = clock.gcMillis(); + } catch (RuntimeException e) { + stop("its clock failed", e); + } + } + + /** off is a Pacer that never waits: write's, outside a send. */ + static Pacer off() { + Pacer p = new Pacer(JvmClock.NONE); + p.off = true; + return p; + } + + /** jvm paces by this JVM's collectors; off when the runtime has no collector beans to read. */ + static Pacer jvm() { + List beans; + try { + beans = ManagementFactory.getGarbageCollectorMXBeans(); + } catch (RuntimeException | LinkageError e) { + log(Level.FINE, "Keploy coverage: the build table is sent unpaced: no collector beans", e); + return off(); + } + return new Pacer(new JvmClock(beans)); + } + + /** + * deadlineNanos is the most the pacing lets a table of classFiles take: 50 s, and 1 ms a class file past + * 50,000 (the table's bound, 60 s at 50,000, scales the same way). + */ + static long deadlineNanos(long classFiles) { + return Math.max(DEADLINE_NANOS, DEADLINE_NANOS / DEADLINE_CLASS_FILES * classFiles); + } + + /** begin says how many class files the send analyzes, for its deadline; not called, the deadline is 50 s. */ + void begin(long classFiles) { + total = Math.max(0, classFiles); + } + + /** waitedNanos is how long the thread waited, in all. */ + long waitedNanos() { + return waited; + } + + /** + * unitDone is called after each unit, whose classFiles were analyzed: it waits what is owed and may be. + * It throws on an interrupt, which it keeps for the thread: an interrupt ends the send. + */ + void unitDone(int classFiles) throws InterruptedIOException { + if (Thread.currentThread().isInterrupted()) { + throw interrupted(); + } + if (off) { + return; + } + try { + long now = clock.nanoTime(); + long g = clock.gcMillis(); + if (g > gc) { + long perMs = WAIT_PER_GC * 1_000_000L; + debt = Math.min(MAX_DEBT_NANOS, debt + Math.min(g - gc, MAX_DEBT_NANOS / perMs) * perMs); + gc = g; + } + debt = Math.max(0, debt - Math.max(0, now - last)); + int files = Math.max(0, classFiles); + bank = Math.min(MAX_BANK_NANOS, bank + ALLOWANCE_NANOS_PER_CLASS * files); + done += files; + last = now; + long wait = Math.min(Math.min(debt, bank), room(now)); + if (wait < MIN_WAIT_NANOS) { + return; + } + try { + clock.sleep(wait); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw interrupted(); + } + long after = clock.nanoTime(); + long took = Math.max(0, after - now); + // What the sleep overran comes off the bank too, so the bound holds for the time slept; and at + // least the wait asked for, so it holds on a clock that does not move across a sleep. + long charged = Math.max(took, wait); + bank -= charged; + debt = Math.max(0, debt - charged); + waited += took; + last = after; + } catch (RuntimeException e) { + stop("its clock failed", e); + } + } + + /** room is how long the thread may wait at now and still end the table by its deadline. */ + private long room(long now) { + long elapsed = Math.max(0, now - start); + double rest = 0; + if (done > 0 && total > done) { + rest = (double) Math.max(0, elapsed - waited) / done * (total - done); + } + return (long) Math.max(-1.0, deadlineNanos(total) - elapsed - rest); + } + + static InterruptedIOException interrupted() { + return new InterruptedIOException("the build table's thread was interrupted"); + } + + private void stop(String why, Throwable thrown) { + off = true; + log(Level.FINE, "Keploy coverage: the build table is sent unpaced: " + why, thrown); + } + + /** JvmClock reads System.nanoTime and the JVM's collector beans, and sleeps the thread. */ + static final class JvmClock implements Clock { + static final JvmClock NONE = new JvmClock(Collections.emptyList()); + private final List beans; + + JvmClock(List beans) { + this.beans = beans; + } + + @Override + public long nanoTime() { + return System.nanoTime(); + } + + @Override + public long gcMillis() { + long sum = 0; + for (GarbageCollectorMXBean b : beans) { + // -1: this collector does not report its time. + long t = b.getCollectionTime(); + if (t > 0) { + sum += t; + } + } + return sum; + } + + @Override + public void sleep(long nanos) throws InterruptedException { + Thread.sleep(nanos / 1_000_000L, (int) (nanos % 1_000_000L)); + } + } + } + // ---------------------------------------------------------------- the table /** @@ -589,6 +801,12 @@ static Summary write(List classes, RecordSink sink, RecordSink probes /** write with the bounds of a unit's table record (maxRecord) and probe map (maxMap bytes, maxSteps). */ static Summary write(List classes, RecordSink sink, RecordSink probes, int maxRecord, int maxMap, long maxSteps) throws IOException { + return write(classes, sink, probes, maxRecord, maxMap, maxSteps, Pacer.off()); + } + + /** write with the bounds, its thread paced by pacer after each unit. */ + static Summary write(List classes, RecordSink sink, RecordSink probes, int maxRecord, int maxMap, + long maxSteps, Pacer pacer) throws IOException { Grouped g = group(classes); if (g.build.isEmpty()) { return new Summary("", 0, 0, 0, g.unnamed, 0, 0, 0); @@ -600,6 +818,11 @@ static Summary write(List classes, RecordSink sink, RecordSink probes int notMapped = 0; int tooLarge = 0; int probeRecords = 0; + long classFiles = 0; + for (Unit u : units) { + classFiles += u.files.size(); + } + pacer.begin(classFiles); for (int i = 0; i < units.size(); i++) { // A unit written is dropped: what stays is the class list (names, ids), not the table. Unit u = units.set(i, null); @@ -625,6 +848,7 @@ static Summary write(List classes, RecordSink sink, RecordSink probes } } sink.record(build, rec); + pacer.unitDone(u.files.size()); } return new Summary(build, units.size(), notMeasured, noLines, g.unnamed, notMapped, tooLarge, probeRecords); @@ -1764,6 +1988,14 @@ static void string(StringBuilder sb, String s) { /** Sends one whole frame; it throws when the receiver did not take it. */ interface FrameSender { void send(byte[] frame) throws IOException; + + /** + * send sends the frame in b[off, off + len). The bytes are the writer's own and change once it returns, + * so a sender that keeps them copies them (as this default does); FrameWriter sends through this one. + */ + default void send(byte[] b, int off, int len) throws IOException { + send(Arrays.copyOfRange(b, off, off + len)); + } } /** @@ -1774,7 +2006,12 @@ static final class FrameWriter implements RecordSink { private final FrameSender sender; private final String part; private final int bodyCap; - private final ByteArrayOutputStream body = new ByteArrayOutputStream(); + // The frame being filled, made at the first write and reused for every frame: its body from + // HEADER_ROOM on, its magic, length and header just before it at each flush. A frame allocated anew + // (twice: the body's copy and the frame) is a G1 humongous object at 1 MiB, for which G1 grows the + // heap when its free regions run short: 70-150 MiB of a 4 GiB pod's heap at 50,000 classes. + private byte[] frame; + private int bodyLen; private final MessageDigest digest; int seq; private String build; @@ -1812,18 +2049,26 @@ public void record(String build, byte[] json) throws IOException { private void write(byte[] b, int off, int len) throws IOException { while (len > 0) { - int room = bodyCap - body.size(); + int room = bodyCap - bodyLen; if (room == 0) { flush(false, -1); continue; } int take = Math.min(room, len); - body.write(b, off, take); + System.arraycopy(b, off, frame(), HEADER_ROOM + bodyLen, take); + bodyLen += take; off += take; len -= take; } } + private byte[] frame() { + if (frame == null) { + frame = new byte[HEADER_ROOM + bodyCap]; + } + return frame; + } + /** bytes adds whole records, each ended by its newline, to the stream as they are. */ void bytes(String build, byte[] b, int off, int len) throws IOException { this.build = build; @@ -1848,23 +2093,26 @@ private void flush(boolean last, int records) throws IOException { } h.append("}\n"); byte[] header = h.toString().getBytes(StandardCharsets.UTF_8); - byte[] payload = body.toByteArray(); - int n = header.length + payload.length; - byte[] frame = new byte[8 + n]; - System.arraycopy(MAGIC.getBytes(StandardCharsets.US_ASCII), 0, frame, 0, 4); - frame[4] = (byte) (n >>> 24); - frame[5] = (byte) (n >>> 16); - frame[6] = (byte) (n >>> 8); - frame[7] = (byte) n; - System.arraycopy(header, 0, frame, 8, header.length); - System.arraycopy(payload, 0, frame, 8 + header.length, payload.length); + int start = HEADER_ROOM - header.length - 8; + if (start < 0) { + // A build id or record count no header can carry: never one this class makes. + throw new IOException("TABLE frame " + seq + ": a header of " + header.length + " bytes"); + } + int n = header.length + bodyLen; + byte[] f = frame(); + System.arraycopy(MAGIC.getBytes(StandardCharsets.US_ASCII), 0, f, start, 4); + f[start + 4] = (byte) (n >>> 24); + f[start + 5] = (byte) (n >>> 16); + f[start + 6] = (byte) (n >>> 8); + f[start + 7] = (byte) n; + System.arraycopy(header, 0, f, start + 8, header.length); try { - sender.send(frame); + sender.send(f, start, 8 + n); } catch (IOException e) { throw new IOException("TABLE frame " + (PART_TABLE.equals(part) ? "" : part + " ") + seq + ": " + e.getMessage(), e); } - body.reset(); + bodyLen = 0; seq++; } } @@ -1970,11 +2218,16 @@ static final class SocketSender implements FrameSender { @Override public void send(byte[] frame) throws IOException { + send(frame, 0, frame.length); + } + + @Override + public void send(byte[] b, int off, int len) throws IOException { AFUNIXSocket s = connect(); try { s.setSoTimeout(REPLY_TIMEOUT_MILLIS); OutputStream out = s.getOutputStream(); - out.write(frame); + out.write(b, off, len); out.flush(); // Through the channel: junixsocket's Socket.shutdownOutput says "not connected" after connect. s.getChannel().shutdownOutput(); diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableBudgetTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableBudgetTest.java index afc3ce7f..819313b5 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableBudgetTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableBudgetTest.java @@ -37,8 +37,12 @@ * app (classes of one if/else chain method each, as the large-app profile generates them), on one class with a * single long method, and on a real library. -Dkeploy.budget.classes=50000 and -Dkeploy.budget.branches=2000 * measure the large app and a 2000-branch method; the defaults keep the rows quick. It checks that every class - * is measured, what a generated class allocates (bounded below), and that a 3000-branch method maps in under a - * second of CPU; the other numbers, the committed heap's growth among them (BUDGET-COMMITTED), are for reading. + * is measured, what a generated class allocates (bounded below), that the generated app's send, paced by the + * collections, grows the committed heap by no more than maxCommittedGrowth (BUDGET-COMMITTED committed_growth_mb, + * over the heap just after the row's own full collection) within 60 s at 50,000 classes with no frame more than + * 10 s after the one before, and that a 3000-branch method maps in under a second of CPU; the other numbers are + * for reading. committed_growth_from_pre_gc_mb is over the heap before that collection, which the row's own + * class generation sized: it is not asserted. * *

heap_after_gc_max_mb is the most heap any collection during the send left in use: an upper bound of what * the analysis keeps live (a young collection also counts old garbage not collected yet). It is -1 when no @@ -72,6 +76,50 @@ public void aGeneratedAppsTableIsMeasured() throws Exception { // since one analysis gives both (and the records' text). long perClass = b.allocated / n; assertTrue("allocated " + perClass + " bytes a class", perClass < ALLOCATED_PER_CLASS); + // Paced by the collections, the send grows the committed heap over what it was just after a full + // collection by no more than maxCommittedGrowth, within 60 s at 50,000 classes, and its frames keep + // coming. A live run collects at each frame, which the pacing waits out: read its numbers only. + if (Boolean.getBoolean("keploy.budget.live")) { + return; + } + long max = maxCommittedGrowth(gcNames().contains("G1"), Runtime.getRuntime().maxMemory(), + System.getProperty("java.specification.version")); + assertTrue("committed heap growth " + (b.committedGrowth >> 20) + " MiB, over " + (max >> 20), + b.committedGrowth <= max); + assertTrue("took " + b.wallNanos / 1_000_000 + " ms", b.wallNanos < maxWallNanos(n)); + assertTrue("a frame came " + b.maxFrameGapNanos / 1_000_000 + " ms after the one before", + b.maxFrameGapNanos < MAX_FRAME_GAP_NANOS); + } + + /** + * maxCommittedGrowth is the most the send may add to an app pod's committed heap over the heap just after a + * full collection: 128 MiB, but more on G1 with a heap cap of 512 MiB or more before JDK 21. There the + * collection has shrunk G1's heap, its pause raises G1's pause-time ratio, and G1 grows the heap back while + * the table is sent, paced or not: at 50,000 classes in a 4 GiB pod, 420-421 MiB on JDK 8 (unpaced 422-707) and + * 13-166 MiB on JDK 17. Unpaced, JDK 21's G1 there grew 175-343 MiB (paced, 19-20), past the bound. + */ + static long maxCommittedGrowth(boolean g1, long maxHeap, String javaSpec) { + if (g1 && maxHeap >= 512L << 20) { + if (javaSpec.startsWith("1.")) { + return 448L << 20; + } + if (Integer.parseInt(javaSpec) < 21) { + return 192L << 20; + } + } + return 128L << 20; + } + + /** The longest a frame may come after the one before (or the send's start): Keploy waits 30 s for one. */ + static final long MAX_FRAME_GAP_NANOS = 10_000_000_000L; + + /** + * maxWallNanos is the table's time bound: 60 s at 50,000 classes, at least 60 s for fewer, and 1.2 ms a class + * past 50,000. At 50,000 that is 10 s past the pacing's own deadline (BuildTable.Pacer.deadlineNanos), which + * leaves the probe map's frames that time; past 50,000 it leaves more. + */ + static long maxWallNanos(int classes) { + return Math.max(60_000_000_000L, 60_000_000_000L / 50_000 * classes); } /** The most a generated class's table and probe map may allocate on the send's thread: below two analyses'. */ @@ -181,15 +229,54 @@ static byte[] gen(String name, int i, int branches) { return cw.toByteArray(); } - /** count is how many times s holds what, ending past its first carried chars (counted with the frame before). */ - static int count(String s, String what, int carried) { - int n = 0; - for (int k = s.indexOf(what); k >= 0; k = s.indexOf(what, k + 1)) { - if (k + what.length() > carried) { - n++; + /** Matches counts an ASCII text over a stream of byte chunks, a match split over chunks included. */ + static final class Matches { + private final byte[] what; + // The stream's last what.length - 1 bytes (fewer at its start), where a split match begins. + private final byte[] tail; + private int tailLen; + int n; + + Matches(String what) { + this.what = what.getBytes(StandardCharsets.US_ASCII); + this.tail = new byte[this.what.length - 1]; + } + + void add(byte[] b, int from, int to) { + // Matches that begin in the tail and end in this chunk. + for (int i = 0; i < tailLen; i++) { + int inTail = tailLen - i; + if (inTail + (to - from) >= what.length && same(tail, i, 0, inTail) + && same(b, from, inTail, what.length - inTail)) { + n++; + } + } + for (int i = from; i + what.length <= to; i++) { + if (same(b, i, 0, what.length)) { + n++; + } + } + int len = to - from; + if (len >= tail.length) { + System.arraycopy(b, to - tail.length, tail, 0, tail.length); + tailLen = tail.length; + } else { + int keep = Math.min(tailLen, tail.length - len); + System.arraycopy(tail, tailLen - keep, tail, 0, keep); + System.arraycopy(b, from, tail, keep, len); + tailLen = keep + len; } } - return n; + + // same says whether b[at, at + len) is what[whatAt, whatAt + len). + private boolean same(byte[] b, int at, int whatAt, int len) { + for (int k = 0; k < len; k++) { + if (b[at + k] != what[whatAt + k]) { + return false; + } + } + return true; + } } /** returns is a class with a static run(int) of n lines "if (x == k) return k;", then "return -1;". */ @@ -232,6 +319,12 @@ static final class Budget { long allocated; /** The send thread's CPU time, in nanoseconds. */ long cpuNanos; + /** The send's wall time, in nanoseconds. */ + long wallNanos; + /** The most the heap's committed size grew over what it was just before the send (after a collection). */ + long committedGrowth; + /** The longest time between two frames, or the send's start and its first frame. */ + long maxFrameGapNanos; } /** measure sends the table of classes to a sink that counts, as the table's thread does, and reports. */ @@ -331,10 +424,11 @@ private static Budget measure(String label, List classes, boolean live = Boolean.getBoolean("keploy.budget.live"); long[] liveMax = {-1}; // A record may run on into the next frame: not_measured and record_too_large are counted over the - // table's stream, keeping the end of each frame that could begin a split match. - String notMeasured = "\"state\":\"not_measured\""; - String tooLarge = "\"reason\":{\"code\":\"record_too_large\"}"; - String[] carry = {""}; + // table's stream, a match split over two frames included. The sink reads each frame where the writer + // keeps it, as the socket's does: a copy or a String of a 1 MiB frame is a G1 humongous object, which + // grows the heap this row measures. + Matches notMeasured = new Matches("\"state\":\"not_measured\""); + Matches tooLarge = new Matches("\"reason\":{\"code\":\"record_too_large\"}"); Budget b = new Budget(); long gc0 = gcMillis(); long process0 = processCpuNanos(); @@ -344,40 +438,56 @@ private static Budget measure(String label, List classes, long allocated0 = allocation.getThreadAllocatedBytes(self); long cpu0 = threads.getCurrentThreadCpuTime(); long t0 = System.nanoTime(); + long[] lastFrame = {t0, 0}; sampler.start(); - BuildTable.send(() -> classes, frame -> { - long sink0 = allocation.getThreadAllocatedBytes(self); - if (live) { - System.gc(); - long used = 0; - for (MemoryPoolMXBean p : ManagementFactory.getMemoryPoolMXBeans()) { - if (heapPools.contains(p.getName())) { - used += p.getUsage().getUsed(); + BuildTable.send(() -> classes, new BuildTable.FrameSender() { + @Override + public void send(byte[] frame) throws java.io.IOException { + send(frame, 0, frame.length); + } + + @Override + public void send(byte[] frame, int off, int len) { + long sink0 = allocation.getThreadAllocatedBytes(self); + long at0 = System.nanoTime(); + lastFrame[1] = Math.max(lastFrame[1], at0 - lastFrame[0]); + if (live) { + System.gc(); + long used = 0; + for (MemoryPoolMXBean p : ManagementFactory.getMemoryPoolMXBeans()) { + if (heapPools.contains(p.getName())) { + used += p.getUsage().getUsed(); + } } + liveMax[0] = Math.max(liveMax[0], used); } - liveMax[0] = Math.max(liveMax[0], used); - } - String f = new String(frame, StandardCharsets.UTF_8); - int at = f.indexOf("{\"v\":1,"); - String header = f.substring(at, f.indexOf('\n', at)); - boolean probes = header.contains("\"part\":\"probes\""); - bytes[probes ? 1 : 0] += frame.length; - if (header.contains("\"last\":true")) { - int records = Integer.parseInt(header.replaceAll(".*\"records\":(\\d+).*", "$1")); - if (probes) { - b.probeRecords = records; - } else { - b.tableRecords = records; + // The header is the line after the 8 bytes of magic and length. + int nl = off + 8; + while (frame[nl] != '\n') { + nl++; } + String header = new String(frame, off + 8, nl - off - 8, StandardCharsets.UTF_8); + boolean probes = header.contains("\"part\":\"probes\""); + bytes[probes ? 1 : 0] += len; + if (header.contains("\"last\":true")) { + int records = Integer.parseInt(header.replaceAll(".*\"records\":(\\d+).*", "$1")); + if (probes) { + b.probeRecords = records; + } else { + b.tableRecords = records; + } + } + if (!probes) { + notMeasured.add(frame, nl + 1, off + len); + tooLarge.add(frame, nl + 1, off + len); + } + sinkAllocated[0] += allocation.getThreadAllocatedBytes(self) - sink0; + // The sink's own time (a live run's collection) is not the send's gap. + lastFrame[0] = System.nanoTime(); } - if (!probes) { - String body = carry[0] + f.substring(f.indexOf('\n', at) + 1); - b.notMeasured += count(body, notMeasured, carry[0].length()); - b.tooLarge += count(body, tooLarge, carry[0].length()); - carry[0] = body.substring(Math.max(0, body.length() - (tooLarge.length() - 1))); - } - sinkAllocated[0] += allocation.getThreadAllocatedBytes(self) - sink0; }); + b.notMeasured = notMeasured.n; + b.tooLarge = tooLarge.n; long wall = System.nanoTime() - t0; long allocated = allocation.getThreadAllocatedBytes(self) - allocated0 - sinkAllocated[0]; sending.set(false); @@ -406,13 +516,16 @@ private static Budget measure(String label, List classes, bytes[0], bytes[1], b.notMeasured, allocated >> 20, (allocated / Math.max(1, classes.size())) >> 10); b.allocated = allocated; b.cpuNanos = cpu; + b.wallNanos = wall; + b.maxFrameGapNanos = lastFrame[1]; long committedPeak = Math.max(committedSampled.get(), committedGcMax.get()); + b.committedGrowth = committedPeak - committedBase; System.out.printf(Locale.ROOT, "BUDGET-COMMITTED %s java=%s gc=%s heap_max_mb=%d committed_pre_gc_mb=%d" + " committed_before_mb=%d committed_peak_mb=%d committed_end_mb=%d committed_growth_mb=%d" - + " committed_growth_from_pre_gc_mb=%d%n", + + " committed_growth_from_pre_gc_mb=%d max_frame_gap_ms=%d%n", label, System.getProperty("java.version"), gcNames(), Runtime.getRuntime().maxMemory() >> 20, committedPre >> 20, committedBase >> 20, committedPeak >> 20, committedEnd >> 20, - (committedPeak - committedBase) >> 20, (committedPeak - committedPre) >> 20); + (committedPeak - committedBase) >> 20, (committedPeak - committedPre) >> 20, lastFrame[1] / 1_000_000); return b; } diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTablePacerTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTablePacerTest.java new file mode 100644 index 00000000..ef8f3644 --- /dev/null +++ b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTablePacerTest.java @@ -0,0 +1,391 @@ +package io.keploy.dedup; + +import org.junit.Test; + +import java.io.InterruptedIOException; +import java.util.ArrayList; +import java.util.List; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +/** + * The table's pacing (BuildTable.Pacer) on a clock the rows drive: it waits out 200 times the collections' + * time, only after a collection, never more than its allowance of 0.8 ms a class file, saved up to a second, + * and never past the table's deadline. + */ +public class BuildTablePacerTest { + + private static final long MS = 1_000_000L; + private static final long A = BuildTable.Pacer.ALLOWANCE_NANOS_PER_CLASS; + private static final long MIN = BuildTable.Pacer.MIN_WAIT_NANOS; + // The class files whose allowance first reaches the smallest wait, after a wait used it all. + private static final long K = (MIN + A - 1) / A; + + /** A clock that moves only as a row says and by what the pacer sleeps (plus overrun a sleep; or not). */ + static final class Clock implements BuildTable.Pacer.Clock { + long now = 1_000 * MS; + long gc = 7; + long overrun; + /** A sleep does not move the clock (one too coarse to see it). */ + boolean frozen; + /** The time the last sleep ended. */ + long lastWake; + int gcReads; + RuntimeException failGc; + boolean interrupt; + final List sleeps = new ArrayList<>(); + /** The unit after which each sleep came (the count of unitDone calls made by then). */ + final List sleptAt = new ArrayList<>(); + int units; + + @Override + public long nanoTime() { + return now; + } + + @Override + public long gcMillis() { + gcReads++; + if (failGc != null) { + throw failGc; + } + return gc; + } + + @Override + public void sleep(long nanos) throws InterruptedException { + if (interrupt) { + throw new InterruptedException("row"); + } + sleeps.add(nanos); + sleptAt.add(units); + if (!frozen) { + now += nanos + overrun; + } + lastWake = now; + } + + long slept() { + long s = 0; + for (long n : sleeps) { + s += n; + } + return s; + } + } + + /** run makes n units of one class file each, workNanos apiece, collecting gcPerUnit ms in each. */ + private static void run(BuildTable.Pacer p, Clock c, int n, long workNanos, long gcPerUnit) + throws InterruptedIOException { + for (int i = 0; i < n; i++) { + c.now += workNanos; + c.gc += gcPerUnit; + c.units++; + p.unitDone(1); + } + } + + @Test + public void aJvmThatDoesNotCollectIsNotPaced() throws Exception { + Clock c = new Clock(); + BuildTable.Pacer p = new BuildTable.Pacer(c); + run(p, c, 50_000, 50_000, 0); + assertEquals(0, c.sleeps.size()); + assertEquals(0, p.waitedNanos()); + } + + @Test + public void aCollectionIsWaitedOutTo200TimesItsTime() throws Exception { + Clock c = new Clock(); + BuildTable.Pacer p = new BuildTable.Pacer(c); + run(p, c, 1_000, 50_000, 0); + assertEquals("no wait before a collection", 0, c.sleeps.size()); + // A 4 ms collection during the next unit: 800 ms of the send's time, less that unit's 50 us. + c.now += 50_000; + c.gc += 4; + c.units++; + p.unitDone(1); + assertEquals(1, c.sleeps.size()); + assertEquals(800 * MS - 50_000, (long) c.sleeps.get(0)); + run(p, c, 1_000, 50_000, 0); + assertEquals("paid off: no wait after it", 1, c.sleeps.size()); + assertEquals(800 * MS - 50_000, p.waitedNanos()); + } + + @Test + public void workSinceACollectionCountsTowardItsWait() throws Exception { + Clock c = new Clock(); + BuildTable.Pacer p = new BuildTable.Pacer(c); + run(p, c, 2_000, 50_000, 0); + // A 1 ms collection (200 ms owed) during a unit that took 150 ms: 50 ms are left to wait. + c.now += 150 * MS; + c.gc += 1; + c.units++; + p.unitDone(1); + assertEquals(1, c.sleeps.size()); + assertEquals(50 * MS, (long) c.sleeps.get(0)); + // Owed less than the smallest wait is never slept. + c.now += 195 * MS + 1; + c.gc += 1; + c.units++; + p.unitDone(1); + assertEquals(1, c.sleeps.size()); + } + + @Test + public void theWaitsAreBoundedByTheAllowanceOfEachClassFile() throws Exception { + Clock c = new Clock(); + BuildTable.Pacer p = new BuildTable.Pacer(c); + long start = c.now; + // A JVM that collects 10 ms in every unit owes 2 s a unit: only the allowance bounds the waits. + run(p, c, 50_000, 20_000, 10); + long slept = c.slept(); + assertTrue("slept " + slept, slept <= 50_000 * A); + assertTrue("slept " + slept, slept > 50_000 * A - MIN); + assertEquals(slept, p.waitedNanos()); + assertTrue("the table's time", c.now - start <= 50_000 * A + 50_000 * 20_000L); + assertEquals("40 s at 50,000 classes", 40_000 * MS, 50_000 * A); + // Over any run of units, the waits add up to at most a second plus the allowance of its class files. + long[] sum = new long[c.units + 1]; + for (int i = 0; i < c.sleeps.size(); i++) { + sum[c.sleptAt.get(i)] += c.sleeps.get(i); + } + long low = 0; + long prefix = 0; + for (int j = 1; j <= c.units; j++) { + prefix += sum[j]; + long at = prefix - j * A; + assertTrue("waits over a run of units, up to unit " + j, at - low <= BuildTable.Pacer.MAX_BANK_NANOS); + low = Math.min(low, at); + } + } + + @Test + public void aQuietStretchSavesUpAtMostASecond() throws Exception { + Clock c = new Clock(); + BuildTable.Pacer p = new BuildTable.Pacer(c); + run(p, c, 5_000, 20_000, 0); + // 100 ms of collection owes 20 s; 4 s of allowance went by, but a second of it was kept. + c.now += 20_000; + c.gc += 100; + c.units++; + p.unitDone(1); + assertEquals(1, c.sleeps.size()); + assertEquals(BuildTable.Pacer.MAX_BANK_NANOS, (long) c.sleeps.get(0)); + // Then a wait each time the allowance reaches the smallest wait: K class files' worth every K of them. + run(p, c, 100, 20_000, 0); + assertEquals(1 + 100 / K, c.sleeps.size()); + for (int i = 1; i < c.sleeps.size(); i++) { + assertEquals(K * A, (long) c.sleeps.get(i)); + } + } + + @Test + public void aUnitOfManyClassFilesEarnsItsAllowance() throws Exception { + Clock c = new Clock(); + BuildTable.Pacer p = new BuildTable.Pacer(c); + c.now += 20_000; + c.gc += 1_000; + p.unitDone(300); + assertEquals(1, c.sleeps.size()); + assertEquals(300 * A, (long) c.sleeps.get(0)); + } + + @Test + public void aSleepsOverrunComesOffTheAllowance() throws Exception { + Clock c = new Clock(); + c.overrun = 5 * MS; + BuildTable.Pacer p = new BuildTable.Pacer(c); + long start = c.now; + run(p, c, 10_000, 20_000, 10); + long waited = c.now - start - 10_000 * 20_000L; + assertEquals(waited, p.waitedNanos()); + // The time asleep, overruns included, stays within the allowance (and one overrun past it). + assertTrue("waited " + waited, waited <= 10_000 * A + c.overrun); + assertTrue("waited " + waited, waited > 10_000 * A - MIN - c.overrun); + } + + @Test + public void anInterruptEndsTheSendAndIsKept() throws Exception { + // During a wait. + Clock c = new Clock(); + BuildTable.Pacer p = new BuildTable.Pacer(c); + c.interrupt = true; + try { + run(p, c, 100, 20_000, 10); + fail("an interrupt during a wait ends the send"); + } catch (InterruptedIOException e) { + assertEquals("the build table's thread was interrupted", e.getMessage()); + assertTrue("the interrupt is kept for the thread", Thread.interrupted()); + } + assertEquals("it came at the first wait", K, c.units); + // Between waits, and on a Pacer that never waits. + for (BuildTable.Pacer q : new BuildTable.Pacer[] {new BuildTable.Pacer(new Clock()), BuildTable.Pacer.off()}) { + q.unitDone(1); + Thread.currentThread().interrupt(); + try { + q.unitDone(1); + fail("an interrupt ends the send at the next unit"); + } catch (InterruptedIOException e) { + assertTrue("the interrupt is kept for the thread", Thread.interrupted()); + } + } + } + + @Test + public void theWaitsGiveWayToTheDeadline() throws Exception { + // 50,000 class files of 0.5 ms work each (25 s) in a JVM that collects 10 ms in each: the allowance + // alone would wait 40 s, a table of 65 s. The deadline holds it to 50 s, waiting all the room left. + Clock c = new Clock(); + BuildTable.Pacer p = new BuildTable.Pacer(c); + p.begin(50_000); + long start = c.now; + run(p, c, 50_000, MS / 2, 10); + long took = c.now - start; + assertTrue("took " + took, took <= BuildTable.Pacer.deadlineNanos(50_000)); + assertTrue("waited " + p.waitedNanos(), p.waitedNanos() > 25_000 * MS - 2 * MIN); + // Not told its class files, a Pacer still waits no later than 50 s into the table. + Clock d = new Clock(); + BuildTable.Pacer q = new BuildTable.Pacer(d); + start = d.now; + run(q, d, 50_000, MS / 2, 10); + assertTrue("woke at " + (d.lastWake - start), d.lastWake - start <= BuildTable.Pacer.deadlineNanos(0)); + assertTrue("waited " + q.waitedNanos(), q.waitedNanos() > 25_000 * MS); + assertEquals(50_000 * MS, BuildTable.Pacer.deadlineNanos(50_000)); + assertEquals("1 ms a class file past 50,000", 100_000 * MS, BuildTable.Pacer.deadlineNanos(100_000)); + assertEquals(50_000 * MS, BuildTable.Pacer.deadlineNanos(10)); + } + + @Test + public void aTableWhoseWorkFillsItsDeadlineIsNotPaced() throws Exception { + Clock c = new Clock(); + BuildTable.Pacer p = new BuildTable.Pacer(c); + p.begin(50_000); + run(p, c, 50_000, MS, 10); + assertEquals(0, c.sleeps.size()); + } + + @Test + public void aClockThatDoesNotMoveAcrossASleepIsChargedTheWait() throws Exception { + Clock c = new Clock(); + c.frozen = true; + BuildTable.Pacer p = new BuildTable.Pacer(c); + run(p, c, 10_000, 20_000, 10); + long slept = c.slept(); + assertTrue("slept " + slept, slept <= 10_000 * A); + assertTrue("slept " + slept, slept > 10_000 * A - MIN); + } + + @Test + public void aCollectorBeanThatFailsEndsThePacing() throws Exception { + Clock c = new Clock(); + BuildTable.Pacer p = new BuildTable.Pacer(c); + run(p, c, 100, 20_000, 10); + int before = c.sleeps.size(); + assertTrue(before > 0); + c.failGc = new UnsupportedOperationException("row"); + run(p, c, 1, 20_000, 10); + c.failGc = null; + run(p, c, 1_000, 20_000, 10); + assertEquals(before, c.sleeps.size()); + } + + @Test + public void aPacerOffNeverReadsOrWaits() throws Exception { + BuildTable.Pacer p = BuildTable.Pacer.off(); + for (int i = 0; i < 1_000; i++) { + p.unitDone(1_000); + } + assertEquals(0, p.waitedNanos()); + } + + @Test + public void theJvmsPacerReadsItsCollectors() throws Exception { + BuildTable.Pacer.JvmClock jvm = new BuildTable.Pacer.JvmClock( + java.lang.management.ManagementFactory.getGarbageCollectorMXBeans()); + long before = jvm.gcMillis(); + assertTrue("a collection's time is read: " + before, before >= 0); + System.gc(); + assertTrue(jvm.gcMillis() >= before); + assertEquals(0, BuildTable.Pacer.JvmClock.NONE.gcMillis()); + } + + @Test + public void writePacesAfterEachUnitByItsClassFiles() throws Exception { + List app = new ArrayList<>(); + for (int i = 0; i < 100; i++) { + app.add(new BuildTable.ClassFile(BuildTableBudgetTest.name(i), + BuildTableBudgetTest.gen(BuildTableBudgetTest.name(i), i, 3))); + } + Clock c = new Clock(); + int[] records = {0}; + BuildTable.Pacer.Clock collecting = new BuildTable.Pacer.Clock() { + @Override + public long nanoTime() { + return c.nanoTime(); + } + + @Override + public long gcMillis() { + // Each read finds a second more of collection: the allowance alone bounds the waits. + c.gc += 1_000; + return c.gcMillis(); + } + + @Override + public void sleep(long nanos) throws InterruptedException { + c.units = records[0]; + c.sleep(nanos); + } + }; + BuildTable.Summary s = BuildTable.write(app, (build, json) -> records[0]++, BuildTable.DROP, + BuildTable.MAX_RECORD_BYTES, BuildTable.MAX_MAP_BYTES, BuildTable.MAX_MAP_STEPS, + new BuildTable.Pacer(collecting)); + assertEquals(100, s.records); + assertEquals("one read at the start, one after each unit", 101, c.gcReads); + // The allowance of K class files at a time: after every Kth unit's record. + assertEquals(100 / K, c.sleeps.size()); + for (int i = 0; i < c.sleeps.size(); i++) { + assertEquals(K * A, (long) c.sleeps.get(i)); + assertEquals((i + 1) * K, (int) c.sleptAt.get(i)); + } + } + + @Test + public void writeTellsThePacerItsClassFilesForTheDeadline() throws Exception { + // 100 classes of 0.9 s work each: 90 s, past the 50 s deadline from the first, so never a wait. A Pacer + // not told how many class files come would wait through its first 50 s. + List app = new ArrayList<>(); + for (int i = 0; i < 100; i++) { + app.add(new BuildTable.ClassFile(BuildTableBudgetTest.name(i), + BuildTableBudgetTest.gen(BuildTableBudgetTest.name(i), i, 3))); + } + Clock c = new Clock(); + BuildTable.Pacer.Clock slow = new BuildTable.Pacer.Clock() { + @Override + public long nanoTime() { + return c.nanoTime(); + } + + @Override + public long gcMillis() { + // Read once a unit: a unit's work, and a second more of collection. + c.now += 900 * MS; + c.gc += 1_000; + return c.gcMillis(); + } + + @Override + public void sleep(long nanos) throws InterruptedException { + c.sleep(nanos); + } + }; + BuildTable.Summary s = BuildTable.write(app, (build, json) -> { }, BuildTable.DROP, + BuildTable.MAX_RECORD_BYTES, BuildTable.MAX_MAP_BYTES, BuildTable.MAX_MAP_STEPS, + new BuildTable.Pacer(slow)); + assertEquals(100, s.records); + assertEquals(0, c.sleeps.size()); + } +} diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java index b62eb3b5..05ef0c83 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java @@ -869,6 +869,55 @@ public void noReceiverIsSaidSo() throws Exception { } } + @Test(timeout = 30_000) + public void anInterruptEndsTheSendAndIsSaidSo() throws Exception { + // Before the send, the next unit ends it, so no frame goes. After the table's last frame, the interrupt + // closes the probe map's temp file (an interruptible channel) as it is read back. Either way the WARNING + // names the interrupt, not the temp file, the interrupt is kept, and no temp file is left. + List app = javaDedup(); + for (boolean afterTable : new boolean[] {false, true}) { + File dir = tmp.newFolder("spill-" + afterTable); + List heads = new ArrayList<>(); + BuildTable.FrameSender sender = frame -> { + int nl = 8; + while (frame[nl] != '\n') { + nl++; + } + String h = new String(frame, 8, nl - 8, StandardCharsets.UTF_8); + heads.add(h); + if (afterTable && h.contains("\"part\":\"table\"") && h.contains("\"last\":true")) { + Thread.currentThread().interrupt(); + } + }; + try (Logs logs = new Logs()) { + if (!afterTable) { + Thread.currentThread().interrupt(); + } + try { + BuildTable.send(() -> app, sender, dir); + assertTrue(afterTable + ": the interrupt is kept", Thread.currentThread().isInterrupted()); + } finally { + Thread.interrupted(); + } + LogRecord last = logs.records.get(logs.records.size() - 1); + assertEquals(Level.WARNING, last.getLevel()); + assertTrue(last.getMessage(), last.getMessage().endsWith("the build table was not sent: its thread was" + + " interrupted; it goes again when Keploy asks for it again")); + assertNull(last.getThrown()); + } + if (afterTable) { + assertFalse(heads.isEmpty()); + assertTrue(heads.toString(), heads.get(heads.size() - 1).contains("\"last\":true")); + for (String h : heads) { + assertTrue("no probe map frame: " + h, h.contains("\"part\":\"table\"")); + } + } else { + assertEquals(heads.toString(), 0, heads.size()); + } + assertArrayEquals("no temp file left", new String[0], dir.list()); + } + } + @Test(timeout = 30_000) public void aRequestedTableArrivesWithTheBuildId() throws Exception { File sock = new File(tmp.getRoot(), "table.sock"); From ddd0cf2d194a74bda7d5076d4afa7b2d32b1b44e Mon Sep 17 00:00:00 2001 From: slayerjain Date: Fri, 20 Nov 2026 13:59:45 +0530 Subject: [PATCH 10/14] fix(dedup): run a table request after an interrupt on a new thread An interrupt ends a build table send, and the interrupt is kept for its thread. Before, Sends ran the request made during that send on the same thread with the interrupt still set. That request names the agent's newest index, which may not be read yet. On JDK 8 a class file's read fails on an interrupted thread (ClosedByInterruptException), so the index would keep only the classes it got, for the JVM's life (a wrong build id). The request then ended at its first unit with a second WARNING. - Sends hands the request made during an interrupted send to a new thread, uninterrupted, and ends the interrupted one. Sends stays running while that thread starts, so a request made meanwhile waits to run after it. If that thread cannot start, the request is dropped (and any request made while it tried) with the same WARNING as a first thread that cannot start, and the next request starts a thread. - send() checks the interrupt before it reads any class: on an interrupted thread it reads none, sends no frame, and logs the WARNING that names the interrupt. - Pacer takes the analysis's rate from begin(), after the index's read, not from the send's start. Before, a slow index read counted as the analysis's work, so the rate was overestimated and the first units went unpaced. The 50 s deadline still counts from the send's start, the index's read included. Rows: a request made while a send is interrupted runs on a new, uninterrupted thread; one made while that thread starts starts no thread of its own and runs after the first on that thread; one whose new thread cannot start is dropped with a WARNING, as is one made while it tried, and the next request runs; a send on an interrupted thread reads no class; after a 10 s read the first wait comes as soon as the allowance allows and the waits end by the deadline. Each fails on the code before this change. The Pacer's interrupt row clears the flag in a finally, so a failing row cannot leave it set for the rows after it. Signed-off-by: slayerjain --- .../main/java/io/keploy/dedup/BuildTable.java | 61 ++++-- .../io/keploy/dedup/BuildTablePacerTest.java | 57 ++++-- .../java/io/keploy/dedup/BuildTableTest.java | 183 +++++++++++++++++- 3 files changed, 268 insertions(+), 33 deletions(-) diff --git a/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java b/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java index d4cb7289..d14c1df9 100644 --- a/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java +++ b/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java @@ -393,9 +393,15 @@ static Thread thread(Runnable r) { * Sends runs one send at a time. Requests made while one runs are run once after it, as the newest of them: * each request names its own index and data socket, and only the newest is the agent's now (a stop() and * start() during a send makes a new index and socket). A send that throws is logged, and the request made - * while it ran still runs. It assumes one caller at a time (the control server handles one command at a - * time): a thread that fails to start clears {@code pending}, which would drop a request that raced in from - * a second caller. + * while it ran still runs. A send that leaves its thread interrupted (an interrupt ends a send: see + * {@link Pacer}) ends that thread: the request made while it ran goes to a new thread, so it never reads the + * index's class files on an interrupted thread (on JDK 8 a read of a class file there fails, and the index + * would keep the classes it got). Sends stays running while that thread starts, so a request made meanwhile + * waits to run after it. A thread that fails to start clears {@code pending}: a request made while the + * handover's thread was starting is dropped with the one it was starting for (ACKed, never sent; Keploy's + * wait for the table ends it, and Keploy asks again on its next run). On the control path the failure + * answers IGNORED and drops no other request, as Sends assumes one caller at a time there (the control + * server handles one command at a time). */ static final class Sends { private final ThreadFactory threads; @@ -421,12 +427,17 @@ boolean request(Runnable send) { } running = true; } + // Called on the control path: a thread that cannot start (a pids limit, a security manager) must not + // end the control server. Keploy reads IGNORED as no table from this SDK. + return start(send); + } + + /** start runs send and the requests after it on a new thread; false, with a WARNING, if none starts. */ + private boolean start(Runnable send) { try { threads.newThread(() -> loop(send)).start(); return true; } catch (Throwable t) { - // Called on the control path: a thread that cannot start (a pids limit, a security manager) - // must not end the control server. Keploy reads IGNORED as no table from this SDK. synchronized (this) { running = false; pending = null; @@ -454,6 +465,13 @@ private void loop(Runnable first) { send = pending; pending = null; } + if (Thread.currentThread().isInterrupted()) { + // The interrupt ended this thread's send and is kept for it: the next one runs on a new + // thread, uninterrupted. Sends stays running meanwhile, so a request made now waits for it. + // start's factory and log calls run on this interrupted thread: keep them non-interruptible. + start(send); + return; + } } } } @@ -470,6 +488,10 @@ static void send(Supplier> classes, FrameSender sender) { /** send with the probe map's temp file made in spillDir. */ static void send(Supplier> classes, FrameSender sender, File spillDir) { try { + if (Thread.currentThread().isInterrupted()) { + // An interrupt ends a send: read no class file on this thread (on JDK 8 that read fails). + throw Pacer.interrupted(); + } long cpu0 = threadCpuNanos(); long t0 = System.nanoTime(); FrameWriter frames = new FrameWriter(sender, PART_TABLE, MAX_FRAME_BYTES); @@ -544,11 +566,11 @@ private static long threadCpuNanos() { * the waits over any k class files add up to at most a second plus 0.8 k ms, so frames keep coming (Keploy's * wait for the table is a bound on the time since its last frame), and all of them to at most 0.8 ms a * class file (40 s at 50,000). And no wait is taken that would end the table past {@link #deadlineNanos} - * (50 s at 50,000 class files), reckoned as the time so far plus the rest of its class files at the rate - * the work went so far: on a slow or busy CPU the pacing gives way to the table's 60 s bound, with 10 s - * left for the probe map's frames and the estimate's error. A clock or collector bean that fails ends the - * pacing, and the rest is sent unpaced. An interrupt ends the send: unitDone throws, and the interrupt is - * kept for the thread. + * (50 s at 50,000 class files), reckoned as the time since the send began (the index's read included) plus + * the rest of its class files at the rate the analysis went so far (from {@link #begin}, its waits left + * out): on a slow or busy CPU the pacing gives way to the table's 60 s bound, with 10 s left for the probe + * map's frames and the estimate's error. A clock or collector bean that fails ends the pacing, and the rest + * is sent unpaced. An interrupt ends the send: unitDone throws, and the interrupt is kept for the thread. */ static final class Pacer { /** Each millisecond of collection asks for this many of the send's: at most 0.5% of it collecting. */ @@ -578,6 +600,11 @@ interface Clock { private final Clock clock; private boolean off; private long start; + /** + * When the analysis began, for the rate: set by begin, or at construction if begin is never reached (then + * total is 0 and the rate is never read). start is the send's, for the deadline. + */ + private long begun; private long total; private long done; private long last; @@ -591,6 +618,7 @@ interface Clock { try { last = clock.nanoTime(); start = last; + begun = last; gc = clock.gcMillis(); } catch (RuntimeException e) { stop("its clock failed", e); @@ -624,9 +652,18 @@ static long deadlineNanos(long classFiles) { return Math.max(DEADLINE_NANOS, DEADLINE_NANOS / DEADLINE_CLASS_FILES * classFiles); } - /** begin says how many class files the send analyzes, for its deadline; not called, the deadline is 50 s. */ + /** + * begin is called as the analysis begins, after the index's read: it says how many class files the send + * analyzes, for its deadline (not called, the deadline is 50 s), and reads the clock the rate is taken + * from, so the index's read does not count as the analysis's work. + */ void begin(long classFiles) { total = Math.max(0, classFiles); + try { + begun = clock.nanoTime(); + } catch (RuntimeException e) { + stop("its clock failed", e); + } } /** waitedNanos is how long the thread waited, in all. */ @@ -687,7 +724,7 @@ private long room(long now) { long elapsed = Math.max(0, now - start); double rest = 0; if (done > 0 && total > done) { - rest = (double) Math.max(0, elapsed - waited) / done * (total - done); + rest = (double) Math.max(0, now - begun - waited) / done * (total - done); } return (long) Math.max(-1.0, deadlineNanos(total) - elapsed - rest); } diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTablePacerTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTablePacerTest.java index ef8f3644..587438b9 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTablePacerTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTablePacerTest.java @@ -209,28 +209,34 @@ public void aSleepsOverrunComesOffTheAllowance() throws Exception { @Test public void anInterruptEndsTheSendAndIsKept() throws Exception { - // During a wait. - Clock c = new Clock(); - BuildTable.Pacer p = new BuildTable.Pacer(c); - c.interrupt = true; try { - run(p, c, 100, 20_000, 10); - fail("an interrupt during a wait ends the send"); - } catch (InterruptedIOException e) { - assertEquals("the build table's thread was interrupted", e.getMessage()); - assertTrue("the interrupt is kept for the thread", Thread.interrupted()); - } - assertEquals("it came at the first wait", K, c.units); - // Between waits, and on a Pacer that never waits. - for (BuildTable.Pacer q : new BuildTable.Pacer[] {new BuildTable.Pacer(new Clock()), BuildTable.Pacer.off()}) { - q.unitDone(1); - Thread.currentThread().interrupt(); + // During a wait. + Clock c = new Clock(); + BuildTable.Pacer p = new BuildTable.Pacer(c); + c.interrupt = true; try { - q.unitDone(1); - fail("an interrupt ends the send at the next unit"); + run(p, c, 100, 20_000, 10); + fail("an interrupt during a wait ends the send"); } catch (InterruptedIOException e) { + assertEquals("the build table's thread was interrupted", e.getMessage()); assertTrue("the interrupt is kept for the thread", Thread.interrupted()); } + assertEquals("it came at the first wait", K, c.units); + // Between waits, and on a Pacer that never waits. + for (BuildTable.Pacer q : new BuildTable.Pacer[] {new BuildTable.Pacer(new Clock()), + BuildTable.Pacer.off()}) { + q.unitDone(1); + Thread.currentThread().interrupt(); + try { + q.unitDone(1); + fail("an interrupt ends the send at the next unit"); + } catch (InterruptedIOException e) { + assertTrue("the interrupt is kept for the thread", Thread.interrupted()); + } + } + } finally { + // A failing row must not leave the runner's thread interrupted for the rows after it. + Thread.interrupted(); } } @@ -258,6 +264,23 @@ public void theWaitsGiveWayToTheDeadline() throws Exception { assertEquals(50_000 * MS, BuildTable.Pacer.deadlineNanos(10)); } + @Test + public void theRateLeavesOutTheTimeBeforeTheAnalysis() throws Exception { + // The index's read takes 10 s before the analysis begins; then 50,000 class files of 0.5 ms work each + // (25 s), 10 ms of collection in each. The rate is the analysis's own (0.5 ms a class file), so the first + // wait comes as soon as the allowance reaches the smallest wait; the deadline still counts the 10 s, so + // the waits fill the 15 s left and no more. + Clock c = new Clock(); + BuildTable.Pacer p = new BuildTable.Pacer(c); + long start = c.now; + c.now += 10_000 * MS; + p.begin(50_000); + run(p, c, 50_000, MS / 2, 10); + assertEquals("the first wait as soon as the allowance allows", K, (long) c.sleptAt.get(0)); + assertTrue("took " + (c.now - start), c.now - start <= BuildTable.Pacer.deadlineNanos(50_000)); + assertTrue("waited " + p.waitedNanos(), p.waitedNanos() > 15_000 * MS - 2 * MIN); + } + @Test public void aTableWhoseWorkFillsItsDeadlineIsNotPaced() throws Exception { Clock c = new Clock(); diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java index 05ef0c83..b02b204a 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java @@ -42,7 +42,9 @@ import java.util.stream.Collectors; import java.util.concurrent.CountDownLatch; import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; import java.util.logging.Handler; import java.util.logging.Level; import java.util.logging.LogRecord; @@ -53,7 +55,9 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertNotSame; import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertSame; import static org.junit.Assert.assertTrue; import static org.junit.Assert.fail; @@ -871,11 +875,13 @@ public void noReceiverIsSaidSo() throws Exception { @Test(timeout = 30_000) public void anInterruptEndsTheSendAndIsSaidSo() throws Exception { - // Before the send, the next unit ends it, so no frame goes. After the table's last frame, the interrupt - // closes the probe map's temp file (an interruptible channel) as it is read back. Either way the WARNING - // names the interrupt, not the temp file, the interrupt is kept, and no temp file is left. + // Before the send, it reads no class (on JDK 8 a class file's read fails on an interrupted thread, and the + // index would keep what it got), so no frame goes. After the table's last frame, the interrupt closes the + // probe map's temp file (an interruptible channel) as it is read back. Either way the WARNING names the + // interrupt, not the temp file, the interrupt is kept, and no temp file is left. List app = javaDedup(); for (boolean afterTable : new boolean[] {false, true}) { + AtomicInteger read = new AtomicInteger(); File dir = tmp.newFolder("spill-" + afterTable); List heads = new ArrayList<>(); BuildTable.FrameSender sender = frame -> { @@ -894,7 +900,10 @@ public void anInterruptEndsTheSendAndIsSaidSo() throws Exception { Thread.currentThread().interrupt(); } try { - BuildTable.send(() -> app, sender, dir); + BuildTable.send(() -> { + read.incrementAndGet(); + return app; + }, sender, dir); assertTrue(afterTable + ": the interrupt is kept", Thread.currentThread().isInterrupted()); } finally { Thread.interrupted(); @@ -914,6 +923,7 @@ public void anInterruptEndsTheSendAndIsSaidSo() throws Exception { } else { assertEquals(heads.toString(), 0, heads.size()); } + assertEquals(afterTable + ": the classes read", afterTable ? 1 : 0, read.get()); assertArrayEquals("no temp file left", new String[0], dir.list()); } } @@ -1014,6 +1024,171 @@ public void aRequestMadeWhileASendFailsStillRuns() throws Exception { assertEquals(Arrays.asList("failing", "next", "after"), new ArrayList<>(ran)); } + @Test(timeout = 30_000) + public void aRequestMadeWhileASendIsInterruptedRunsOnANewThread() throws Exception { + // An interrupt ends a send and is kept for its thread. The request made while it ran names the agent's + // newest index, not read yet: it must not run on that thread, where on JDK 8 the index's read of a class + // file fails. It runs on a new thread, uninterrupted, and Sends takes requests after it. + CountDownLatch inFirst = new CountDownLatch(1); + CountDownLatch release = new CountDownLatch(1); + CountDownLatch next = new CountDownLatch(1); + CountDownLatch after = new CountDownLatch(1); + AtomicReference first = new AtomicReference<>(); + AtomicReference second = new AtomicReference<>(); + AtomicReference secondInterrupted = new AtomicReference<>(); + AtomicInteger made = new AtomicInteger(); + BuildTable.Sends sends = new BuildTable.Sends(r -> { + made.incrementAndGet(); + return BuildTable.thread(r); + }); + assertTrue(sends.request(() -> { + first.set(Thread.currentThread()); + inFirst.countDown(); + try { + release.await(10, TimeUnit.SECONDS); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + Thread.currentThread().interrupt(); + })); + assertTrue(inFirst.await(10, TimeUnit.SECONDS)); + assertTrue(sends.request(() -> { + second.set(Thread.currentThread()); + secondInterrupted.set(Thread.currentThread().isInterrupted()); + next.countDown(); + })); + release.countDown(); + assertTrue("the request made while it ran ran", next.await(10, TimeUnit.SECONDS)); + assertNotSame("on a new thread", first.get(), second.get()); + assertEquals("not interrupted", Boolean.FALSE, secondInterrupted.get()); + assertEquals(2, made.get()); + assertTrue(sends.request(after::countDown)); + assertTrue("a request after them runs", after.await(10, TimeUnit.SECONDS)); + } + + @Test(timeout = 30_000) + public void aRequestMadeWhileTheNextThreadStartsRunsAfterItOnThatThread() throws Exception { + // Sends stays running while the interrupted thread starts the next one, so sends stay one at a time: a + // request made in that window starts no thread of its own and runs after the one being started, on the + // new thread. The factory holds the window open; it runs on the interrupted thread, so it spins rather + // than waits. + CountDownLatch inFirst = new CountDownLatch(1); + CountDownLatch release = new CountDownLatch(1); + CountDownLatch inHandover = new CountDownLatch(1); + CountDownLatch done = new CountDownLatch(1); + AtomicBoolean go = new AtomicBoolean(); + AtomicInteger made = new AtomicInteger(); + List ran = Collections.synchronizedList(new ArrayList<>()); + List on = Collections.synchronizedList(new ArrayList<>()); + BuildTable.Sends sends = new BuildTable.Sends(r -> { + if (made.incrementAndGet() == 2) { + inHandover.countDown(); + spin(go); + } + return BuildTable.thread(r); + }); + assertTrue(sends.request(() -> { + on.add(Thread.currentThread()); + inFirst.countDown(); + try { + release.await(10, TimeUnit.SECONDS); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + Thread.currentThread().interrupt(); + })); + assertTrue(inFirst.await(10, TimeUnit.SECONDS)); + assertTrue(sends.request(() -> { + on.add(Thread.currentThread()); + ran.add("next"); + })); + release.countDown(); + assertTrue("the interrupted thread starts the next one", inHandover.await(10, TimeUnit.SECONDS)); + try { + assertTrue("a request made meanwhile is taken", sends.request(() -> { + on.add(Thread.currentThread()); + ran.add("meanwhile"); + done.countDown(); + })); + assertEquals("and starts no thread of its own", 2, made.get()); + } finally { + go.set(true); + } + assertTrue("it runs", done.await(10, TimeUnit.SECONDS)); + assertEquals("after the one being started", Arrays.asList("next", "meanwhile"), new ArrayList<>(ran)); + assertEquals(2, made.get()); + assertNotSame("on the new thread", on.get(0), on.get(1)); + assertSame("both", on.get(1), on.get(2)); + } + + /** spin waits, without being interruptible, until go is set or 10 s pass. */ + private static void spin(AtomicBoolean go) { + long end = System.nanoTime() + 10_000_000_000L; + while (!go.get() && System.nanoTime() - end < 0) { + Thread.yield(); + } + } + + @Test(timeout = 30_000) + public void anInterruptedSendWhoseNextThreadCannotStartLetsSendsTakeRequests() throws Exception { + // The new thread for the request made during an interrupted send cannot start: that request is dropped + // with a WARNING (Keploy asks again), and so is one made while the thread was being started (ACKed, never + // sent). The next request starts a thread. The factory holds that window open by spinning: it runs on the + // interrupted thread. + CountDownLatch inFirst = new CountDownLatch(1); + CountDownLatch release = new CountDownLatch(1); + CountDownLatch inHandover = new CountDownLatch(1); + CountDownLatch after = new CountDownLatch(1); + AtomicBoolean go = new AtomicBoolean(); + AtomicInteger made = new AtomicInteger(); + AtomicInteger dropped = new AtomicInteger(); + BuildTable.Sends sends = new BuildTable.Sends(r -> { + if (made.incrementAndGet() == 2) { + inHandover.countDown(); + spin(go); + throw new OutOfMemoryError("unable to create native thread"); + } + return BuildTable.thread(r); + }); + try (Logs logs = new Logs()) { + assertTrue(sends.request(() -> { + inFirst.countDown(); + try { + release.await(10, TimeUnit.SECONDS); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + Thread.currentThread().interrupt(); + })); + assertTrue(inFirst.await(10, TimeUnit.SECONDS)); + assertTrue(sends.request(dropped::incrementAndGet)); + release.countDown(); + assertTrue("the interrupted thread tries to start one", inHandover.await(10, TimeUnit.SECONDS)); + try { + assertTrue("a request made meanwhile is ACKed", sends.request(dropped::incrementAndGet)); + assertEquals(2, made.get()); + } finally { + go.set(true); + } + // The WARNING is logged once Sends has let go of the dropped requests. + long end = System.nanoTime() + 10_000_000_000L; + boolean warned = false; + while (!warned && System.nanoTime() < end) { + for (LogRecord rec : logs.records) { + warned |= rec.getLevel() == Level.WARNING + && rec.getMessage().contains("the build table's thread did not start"); + } + Thread.sleep(10); + } + assertTrue("a WARNING says so", warned); + assertEquals(2, made.get()); + assertTrue("the next request starts a thread", sends.request(after::countDown)); + assertTrue(after.await(10, TimeUnit.SECONDS)); + assertEquals("the dropped requests never ran", 0, dropped.get()); + assertEquals(3, made.get()); + } + } + /** Throwing is a JUL handler on BuildTable's logger that throws on every record, as an app's bridge can. */ static final class Throwing extends Handler implements AutoCloseable { final AtomicInteger published = new AtomicInteger(); From 895dc707d6955da46d57121ba8b850a460b59016 Mon Sep 17 00:00:00 2001 From: slayerjain Date: Fri, 20 Nov 2026 15:07:41 +0530 Subject: [PATCH 11/14] fix(dedup): log when the build table is deferred or cut short by exit A native keploy test stops the app at each test set's end. A build table send still running then ended with no line in the app's log, so a run whose table never arrived could not tell why from the app's side. Two ways the SDK puts the table off were silent too. - An exit notice: while a send runs, a shutdown hook is held for it and removed when the send ends. If the JVM exits first, the hook prints one line to System.err: how many class files were analyzed of how many (or that the class files were still being read), the frames sent, the time since the send began and how much of it was paced. It writes to System.err, not java.util.logging, because shutdown hooks run at once and in no order, and the logging handlers may already be closed. A JVM that is killed (SIGKILL) or halted runs no hook and prints nothing. - A TABLE ask made while a send runs is logged at INFO, once per ask: it is sent again when that send ends, and an earlier ask that waited for it is dropped (the newest names Keploy's current socket). - A paced send logs one INFO line at its first wait: the collection time since the send began, the wait per millisecond of it, the most the waits may add up to for its class files, and the deadline no wait may push the table's end past. The Pacer now counts every analyzed class file, also when the pacing is off, so the notice can say how far the send got. Rows: a JVM that exits mid-send prints the notice exactly once; a JVM whose send ends before its exit prints none (each in its own JVM); an ask made during a send logs one INFO line each, the second naming the dropped ask, and only the newest ask runs; a paced send logs one line however many times it waits; a send that never waits logs none. The first, third and fourth fail on the code before this change. Signed-off-by: slayerjain --- .../main/java/io/keploy/dedup/BuildTable.java | 157 +++++++++++-- .../io/keploy/dedup/BuildTableNoticeTest.java | 208 ++++++++++++++++++ 2 files changed, 351 insertions(+), 14 deletions(-) create mode 100644 keploy-sdk/src/test/java/io/keploy/dedup/BuildTableNoticeTest.java diff --git a/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java b/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java index d14c1df9..190d2009 100644 --- a/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java +++ b/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java @@ -19,6 +19,7 @@ import java.io.InputStream; import java.io.InputStreamReader; import java.io.OutputStream; +import java.io.PrintStream; import java.lang.management.GarbageCollectorMXBean; import java.lang.management.ManagementFactory; import java.lang.management.MemoryPoolMXBean; @@ -419,13 +420,24 @@ synchronized int requests() { } boolean request(Runnable send) { + boolean deferred; + boolean dropped = false; synchronized (this) { requests++; - if (running) { + deferred = running; + if (deferred) { + dropped = pending != null; pending = send; - return true; + } else { + running = true; } - running = true; + } + if (deferred) { + // Said once per ask, off the lock, so a table that comes late names why. + log(Level.INFO, "Keploy coverage: Keploy asked for the build table while this JVM sends one: it is" + + " sent again when that send ends" + (dropped ? " (an earlier ask that waited for it is" + + " dropped, as this newer one names Keploy's current socket)" : ""), null); + return true; } // Called on the control path: a thread that cannot start (a pids limit, a security manager) must not // end the control server. Keploy reads IGNORED as no table from this SDK. @@ -498,12 +510,17 @@ static void send(Supplier> classes, FrameSender sender, File spi FrameWriter maps = new FrameWriter(sender, PART_PROBES, MAX_FRAME_BYTES); Summary s; Pacer pacer = Pacer.jvm(); - try (Spill spill = Spill.open(spillDir)) { - s = write(classes.get(), frames, spill, MAX_RECORD_BYTES, MAX_MAP_BYTES, MAX_MAP_STEPS, pacer); - frames.finish(s.build, s.records); - spill.sendTo(maps, s.build); + ExitNotice exit = ExitNotice.add(t0, pacer, frames, maps); + try { + try (Spill spill = Spill.open(spillDir)) { + s = write(classes.get(), frames, spill, MAX_RECORD_BYTES, MAX_MAP_BYTES, MAX_MAP_STEPS, pacer); + frames.finish(s.build, s.records); + spill.sendTo(maps, s.build); + } + maps.finish(s.build, s.probeRecords); + } finally { + exit.remove(); } - maps.finish(s.build, s.probeRecords); StringBuilder m = new StringBuilder("Keploy coverage: build table "); m.append(s.build.isEmpty() ? "(no app classes)" : s.build).append(" sent (").append(s.records) .append(" classes, ").append(s.notMeasured).append(" not measured"); @@ -542,6 +559,90 @@ static void send(Supplier> classes, FrameSender sender, File spi } } + /** + * ExitNotice says, once, on System.err, that the JVM exits while a send runs: a native keploy test stops the + * app at each test set's end, and a send cut short there would otherwise leave no line in the app's log. + * It is a shutdown hook added for the send and removed when it ends. It writes to System.err, not + * java.util.logging: the JVM runs its shutdown hooks at once and in no order, and LogManager's own hook (and + * an app's logging framework's) may have closed the handlers already. A JVM that is killed (SIGKILL) or + * halted runs no hook and says nothing. The hook reads the send's progress, which the send's thread writes: + * the Pacer's class-file counts and wait, and each FrameWriter's frame count. + */ + static final class ExitNotice implements Runnable { + static final String NAME = "keploy-build-table-exit"; + private final long t0; + private final Pacer pacer; + private final FrameWriter table; + private final FrameWriter probes; + private final PrintStream out; + private volatile boolean ended; + private Thread hook; + + ExitNotice(long t0, Pacer pacer, FrameWriter table, FrameWriter probes, PrintStream out) { + this.t0 = t0; + this.pacer = pacer; + this.table = table; + this.probes = probes; + this.out = out; + } + + /** add makes the notice and adds its hook; a JVM that takes no hook (exiting, a security manager) has none. */ + static ExitNotice add(long t0, Pacer pacer, FrameWriter table, FrameWriter probes) { + ExitNotice n = new ExitNotice(t0, pacer, table, probes, System.err); + Thread h = new Thread(n, NAME); + try { + Runtime.getRuntime().addShutdownHook(h); + n.hook = h; + } catch (Throwable t) { + log(Level.FINE, "Keploy coverage: no exit notice for the build table's send", t); + } + return n; + } + + /** remove ends the notice: the send is over, so its hook says nothing even when it runs (an exit now). */ + void remove() { + ended = true; + if (hook == null) { + return; + } + try { + Runtime.getRuntime().removeShutdownHook(hook); + } catch (Throwable t) { + // The JVM exits already (IllegalStateException): the hook runs and, as ended is set, says nothing. + } + } + + @Override + public void run() { + if (ended) { + return; + } + try { + out.println(line()); + } catch (Throwable ignored) { + // Nowhere left to say it. + } + } + + /** line is what the notice says, with the send's progress read now. */ + String line() { + long total = pacer.totalClassFiles(); + long done = pacer.doneClassFiles(); + String where; + if (total == 0) { + where = "it was still reading the app's class files"; + } else if (done < total) { + where = done + " of " + total + " class files were analyzed"; + } else { + where = "all " + total + " class files were analyzed"; + } + return "Keploy coverage: the JVM exits before its build table is sent whole: " + where + "; " + + (table.seq + probes.seq) + " frames sent in " + (System.nanoTime() - t0) / 1_000_000 + + " ms (" + pacer.waitedNanos() / 1_000_000 + " ms of it paced). Keploy gets this build's table" + + " only from a JVM that runs until it is sent"; + } + } + private static long threadCpuNanos() { try { return java.lang.management.ManagementFactory.getThreadMXBean().getCurrentThreadCpuTime(); @@ -605,13 +706,18 @@ interface Clock { * total is 0 and the rate is never read). start is the send's, for the deadline. */ private long begun; - private long total; - private long done; + // total, done and waited are volatile for ExitNotice, which reads them on its own thread; only the + // send's thread writes them. + private volatile long total; + private volatile long done; private long last; private long gc; + /** The collections' time when the send began, for the line that says the send is paced. */ + private long gc0; + private boolean told; private long debt; private long bank; - private long waited; + private volatile long waited; Pacer(Clock clock) { this.clock = clock; @@ -620,6 +726,7 @@ interface Clock { start = last; begun = last; gc = clock.gcMillis(); + gc0 = gc; } catch (RuntimeException e) { stop("its clock failed", e); } @@ -671,11 +778,23 @@ long waitedNanos() { return waited; } + /** totalClassFiles is the class files the send analyzes, from begin; 0 before it. */ + long totalClassFiles() { + return total; + } + + /** doneClassFiles is the class files analyzed so far, paced or not. */ + long doneClassFiles() { + return done; + } + /** * unitDone is called after each unit, whose classFiles were analyzed: it waits what is owed and may be. * It throws on an interrupt, which it keeps for the thread: an interrupt ends the send. */ void unitDone(int classFiles) throws InterruptedIOException { + int files = Math.max(0, classFiles); + done += files; if (Thread.currentThread().isInterrupted()) { throw interrupted(); } @@ -691,14 +810,23 @@ void unitDone(int classFiles) throws InterruptedIOException { gc = g; } debt = Math.max(0, debt - Math.max(0, now - last)); - int files = Math.max(0, classFiles); bank = Math.min(MAX_BANK_NANOS, bank + ALLOWANCE_NANOS_PER_CLASS * files); - done += files; last = now; long wait = Math.min(Math.min(debt, bank), room(now)); if (wait < MIN_WAIT_NANOS) { return; } + if (!told) { + // Deferred: said once a send, at its first wait, so a table that comes late names why. + told = true; + log(Level.INFO, "Keploy coverage: the build table's send is paced, so its garbage does not grow" + + " the app's heap: the JVM's collections took " + (g - gc0) + " ms in the " + + Math.max(0, now - start) / 1_000_000 + " ms since it began, and it waits " + WAIT_PER_GC + + " times that; its waits add up to at most " + ALLOWANCE_NANOS_PER_CLASS / 1000 + + " us a class file (" + ALLOWANCE_NANOS_PER_CLASS * total / 1_000_000 + " ms for its " + + total + "), and none ends the table past " + deadlineNanos(total) / 1_000_000 + + " ms from its start", null); + } try { clock.sleep(wait); } catch (InterruptedException e) { @@ -2050,7 +2178,8 @@ static final class FrameWriter implements RecordSink { private byte[] frame; private int bodyLen; private final MessageDigest digest; - int seq; + /** The frames sent; volatile for ExitNotice, which reads it on its own thread (only the send's writes it). */ + volatile int seq; private String build; FrameWriter(FrameSender sender) { diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableNoticeTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableNoticeTest.java new file mode 100644 index 00000000..8952138e --- /dev/null +++ b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableNoticeTest.java @@ -0,0 +1,208 @@ +package io.keploy.dedup; + +import org.junit.Test; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.logging.Level; +import java.util.logging.LogRecord; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; + +/** + * The lines the SDK logs when it defers the build table's send or the JVM cuts it short: a native keploy test + * stops the app at each test set's end, and a table that comes late, or never, must say why in the app's log. + */ +public class BuildTableNoticeTest { + + private static final String EXIT_LINE = "Keploy coverage: the JVM exits before its build table is sent whole: "; + private static final String DEFERRED = "Keploy asked for the build table while this JVM sends one"; + private static final String PACED = "Keploy coverage: the build table's send is paced"; + + @Test(timeout = 30_000) + public void anAskWhileASendRunsSaysItIsDeferredOnceEach() throws Exception { + CountDownLatch inFirst = new CountDownLatch(1); + CountDownLatch release = new CountDownLatch(1); + CountDownLatch ended = new CountDownLatch(2); + List ran = java.util.Collections.synchronizedList(new ArrayList<>()); + BuildTable.Sends sends = new BuildTable.Sends(BuildTable::thread); + try (BuildTableTest.Logs logs = new BuildTableTest.Logs()) { + assertTrue(sends.request(() -> { + inFirst.countDown(); + try { + release.await(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + ran.add("first"); + ended.countDown(); + })); + assertTrue(inFirst.await(10, TimeUnit.SECONDS)); + assertEquals(0, lines(logs, DEFERRED).size()); + assertTrue(sends.request(() -> ran.add("second"))); + List one = lines(logs, DEFERRED); + assertEquals(one.toString(), 1, one.size()); + assertTrue(one.get(0), !one.get(0).contains("dropped")); + assertTrue(sends.request(() -> { + ran.add("third"); + ended.countDown(); + })); + List two = lines(logs, DEFERRED); + assertEquals(two.toString(), 2, two.size()); + assertTrue(two.get(1), two.get(1).contains("(an earlier ask that waited for it is dropped")); + assertEquals("[INFO, INFO]", levels(logs, DEFERRED)); + release.countDown(); + assertTrue(ended.await(10, TimeUnit.SECONDS)); + // The first send and the newest ask ran; the ask it replaced did not, as its line said. + assertEquals("[first, third]", ran.toString()); + } + } + + @Test(timeout = 30_000) + public void aPacedSendSaysSoOnceAtItsFirstWait() throws Exception { + BuildTablePacerTest.Clock c = new BuildTablePacerTest.Clock(); + try (BuildTableTest.Logs logs = new BuildTableTest.Logs()) { + BuildTable.Pacer p = new BuildTable.Pacer(c); + p.begin(1_000); + for (int i = 0; i < 1_000; i++) { + c.now += 1_000_000L; + c.gc += 1; // 1 ms of collections a unit: over the 0.5% goal, so it waits after many units + c.units++; + p.unitDone(1); + } + assertTrue("the row's pacer waited " + c.sleeps.size() + " times", c.sleeps.size() > 1); + List got = lines(logs, PACED); + assertEquals(got.toString(), 1, got.size()); + assertTrue(got.get(0), got.get(0).contains("for its 1000), and none ends the table past 50000 ms")); + } + } + + @Test(timeout = 30_000) + public void aSendThatIsNotPacedSaysNothingOfIt() throws Exception { + BuildTablePacerTest.Clock c = new BuildTablePacerTest.Clock(); + try (BuildTableTest.Logs logs = new BuildTableTest.Logs()) { + BuildTable.Pacer p = new BuildTable.Pacer(c); + p.begin(1_000); + for (int i = 0; i < 1_000; i++) { + c.now += 1_000_000L; + c.units++; + p.unitDone(1); + } + assertEquals(0, c.sleeps.size()); + assertEquals(0, lines(logs, PACED).size()); + } + } + + @Test(timeout = 120_000) + public void aJvmThatExitsMidSendSaysSoOnce() throws Exception { + String out = runChild("mid"); + assertEquals(out, 1, count(out, EXIT_LINE)); + assertTrue(out, out.contains(" class files were analyzed; 0 frames sent in ")); + assertTrue(out, out.contains("CHILD exiting")); + } + + @Test(timeout = 120_000) + public void aSendThatEndsBeforeTheJvmExitsSaysNothing() throws Exception { + String out = runChild("after"); + assertEquals(out, 0, count(out, EXIT_LINE)); + assertTrue(out, out.contains("CHILD sent")); + } + + /** runChild runs ExitMain in a JVM of its own (an exit ends it), and returns its stdout and stderr. */ + private static String runChild(String mode) throws Exception { + String javaBin = new java.io.File(System.getProperty("java.home"), "bin/java").getPath(); + Process process = new ProcessBuilder(javaBin, "-Xmx512m", "-cp", System.getProperty("java.class.path"), + ExitMain.class.getName(), mode).redirectErrorStream(true).start(); + String out = new String(readAll(process.getInputStream()), StandardCharsets.UTF_8); + assertTrue(out, process.waitFor(60, TimeUnit.SECONDS)); + assertEquals(out, 0, process.exitValue()); + return out; + } + + /** + * ExitMain sends a table of generated classes on the SDK's own thread. "mid": its first frame never returns, + * and the JVM exits mid-send. "after": the send ends, then the JVM exits. + */ + public static final class ExitMain { + public static void main(String[] args) throws Exception { + boolean mid = "mid".equals(args[0]); + List app = new ArrayList<>(); + for (int i = 0; i < 300; i++) { + String n = BuildTableBudgetTest.name(i); + app.add(new BuildTable.ClassFile(n, BuildTableBudgetTest.gen(n, i, BuildTableBudgetTest.BRANCHES))); + } + CountDownLatch inFrame = new CountDownLatch(1); + CountDownLatch sent = new CountDownLatch(1); + BuildTable.thread(() -> { + BuildTable.send(() -> app, frame -> { + inFrame.countDown(); + if (mid) { + try { + new CountDownLatch(1).await(); // the frame never returns: the exit cuts the send + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + } + }); + sent.countDown(); + }).start(); + if (mid) { + if (!inFrame.await(30, TimeUnit.SECONDS)) { + throw new IllegalStateException("no frame"); + } + System.out.println("CHILD exiting"); + } else { + if (!sent.await(30, TimeUnit.SECONDS)) { + throw new IllegalStateException("not sent"); + } + System.out.println("CHILD sent"); + } + System.out.flush(); + System.exit(0); + } + } + + private static List lines(BuildTableTest.Logs logs, String part) { + List out = new ArrayList<>(); + for (LogRecord r : logs.records) { + if (r.getMessage() != null && r.getMessage().contains(part)) { + out.add(r.getMessage()); + } + } + return out; + } + + private static String levels(BuildTableTest.Logs logs, String part) { + List out = new ArrayList<>(); + for (LogRecord r : logs.records) { + if (r.getMessage() != null && r.getMessage().contains(part)) { + out.add(r.getLevel()); + } + } + return out.toString(); + } + + private static int count(String s, String part) { + int n = 0; + for (int i = s.indexOf(part); i >= 0; i = s.indexOf(part, i + 1)) { + n++; + } + return n; + } + + private static byte[] readAll(InputStream in) throws IOException { + ByteArrayOutputStream out = new ByteArrayOutputStream(); + byte[] b = new byte[8192]; + for (int n; (n = in.read(b)) > 0; ) { + out.write(b, 0, n); + } + return out.toByteArray(); + } +} From 158915e5d00a796e1e8ed6495b23dadc54356d74 Mon Sep 17 00:00:00 2001 From: slayerjain Date: Fri, 20 Nov 2026 16:14:01 +0530 Subject: [PATCH 12/14] fix(dedup): bound the table's pacing by the send's own work time The build table's send was paced by owing 200 times all the collection time since it began, the app's own collections included, and its waits were bounded by 0.8 ms a class file (40 s at 50,000) and a 50 s deadline. A busy app collects past that 0.5% goal for its own sake (2.7% of the time while it allocates 400 MB/s), so every send in a busy app waited out its whole allowance. A native keploy test restarts the app at each test set, and each new JVM starts the send over: a 50,000-class app under test never got its table to Keploy within a set, so the whole run had no coverage. - The waits are drawn from the send's own work time, not from its class files: each unit's wall time since the one before (its socket writes included, its waits left out; the first unit's includes the index's read) is what the thread may wait, at most a second of it saved up. All the waits add up to at most the send's own work (plus one sleep's overrun), so a paced table takes at most about twice as long as unpaced, however much the app collects. No one wait is over a second, as before, so frames keep coming. - No wait may push the table's end past 20 s at 50,000 class files (0.4 ms a class file past that), down from 50 s. - The paced line says what is owed when the first wait is taken ("so it owes N ms now (each ms of collection adds 200, and each ms the send works or waits takes one off, never below 0)") and the bound. - The exit notice says what it knows when the exit starts ("the JVM is exiting while its build table is being sent. So far: ..."): the send's daemon thread may still finish while the other hooks run. Its hook thread is made inside the try, so a notice that fails never ends the send, and a JVM already exiting when a send starts prints the line at once. - When the thread for a request that waited out an interrupted send fails to start, the WARNING says that request is not sent. Rows: - A busy app's table: two threads allocate 400 MB/s, keeping a ring of arrays, while the table is sent. Its waits stay within the send's own work time (plus 0.5 s for one sleep's overrun), the table is whole, within 60 s, with no frame more than 10 s after the one before. 5,000 classes by default, 50,000 with -Dkeploy.budget.classes. It fails on the code before this change: at 50,000 classes on JDK 21 with -Xmx4g and G1 the send took 43,498 ms, 39,991 of them paced, for 3,507 ms of work; at 5,000 classes, 3,990 ms paced for 655 ms of work. - The Pacer rows are recast for the work bound: the waits add up to at most the work, and over any run of units to at most a second plus those units' time; a unit may wait its own time, not its class files; the deadline is 20 s; the Pacer counts its class files paced or not. - A send removes its exit hook, after a whole send and after one whose socket fails (it fails if the hook is left registered). The hook the row reads is kept per thread, so the app keeps no send's buffers past the thread that sent it. The paced line names the owed time, after an index read longer than what the first unit owes, so the floor is hit and nothing carries over. The rows read only their own thread's log lines. Measured at 50,000 generated classes, one run each, on JDK 21, 17 and 8, G1 and Parallel, in 1 and 4 GiB pods (-XX:MaxRAM, no -Xmx): - A quiet app: the table took 5.0-8.1 s (2.5-4.0 s of it paced; thread CPU 2.4-3.7 s; the longest gap between frames 0.74-1.21 s). Committed heap growth over the heap after a full collection, the row's bound in brackets: JDK 21 G1 20 and 59 MiB (128), Parallel 54 and 23 (128); JDK 17 G1 11 (128) and 145 (192), Parallel 33 and 32 (128); JDK 8 G1 79 (128) and 300 (448), Parallel 47 and 32 (128), in 1 and 4 GiB pods. - A busy app (the same cells, plus -Xmx4g G1 on each JDK): the table took 4.9-8.5 s, 2.4-4.2 s of it paced, each send's waits within its own 2.5-4.3 s of work. The committed heap grew 0-280 MiB in the pods and 0-816 MiB at -Xmx4g over the send's start; that is the app's own allocation as much as the send's, and it is not asserted. Signed-off-by: slayerjain --- .../main/java/io/keploy/dedup/BuildTable.java | 118 ++++++++----- .../io/keploy/dedup/BuildTableBudgetTest.java | 154 ++++++++++++++++- .../io/keploy/dedup/BuildTableNoticeTest.java | 50 +++++- .../io/keploy/dedup/BuildTablePacerTest.java | 163 ++++++++++-------- .../java/io/keploy/dedup/BuildTableTest.java | 13 ++ 5 files changed, 374 insertions(+), 124 deletions(-) diff --git a/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java b/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java index 190d2009..ecca41c7 100644 --- a/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java +++ b/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java @@ -441,11 +441,15 @@ boolean request(Runnable send) { } // Called on the control path: a thread that cannot start (a pids limit, a security manager) must not // end the control server. Keploy reads IGNORED as no table from this SDK. - return start(send); + return start(send, false); } - /** start runs send and the requests after it on a new thread; false, with a WARNING, if none starts. */ - private boolean start(Runnable send) { + /** + * start runs send and the requests after it on a new thread; false, with a WARNING, if none starts. + * handover is true when send is an ask that waited for the send before it (its thread was interrupted): + * the WARNING then says that ask is not sent, as it was ACKed and logged as sent later. + */ + private boolean start(Runnable send, boolean handover) { try { threads.newThread(() -> loop(send)).start(); return true; @@ -454,7 +458,9 @@ private boolean start(Runnable send) { running = false; pending = null; } - log(Level.WARNING, "Keploy coverage: the build table's thread did not start", t); + log(Level.WARNING, "Keploy coverage: the build table's thread did not start" + (handover + ? ": the ask for the build table that waited for the send before it is not sent (Keploy" + + " asks again on its next run)" : ""), t); return false; } } @@ -481,7 +487,7 @@ private void loop(Runnable first) { // The interrupt ended this thread's send and is kept for it: the next one runs on a new // thread, uninterrupted. Sends stays running meanwhile, so a request made now waits for it. // start's factory and log calls run on this interrupted thread: keep them non-interruptible. - start(send); + start(send, true); return; } } @@ -560,16 +566,24 @@ static void send(Supplier> classes, FrameSender sender, File spi } /** - * ExitNotice says, once, on System.err, that the JVM exits while a send runs: a native keploy test stops the - * app at each test set's end, and a send cut short there would otherwise leave no line in the app's log. + * ExitNotice says, once, on System.err, that the JVM is exiting while a send runs: a native keploy test stops + * the app at each test set's end, and a send cut short there would otherwise leave no line in the app's log. * It is a shutdown hook added for the send and removed when it ends. It writes to System.err, not * java.util.logging: the JVM runs its shutdown hooks at once and in no order, and LogManager's own hook (and * an app's logging framework's) may have closed the handlers already. A JVM that is killed (SIGKILL) or * halted runs no hook and says nothing. The hook reads the send's progress, which the send's thread writes: - * the Pacer's class-file counts and wait, and each FrameWriter's frame count. + * the Pacer's class-file counts and wait, and each FrameWriter's frame count. What it says is that progress + * at the exit's start, not the send's outcome: the send's daemon thread runs on while the other hooks run, and + * may still send the rest. */ static final class ExitNotice implements Runnable { static final String NAME = "keploy-build-table-exit"; + /** + * The hook the last add on this thread made, for the rows that check a send removes it. Per thread, so a + * row reads its own send's hook, not another class's, and the app keeps none past its table's thread: + * the value (and the send it holds) goes when that thread ends, with its last send. + */ + static final ThreadLocal LAST_HOOK = new ThreadLocal<>(); private final long t0; private final Pacer pacer; private final FrameWriter table; @@ -586,13 +600,20 @@ static final class ExitNotice implements Runnable { this.out = out; } - /** add makes the notice and adds its hook; a JVM that takes no hook (exiting, a security manager) has none. */ + /** + * add makes the notice and adds its hook. A JVM that takes no hook (a security manager) has none; one that + * is exiting already has none either, and the notice says its line at once, as the exit may cut the send. + * A notice that fails never ends the send. + */ static ExitNotice add(long t0, Pacer pacer, FrameWriter table, FrameWriter probes) { ExitNotice n = new ExitNotice(t0, pacer, table, probes, System.err); - Thread h = new Thread(n, NAME); try { + Thread h = new Thread(n, NAME); Runtime.getRuntime().addShutdownHook(h); n.hook = h; + LAST_HOOK.set(h); + } catch (IllegalStateException exiting) { + n.run(); } catch (Throwable t) { log(Level.FINE, "Keploy coverage: no exit notice for the build table's send", t); } @@ -624,22 +645,23 @@ public void run() { } } - /** line is what the notice says, with the send's progress read now. */ + /** line is what the notice says: the send's progress, read now. */ String line() { long total = pacer.totalClassFiles(); long done = pacer.doneClassFiles(); String where; if (total == 0) { - where = "it was still reading the app's class files"; + where = "reading the app's class files"; } else if (done < total) { - where = done + " of " + total + " class files were analyzed"; + where = done + " of " + total + " class files analyzed"; } else { - where = "all " + total + " class files were analyzed"; + where = "all " + total + " class files analyzed"; } - return "Keploy coverage: the JVM exits before its build table is sent whole: " + where + "; " + return "Keploy coverage: the JVM is exiting while its build table is being sent. So far: " + where + "; " + (table.seq + probes.seq) + " frames sent in " + (System.nanoTime() - t0) / 1_000_000 - + " ms (" + pacer.waitedNanos() / 1_000_000 + " ms of it paced). Keploy gets this build's table" - + " only from a JVM that runs until it is sent"; + + " ms (" + pacer.waitedNanos() / 1_000_000 + " ms of it paced). A table the exit cuts short" + + " does not reach Keploy: Keploy gets this build's table only from a JVM that runs until it is" + + " sent"; } } @@ -656,36 +678,37 @@ private static long threadCpuNanos() { * the app's heap. The analysis allocates fast (about 80-90 KiB a class, over 1 GB/s at full speed, nearly * all of it short-lived), and G1 and Parallel commit more heap when collecting takes more than their goal * of the time (Parallel 1%; G1 a few percent, as low as 1% while it has committed a small part of its cap). - * So after each unit the thread owes {@link #WAIT_PER_GC} times the time the collections took since the - * send began (the app's and its own) less the time since, and waits it out: collections then take at most - * 1/200 of the send's time, and a JVM that does not collect is not paced at all. The time read is every - * collector bean's: on a collector whose beans count concurrent cycles (ZGC, Shenandoah) the send is paced - * at its full allowance. + * So the thread owes {@link #WAIT_PER_GC} ms for each ms the collections take while it sends (the app's and + * its own), each ms it works or waits pays one off (never below 0), and after each unit it waits what it owes + * as far as its bounds let it. The time read is every collector bean's: a busy app's own collections (or + * ZGC's and Shenandoah's concurrent cycles, which their beans count) make the thread owe more than its bounds + * let it wait, and so do the send's own collections on every JVM measured (G1 and Parallel, JDK 8 to 21, + * quiet or busy): in practice the send waits about as long as it works. * - *

The waits are bounded. They are drawn from an allowance of {@link #ALLOWANCE_NANOS_PER_CLASS} per - * class file analyzed, of which at most {@link #MAX_BANK_NANOS} is saved up: no one wait is over a second, - * the waits over any k class files add up to at most a second plus 0.8 k ms, so frames keep coming (Keploy's - * wait for the table is a bound on the time since its last frame), and all of them to at most 0.8 ms a - * class file (40 s at 50,000). And no wait is taken that would end the table past {@link #deadlineNanos} - * (50 s at 50,000 class files), reckoned as the time since the send began (the index's read included) plus - * the rest of its class files at the rate the analysis went so far (from {@link #begin}, its waits left - * out): on a slow or busy CPU the pacing gives way to the table's 60 s bound, with 10 s left for the probe - * map's frames and the estimate's error. A clock or collector bean that fails ends the pacing, and the rest - * is sent unpaced. An interrupt ends the send: unitDone throws, and the interrupt is kept for the thread. + *

The waits are bounded by the send's own work: each unit's wall time (the time since the unit before, + * its frames' socket writes included and its waits left out; the first unit's includes the index's read) is + * added to what the thread may wait, and at most {@link #MAX_BANK_NANOS} of it is saved up. So all the + * waits add up to at most the send's own work time plus one sleep's overrun (a table paced takes at most + * about twice as long as unpaced, whatever the app's collections), no one wait is over a second, and the + * waits over any run of units add up to at most a second plus those units' own time, so frames keep coming + * (Keploy's wait for the table is a bound on the time since its last frame). + * And no wait is taken that would end the table past {@link #deadlineNanos} (20 s at 50,000 class files), + * reckoned as the time since the send began (the index's read included) plus the rest of its class files at + * the rate the analysis went so far (from {@link #begin}, its waits left out): on a slow or busy CPU the + * pacing gives way first. A clock or collector bean that fails ends the pacing, and the rest is sent + * unpaced. An interrupt ends the send: unitDone throws, and the interrupt is kept for the thread. */ static final class Pacer { /** Each millisecond of collection asks for this many of the send's: at most 0.5% of it collecting. */ static final long WAIT_PER_GC = 200; - /** The wait each analyzed class file adds to what the thread may wait: the table's time bound. */ - static final long ALLOWANCE_NANOS_PER_CLASS = 800_000L; - /** The most of the allowance kept for a later wait: the longest wait after a quiet stretch. */ + /** The most of the send's own work time kept for a later wait: the longest wait after a quiet stretch. */ static final long MAX_BANK_NANOS = 1_000_000_000L; /** Waits shorter than this are put off to the next unit, so the thread does not sleep per class. */ static final long MIN_WAIT_NANOS = 10_000_000L; /** The pacing's time bound up to {@link #DEADLINE_CLASS_FILES} class files: see deadlineNanos. */ - static final long DEADLINE_NANOS = 50_000_000_000L; + static final long DEADLINE_NANOS = 20_000_000_000L; static final long DEADLINE_CLASS_FILES = 50_000; - // The most the thread may owe (an hour), so the sum cannot overflow; the allowance bounds it anyway. + // The most the thread may owe (an hour), so the sum cannot overflow; the work-time bound limits it anyway. private static final long MAX_DEBT_NANOS = 3_600_000_000_000L; /** The clock, the collections' time and the sleep a Pacer reads; the JVM's, or a row's. */ @@ -752,7 +775,7 @@ static Pacer jvm() { } /** - * deadlineNanos is the most the pacing lets a table of classFiles take: 50 s, and 1 ms a class file past + * deadlineNanos is the most the pacing lets a table of classFiles take: 20 s, and 0.4 ms a class file past * 50,000 (the table's bound, 60 s at 50,000, scales the same way). */ static long deadlineNanos(long classFiles) { @@ -761,7 +784,7 @@ static long deadlineNanos(long classFiles) { /** * begin is called as the analysis begins, after the index's read: it says how many class files the send - * analyzes, for its deadline (not called, the deadline is 50 s), and reads the clock the rate is taken + * analyzes, for its deadline (not called, the deadline is 20 s), and reads the clock the rate is taken * from, so the index's read does not count as the analysis's work. */ void begin(long classFiles) { @@ -809,8 +832,10 @@ void unitDone(int classFiles) throws InterruptedIOException { debt = Math.min(MAX_DEBT_NANOS, debt + Math.min(g - gc, MAX_DEBT_NANOS / perMs) * perMs); gc = g; } - debt = Math.max(0, debt - Math.max(0, now - last)); - bank = Math.min(MAX_BANK_NANOS, bank + ALLOWANCE_NANOS_PER_CLASS * files); + // The send's own time since the last unit pays off what it owes, and is what it may wait. + long worked = Math.max(0, now - last); + debt = Math.max(0, debt - worked); + bank = Math.min(MAX_BANK_NANOS, bank + worked); last = now; long wait = Math.min(Math.min(debt, bank), room(now)); if (wait < MIN_WAIT_NANOS) { @@ -821,11 +846,12 @@ void unitDone(int classFiles) throws InterruptedIOException { told = true; log(Level.INFO, "Keploy coverage: the build table's send is paced, so its garbage does not grow" + " the app's heap: the JVM's collections took " + (g - gc0) + " ms in the " - + Math.max(0, now - start) / 1_000_000 + " ms since it began, and it waits " + WAIT_PER_GC - + " times that; its waits add up to at most " + ALLOWANCE_NANOS_PER_CLASS / 1000 - + " us a class file (" + ALLOWANCE_NANOS_PER_CLASS * total / 1_000_000 + " ms for its " - + total + "), and none ends the table past " + deadlineNanos(total) / 1_000_000 - + " ms from its start", null); + + Math.max(0, now - start) / 1_000_000 + " ms since it began, so it owes " + + debt / 1_000_000 + " ms now (each ms of collection adds " + WAIT_PER_GC + ", and each" + + " ms the send works or waits takes one off, never below 0); its" + + " waits add up to at most its own work time (plus one sleep's overrun), so the table" + + " takes about twice as long as unpaced at most, and none ends the table past " + + deadlineNanos(total) / 1_000_000 + " ms from its start", null); } try { clock.sleep(wait); diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableBudgetTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableBudgetTest.java index 819313b5..f0ed5a77 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableBudgetTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableBudgetTest.java @@ -23,8 +23,12 @@ import java.util.Locale; import java.util.Map; import java.util.Set; +import java.util.concurrent.atomic.AtomicBoolean; import java.util.concurrent.atomic.AtomicInteger; import java.util.concurrent.atomic.AtomicLong; +import java.util.logging.LogRecord; +import java.util.regex.Matcher; +import java.util.regex.Pattern; import javax.management.NotificationEmitter; import javax.management.NotificationListener; import javax.management.openmbean.CompositeData; @@ -95,8 +99,9 @@ public void aGeneratedAppsTableIsMeasured() throws Exception { * maxCommittedGrowth is the most the send may add to an app pod's committed heap over the heap just after a * full collection: 128 MiB, but more on G1 with a heap cap of 512 MiB or more before JDK 21. There the * collection has shrunk G1's heap, its pause raises G1's pause-time ratio, and G1 grows the heap back while - * the table is sent, paced or not: at 50,000 classes in a 4 GiB pod, 420-421 MiB on JDK 8 (unpaced 422-707) and - * 13-166 MiB on JDK 17. Unpaced, JDK 21's G1 there grew 175-343 MiB (paced, 19-20), past the bound. + * the table is sent, paced or not: at 50,000 classes in a 4 GiB pod, 300 MiB on JDK 8 (unpaced 422-707) and + * 145 MiB on JDK 17, paced by the send's own work time (one run each; earlier pacers saw 420-421 and 13-166). + * Unpaced, JDK 21's G1 there grew 175-343 MiB (paced, 59), past the bound. */ static long maxCommittedGrowth(boolean g1, long maxHeap, String javaSpec) { if (g1 && maxHeap >= 512L << 20) { @@ -115,8 +120,8 @@ static long maxCommittedGrowth(boolean g1, long maxHeap, String javaSpec) { /** * maxWallNanos is the table's time bound: 60 s at 50,000 classes, at least 60 s for fewer, and 1.2 ms a class - * past 50,000. At 50,000 that is 10 s past the pacing's own deadline (BuildTable.Pacer.deadlineNanos), which - * leaves the probe map's frames that time; past 50,000 it leaves more. + * past 50,000. At 50,000 that is 40 s past the pacing's own deadline (BuildTable.Pacer.deadlineNanos), which + * leaves a slow or busy CPU's own work that time; past 50,000 it leaves more. */ static long maxWallNanos(int classes) { return Math.max(60_000_000_000L, 60_000_000_000L / 50_000 * classes); @@ -125,6 +130,147 @@ static long maxWallNanos(int classes) { /** The most a generated class's table and probe map may allocate on the send's thread: below two analyses'. */ static final long ALLOCATED_PER_CLASS = 112 << 10; + /** The busy app's allocation, in MB/s over its threads: a JVM serving a test run's requests. */ + static final int BUSY_MBPS = 400; + static final int BUSY_THREADS = 2; + static final int BUSY_ARRAY = 64 << 10; + /** How long the app's load runs before Keploy asks for the table. */ + static final long BUSY_SETTLE_MS = 2_000; + /** What the waits may run past the send's own work: one sleep's overrun (a collection's pause in it). */ + static final long BUSY_OVERRUN_MS = 500; + /** The send's INFO line: its classes, those not measured, its time, its CPU and its waits. */ + static final Pattern SENT = Pattern.compile("sent \\((\\d+) classes, (\\d+) not measured.*; \\d+ frames," + + " (\\d+) ms, cpu -?\\d+ ms, paced (\\d+) ms\\)$"); + + /** + * A busy app's table, the native keploy test's case: app threads allocate BUSY_MBPS of short-lived garbage + * while the table is sent, so the JVM collects often for the app's own sake, and the pacing (which reads + * every collection, the app's too) is owed far more than it may wait. Its waits must stay within the send's + * own work time (one sleep's overrun past it at most), so a busy app's table takes at most about twice as + * long as unpaced: before, they took the whole allowance of 0.8 ms a class (40 s at 50,000), and the app's + * test set ended before its table was sent. The class count is -Dkeploy.budget.classes (5,000 by default). + */ + @Test(timeout = 900_000) + public void aBusyAppsTableIsPacedAtMostItsOwnWork() throws Exception { + int n = Integer.getInteger("keploy.budget.classes", 5000); + List app = new ArrayList<>(n); + for (int i = 0; i < n; i++) { + app.add(new BuildTable.ClassFile(name(i), gen(name(i), i, BRANCHES))); + } + AtomicBoolean stop = new AtomicBoolean(); + AtomicLong appBytes = new AtomicLong(); + // Each thread keeps a ring of its newest arrays, a churning retained set as request handling has: 64 MiB + // a thread, and at most 1/16 of the heap's cap in all. + int slots = (int) Math.min(1024, Runtime.getRuntime().maxMemory() / 16 / BUSY_THREADS / BUSY_ARRAY); + // The load's threads and the sampler: each ends at stop, and finally stops and joins them all. + List threads = new ArrayList<>(); + try (BuildTableTest.Logs logs = new BuildTableTest.Logs()) { + for (int t = 0; t < BUSY_THREADS; t++) { + Thread th = new Thread(() -> allocate(stop, appBytes, slots), "busy-app-" + t); + th.setDaemon(true); + th.start(); + threads.add(th); + } + Thread.sleep(BUSY_SETTLE_MS); + long committed0 = heapCommitted(); + AtomicLong committedMax = new AtomicLong(committed0); + AtomicBoolean sending = new AtomicBoolean(true); + Thread sampler = new Thread(() -> { + while (sending.get() && !stop.get()) { + committedMax.accumulateAndGet(heapCommitted(), Math::max); + try { + Thread.sleep(50); + } catch (InterruptedException e) { + return; + } + } + }, "busy-committed"); + sampler.setDaemon(true); + sampler.start(); + threads.add(sampler); + long gc0 = gcMillis(); + long app0 = appBytes.get(); + long t0 = System.nanoTime(); + long[] frames = {t0, 0}; // the last frame's time, the longest gap + BuildTable.send(() -> app, new BuildTable.FrameSender() { + @Override + public void send(byte[] frame) { + send(frame, 0, frame.length); + } + + @Override + public void send(byte[] frame, int off, int len) { + long now = System.nanoTime(); + frames[1] = Math.max(frames[1], now - frames[0]); + frames[0] = now; + } + }); + long wall = System.nanoTime() - t0; + long gc = gcMillis() - gc0; + sending.set(false); + sampler.join(); + String sent = null; + for (LogRecord r : logs.mine()) { + if (r.getMessage() != null && r.getMessage().contains(" sent (")) { + sent = r.getMessage(); + } + } + assertTrue("the send's INFO line", sent != null); + Matcher m = SENT.matcher(sent); + assertTrue(sent, m.find()); + long sendMs = Long.parseLong(m.group(3)); + long pacedMs = Long.parseLong(m.group(4)); + long workMs = sendMs - pacedMs; + System.out.printf(Locale.ROOT, "BUSY app-%d java=%s gc=%s heap_max_mb=%d app_mbps=%d wall_ms=%d" + + " send_ms=%d paced_ms=%d work_ms=%d gc_ms=%d gc_pct=%.2f max_frame_gap_ms=%d" + + " committed_start_mb=%d committed_max_mb=%d committed_growth_mb=%d app_mb=%d%n", + n, System.getProperty("java.version"), gcNames(), Runtime.getRuntime().maxMemory() >> 20, + BUSY_MBPS, wall / 1_000_000, sendMs, pacedMs, workMs, gc, + 100.0 * gc / Math.max(1, wall / 1_000_000), frames[1] / 1_000_000, committed0 >> 20, + committedMax.get() >> 20, (committedMax.get() - committed0) >> 20, + (appBytes.get() - app0) >> 20); + assertEquals(sent, n, Integer.parseInt(m.group(1))); + assertEquals(sent, 0, Integer.parseInt(m.group(2))); + assertTrue("paced " + pacedMs + " ms, past the send's own " + workMs + " ms of work", + pacedMs <= workMs + BUSY_OVERRUN_MS); + assertTrue("took " + wall / 1_000_000 + " ms", wall < maxWallNanos(n)); + assertTrue("a frame came " + frames[1] / 1_000_000 + " ms after the one before", + frames[1] < MAX_FRAME_GAP_NANOS); + } finally { + stop.set(true); + for (Thread th : threads) { + th.join(); + } + } + } + + /** allocate is a busy app's thread: BUSY_MBPS / BUSY_THREADS of BUSY_ARRAY arrays, a ring of slots kept. */ + private static void allocate(AtomicBoolean stop, AtomicLong appBytes, int slots) { + long perMs = (long) BUSY_MBPS * 1_000_000 / 1000 / BUSY_THREADS; + byte[][] ring = new byte[Math.max(1, slots)][]; + long t0 = System.nanoTime(); + long made = 0; + int k = 0; + while (!stop.get()) { + for (int j = 0; j < 16; j++) { + byte[] b = new byte[BUSY_ARRAY]; + b[0] = (byte) k; + ring[k] = b; + k = k + 1 == ring.length ? 0 : k + 1; + } + made += 16L * BUSY_ARRAY; + appBytes.addAndGet(16L * BUSY_ARRAY); + long ahead = made / perMs - (System.nanoTime() - t0) / 1_000_000; + if (ahead > 0) { + try { + Thread.sleep(ahead); + } catch (InterruptedException e) { + return; + } + } + } + } + @Test(timeout = 600_000) public void aClassWithOneLongMethodIsMeasured() throws Exception { int branches = Integer.getInteger("keploy.budget.branches", 500); diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableNoticeTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableNoticeTest.java index 8952138e..35927b71 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableNoticeTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableNoticeTest.java @@ -14,6 +14,7 @@ import java.util.logging.LogRecord; import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertTrue; /** @@ -22,7 +23,8 @@ */ public class BuildTableNoticeTest { - private static final String EXIT_LINE = "Keploy coverage: the JVM exits before its build table is sent whole: "; + private static final String EXIT_LINE = "Keploy coverage: the JVM is exiting while its build table is being sent." + + " So far: "; private static final String DEFERRED = "Keploy asked for the build table while this JVM sends one"; private static final String PACED = "Keploy coverage: the build table's send is paced"; @@ -70,17 +72,25 @@ public void aPacedSendSaysSoOnceAtItsFirstWait() throws Exception { BuildTablePacerTest.Clock c = new BuildTablePacerTest.Clock(); try (BuildTableTest.Logs logs = new BuildTableTest.Logs()) { BuildTable.Pacer p = new BuildTable.Pacer(c); + c.now += 400_000_000L; // the index's read: 400 ms, longer than the 1st unit owes, so the floor is hit p.begin(1_000); for (int i = 0; i < 1_000; i++) { c.now += 1_000_000L; - c.gc += 1; // 1 ms of collections a unit: over the 0.5% goal, so it waits after many units + c.gc += 1; // 1 ms of collections a unit: over the 0.5% goal, so it waits c.units++; p.unitDone(1); } assertTrue("the row's pacer waited " + c.sleeps.size() + " times", c.sleeps.size() > 1); List got = lines(logs, PACED); assertEquals(got.toString(), 1, got.size()); - assertTrue(got.get(0), got.get(0).contains("for its 1000), and none ends the table past 50000 ms")); + // The 1st unit owes 200 ms, less its 401 ms: 0, never below. The 2nd owes 200 ms less its 1 ms: 199 ms, + // its first wait (200 times the 2 ms of collections less the 402 ms since the start is below 0). + assertTrue(got.get(0), got.get(0).contains("the JVM's collections took 2 ms in the 402 ms since it began," + + " so it owes 199 ms now (each ms of collection adds 200, and each ms the send works or waits" + + " takes one off, never below 0)")); + assertTrue(got.get(0), got.get(0).contains("its waits add up to at most its own work time (plus one" + + " sleep's overrun), so the table takes about twice as long as unpaced at most, and none ends the" + + " table past 20000 ms from its start")); } } @@ -104,7 +114,7 @@ public void aSendThatIsNotPacedSaysNothingOfIt() throws Exception { public void aJvmThatExitsMidSendSaysSoOnce() throws Exception { String out = runChild("mid"); assertEquals(out, 1, count(out, EXIT_LINE)); - assertTrue(out, out.contains(" class files were analyzed; 0 frames sent in ")); + assertTrue(out, out.contains(" class files analyzed; 0 frames sent in ")); assertTrue(out, out.contains("CHILD exiting")); } @@ -115,6 +125,33 @@ public void aSendThatEndsBeforeTheJvmExitsSaysNothing() throws Exception { assertTrue(out, out.contains("CHILD sent")); } + @Test(timeout = 60_000) + public void aSendRemovesItsExitHookWhenItEnds() throws Exception { + List app = new ArrayList<>(); + for (int i = 0; i < 20; i++) { + String n = BuildTableBudgetTest.name(i); + app.add(new BuildTable.ClassFile(n, BuildTableBudgetTest.gen(n, i, 3))); + } + // A send that ends whole, and one whose socket fails at its first frame. + BuildTable.FrameSender whole = frame -> { }; + BuildTable.FrameSender failing = frame -> { + throw new IOException("row: the data socket is closed"); + }; + try { + for (BuildTable.FrameSender sender : new BuildTable.FrameSender[] {whole, failing}) { + BuildTable.ExitNotice.LAST_HOOK.remove(); + BuildTable.send(() -> app, sender); // on this thread, so LAST_HOOK is this send's + Thread hook = BuildTable.ExitNotice.LAST_HOOK.get(); + assertTrue("the send added an exit hook", hook != null); + // Removing it again finds none: the send removed it (else this removes it, for the rows after). + assertFalse("the send left its exit hook registered", + Runtime.getRuntime().removeShutdownHook(hook)); + } + } finally { + BuildTable.ExitNotice.LAST_HOOK.remove(); + } + } + /** runChild runs ExitMain in a JVM of its own (an exit ends it), and returns its stdout and stderr. */ private static String runChild(String mode) throws Exception { String javaBin = new java.io.File(System.getProperty("java.home"), "bin/java").getPath(); @@ -169,9 +206,10 @@ public static void main(String[] args) throws Exception { } } + /** lines are the messages with part that this row's thread logged (a send of another class may log too). */ private static List lines(BuildTableTest.Logs logs, String part) { List out = new ArrayList<>(); - for (LogRecord r : logs.records) { + for (LogRecord r : logs.mine()) { if (r.getMessage() != null && r.getMessage().contains(part)) { out.add(r.getMessage()); } @@ -181,7 +219,7 @@ private static List lines(BuildTableTest.Logs logs, String part) { private static String levels(BuildTableTest.Logs logs, String part) { List out = new ArrayList<>(); - for (LogRecord r : logs.records) { + for (LogRecord r : logs.mine()) { if (r.getMessage() != null && r.getMessage().contains(part)) { out.add(r.getLevel()); } diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTablePacerTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTablePacerTest.java index 587438b9..5a52dfae 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTablePacerTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTablePacerTest.java @@ -12,16 +12,15 @@ /** * The table's pacing (BuildTable.Pacer) on a clock the rows drive: it waits out 200 times the collections' - * time, only after a collection, never more than its allowance of 0.8 ms a class file, saved up to a second, - * and never past the table's deadline. + * time, only after a collection, never more in all than the send's own work time, saved up to a second, and + * never past the table's deadline. */ public class BuildTablePacerTest { private static final long MS = 1_000_000L; - private static final long A = BuildTable.Pacer.ALLOWANCE_NANOS_PER_CLASS; private static final long MIN = BuildTable.Pacer.MIN_WAIT_NANOS; - // The class files whose allowance first reaches the smallest wait, after a wait used it all. - private static final long K = (MIN + A - 1) / A; + // The units of 1 ms work whose time first reaches the smallest wait, after a wait used it all. + private static final long K = MIN / MS; /** A clock that moves only as a row says and by what the pacer sleeps (plus overrun a sleep; or not). */ static final class Clock implements BuildTable.Pacer.Clock { @@ -100,18 +99,19 @@ public void aJvmThatDoesNotCollectIsNotPaced() throws Exception { public void aCollectionIsWaitedOutTo200TimesItsTime() throws Exception { Clock c = new Clock(); BuildTable.Pacer p = new BuildTable.Pacer(c); - run(p, c, 1_000, 50_000, 0); + run(p, c, 1_000, MS, 0); assertEquals("no wait before a collection", 0, c.sleeps.size()); - // A 4 ms collection during the next unit: 800 ms of the send's time, less that unit's 50 us. - c.now += 50_000; + // A 4 ms collection during the next unit: 800 ms of the send's time, less that unit's 1 ms (a second of + // the send's own work came before, so its bound lets it all be waited). + c.now += MS; c.gc += 4; c.units++; p.unitDone(1); assertEquals(1, c.sleeps.size()); - assertEquals(800 * MS - 50_000, (long) c.sleeps.get(0)); - run(p, c, 1_000, 50_000, 0); + assertEquals(799 * MS, (long) c.sleeps.get(0)); + run(p, c, 1_000, MS, 0); assertEquals("paid off: no wait after it", 1, c.sleeps.size()); - assertEquals(800 * MS - 50_000, p.waitedNanos()); + assertEquals(799 * MS, p.waitedNanos()); } @Test @@ -135,19 +135,20 @@ public void workSinceACollectionCountsTowardItsWait() throws Exception { } @Test - public void theWaitsAreBoundedByTheAllowanceOfEachClassFile() throws Exception { + public void theWaitsAreBoundedByTheSendsOwnWork() throws Exception { Clock c = new Clock(); BuildTable.Pacer p = new BuildTable.Pacer(c); long start = c.now; - // A JVM that collects 10 ms in every unit owes 2 s a unit: only the allowance bounds the waits. - run(p, c, 50_000, 20_000, 10); + // A JVM that collects 10 ms in every unit (a busy app's own collections) owes 2 s a unit: only the send's + // own work, 0.1 ms a unit (5 s in all), bounds the waits. + long w = MS / 10; + run(p, c, 50_000, w, 10); long slept = c.slept(); - assertTrue("slept " + slept, slept <= 50_000 * A); - assertTrue("slept " + slept, slept > 50_000 * A - MIN); + assertTrue("slept " + slept, slept <= 50_000 * w); + assertTrue("slept " + slept, slept > 50_000 * w - MIN); assertEquals(slept, p.waitedNanos()); - assertTrue("the table's time", c.now - start <= 50_000 * A + 50_000 * 20_000L); - assertEquals("40 s at 50,000 classes", 40_000 * MS, 50_000 * A); - // Over any run of units, the waits add up to at most a second plus the allowance of its class files. + assertTrue("the table takes at most twice its work", c.now - start <= 2 * 50_000 * w); + // Over any run of units, the waits add up to at most a second plus those units' own time. long[] sum = new long[c.units + 1]; for (int i = 0; i < c.sleeps.size(); i++) { sum[c.sleptAt.get(i)] += c.sleeps.get(i); @@ -156,7 +157,7 @@ public void theWaitsAreBoundedByTheAllowanceOfEachClassFile() throws Exception { long prefix = 0; for (int j = 1; j <= c.units; j++) { prefix += sum[j]; - long at = prefix - j * A; + long at = prefix - j * w; assertTrue("waits over a run of units, up to unit " + j, at - low <= BuildTable.Pacer.MAX_BANK_NANOS); low = Math.min(low, at); } @@ -166,45 +167,52 @@ public void theWaitsAreBoundedByTheAllowanceOfEachClassFile() throws Exception { public void aQuietStretchSavesUpAtMostASecond() throws Exception { Clock c = new Clock(); BuildTable.Pacer p = new BuildTable.Pacer(c); - run(p, c, 5_000, 20_000, 0); - // 100 ms of collection owes 20 s; 4 s of allowance went by, but a second of it was kept. + run(p, c, 5_000, MS, 0); + // 100 ms of collection owes 20 s; 5 s of the send's work went by, but a second of it was kept. c.now += 20_000; c.gc += 100; c.units++; p.unitDone(1); assertEquals(1, c.sleeps.size()); assertEquals(BuildTable.Pacer.MAX_BANK_NANOS, (long) c.sleeps.get(0)); - // Then a wait each time the allowance reaches the smallest wait: K class files' worth every K of them. - run(p, c, 100, 20_000, 0); + // Then a wait each time the send's work since reaches the smallest wait: K units' time every K of them. + run(p, c, 100, MS, 0); assertEquals(1 + 100 / K, c.sleeps.size()); for (int i = 1; i < c.sleeps.size(); i++) { - assertEquals(K * A, (long) c.sleeps.get(i)); + assertEquals(K * MS, (long) c.sleeps.get(i)); } } @Test - public void aUnitOfManyClassFilesEarnsItsAllowance() throws Exception { + public void aUnitMayWaitItsOwnTimeNotItsClassFiles() throws Exception { Clock c = new Clock(); BuildTable.Pacer p = new BuildTable.Pacer(c); + // A unit of 300 class files that took 20 us: a second of collection owes 200 s, and it may wait 20 us. c.now += 20_000; c.gc += 1_000; p.unitDone(300); + assertEquals(0, c.sleeps.size()); + // One class file that took 300 ms: it may wait that, and the 20 us before it. + c.now += 300 * MS; + c.gc += 1_000; + p.unitDone(1); assertEquals(1, c.sleeps.size()); - assertEquals(300 * A, (long) c.sleeps.get(0)); + assertEquals(300 * MS + 20_000, (long) c.sleeps.get(0)); } @Test - public void aSleepsOverrunComesOffTheAllowance() throws Exception { + public void aSleepsOverrunComesOffWhatItMayWait() throws Exception { Clock c = new Clock(); c.overrun = 5 * MS; BuildTable.Pacer p = new BuildTable.Pacer(c); long start = c.now; - run(p, c, 10_000, 20_000, 10); - long waited = c.now - start - 10_000 * 20_000L; + long w = MS / 10; + run(p, c, 10_000, w, 10); + long waited = c.now - start - 10_000 * w; assertEquals(waited, p.waitedNanos()); - // The time asleep, overruns included, stays within the allowance (and one overrun past it). - assertTrue("waited " + waited, waited <= 10_000 * A + c.overrun); - assertTrue("waited " + waited, waited > 10_000 * A - MIN - c.overrun); + // The time asleep, overruns included, stays within the send's own work (and one overrun past it). + assertTrue("waited " + waited, waited <= 10_000 * w + c.overrun); + assertTrue("waited " + waited, waited > 10_000 * w - MIN - c.overrun); } @Test @@ -215,7 +223,7 @@ public void anInterruptEndsTheSendAndIsKept() throws Exception { BuildTable.Pacer p = new BuildTable.Pacer(c); c.interrupt = true; try { - run(p, c, 100, 20_000, 10); + run(p, c, 100, MS, 10); fail("an interrupt during a wait ends the send"); } catch (InterruptedIOException e) { assertEquals("the build table's thread was interrupted", e.getMessage()); @@ -242,43 +250,47 @@ public void anInterruptEndsTheSendAndIsKept() throws Exception { @Test public void theWaitsGiveWayToTheDeadline() throws Exception { - // 50,000 class files of 0.5 ms work each (25 s) in a JVM that collects 10 ms in each: the allowance - // alone would wait 40 s, a table of 65 s. The deadline holds it to 50 s, waiting all the room left. + // 50,000 class files of 0.3 ms work each (15 s) in a JVM that collects 10 ms in each: the work bound + // alone would wait 15 s, a table of 30 s. The deadline holds it to 20 s, waiting all the room left. + long w = 3 * MS / 10; Clock c = new Clock(); BuildTable.Pacer p = new BuildTable.Pacer(c); p.begin(50_000); long start = c.now; - run(p, c, 50_000, MS / 2, 10); + run(p, c, 50_000, w, 10); long took = c.now - start; assertTrue("took " + took, took <= BuildTable.Pacer.deadlineNanos(50_000)); - assertTrue("waited " + p.waitedNanos(), p.waitedNanos() > 25_000 * MS - 2 * MIN); - // Not told its class files, a Pacer still waits no later than 50 s into the table. + assertTrue("waited " + p.waitedNanos(), p.waitedNanos() > 5_000 * MS - 2 * MIN); + // Not told its class files, a Pacer still waits no later than 20 s into the table. Clock d = new Clock(); BuildTable.Pacer q = new BuildTable.Pacer(d); start = d.now; - run(q, d, 50_000, MS / 2, 10); + run(q, d, 50_000, w, 10); assertTrue("woke at " + (d.lastWake - start), d.lastWake - start <= BuildTable.Pacer.deadlineNanos(0)); - assertTrue("waited " + q.waitedNanos(), q.waitedNanos() > 25_000 * MS); - assertEquals(50_000 * MS, BuildTable.Pacer.deadlineNanos(50_000)); - assertEquals("1 ms a class file past 50,000", 100_000 * MS, BuildTable.Pacer.deadlineNanos(100_000)); - assertEquals(50_000 * MS, BuildTable.Pacer.deadlineNanos(10)); + assertTrue("waited " + q.waitedNanos(), q.waitedNanos() > 5_000 * MS); + assertEquals(20_000 * MS, BuildTable.Pacer.deadlineNanos(50_000)); + assertEquals("0.4 ms a class file past 50,000", 40_000 * MS, BuildTable.Pacer.deadlineNanos(100_000)); + assertEquals(20_000 * MS, BuildTable.Pacer.deadlineNanos(10)); } @Test public void theRateLeavesOutTheTimeBeforeTheAnalysis() throws Exception { - // The index's read takes 10 s before the analysis begins; then 50,000 class files of 0.5 ms work each - // (25 s), 10 ms of collection in each. The rate is the analysis's own (0.5 ms a class file), so the first - // wait comes as soon as the allowance reaches the smallest wait; the deadline still counts the 10 s, so - // the waits fill the 15 s left and no more. + // The index's read takes 4 s before the analysis begins; then 50,000 class files of 0.2 ms work each + // (10 s), 10 ms of collection in each. The rate is the analysis's own (0.2 ms a class file), so the waits + // begin at once (the read is the send's own work too); the deadline still counts the 4 s, so the waits + // fill the 6 s left and no more. Clock c = new Clock(); BuildTable.Pacer p = new BuildTable.Pacer(c); long start = c.now; - c.now += 10_000 * MS; + c.now += 4_000 * MS; p.begin(50_000); - run(p, c, 50_000, MS / 2, 10); - assertEquals("the first wait as soon as the allowance allows", K, (long) c.sleptAt.get(0)); + run(p, c, 50_000, MS / 5, 10); + // The read's 4 s paid off the first unit's 2 s owed, and saved a second up: the first wait is at the + // second unit, a second long. Had the rate counted the read, the deadline would have left no room. + assertEquals("the first wait at the second unit", 2, (long) c.sleptAt.get(0)); + assertEquals(BuildTable.Pacer.MAX_BANK_NANOS, (long) c.sleeps.get(0)); assertTrue("took " + (c.now - start), c.now - start <= BuildTable.Pacer.deadlineNanos(50_000)); - assertTrue("waited " + p.waitedNanos(), p.waitedNanos() > 15_000 * MS - 2 * MIN); + assertTrue("waited " + p.waitedNanos(), p.waitedNanos() > 6_000 * MS - 2 * MIN); } @Test @@ -297,21 +309,21 @@ public void aClockThatDoesNotMoveAcrossASleepIsChargedTheWait() throws Exception BuildTable.Pacer p = new BuildTable.Pacer(c); run(p, c, 10_000, 20_000, 10); long slept = c.slept(); - assertTrue("slept " + slept, slept <= 10_000 * A); - assertTrue("slept " + slept, slept > 10_000 * A - MIN); + assertTrue("slept " + slept, slept <= 10_000 * 20_000L); + assertTrue("slept " + slept, slept > 10_000 * 20_000L - MIN); } @Test public void aCollectorBeanThatFailsEndsThePacing() throws Exception { Clock c = new Clock(); BuildTable.Pacer p = new BuildTable.Pacer(c); - run(p, c, 100, 20_000, 10); + run(p, c, 100, MS, 10); int before = c.sleeps.size(); assertTrue(before > 0); c.failGc = new UnsupportedOperationException("row"); - run(p, c, 1, 20_000, 10); + run(p, c, 1, MS, 10); c.failGc = null; - run(p, c, 1_000, 20_000, 10); + run(p, c, 1_000, MS, 10); assertEquals(before, c.sleeps.size()); } @@ -324,6 +336,18 @@ public void aPacerOffNeverReadsOrWaits() throws Exception { assertEquals(0, p.waitedNanos()); } + @Test + public void aPacerCountsItsClassFilesPacedOrNot() throws Exception { + for (BuildTable.Pacer p : new BuildTable.Pacer[] {BuildTable.Pacer.off(), new BuildTable.Pacer(new Clock())}) { + assertEquals(0, p.totalClassFiles()); + p.begin(5); + p.unitDone(2); + p.unitDone(3); + assertEquals(5, p.doneClassFiles()); + assertEquals(5, p.totalClassFiles()); + } + } + @Test public void theJvmsPacerReadsItsCollectors() throws Exception { BuildTable.Pacer.JvmClock jvm = new BuildTable.Pacer.JvmClock( @@ -336,7 +360,7 @@ public void theJvmsPacerReadsItsCollectors() throws Exception { } @Test - public void writePacesAfterEachUnitByItsClassFiles() throws Exception { + public void writePacesAfterEachUnitByItsTime() throws Exception { List app = new ArrayList<>(); for (int i = 0; i < 100; i++) { app.add(new BuildTable.ClassFile(BuildTableBudgetTest.name(i), @@ -352,7 +376,7 @@ public long nanoTime() { @Override public long gcMillis() { - // Each read finds a second more of collection: the allowance alone bounds the waits. + // Each read finds a second more of collection: the send's own work alone bounds the waits. c.gc += 1_000; return c.gcMillis(); } @@ -363,23 +387,26 @@ public void sleep(long nanos) throws InterruptedException { c.sleep(nanos); } }; - BuildTable.Summary s = BuildTable.write(app, (build, json) -> records[0]++, BuildTable.DROP, - BuildTable.MAX_RECORD_BYTES, BuildTable.MAX_MAP_BYTES, BuildTable.MAX_MAP_STEPS, + // Each unit's record takes 1 ms of the send's work. + BuildTable.Summary s = BuildTable.write(app, (build, json) -> { + records[0]++; + c.now += MS; + }, BuildTable.DROP, BuildTable.MAX_RECORD_BYTES, BuildTable.MAX_MAP_BYTES, BuildTable.MAX_MAP_STEPS, new BuildTable.Pacer(collecting)); assertEquals(100, s.records); assertEquals("one read at the start, one after each unit", 101, c.gcReads); - // The allowance of K class files at a time: after every Kth unit's record. + // The work of K units at a time: after every Kth unit's record. assertEquals(100 / K, c.sleeps.size()); for (int i = 0; i < c.sleeps.size(); i++) { - assertEquals(K * A, (long) c.sleeps.get(i)); + assertEquals(K * MS, (long) c.sleeps.get(i)); assertEquals((i + 1) * K, (int) c.sleptAt.get(i)); } } @Test public void writeTellsThePacerItsClassFilesForTheDeadline() throws Exception { - // 100 classes of 0.9 s work each: 90 s, past the 50 s deadline from the first, so never a wait. A Pacer - // not told how many class files come would wait through its first 50 s. + // 100 classes of 0.9 s work each: 90 s, past the 20 s deadline from the first, so never a wait. A Pacer + // not told how many class files come would wait through its first 20 s. List app = new ArrayList<>(); for (int i = 0; i < 100; i++) { app.add(new BuildTable.ClassFile(BuildTableBudgetTest.name(i), @@ -394,8 +421,7 @@ public long nanoTime() { @Override public long gcMillis() { - // Read once a unit: a unit's work, and a second more of collection. - c.now += 900 * MS; + // Read once a unit: a second more of collection. c.gc += 1_000; return c.gcMillis(); } @@ -405,7 +431,8 @@ public void sleep(long nanos) throws InterruptedException { c.sleep(nanos); } }; - BuildTable.Summary s = BuildTable.write(app, (build, json) -> { }, BuildTable.DROP, + // Each unit's record takes 0.9 s of the send's work. + BuildTable.Summary s = BuildTable.write(app, (build, json) -> c.now += 900 * MS, BuildTable.DROP, BuildTable.MAX_RECORD_BYTES, BuildTable.MAX_MAP_BYTES, BuildTable.MAX_MAP_STEPS, new BuildTable.Pacer(slow)); assertEquals(100, s.records); diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java index b02b204a..06025f9e 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java @@ -794,6 +794,19 @@ public void publish(LogRecord record) { records.add(record); } + /** mine is the records the calling thread logged: a send of another row's class may log meanwhile. */ + @SuppressWarnings("deprecation") // getLongThreadID is JDK 16+; this SDK builds for Java 8 + List mine() { + int me = (int) Thread.currentThread().getId(); + List out = new ArrayList<>(); + for (LogRecord r : records) { + if (r.getThreadID() == me) { + out.add(r); + } + } + return out; + } + @Override public void flush() { } From 1d88997fb6b007bd6a2a30fee20acc21ffbadb1b Mon Sep 17 00:00:00 2001 From: slayerjain Date: Fri, 20 Nov 2026 17:00:28 +0530 Subject: [PATCH 13/14] feat(dedup): send a class too complex to map as unit_too_complex A class whose probe map passes its bound (8 MiB of records, or 64 Mi walk steps) has no probe map, so no test's leaves can be named in it. The table sent it as not_measured record_too_large, the code for a table record past 8 MiB, so Keploy could not tell a record too large from a class too complex to map. It is now not_measured with the contract's code unit_too_complex. It has no methods, as any not_measured record, so Keploy leaves its lines out of the totals and never counts them as missed; the rest of the build is measured as usual. - record_too_large is now only a table record past 8 MiB with its newline. The send's INFO line counts the two apart: "N not measured as too complex to map (unit_too_complex: a probe map over 8 MiB or 64 Mi steps; ...)". - The table format stays 1: no released SDK writes a table, so the build-id row's pinned hash of the capped record is updated alone, as that row says to do before the first release. - A RuntimeException in Keploy's own steps of the class's one analysis (setting a method's instructions from JaCoCo's graph, and walking each probe; both under one wrap) is now NotMapped, with its cause. Before, an index error there was logged as "JaCoCo could not analyze" the class and was not counted with the probe maps that failed, though the class is the same analyzer_failed either way. JaCoCo's own steps (its instruction graph, filters and coverage calculator, with the hooks Keploy runs inside them) still throw as they do, and what the caller's callbacks (wants, and each probe) throw passes as it is, so a map past its bound stays unit_too_complex. A row that a throwing log handler fails no send counted every record published on the build table's logger, and expected two. A collection during its first send adds the paced line (INFO), so it failed once with three. It now counts the two sends' outcome lines, from its own thread. Rows: a probe map past its bytes or steps is unit_too_complex with no probe record and not record_too_large; the send's line counts it apart; a record past its bound is record_too_large and not unit_too_complex; an index error in the probe walk is NotMapped naming the class and method, while what the callbacks throw passes as it is. Each failed on the previous code except two: the record-too-large row is a guard (the previous code had no unit_too_complex to count), and the throwing-handler row is the fix of a row that failed by timing. In the NotMapped row only the walk's part failed there; its callback parts guard against wrapping them, and since the walk and the setting of instructions share one wrap, removing that wrap fails the row. The early-returns budget row now counts unit_too_complex. Signed-off-by: slayerjain --- .../main/java/io/keploy/dedup/BuildTable.java | 80 +++++++++++------- .../analysis/KeployBranchCoverage.java | 79 +++++++++++++----- .../io/keploy/dedup/BuildTableBudgetTest.java | 18 ++-- .../java/io/keploy/dedup/BuildTableTest.java | 82 +++++++++++++++++-- .../java/io/keploy/dedup/ProbeMapTest.java | 54 ++++++++++++ 5 files changed, 246 insertions(+), 67 deletions(-) diff --git a/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java b/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java index ecca41c7..1a00e903 100644 --- a/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java +++ b/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java @@ -108,7 +108,7 @@ static void log(Level level, String message, Throwable thrown) { * The table's format, F of the build id (IndexReport.buildId): the contract's section 3.3 records as this * class writes them and the probe map of leaf ranges, with every bound that decides what a class's records * are ({@link #MAX_RECORD_BYTES}, {@link #MAX_MAP_BYTES}, {@link #MAX_MAP_STEPS}, how - * KeployBranchCoverage.Hits counts its steps, and the reason a unit past one is not measured). Raise it in + * KeployBranchCoverage.Hits counts its steps, and the reason a unit past each is not measured). Raise it in * any release where the table or probe-map records written for the same class bytes can change, so they * name another build. The KTB1 frames that carry the records are not in it: their header's "v" names them. */ @@ -117,12 +117,18 @@ static void log(Level level, String message, Throwable thrown) { static final int MAX_FRAME_BYTES = 1 << 20; /** A record (with its newline) larger than a whole inflated chunk can never be sent: record_too_large. */ static final int MAX_RECORD_BYTES = 8 << 20; - /** The reason a unit's record is sent without its methods when it, or its probe map, is past a bound. */ + /** The reason a unit's record is sent without its methods when the record is past MAX_RECORD_BYTES. */ static final String RECORD_TOO_LARGE = "record_too_large"; + /** + * The reason a unit's record is sent without its methods when its probe map is past MAX_MAP_BYTES or + * MAX_MAP_STEPS (the contract's section 2.4): no test's leaves can be named in it, so Keploy shows it as + * too complex to map, not as missed. + */ + static final String UNIT_TOO_COMPLEX = "unit_too_complex"; /** * The most a unit's probe map may take: its records' bytes, newlines included (a table record's bound), * and the steps its probes' walks take (KeployBranchCoverage.Hits#steps, about a second of CPU). A unit - * past either is sent not_measured record_too_large with no probe record, and its map is given up as + * past either is sent not_measured unit_too_complex with no probe record, and its map is given up as * soon as it is past one, so neither the heap nor the time it takes grows with it: a method of n early * returns maps to about n * n ranges (its probe k covers the k checks before it), 4.4 MB at 1,000 and * 43 MB at 3,000. The bound is on the text; what a unit's map holds live until its records are made is @@ -537,8 +543,13 @@ static void send(Supplier> classes, FrameSender sender, File spi } if (s.tooLarge > 0) { m.append(", ").append(s.tooLarge).append(" not measured as too large (").append(RECORD_TOO_LARGE) - .append(": a record over ").append(MAX_RECORD_BYTES >> 20).append(" MiB, or a probe map over ") - .append(MAX_MAP_BYTES >> 20).append(" MiB or ").append(MAX_MAP_STEPS >> 20) + .append(": a record over ").append(MAX_RECORD_BYTES >> 20) + .append(" MiB; the FINE log names each)"); + } + if (s.tooComplex > 0) { + m.append(", ").append(s.tooComplex).append(" not measured as too complex to map (") + .append(UNIT_TOO_COMPLEX).append(": a probe map over ").append(MAX_MAP_BYTES >> 20) + .append(" MiB or ").append(MAX_MAP_STEPS >> 20) .append(" Mi steps; the FINE log names each)"); } if (s.noLines > 0) { @@ -940,13 +951,15 @@ static final class Summary { final int unnamed; /** How many of the not measured records JaCoCo analyzed but whose probes could not be mapped. */ final int notMapped; - /** How many of the not measured records are record_too_large: the record or its probe map too large. */ + /** How many of the not measured records are record_too_large: the table record too large. */ final int tooLarge; + /** How many of the not measured records are unit_too_complex: the probe map past its bound. */ + final int tooComplex; /** How many probe-map records went to the probes sink: one per class name of each measured unit. */ final int probeRecords; Summary(String build, int records, int notMeasured, int noLines, int unnamed, int notMapped, int tooLarge, - int probeRecords) { + int tooComplex, int probeRecords) { this.build = build; this.records = records; this.notMeasured = notMeasured; @@ -954,6 +967,7 @@ static final class Summary { this.unnamed = unnamed; this.notMapped = notMapped; this.tooLarge = tooLarge; + this.tooComplex = tooComplex; this.probeRecords = probeRecords; } } @@ -1000,7 +1014,7 @@ static Summary write(List classes, RecordSink sink, RecordSink probes long maxSteps, Pacer pacer) throws IOException { Grouped g = group(classes); if (g.build.isEmpty()) { - return new Summary("", 0, 0, 0, g.unnamed, 0, 0, 0); + return new Summary("", 0, 0, 0, g.unnamed, 0, 0, 0, 0); } String build = g.build; List units = g.units; @@ -1008,6 +1022,7 @@ static Summary write(List classes, RecordSink sink, RecordSink probes int noLines = 0; int notMapped = 0; int tooLarge = 0; + int tooComplex = 0; int probeRecords = 0; long classFiles = 0; for (Unit u : units) { @@ -1031,6 +1046,9 @@ static Summary write(List classes, RecordSink sink, RecordSink probes if (u.tooLarge) { tooLarge++; } + if (u.tooComplex) { + tooComplex++; + } } else { noLines += u.noLines; for (byte[] p : u.probes) { @@ -1042,7 +1060,7 @@ static Summary write(List classes, RecordSink sink, RecordSink probes pacer.unitDone(u.files.size()); } return new Summary(build, units.size(), notMeasured, noLines, g.unnamed, notMapped, tooLarge, - probeRecords); + tooComplex, probeRecords); } /** What group makes of the index's classes: the build id ("" with none), the units in table order. */ @@ -1175,8 +1193,10 @@ static final class Unit { boolean failed; // Set when JaCoCo analyzed the unit but its probes could not be mapped (so failed is set too). boolean notMapped; - // Set when the unit's record or its probe map is past its bound: record_too_large (failed is set too). + // Set when the unit's record is past its bound: record_too_large (failed is set too). boolean tooLarge; + // Set when the unit's probe map is past its bound: unit_too_complex (failed is set too). + boolean tooComplex; // Class files with code but no line numbers: their methods have no leaf, so they are left out. int noLines; // The probe map's records for the unit's class files, in file order; set when record() succeeds. @@ -1258,9 +1278,9 @@ private List sortedFiles() { /** * record analyzes each of the unit's class files once, all of them first, in name order, then makes * the unit's records. The first analysis that fails decides: analyzer_failed when JaCoCo refuses the - * class file or its probes cannot be mapped, record_too_large when its probe map is past the budget. + * class file or its probes cannot be mapped, unit_too_complex when its probe map is past the budget. * Only then are the probe-map records made and each charged in place of its text: a record past the - * budget makes the unit record_too_large, and a second class file of one name that maps its probes + * budget makes the unit unit_too_complex, and a second class file of one name that maps its probes * otherwise makes it analyzer_failed. A unit with two failures of different kinds gets the first one's * reason in that order; either way it is not measured and has no probe record. */ @@ -1286,12 +1306,12 @@ byte[] record(MapBudget budget) { ranges.add(r); lastName = f.file.name; } - } catch (TooLarge e) { + } catch (TooComplex e) { log(Level.FINE, "Keploy coverage: the probe map of " + path() + " is past its bound: " + e.getMessage(), null); failed = true; - tooLarge = true; - return notMeasured(RECORD_TOO_LARGE); + tooComplex = true; + return notMeasured(UNIT_TOO_COMPLEX); } catch (KeployBranchCoverage.NotMapped e) { // A class whose probes cannot be mapped exactly is not measured: its hits could not be read. log(Level.FINE, "Keploy coverage: the probes of " + path() + " could not be mapped", e); @@ -1339,12 +1359,12 @@ byte[] record(MapBudget budget) { lastName = f.file.name; last = p; } - } catch (TooLarge e) { + } catch (TooComplex e) { log(Level.FINE, "Keploy coverage: the probe map of " + path() + " is past its bound: " + e.getMessage(), null); failed = true; - tooLarge = true; - return notMeasured(RECORD_TOO_LARGE); + tooComplex = true; + return notMeasured(UNIT_TOO_COMPLEX); } catch (Exception | LinkageError e) { log(Level.FINE, "Keploy coverage: the probes of " + path() + " could not be mapped", e); failed = true; @@ -1616,7 +1636,7 @@ void hits(KeployBranchCoverage.Method a, KeployBranchCoverage.Hits h, BitSet out * off that analysis (KeployBranchCoverage walks back from the probe over the edges JaCoCo's builder wired), * so the map costs the class's probes' walks and their ranges, with no object per probe. The * walks and the text are quadratic in a chain of early returns, so a unit's map has a budget - * ({@link MapBudget}): past it the unit is record_too_large. + * ({@link MapBudget}): past it the unit is unit_too_complex. */ static final class ProbeMap { private ProbeMap() { @@ -1685,7 +1705,7 @@ private static int indexOf(byte[] a, byte[] key) { * of its method's leaf indices, ascending, no two ranges touching; its record adds the method's base * once the unit's methods are ordered ({@link Ranges#json}). It throws NotMapped when the class's lines * or branches differ from the unit's methods (the same name under two roots), when a method with lines - * is in one and not in the other, or when a method's flow is not one KeployBranchCoverage maps; TooLarge + * is in one and not in the other, or when a method's flow is not one KeployBranchCoverage maps; TooComplex * as soon as the unit's map is past its budget's bytes or steps; and what JaCoCo throws for a class it * does not read. */ @@ -1774,11 +1794,11 @@ public void probe(KeployBranchCoverage.Method s, int r, KeployBranchCoverage.Hit } /** - * notMapped is e as a map that failed: TooLarge and NotMapped as they are, a check that failed as + * notMapped is e as a map that failed: TooComplex and NotMapped as they are, a check that failed as * NotMapped. */ private static RuntimeException notMapped(RuntimeException e) { - if (e instanceof TooLarge || e instanceof KeployBranchCoverage.NotMapped) { + if (e instanceof TooComplex || e instanceof KeployBranchCoverage.NotMapped) { return e; } return new KeployBranchCoverage.NotMapped(String.valueOf(e.getMessage()), e); @@ -1786,13 +1806,13 @@ private static RuntimeException notMapped(RuntimeException e) { } /** - * TooLarge ends a unit's probe map that is past its budget (MapBudget): the unit is sent not_measured - * record_too_large. It carries no stack trace: it is an answer, not a bug. + * TooComplex ends a unit's probe map that is past its budget (MapBudget): the unit is sent not_measured + * unit_too_complex. It carries no stack trace: it is an answer, not a bug. */ - static final class TooLarge extends RuntimeException { + static final class TooComplex extends RuntimeException { private static final long serialVersionUID = 1L; - TooLarge(String message) { + TooComplex(String message) { super(message, null, false, false); } } @@ -1800,7 +1820,7 @@ static final class TooLarge extends RuntimeException { /** * MapBudget is what one unit's probe map may still take: bytes (its records', newlines included; its * ranges' text counts as it is written, since a record holds it) and walk steps. Past either it throws - * TooLarge, so a map is given up as soon as it is past, not once it is made. + * TooComplex, so a map is given up as soon as it is past, not once it is made. */ static final class MapBudget { private final long maxBytes; @@ -1818,7 +1838,7 @@ static final class MapBudget { void step(int n) { steps += n; if (steps > maxSteps) { - throw new TooLarge("its probes' walks take more than " + maxSteps + " steps"); + throw new TooComplex("its probes' walks take more than " + maxSteps + " steps"); } } @@ -1829,7 +1849,7 @@ void step(int n) { void grew(int n) { text += n; if (bytes + text > maxBytes) { - throw new TooLarge("its probe map is more than " + maxBytes + " bytes"); + throw new TooComplex("its probe map is more than " + maxBytes + " bytes"); } } @@ -1843,7 +1863,7 @@ void take(int n, long counted) { text -= counted; bytes += n; if (bytes + text > maxBytes) { - throw new TooLarge("its probe records are more than " + maxBytes + " bytes"); + throw new TooComplex("its probe records are more than " + maxBytes + " bytes"); } } } diff --git a/keploy-sdk/src/main/java/org/jacoco/core/internal/analysis/KeployBranchCoverage.java b/keploy-sdk/src/main/java/org/jacoco/core/internal/analysis/KeployBranchCoverage.java index 67a39552..e466f6cb 100644 --- a/keploy-sdk/src/main/java/org/jacoco/core/internal/analysis/KeployBranchCoverage.java +++ b/keploy-sdk/src/main/java/org/jacoco/core/internal/analysis/KeployBranchCoverage.java @@ -95,9 +95,9 @@ public static int analyze(byte[] bytes, boolean[] probes, Map ou /** * NotMapped is a class file whose probes cannot be mapped exactly from this pass: a method's flow has a - * shape the walk does not map, its per-instruction coverage differs from JaCoCo's per-line sums, or a - * check of the caller's fails. The caller marks the class not measured; it is not JaCoCo failing to read - * the class. + * shape the walk does not map, its per-instruction coverage differs from JaCoCo's per-line sums, a check + * of the caller's fails, or Keploy's own steps of the pass fail (a bug of Keploy's, here as NotMapped with + * its cause). The caller marks the class not measured; it is not JaCoCo failing to read the class. */ public static final class NotMapped extends IllegalStateException { private static final long serialVersionUID = 1L; @@ -111,6 +111,17 @@ public NotMapped(String message, Throwable cause) { } } + /** Callers carries what a callback of EachProbe threw out of Keploy's steps, so it passes as it is. */ + private static final class Callers extends RuntimeException { + private static final long serialVersionUID = 1L; + final RuntimeException thrown; + + Callers(RuntimeException thrown) { + super(null, null, false, false); + this.thrown = thrown; + } + } + /** * Hits is what one probe covers in its method: the method's counted instructions that a class analysis * with only that probe executed marks covered, and which of their branches. It is valid during @@ -180,7 +191,9 @@ default boolean wants(Method method) { * probe on each of those edges), and quadratic in a chain of early returns, where each probe walks back * over every check before it. A method whose * flow does not have that shape (an instruction two edges without a probe reach, a replacement that reads - * another replaced instruction) throws. + * another replaced instruction) throws. What setting a method's counted instructions and the probe walk + * throw is NotMapped; JaCoCo's steps (its instruction graph, filters and coverage calculator, with the hooks + * Keploy's builder and filter output run inside them) and each's callbacks throw as they do. */ public static int analyze(byte[] bytes, Map out, EachProbe each) { return analyze(bytes, null, out, each); @@ -233,17 +246,48 @@ public void accept(MethodNode node, MethodVisitor visitor) { } private void collect(Method m, Builder builder, MethodNode node, String signature) { + // JaCoCo's steps, as ClassAnalyzer takes them: the filters, then the coverage calculator. Output output = new Output(); filter.filter(node, this, output); - if (!fill(m, builder, node, signature, output)) { + MethodCoverageCalculator calculator = new MethodCoverageCalculator(builder.getInstructions()); + output.applyTo(calculator); + MethodCoverageImpl coverage = new MethodCoverageImpl(m.name, m.desc, signature); + calculator.calculate(coverage); + if (!coverage.containsCode()) { return; } - out.put(m.name + m.desc, m); - if (each != null && each.wants(m)) { - map(m, builder, node, output); + // Keploy's steps, under one wrap: what fails in them is NotMapped, so a bug of Keploy's is not + // blamed on JaCoCo. What each's callbacks throw is carried out in a Callers and passes as it is. + try { + fill(m, builder, node, output, coverage); + out.put(m.name + m.desc, m); + if (each != null && wants(m)) { + map(m, builder, node, output); + } + } catch (Callers e) { + throw e.thrown; + } catch (RuntimeException e) { + throw keploys(m, e); } } + /** wants is each.wants(m), with what it throws carried out in a Callers. */ + private boolean wants(Method m) { + try { + return each.wants(m); + } catch (RuntimeException e) { + throw new Callers(e); + } + } + + /** keploys is e, thrown by Keploy's own steps for m, as NotMapped naming the class and method. */ + private NotMapped keploys(Method m, RuntimeException e) { + if (e instanceof NotMapped) { + return (NotMapped) e; + } + return new NotMapped(getClassName() + "#" + m.name + m.desc + ": " + e, e); + } + /** * map hands each of m's probes to each with what that probe alone covers, from this pass: the edges * builder wired (each instruction's predecessor) and the filters' output. @@ -389,7 +433,11 @@ private void map(Method m, Builder builder, MethodNode node, Output output) { } } hits.steps = w + fed; - each.probe(m, r, hits); + try { + each.probe(m, r, hits); + } catch (RuntimeException e) { + throw new Callers(e); + } hits.clear(); } } @@ -451,17 +499,11 @@ private NotMapped shape(Method m, String what) { /** * fill sets m's counted instructions from builder's graph and the filters' output, as ClassAnalyzer - * computes a method's coverage. It returns false for a method without code. + * computes a method's coverage, and checks them against coverage, the method's coverage JaCoCo + * computed from them (it has code). */ - private boolean fill(Method m, Builder builder, MethodNode node, String signature, Output output) { + private void fill(Method m, Builder builder, MethodNode node, Output output, MethodCoverageImpl coverage) { Map instructions = builder.getInstructions(); - MethodCoverageCalculator calculator = new MethodCoverageCalculator(instructions); - output.applyTo(calculator); - MethodCoverageImpl coverage = new MethodCoverageImpl(m.name, m.desc, signature); - calculator.calculate(coverage); - if (!coverage.containsCode()) { - return false; - } int[] p = new int[m.probeCount]; System.arraycopy(m.probes, 0, p, 0, m.probeCount); m.probes = p; @@ -532,7 +574,6 @@ private boolean fill(Method m, Builder builder, MethodNode node, String signatur m.covered = Arrays.copyOf(covered, c); m.branches = Arrays.copyOf(branches, c); m.coverage = coverage; - return true; } private static boolean same(ICounter c, int total, int covered) { diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableBudgetTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableBudgetTest.java index f0ed5a77..6ce23aee 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableBudgetTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableBudgetTest.java @@ -295,11 +295,11 @@ public void aMethodOf3000BranchesMapsInUnderASecond() throws Exception { } @Test(timeout = 600_000) - public void aChainOfEarlyReturnsIsMappedOrTooLargeAndNeverUnbounded() throws Exception { + public void aChainOfEarlyReturnsIsMappedOrTooComplexAndNeverUnbounded() throws Exception { // Probe k of a chain of n early returns walks back over the k checks before it and covers each one's // line and branch, so the map is about n * n ranges: 4.4 MB at 1,000 and 43 MB at 3,000; 4,500 ran a // 256 MiB heap out of memory (and 3,000 a 128 MiB one) before the map had a bound. Past it the class - // is record_too_large, given up as its map grows past the bound. + // is unit_too_complex, given up as its map grows past the bound. for (int n : new int[] {1000, 3000, 4500}) { String name = "gen/big/Returns" + n; List one = new ArrayList<>(); @@ -309,7 +309,7 @@ public void aChainOfEarlyReturnsIsMappedOrTooLargeAndNeverUnbounded() throws Exc assertEquals(1, b.tableRecords); assertEquals(name, mapped ? 1 : 0, b.probeRecords); assertEquals(name, mapped ? 0 : 1, b.notMeasured); - assertEquals(name, mapped ? 0 : 1, b.tooLarge); + assertEquals(name, mapped ? 0 : 1, b.tooComplex); assertTrue(name + ": the send took " + (b.cpuNanos / 1_000_000) + " ms of CPU", b.cpuNanos < 1_000_000_000L); assertTrue(name + ": the send allocated " + (b.allocated >> 20) + " MiB", @@ -459,8 +459,8 @@ static final class Budget { int tableRecords; int probeRecords; int notMeasured; - /** How many of the table's records are record_too_large. */ - int tooLarge; + /** How many of the table's records are unit_too_complex. */ + int tooComplex; /** Bytes the send allocated on its thread (the sink's own allocation left out). */ long allocated; /** The send thread's CPU time, in nanoseconds. */ @@ -569,12 +569,12 @@ private static Budget measure(String label, List classes, long[] bytes = new long[2]; boolean live = Boolean.getBoolean("keploy.budget.live"); long[] liveMax = {-1}; - // A record may run on into the next frame: not_measured and record_too_large are counted over the + // A record may run on into the next frame: not_measured and unit_too_complex are counted over the // table's stream, a match split over two frames included. The sink reads each frame where the writer // keeps it, as the socket's does: a copy or a String of a 1 MiB frame is a G1 humongous object, which // grows the heap this row measures. Matches notMeasured = new Matches("\"state\":\"not_measured\""); - Matches tooLarge = new Matches("\"reason\":{\"code\":\"record_too_large\"}"); + Matches tooComplex = new Matches("\"reason\":{\"code\":\"unit_too_complex\"}"); Budget b = new Budget(); long gc0 = gcMillis(); long process0 = processCpuNanos(); @@ -625,7 +625,7 @@ public void send(byte[] frame, int off, int len) { } if (!probes) { notMeasured.add(frame, nl + 1, off + len); - tooLarge.add(frame, nl + 1, off + len); + tooComplex.add(frame, nl + 1, off + len); } sinkAllocated[0] += allocation.getThreadAllocatedBytes(self) - sink0; // The sink's own time (a live run's collection) is not the send's gap. @@ -633,7 +633,7 @@ public void send(byte[] frame, int off, int len) { } }); b.notMeasured = notMeasured.n; - b.tooLarge = tooLarge.n; + b.tooComplex = tooComplex.n; long wall = System.nanoTime() - t0; long allocated = allocation.getThreadAllocatedBytes(self) - allocated0 - sinkAllocated[0]; sending.set(false); diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java index 06025f9e..c8294087 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java @@ -206,8 +206,9 @@ public void aTableThatChangesForTheSameClassesNamesAnotherBuild() throws Excepti assertTrue(why, mapped.records.get(0).contains("\"state\":\"ok\"")); Collect capped = new Collect(); BuildTable.write(one, capped, new Collect(), BuildTable.MAX_RECORD_BYTES, mapBytes, steps[0] - 1); - assertTrue(why, capped.records.get(0).contains("\"state\":\"not_measured\"")); - assertEquals(why, "3ee7977c74cf5939ab7af3957bfc1bbd4e495614d4dcb7bd066c707618f7ec1c", + assertTrue(why, capped.records.get(0) + .contains("\"state\":\"not_measured\",\"reason\":{\"code\":\"unit_too_complex\"}")); + assertEquals(why, "9b7d25b9091045ff4366ec799490ab35c3b1df10c5fcf8528eaaa66f1f746d26", sha256Hex(capped.ndjson.toByteArray())); Collect chunked = new Collect(); BuildTable.write(javaDedup(), chunked, 1600); @@ -360,6 +361,7 @@ public void aClassJacocoCannotAnalyzeIsNotMeasuredAndSoIsItsUnit() throws Except assertEquals(3, s.notMeasured); assertEquals("JaCoCo's refusals are not counted as maps that failed", 0, s.notMapped); assertEquals(0, s.tooLarge); + assertEquals(0, s.tooComplex); // The other classes are as they were. for (String id : golden.keySet()) { @@ -385,10 +387,12 @@ public void aRecordLargerThanAChunkInflatesToIsRecordTooLarge() throws Exception // DedupController and Product are the two records past 1600 bytes. assertEquals(2, tooLarge); assertEquals(2, s.notMeasured); + assertEquals(2, s.tooLarge); + assertEquals("a record too large is not a unit too complex", 0, s.tooComplex); } @Test - public void aProbeMapPastItsBytesOrStepsIsRecordTooLargeWithNoProbeRecord() throws Exception { + public void aProbeMapPastItsBytesOrStepsIsUnitTooComplexWithNoProbeRecord() throws Exception { String name = "gen/big/Returns"; byte[] bytes = BuildTableBudgetTest.returns(name, 200); List one = Collections.singletonList(new BuildTable.ClassFile(name, bytes)); @@ -414,7 +418,8 @@ public void aProbeMapPastItsBytesOrStepsIsRecordTooLargeWithNoProbeRecord() thro String why = "bytes " + bounds[i][0] + ", steps " + bounds[i][1]; assertEquals(why, 1, s.records); assertEquals(why, at ? 0 : 1, s.notMeasured); - assertEquals(why, at ? 0 : 1, s.tooLarge); + assertEquals(why, at ? 0 : 1, s.tooComplex); + assertEquals(why + ": a map past its bound is not a record too large", 0, s.tooLarge); assertEquals(why, 0, s.notMapped); assertEquals(why, at ? 1 : 0, s.probeRecords); assertEquals(why, at ? 1 : 0, map.records.size()); @@ -423,12 +428,33 @@ public void aProbeMapPastItsBytesOrStepsIsRecordTooLargeWithNoProbeRecord() thro assertTrue(rec, rec.contains("\"state\":\"ok\"")); assertEquals(probes.records.get(0), map.records.get(0)); } else { - assertTrue(rec, rec.contains("\"state\":\"not_measured\",\"reason\":{\"code\":\"record_too_large\"}") + assertTrue(rec, rec.contains("\"state\":\"not_measured\",\"reason\":{\"code\":\"unit_too_complex\"}") && rec.endsWith("\"methods\":[]}")); } } } + @Test + public void theSendsLineCountsUnitsTooComplexApartFromRecordsTooLarge() throws Exception { + // A chain of 3,000 early returns: its probe map (about 43 MB) is past the 8 MiB bound. + String name = "gen/big/Returns3000"; + List one = Collections.singletonList( + new BuildTable.ClassFile(name, BuildTableBudgetTest.returns(name, 3000))); + try (Logs logs = new Logs()) { + BuildTable.send(() -> one, frame -> { }); + String line = null; + for (LogRecord r : logs.mine()) { + if (r.getMessage().contains(" sent (")) { + line = r.getMessage(); + } + } + assertNotNull(line); + assertTrue(line, line.contains("(1 classes, 1 not measured, 1 not measured as too complex to map" + + " (unit_too_complex: a probe map over 8 MiB or 64 Mi steps; the FINE log names each);")); + assertFalse(line, line.contains("record_too_large")); + } + } + @Test public void theTableAnswerBoundsTheHeapByTheIndexCountHeldAndReadsNoClassFile() { long mb = 1 << 20; @@ -795,8 +821,13 @@ public void publish(LogRecord record) { } /** mine is the records the calling thread logged: a send of another row's class may log meanwhile. */ - @SuppressWarnings("deprecation") // getLongThreadID is JDK 16+; this SDK builds for Java 8 List mine() { + return mine(records); + } + + /** mine is those of records the calling thread logged. */ + @SuppressWarnings("deprecation") // getLongThreadID is JDK 16+; this SDK builds for Java 8 + static List mine(List records) { int me = (int) Thread.currentThread().getId(); List out = new ArrayList<>(); for (LogRecord r : records) { @@ -1204,7 +1235,7 @@ public void anInterruptedSendWhoseNextThreadCannotStartLetsSendsTakeRequests() t /** Throwing is a JUL handler on BuildTable's logger that throws on every record, as an app's bridge can. */ static final class Throwing extends Handler implements AutoCloseable { - final AtomicInteger published = new AtomicInteger(); + final List published = new java.util.concurrent.CopyOnWriteArrayList<>(); private final Logger logger = Logger.getLogger(BuildTable.class.getName()); Throwing() { @@ -1213,10 +1244,38 @@ static final class Throwing extends Handler implements AutoCloseable { @Override public void publish(LogRecord record) { - published.incrementAndGet(); + published.add(record); throw new IllegalStateException("a log handler that throws (this row's own)"); } + /** + * mine is the messages of the records the calling thread logged (another row's send may log meanwhile), + * each with its exception's message after " | ". + */ + List mine() { + List out = new ArrayList<>(); + for (LogRecord r : Logs.mine(published)) { + Throwable t = r.getThrown(); + out.add(r.getMessage() + (t == null ? "" : " | " + t.getMessage())); + } + return out; + } + + /** count is how many of mine hold each of parts. */ + int count(String... parts) { + int n = 0; + next: + for (String m : mine()) { + for (String p : parts) { + if (!m.contains(p)) { + continue next; + } + } + n++; + } + return n; + } + @Override public void flush() { } @@ -1238,7 +1297,12 @@ public void aLogHandlerThatThrowsNeitherFailsASendNorWedgesSends() throws Except BuildTable.send(() -> { throw new IllegalStateException("classes that cannot be read (this row's own)"); }, f -> fail("nothing to send")); - assertEquals("both sends logged through the handler", 2, h.published.get()); + // Each send's outcome went through the handler. Only those lines are counted: a collection during + // the first send adds its paced line. + assertEquals("the send logged through the handler: " + h.mine(), 1, + h.count("Keploy coverage: build table ", " sent (5 classes")); + assertEquals("the failed send logged through the handler: " + h.mine(), 1, + h.count("Keploy coverage: the build table was not sent | classes that cannot be read")); // Sends logs a send that throws through the same handler: the request made after it still runs. BuildTable.Sends sends = new BuildTable.Sends(BuildTable::thread); CountDownLatch failing = new CountDownLatch(1); diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/ProbeMapTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/ProbeMapTest.java index 9ce1c665..785fd5c4 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/ProbeMapTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/ProbeMapTest.java @@ -52,7 +52,9 @@ import static org.junit.Assert.assertFalse; import static org.junit.Assert.assertNotEquals; import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertSame; import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; /** * Rows for the probe map: for every class file of the java-dedup sample and of a class of code shapes (if and @@ -619,6 +621,58 @@ public void twoCopiesOfAClassThatDifferAreNotMeasured() throws Exception { } } + @Test + public void aFailureInKeploysOwnStepsIsNotMappedAndTheCallersOwnPassesAsItIs() throws Exception { + byte[] bytes = copy(false, 10, 12); + // An index error in the probe walk, Keploy's step after JaCoCo's: here a caller that empties the + // method's branch rows before its probes are walked. It is NotMapped (the class is not measured and + // counted as a map that failed), not an error that blames JaCoCo for the class. + try { + KeployBranchCoverage.analyze(bytes, new HashMap<>(), new KeployBranchCoverage.EachProbe() { + @Override + public boolean wants(KeployBranchCoverage.Method m) { + m.branches = new boolean[0][]; + return true; + } + + @Override + public void probe(KeployBranchCoverage.Method m, int r, KeployBranchCoverage.Hits h) { + } + }); + fail("an index error in the walk passed"); + } catch (KeployBranchCoverage.NotMapped e) { + assertTrue(e.getMessage(), e.getMessage().startsWith("d/C#m")); + assertTrue(String.valueOf(e.getCause()), e.getCause() instanceof ArrayIndexOutOfBoundsException); + } + // What the caller's own callback throws passes as it is: BuildTable ends a map past its bound so. + IllegalArgumentException mine = new IllegalArgumentException("the caller's"); + try { + KeployBranchCoverage.analyze(bytes, new HashMap<>(), (m, r, h) -> { + throw mine; + }); + fail("the callback's exception was lost"); + } catch (IllegalArgumentException e) { + assertSame(mine, e); + } + // And so does what its wants throws, though it runs under the same wrap as the walk. + IllegalStateException declined = new IllegalStateException("the caller's wants"); + try { + KeployBranchCoverage.analyze(bytes, new HashMap<>(), new KeployBranchCoverage.EachProbe() { + @Override + public boolean wants(KeployBranchCoverage.Method m) { + throw declined; + } + + @Override + public void probe(KeployBranchCoverage.Method m, int r, KeployBranchCoverage.Hits h) { + } + }); + fail("the wants exception was lost"); + } catch (IllegalStateException e) { + assertSame(declined, e); + } + } + @Test public void twoCopiesOfAClassThatMapAlikeShareOneRecord() throws Exception { // Two class files of one name whose bytes differ but whose probes map alike: a test's hits name the From 7d8d2169a637f30aee374bade0e106ed45f5074a Mon Sep 17 00:00:00 2001 From: slayerjain Date: Fri, 20 Nov 2026 17:45:51 +0530 Subject: [PATCH 14/14] fix(dedup): send a class too complex to map as record_too_large A class whose probe map passes its bound (8 MiB of records, or 64 Mi walk steps) is sent as not_measured record_too_large again, as it was before the previous commit, which sent it as unit_too_complex. Keploy's readers (the CLI and the server's table ingest) accept only analyzer_failed and record_too_large as a table reason, and refuse the whole table over a record with any other code. So one class too complex to map cost the run its coverage and dedup; with record_too_large it costs that class only. - The send's INFO line still counts such a class apart: "N not measured as too complex to map (record_too_large: a probe map over 8 MiB or 64 Mi steps; the FINE log names each)". A table record past 8 MiB is counted as before ("not measured as too large"). - Keploy's reason text for record_too_large says the class is too large; in this release it also covers a class too complex to map. - Such a class's record is byte for byte what table format 1 wrote before the previous commit: the build-id row's pinned hash of the capped record is back to its earlier value. - The rest of the previous commit stays: a RuntimeException in Keploy's own steps of a class's analysis is NotMapped, and the throwing-handler row counts only its own sends' outcome lines. Rows: the build-id row (the capped record is record_too_large, its pinned hash back), the bounds row (record_too_large at each bound, counted as too complex to map and not as too large), the send's line row (the class counted as too complex to map under record_too_large, with no unit_too_complex and no "not measured as too large"), and the early-returns budget row (counts record_too_large on the wire). Each failed on the previous commit. Signed-off-by: slayerjain --- .../main/java/io/keploy/dedup/BuildTable.java | 34 +++++++++---------- .../io/keploy/dedup/BuildTableBudgetTest.java | 18 +++++----- .../java/io/keploy/dedup/BuildTableTest.java | 16 +++++---- 3 files changed, 35 insertions(+), 33 deletions(-) diff --git a/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java b/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java index 1a00e903..ac6dfbee 100644 --- a/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java +++ b/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java @@ -117,18 +117,18 @@ static void log(Level level, String message, Throwable thrown) { static final int MAX_FRAME_BYTES = 1 << 20; /** A record (with its newline) larger than a whole inflated chunk can never be sent: record_too_large. */ static final int MAX_RECORD_BYTES = 8 << 20; - /** The reason a unit's record is sent without its methods when the record is past MAX_RECORD_BYTES. */ - static final String RECORD_TOO_LARGE = "record_too_large"; /** - * The reason a unit's record is sent without its methods when its probe map is past MAX_MAP_BYTES or - * MAX_MAP_STEPS (the contract's section 2.4): no test's leaves can be named in it, so Keploy shows it as - * too complex to map, not as missed. + * The reason a unit's record is sent without its methods when the record is past MAX_RECORD_BYTES, or + * when its probe map is past MAX_MAP_BYTES or MAX_MAP_STEPS: a unit too complex to map, in which no + * test's leaves can be named. A table reason is one of the two codes Keploy's readers accept + * (analyzer_failed, record_too_large), so a unit too complex to map is sent as record_too_large; only + * this SDK's own log counts it apart. */ - static final String UNIT_TOO_COMPLEX = "unit_too_complex"; + static final String RECORD_TOO_LARGE = "record_too_large"; /** * The most a unit's probe map may take: its records' bytes, newlines included (a table record's bound), * and the steps its probes' walks take (KeployBranchCoverage.Hits#steps, about a second of CPU). A unit - * past either is sent not_measured unit_too_complex with no probe record, and its map is given up as + * past either is sent not_measured record_too_large with no probe record, and its map is given up as * soon as it is past one, so neither the heap nor the time it takes grows with it: a method of n early * returns maps to about n * n ranges (its probe k covers the k checks before it), 4.4 MB at 1,000 and * 43 MB at 3,000. The bound is on the text; what a unit's map holds live until its records are made is @@ -548,7 +548,7 @@ static void send(Supplier> classes, FrameSender sender, File spi } if (s.tooComplex > 0) { m.append(", ").append(s.tooComplex).append(" not measured as too complex to map (") - .append(UNIT_TOO_COMPLEX).append(": a probe map over ").append(MAX_MAP_BYTES >> 20) + .append(RECORD_TOO_LARGE).append(": a probe map over ").append(MAX_MAP_BYTES >> 20) .append(" MiB or ").append(MAX_MAP_STEPS >> 20) .append(" Mi steps; the FINE log names each)"); } @@ -951,9 +951,9 @@ static final class Summary { final int unnamed; /** How many of the not measured records JaCoCo analyzed but whose probes could not be mapped. */ final int notMapped; - /** How many of the not measured records are record_too_large: the table record too large. */ + /** How many of the not measured records are record_too_large as the table record is too large. */ final int tooLarge; - /** How many of the not measured records are unit_too_complex: the probe map past its bound. */ + /** How many of the not measured records are record_too_large as their probe map is past its bound. */ final int tooComplex; /** How many probe-map records went to the probes sink: one per class name of each measured unit. */ final int probeRecords; @@ -1195,7 +1195,7 @@ static final class Unit { boolean notMapped; // Set when the unit's record is past its bound: record_too_large (failed is set too). boolean tooLarge; - // Set when the unit's probe map is past its bound: unit_too_complex (failed is set too). + // Set when the unit's probe map is past its bound: record_too_large (failed is set too). boolean tooComplex; // Class files with code but no line numbers: their methods have no leaf, so they are left out. int noLines; @@ -1278,9 +1278,9 @@ private List sortedFiles() { /** * record analyzes each of the unit's class files once, all of them first, in name order, then makes * the unit's records. The first analysis that fails decides: analyzer_failed when JaCoCo refuses the - * class file or its probes cannot be mapped, unit_too_complex when its probe map is past the budget. + * class file or its probes cannot be mapped, record_too_large when its probe map is past the budget. * Only then are the probe-map records made and each charged in place of its text: a record past the - * budget makes the unit unit_too_complex, and a second class file of one name that maps its probes + * budget makes the unit record_too_large, and a second class file of one name that maps its probes * otherwise makes it analyzer_failed. A unit with two failures of different kinds gets the first one's * reason in that order; either way it is not measured and has no probe record. */ @@ -1311,7 +1311,7 @@ byte[] record(MapBudget budget) { null); failed = true; tooComplex = true; - return notMeasured(UNIT_TOO_COMPLEX); + return notMeasured(RECORD_TOO_LARGE); } catch (KeployBranchCoverage.NotMapped e) { // A class whose probes cannot be mapped exactly is not measured: its hits could not be read. log(Level.FINE, "Keploy coverage: the probes of " + path() + " could not be mapped", e); @@ -1364,7 +1364,7 @@ byte[] record(MapBudget budget) { null); failed = true; tooComplex = true; - return notMeasured(UNIT_TOO_COMPLEX); + return notMeasured(RECORD_TOO_LARGE); } catch (Exception | LinkageError e) { log(Level.FINE, "Keploy coverage: the probes of " + path() + " could not be mapped", e); failed = true; @@ -1636,7 +1636,7 @@ void hits(KeployBranchCoverage.Method a, KeployBranchCoverage.Hits h, BitSet out * off that analysis (KeployBranchCoverage walks back from the probe over the edges JaCoCo's builder wired), * so the map costs the class's probes' walks and their ranges, with no object per probe. The * walks and the text are quadratic in a chain of early returns, so a unit's map has a budget - * ({@link MapBudget}): past it the unit is unit_too_complex. + * ({@link MapBudget}): past it the unit is record_too_large. */ static final class ProbeMap { private ProbeMap() { @@ -1807,7 +1807,7 @@ private static RuntimeException notMapped(RuntimeException e) { /** * TooComplex ends a unit's probe map that is past its budget (MapBudget): the unit is sent not_measured - * unit_too_complex. It carries no stack trace: it is an answer, not a bug. + * record_too_large. It carries no stack trace: it is an answer, not a bug. */ static final class TooComplex extends RuntimeException { private static final long serialVersionUID = 1L; diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableBudgetTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableBudgetTest.java index 6ce23aee..f0ed5a77 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableBudgetTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableBudgetTest.java @@ -295,11 +295,11 @@ public void aMethodOf3000BranchesMapsInUnderASecond() throws Exception { } @Test(timeout = 600_000) - public void aChainOfEarlyReturnsIsMappedOrTooComplexAndNeverUnbounded() throws Exception { + public void aChainOfEarlyReturnsIsMappedOrTooLargeAndNeverUnbounded() throws Exception { // Probe k of a chain of n early returns walks back over the k checks before it and covers each one's // line and branch, so the map is about n * n ranges: 4.4 MB at 1,000 and 43 MB at 3,000; 4,500 ran a // 256 MiB heap out of memory (and 3,000 a 128 MiB one) before the map had a bound. Past it the class - // is unit_too_complex, given up as its map grows past the bound. + // is record_too_large, given up as its map grows past the bound. for (int n : new int[] {1000, 3000, 4500}) { String name = "gen/big/Returns" + n; List one = new ArrayList<>(); @@ -309,7 +309,7 @@ public void aChainOfEarlyReturnsIsMappedOrTooComplexAndNeverUnbounded() throws E assertEquals(1, b.tableRecords); assertEquals(name, mapped ? 1 : 0, b.probeRecords); assertEquals(name, mapped ? 0 : 1, b.notMeasured); - assertEquals(name, mapped ? 0 : 1, b.tooComplex); + assertEquals(name, mapped ? 0 : 1, b.tooLarge); assertTrue(name + ": the send took " + (b.cpuNanos / 1_000_000) + " ms of CPU", b.cpuNanos < 1_000_000_000L); assertTrue(name + ": the send allocated " + (b.allocated >> 20) + " MiB", @@ -459,8 +459,8 @@ static final class Budget { int tableRecords; int probeRecords; int notMeasured; - /** How many of the table's records are unit_too_complex. */ - int tooComplex; + /** How many of the table's records are record_too_large. */ + int tooLarge; /** Bytes the send allocated on its thread (the sink's own allocation left out). */ long allocated; /** The send thread's CPU time, in nanoseconds. */ @@ -569,12 +569,12 @@ private static Budget measure(String label, List classes, long[] bytes = new long[2]; boolean live = Boolean.getBoolean("keploy.budget.live"); long[] liveMax = {-1}; - // A record may run on into the next frame: not_measured and unit_too_complex are counted over the + // A record may run on into the next frame: not_measured and record_too_large are counted over the // table's stream, a match split over two frames included. The sink reads each frame where the writer // keeps it, as the socket's does: a copy or a String of a 1 MiB frame is a G1 humongous object, which // grows the heap this row measures. Matches notMeasured = new Matches("\"state\":\"not_measured\""); - Matches tooComplex = new Matches("\"reason\":{\"code\":\"unit_too_complex\"}"); + Matches tooLarge = new Matches("\"reason\":{\"code\":\"record_too_large\"}"); Budget b = new Budget(); long gc0 = gcMillis(); long process0 = processCpuNanos(); @@ -625,7 +625,7 @@ public void send(byte[] frame, int off, int len) { } if (!probes) { notMeasured.add(frame, nl + 1, off + len); - tooComplex.add(frame, nl + 1, off + len); + tooLarge.add(frame, nl + 1, off + len); } sinkAllocated[0] += allocation.getThreadAllocatedBytes(self) - sink0; // The sink's own time (a live run's collection) is not the send's gap. @@ -633,7 +633,7 @@ public void send(byte[] frame, int off, int len) { } }); b.notMeasured = notMeasured.n; - b.tooComplex = tooComplex.n; + b.tooLarge = tooLarge.n; long wall = System.nanoTime() - t0; long allocated = allocation.getThreadAllocatedBytes(self) - allocated0 - sinkAllocated[0]; sending.set(false); diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java index c8294087..61bc7cb0 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java @@ -207,8 +207,8 @@ public void aTableThatChangesForTheSameClassesNamesAnotherBuild() throws Excepti Collect capped = new Collect(); BuildTable.write(one, capped, new Collect(), BuildTable.MAX_RECORD_BYTES, mapBytes, steps[0] - 1); assertTrue(why, capped.records.get(0) - .contains("\"state\":\"not_measured\",\"reason\":{\"code\":\"unit_too_complex\"}")); - assertEquals(why, "9b7d25b9091045ff4366ec799490ab35c3b1df10c5fcf8528eaaa66f1f746d26", + .contains("\"state\":\"not_measured\",\"reason\":{\"code\":\"record_too_large\"}")); + assertEquals(why, "3ee7977c74cf5939ab7af3957bfc1bbd4e495614d4dcb7bd066c707618f7ec1c", sha256Hex(capped.ndjson.toByteArray())); Collect chunked = new Collect(); BuildTable.write(javaDedup(), chunked, 1600); @@ -392,7 +392,7 @@ public void aRecordLargerThanAChunkInflatesToIsRecordTooLarge() throws Exception } @Test - public void aProbeMapPastItsBytesOrStepsIsUnitTooComplexWithNoProbeRecord() throws Exception { + public void aProbeMapPastItsBytesOrStepsIsRecordTooLargeWithNoProbeRecord() throws Exception { String name = "gen/big/Returns"; byte[] bytes = BuildTableBudgetTest.returns(name, 200); List one = Collections.singletonList(new BuildTable.ClassFile(name, bytes)); @@ -418,8 +418,9 @@ public void aProbeMapPastItsBytesOrStepsIsUnitTooComplexWithNoProbeRecord() thro String why = "bytes " + bounds[i][0] + ", steps " + bounds[i][1]; assertEquals(why, 1, s.records); assertEquals(why, at ? 0 : 1, s.notMeasured); + // Sent as record_too_large (the code Keploy's readers accept), counted apart in the SDK's own log. assertEquals(why, at ? 0 : 1, s.tooComplex); - assertEquals(why + ": a map past its bound is not a record too large", 0, s.tooLarge); + assertEquals(why + ": a map past its bound is not counted as a record too large", 0, s.tooLarge); assertEquals(why, 0, s.notMapped); assertEquals(why, at ? 1 : 0, s.probeRecords); assertEquals(why, at ? 1 : 0, map.records.size()); @@ -428,7 +429,7 @@ public void aProbeMapPastItsBytesOrStepsIsUnitTooComplexWithNoProbeRecord() thro assertTrue(rec, rec.contains("\"state\":\"ok\"")); assertEquals(probes.records.get(0), map.records.get(0)); } else { - assertTrue(rec, rec.contains("\"state\":\"not_measured\",\"reason\":{\"code\":\"unit_too_complex\"}") + assertTrue(rec, rec.contains("\"state\":\"not_measured\",\"reason\":{\"code\":\"record_too_large\"}") && rec.endsWith("\"methods\":[]}")); } } @@ -450,8 +451,9 @@ public void theSendsLineCountsUnitsTooComplexApartFromRecordsTooLarge() throws E } assertNotNull(line); assertTrue(line, line.contains("(1 classes, 1 not measured, 1 not measured as too complex to map" - + " (unit_too_complex: a probe map over 8 MiB or 64 Mi steps; the FINE log names each);")); - assertFalse(line, line.contains("record_too_large")); + + " (record_too_large: a probe map over 8 MiB or 64 Mi steps; the FINE log names each);")); + assertFalse(line, line.contains("not measured as too large")); + assertFalse(line, line.contains("unit_too_complex")); } }