diff --git a/README.md b/README.md index b53a1a5c..e061d232 100644 --- a/README.md +++ b/README.md @@ -31,7 +31,7 @@ Since 2.2.0 a testcase's record also names, in `"classMismatch":[className, ...] With the JaCoCo agent in the application's JVM (the usual case), steps 2 and 5 are one pass over the probe arrays of the application's classes alone, split across up to four threads (half the cores): a probe is cleared only once it was read as hit, so a hit made while the pass runs is reported by that pass or the next one, never lost. Library classes are neither read nor cleared. If JaCoCo's runtime data cannot be read that way, or the pass fails once, the agent uses JaCoCo's own reset and dump for the rest of the run, as it does when JaCoCo is reached over TCP. -Before the first testcase's coverage, the agent sends Enterprise one INDEX record on `/tmp/coverage_data.sock`: a JSON object starting with `{"kind":"index",` that says how it found the application's own classes (`source`: `override` when the dirs and jars in `KEPLOY_JAVA_CLASS_DIRS` held them, `derived` from the launch or from `target/classes` or `build/classes/java/main` under the working directory, `whole-jar` or `classpath` when it could not tell them from the libraries, `none`), what it found them from in words (`launch`), the dirs and jars it read them from (`roots`, at most 64, and `rootCount`), the package roots it kept (`packages`), how many classes it indexed, the build id of those classes (`jv1-` and 32 hex), which JaCoCo agent measures them (`jacoco`: `embedded`, the one keploy-sdk.jar starts, `attached` by the command, `none`, or `unchecked` when the agent was not started as a `-javaagent`), that embedded agent's `includes=`, and two versions as their jars' manifests name them, or `""` when not known: keploy-sdk.jar's (`sdkVersion`) and that of the JaCoCo agent that measures the classes, embedded or attached (`jacocoVersion`). It carries no class list. It is sent as the agent starts when that path's socket file is there (Enterprise's receiver makes it), and otherwise, or when that send fails, right before the first testcase's coverage; a receiver Enterprise binds anew on that path gets it again before its first record. On the TCP transport used in Kubernetes no INDEX record is sent. +Before the first testcase's coverage, the agent sends Enterprise one INDEX record on `/tmp/coverage_data.sock`: a JSON object starting with `{"kind":"index",` that says how it found the application's own classes (`source`: `override` when the dirs and jars in `KEPLOY_JAVA_CLASS_DIRS` held them, `derived` from the launch or from `target/classes` or `build/classes/java/main` under the working directory, `whole-jar` or `classpath` when it could not tell them from the libraries, `none`), what it found them from in words (`launch`), the dirs and jars it read them from (`roots`, at most 64, and `rootCount`), the package roots it kept (`packages`), how many classes it indexed, the build id of those classes (`jv1-` and 32 hex; it also names the format of keploy-sdk's build table and the JaCoCo release keploy-sdk analyzes the classes with, so an SDK upgrade that changes either gives the same classes a new build id), which JaCoCo agent measures them (`jacoco`: `embedded`, the one keploy-sdk.jar starts, `attached` by the command, `none`, or `unchecked` when the agent was not started as a `-javaagent`), that embedded agent's `includes=`, and two versions as their jars' manifests name them, or `""` when not known: keploy-sdk.jar's (`sdkVersion`) and that of the JaCoCo agent that measures the classes, embedded or attached (`jacocoVersion`). It carries no class list. It is sent as the agent starts when that path's socket file is there (Enterprise's receiver makes it), and otherwise, or when that send fails, right before the first testcase's coverage; a receiver Enterprise binds anew on that path gets it again before its first record. On the TCP transport used in Kubernetes no INDEX record is sent. Since 2.2.0 Keploy can also ask for that background, to keep it out of what it compares: it sends the line `BACKGROUND` on `/tmp/coverage_control.sock`. The agent answers `ACK` when it reads JaCoCo's probes in place and a receiver is on `/tmp/coverage_data.sock`, and `IGNORED` otherwise (`KEPLOY_JAVA_SCAN=off`, JaCoCo reached over TCP, the in-place read failed, or no receiver), as an older agent answers a line it does not know. The ask holds for this run of the application and that receiver only: an application started again (native `keploy test` starts it again for each test set) and a receiver Enterprise binds anew on that path (the next docker-compose test set) get no record until Keploy asks again, so Keploy asks after each INDEX record, which comes once per start of the application and once per new receiver. From then on, each `START` that cleared hits made since the previous testcase's `END` sends one BACKGROUND record on `/tmp/coverage_data.sock`, right after that `START` is answered (after the INDEX record, when that receiver does not have it yet): `{"kind":"background","v":1,"build":"","after":"","before":"","executedLinesByFile":{className:[probeIdx]}}`, under the key and in the shape of a testcase's record (with its `classMismatch`, left out when empty), plus `"lost":N` when the records of the N windows before it could not be made or sent. A window with no hits sends nothing. What ran before the first testcase (the application's start, the warmup) is not background, and neither is what ran after an `END` that was not taken or a `START` whose `END` never came (a testcase's own hits): none of it is sent. A test's own work that goes on after its `END` (an async task it started) lands in the next window, and so is reported as background. A Keploy that never asks gets no record, short of a new receiver that replaces the one that asked within the same instant (the agent tells receivers apart by the socket file's inode and modification time). On the TCP transport used in Kubernetes there is no BACKGROUND. diff --git a/keploy-sdk/pom.xml b/keploy-sdk/pom.xml index 85783e36..40f08847 100644 --- a/keploy-sdk/pom.xml +++ b/keploy-sdk/pom.xml @@ -20,6 +20,10 @@ UTF-8 UTF-8 0.8.15 + + -Xmx1g diff --git a/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java b/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java new file mode 100644 index 00000000..ac6dfbee --- /dev/null +++ b/keploy-sdk/src/main/java/io/keploy/dedup/BuildTable.java @@ -0,0 +1,2493 @@ +package io.keploy.dedup; + +import org.jacoco.core.analysis.ILine; +import org.jacoco.core.analysis.IMethodCoverage; +import org.jacoco.core.analysis.ISourceNode; +import org.jacoco.core.internal.analysis.KeployBranchCoverage; +import org.newsclub.net.unix.AFUNIXSocket; +import org.newsclub.net.unix.AFUNIXSocketAddress; +import org.objectweb.asm.ClassReader; +import org.objectweb.asm.ClassVisitor; +import org.objectweb.asm.MethodVisitor; +import org.objectweb.asm.Opcodes; + +import java.io.BufferedOutputStream; +import java.io.BufferedReader; +import java.io.File; +import java.io.IOException; +import java.io.InterruptedIOException; +import java.io.InputStream; +import java.io.InputStreamReader; +import java.io.OutputStream; +import java.io.PrintStream; +import java.lang.management.GarbageCollectorMXBean; +import java.lang.management.ManagementFactory; +import java.lang.management.MemoryPoolMXBean; +import java.lang.management.MemoryType; +import java.lang.management.MemoryUsage; +import java.nio.channels.Channels; +import java.nio.channels.FileChannel; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.LinkOption; +import java.nio.file.Path; +import java.nio.file.StandardOpenOption; +import java.security.MessageDigest; +import java.security.NoSuchAlgorithmException; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.BitSet; +import java.util.Collections; +import java.util.Comparator; +import java.util.HashMap; +import java.util.HashSet; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.concurrent.ThreadFactory; +import java.util.function.Function; +import java.util.function.IntSupplier; +import java.util.function.Supplier; +import java.util.logging.Level; +import java.util.logging.Logger; + +/** + * BuildTable is the build table of the code-coverage ingest contract (schema 1, section 3.3) for the app's + * classes: one record per top-level class (its nested, inner, anonymous, lambda and Scala module classes + * folded in by name), methods with JaCoCo's per-method lines and branch edges, in the contract's order and + * written byte for byte as Keploy's Go encoder writes them (enterprise pkg/service/codecoverage). + * + *

Keploy asks for it with the control-socket command {@code TABLE}, which it sends once its first START + * is answered ACK. The agent answers ACK at once and never works on the table on the control path: JaCoCo's + * analyzer runs over the indexed class bytes on a daemon thread of its own, which streams the records to + * Keploy as TABLE frames on the data socket. An older SDK answers the command IGNORED, and an older Keploy + * never sends it, so it never gets a frame. A TABLE while a send runs has that send run once more after it. + * + *

A TABLE frame is one connection to the data socket: the 4 bytes {@code KTB1}, a 4-byte big-endian + * length n, then n bytes (n plus 8 is at most {@link #MAX_FRAME_BYTES}): a header JSON line, then the next + * bytes of the table's NDJSON stream (a record may run on into the next frame). The header is + * {@code {"v":1,"part":"table","build":"jv1-...","seq":k,"last":false}}; the last frame also carries + * {@code "records"} (the record count) and {@code "sha256"} (hex, over the whole NDJSON stream). With no app + * class the stream is one frame: build "", last, 0 records. After each frame the SDK shuts its side's output + * and waits for {@code OK} and a newline; a receiver that closes without it ends the stream, and the table + * goes again only when Keploy sends TABLE again. + * + *

After the table's last frame the probe map follows as a second stream of the same frames, with + * {@code "part":"probes"} and seq from 0 again: one record per class file name of each measured class, giving + * for each JaCoCo probe the leaves (indices into its class's leaves, as ranges) that a hit of that probe + * covers. Keploy turns each test's probe hits, which name a class by its name, into its leaves with it; it is + * never uploaded. Two class files of one name whose bytes differ share the record when they map their probes + * alike, and fail their class closed (not measured) when not. + * + *

The send holds no whole table in the heap: each class's table record goes into the frame being filled + * as it is made, and its probe-map records into a temp file (owner-only, unlinked once open where the OS + * allows), which is streamed back as the probe map's frames once the table's last frame is sent. So what a + * send keeps is the class list (names and ids, no bytes of its own), one frame and one class's analysis. + * A TABLE asked when the heap is short is not taken: see {@link #answer}. + */ +final class BuildTable { + + private static final Logger LOGGER = Logger.getLogger(BuildTable.class.getName()); + + /** + * log never throws. java.util.logging does not catch what a Handler throws (an app's logging bridge, an OOM + * while formatting), and a log that throws would end a send, or leave {@link Sends} running with no thread: + * every later TABLE would be ACKed and never sent. + */ + static void log(Level level, String message, Throwable thrown) { + try { + LOGGER.log(level, message, thrown); + } catch (Throwable ignored) { + // Nowhere left to say it: the log is what failed. + } + } + + /** The control-socket command Keploy sends for the build table. */ + static final String COMMAND = "TABLE"; + /** + * The table's format, F of the build id (IndexReport.buildId): the contract's section 3.3 records as this + * class writes them and the probe map of leaf ranges, with every bound that decides what a class's records + * are ({@link #MAX_RECORD_BYTES}, {@link #MAX_MAP_BYTES}, {@link #MAX_MAP_STEPS}, how + * KeployBranchCoverage.Hits counts its steps, and the reason a unit past each is not measured). Raise it in + * any release where the table or probe-map records written for the same class bytes can change, so they + * name another build. The KTB1 frames that carry the records are not in it: their header's "v" names them. + */ + static final int TABLE_FORMAT = 1; + /** A TABLE frame's size cap, magic and length included. */ + static final int MAX_FRAME_BYTES = 1 << 20; + /** A record (with its newline) larger than a whole inflated chunk can never be sent: record_too_large. */ + static final int MAX_RECORD_BYTES = 8 << 20; + /** + * The reason a unit's record is sent without its methods when the record is past MAX_RECORD_BYTES, or + * when its probe map is past MAX_MAP_BYTES or MAX_MAP_STEPS: a unit too complex to map, in which no + * test's leaves can be named. A table reason is one of the two codes Keploy's readers accept + * (analyzer_failed, record_too_large), so a unit too complex to map is sent as record_too_large; only + * this SDK's own log counts it apart. + */ + static final String RECORD_TOO_LARGE = "record_too_large"; + /** + * The most a unit's probe map may take: its records' bytes, newlines included (a table record's bound), + * and the steps its probes' walks take (KeployBranchCoverage.Hits#steps, about a second of CPU). A unit + * past either is sent not_measured record_too_large with no probe record, and its map is given up as + * soon as it is past one, so neither the heap nor the time it takes grows with it: a method of n early + * returns maps to about n * n ranges (its probe k covers the k checks before it), 4.4 MB at 1,000 and + * 43 MB at 3,000. The bound is on the text; what a unit's map holds live until its records are made is + * its ranges' int bounds (4 bytes each, a bound's text at least about 1.5), so up to about 2.7 times + * the bound. + */ + static final int MAX_MAP_BYTES = MAX_RECORD_BYTES; + static final long MAX_MAP_STEPS = 64L << 20; + /** The first 4 bytes of a TABLE frame. */ + static final String MAGIC = "KTB1"; + /** The parts a TABLE answer streams, in this order: the table, then the probe map. */ + static final String PART_TABLE = "table"; + static final String PART_PROBES = "probes"; + + private static final int CONNECT_TIMEOUT_MILLIS = 5_000; + private static final int REPLY_TIMEOUT_MILLIS = 30_000; + private static final int CONNECT_ATTEMPTS = 3; + // Room left in a frame for the magic, the length and the header line. + private static final int HEADER_ROOM = 1024; + + // The JVM's table sends, one at a time. + private static final Sends JVM = new Sends(BuildTable::thread); + + private BuildTable() { + } + + /** One class file the index read: its VM name (org/acme/Foo$Bar) and its bytes. */ + static final class ClassFile { + final String name; + final byte[] bytes; + + ClassFile(String name, byte[] bytes) { + this.name = name; + this.bytes = bytes; + } + } + + /** files maps index entries to class files, so the agent's command hook stays one call. */ + static List files(List entries, Function name, Function bytes) { + List out = new ArrayList<>(entries.size()); + for (T e : entries) { + out.add(new ClassFile(name.apply(e), bytes.apply(e))); + } + return out; + } + + /** The env var, then the system property, that set the free heap a table needs, in MiB; 0 turns it off. */ + static final String MIN_HEAP_ENV = "KEPLOY_COVERAGE_TABLE_MIN_HEAP_MB"; + static final String MIN_HEAP_PROPERTY = "keploy.coverage.table.min.heap.mb"; + /** The free heap a table needs when neither is set: this many MiB, plus {@link #MIN_HEAP_KB_PER_CLASS}. */ + static final long MIN_HEAP_BASE_MB = 64; + /** ...and this many KiB per class the index holds (113 MiB at 50,000 classes). */ + static final long MIN_HEAP_KB_PER_CLASS = 1; + /** The reason a TABLE ask is declined when the heap is short: Keploy marks the run not_measured with it. */ + static final String LOW_HEAP = "low_heap"; + + /** + * answer is the control socket's reply to Keploy's TABLE command. When the heap has less free than the + * table needs ({@link #lowHeap}) the analysis is skipped and the reply is {@code IGNORED proto=2 + * reason=low_heap heap_free_mb=F heap_needed_mb=N}, with a WARNING in the app's log; else it is + * {@link #request}'s ACK or IGNORED. The heap is read once, here, when Keploy asks: a send that runs after + * another one was checked when it was asked. The bound takes loaded, the class count of the index when it + * is read already (the app's start or its first START read it, before Keploy asks for a table); when it is + * not read yet (loaded is -1: no START read it, the index was not warmed up and the JVM started the SDK + * from agentmain or start(), or with a JaCoCo of its own), the bound is the base alone, as the count is + * not known without reading the class files. The answer reads no class file and waits on no lock: classes + * is read on the send's thread. It never blocks. + */ + static String answer(IntSupplier loaded, Supplier> classes, File dataSocket) { + return answer(Heap.read(), loaded.getAsInt(), classes, dataSocket); + } + + /** answer with the heap read and the index's class count (-1: not read yet) given. */ + static String answer(Heap heap, int loaded, Supplier> classes, File dataSocket) { + String low = lowHeap(heap, minHeapMb(System.getenv(MIN_HEAP_ENV), System.getProperty(MIN_HEAP_PROPERTY), + loaded)); + if (low != null) { + return KeployDedupAgent.UNIX_REPLY_IGNORED + " " + low; + } + return request(classes, dataSocket) ? KeployDedupAgent.UNIX_REPLY_TAKEN + : KeployDedupAgent.UNIX_REPLY_IGNORED; + } + + /** + * lowHeap is null when heap has at least minMb MiB free, else what the reply names after IGNORED proto=2 + * ({@code reason=low_heap heap_free_mb=F heap_needed_mb=N}), logged at WARNING. + */ + static String lowHeap(Heap heap, long minMb) { + long free = heap.freeMb(); + if (free >= minMb) { + return null; + } + log(Level.WARNING, "Keploy coverage: the build table is not built, as the heap is short: " + free + + " MiB free of the " + (heap.max >> 20) + " MiB heap cap the JVM reports (in use " + (heap.used >> 20) + + " MiB" + (heap.afterGc ? " after the last collection" : " now") + "), and it needs " + + minMb + " MiB. Keploy marks this run's coverage not measured (" + LOW_HEAP + "). Give the JVM" + + " more heap (-Xmx), or set " + MIN_HEAP_ENV + " (0 turns the check off)", null); + return "reason=" + LOW_HEAP + " heap_free_mb=" + free + " heap_needed_mb=" + minMb; + } + + /** + * minHeapMb is the free heap a table of the given number of classes needs, in MiB: env, else property (as + * the agent reads its settings), else {@link #defaultMinHeapMb}. A value that is not a whole number of 0 or + * more is logged and the default holds. + */ + static long minHeapMb(String env, String property, int classes) { + String v = env == null || env.trim().isEmpty() ? property : env; + if (v == null || v.trim().isEmpty()) { + return defaultMinHeapMb(classes); + } + try { + long mb = Long.parseLong(v.trim()); + if (mb >= 0) { + return mb; + } + } catch (NumberFormatException ignored) { + // logged below + } + log(Level.WARNING, "Keploy coverage: " + MIN_HEAP_ENV + " (or -D" + MIN_HEAP_PROPERTY + ") is '" + v.trim() + + "', not a whole number of MiB (0 or more); the build table needs " + defaultMinHeapMb(classes) + + " MiB of free heap", null); + return defaultMinHeapMb(classes); + } + + /** defaultMinHeapMb is {@link #MIN_HEAP_BASE_MB} MiB plus 1 KiB per class, rounded up to a whole MiB. */ + static long defaultMinHeapMb(int classes) { + return MIN_HEAP_BASE_MB + (Math.max(0, classes) * MIN_HEAP_KB_PER_CLASS + 1023) / 1024; + } + + /** + * Heap is what the low-heap check reads: the heap cap and the heap in use. In use is what the JVM's newest + * collection left (the live set and any garbage it did not reach); with no collection seen, or a JVM that + * does not report one, it is the heap in use now, garbage included. The check never runs a collection: a + * System.gc() would pause the app. + */ + static final class Heap { + /** + * The heap cap in bytes, as Runtime.maxMemory() reports it (a little under -Xmx with the Serial and + * Parallel collectors, which leave out a survivor space); Long.MAX_VALUE when the JVM has none. + */ + final long max; + final long used; + /** afterGc is true when used is what the newest collection left, false when it is the heap in use now. */ + final boolean afterGc; + + Heap(long max, long used, boolean afterGc) { + this.max = max; + this.used = used; + this.afterGc = afterGc; + } + + long freeMb() { + return max == Long.MAX_VALUE ? Long.MAX_VALUE : Math.max(0, max - used) >> 20; + } + + static Heap read() { + Runtime rt = Runtime.getRuntime(); + long now = rt.totalMemory() - rt.freeMemory(); + long after; + try { + after = LastCollection.heapUsedAfter(); + } catch (Throwable t) { + // No com.sun.management (a trimmed runtime) or a JMX error: the heap as it is now. + after = -1; + } + return of(rt.maxMemory(), now, after); + } + + /** + * of is the heap with cap max, now bytes in use now and after bytes in use when the newest collection + * ended (-1 with none seen): the smaller of the two. More in use now is garbage and objects made since + * that collection; less means a collection the JVM did not report ran since, so now is the newer figure. + */ + static Heap of(long max, long now, long after) { + return after < 0 || after > now ? new Heap(max, now, false) : new Heap(max, after, true); + } + + /** + * usedAfterNewest is the heap in use when the newest of ends ended: its use after it, summed over + * heapPools only (the JVM also reports Metaspace and its code cache, which are not the heap); -1 when + * ends is empty. + */ + static long usedAfterNewest(List ends, Set heapPools) { + GcEnd newest = null; + for (GcEnd e : ends) { + if (newest == null || e.endTime > newest.endTime) { + newest = e; + } + } + if (newest == null) { + return -1; + } + long used = 0; + for (Map.Entry p : newest.usedAfter.entrySet()) { + if (heapPools.contains(p.getKey())) { + used += p.getValue(); + } + } + return used; + } + } + + /** GcEnd is one collector's last collection: when it ended (ms since JVM start) and each pool's use after it. */ + static final class GcEnd { + final long endTime; + final Map usedAfter; + + GcEnd(long endTime, Map usedAfter) { + this.endTime = endTime; + this.usedAfter = usedAfter; + } + } + + /** LastCollection reads com.sun.management, in a class of its own so a runtime without it fails only here. */ + static final class LastCollection { + /** heapUsedAfter is the heap in use when the JVM's newest collection ended, or -1 with none seen. */ + static long heapUsedAfter() { + Set heapPools = new HashSet<>(); + for (MemoryPoolMXBean p : ManagementFactory.getMemoryPoolMXBeans()) { + if (p.getType() == MemoryType.HEAP) { + heapPools.add(p.getName()); + } + } + List ends = new ArrayList<>(); + for (GarbageCollectorMXBean g : ManagementFactory.getGarbageCollectorMXBeans()) { + if (g instanceof com.sun.management.GarbageCollectorMXBean) { + com.sun.management.GcInfo info = + ((com.sun.management.GarbageCollectorMXBean) g).getLastGcInfo(); + if (info != null) { + Map usedAfter = new HashMap<>(); + for (Map.Entry e : info.getMemoryUsageAfterGc().entrySet()) { + usedAfter.put(e.getKey(), e.getValue().getUsed()); + } + ends.add(new GcEnd(info.getEndTime(), usedAfter)); + } + } + } + return Heap.usedAfterNewest(ends, heapPools); + } + } + + /** + * request starts the table's send: true (ACK) once it is started on its own thread, or, when a send runs + * already, once this request is set to run after it. It never blocks. + * When no thread can start it answers false (IGNORED) and does not throw; the agent's command hook catches + * anything else. The classes are read on that thread. + */ + static boolean request(Supplier> classes, File dataSocket) { + return JVM.request(() -> send(classes, new SocketSender(dataSocket))); + } + + /** + * requests is how many times the JVM's table send was asked for, started or not: only TABLE asks the heap + * check let through. + */ + static int requests() { + return JVM.requests(); + } + + static Thread thread(Runnable r) { + Thread t = new Thread(r, "keploy-build-table"); + t.setDaemon(true); + // A hint only: HotSpot on Linux runs every Java thread at one OS priority unless told otherwise. + t.setPriority(Thread.MIN_PRIORITY); + return t; + } + + /** + * Sends runs one send at a time. Requests made while one runs are run once after it, as the newest of them: + * each request names its own index and data socket, and only the newest is the agent's now (a stop() and + * start() during a send makes a new index and socket). A send that throws is logged, and the request made + * while it ran still runs. A send that leaves its thread interrupted (an interrupt ends a send: see + * {@link Pacer}) ends that thread: the request made while it ran goes to a new thread, so it never reads the + * index's class files on an interrupted thread (on JDK 8 a read of a class file there fails, and the index + * would keep the classes it got). Sends stays running while that thread starts, so a request made meanwhile + * waits to run after it. A thread that fails to start clears {@code pending}: a request made while the + * handover's thread was starting is dropped with the one it was starting for (ACKed, never sent; Keploy's + * wait for the table ends it, and Keploy asks again on its next run). On the control path the failure + * answers IGNORED and drops no other request, as Sends assumes one caller at a time there (the control + * server handles one command at a time). + */ + static final class Sends { + private final ThreadFactory threads; + private boolean running; + /** The newest request made while a send ran, to run after it; null for none. */ + private Runnable pending; + private int requests; + + Sends(ThreadFactory threads) { + this.threads = threads; + } + + synchronized int requests() { + return requests; + } + + boolean request(Runnable send) { + boolean deferred; + boolean dropped = false; + synchronized (this) { + requests++; + deferred = running; + if (deferred) { + dropped = pending != null; + pending = send; + } else { + running = true; + } + } + if (deferred) { + // Said once per ask, off the lock, so a table that comes late names why. + log(Level.INFO, "Keploy coverage: Keploy asked for the build table while this JVM sends one: it is" + + " sent again when that send ends" + (dropped ? " (an earlier ask that waited for it is" + + " dropped, as this newer one names Keploy's current socket)" : ""), null); + return true; + } + // Called on the control path: a thread that cannot start (a pids limit, a security manager) must not + // end the control server. Keploy reads IGNORED as no table from this SDK. + return start(send, false); + } + + /** + * start runs send and the requests after it on a new thread; false, with a WARNING, if none starts. + * handover is true when send is an ask that waited for the send before it (its thread was interrupted): + * the WARNING then says that ask is not sent, as it was ACKed and logged as sent later. + */ + private boolean start(Runnable send, boolean handover) { + try { + threads.newThread(() -> loop(send)).start(); + return true; + } catch (Throwable t) { + synchronized (this) { + running = false; + pending = null; + } + log(Level.WARNING, "Keploy coverage: the build table's thread did not start" + (handover + ? ": the ask for the build table that waited for the send before it is not sent (Keploy" + + " asks again on its next run)" : ""), t); + return false; + } + } + + private void loop(Runnable first) { + Runnable send = first; + while (true) { + try { + send.run(); + } catch (Throwable t) { + // A failed send does not drop the request made while it ran: that one names the newest + // index and socket. + log(Level.WARNING, "Keploy coverage: the build table was not sent", t); + } + synchronized (this) { + if (pending == null) { + running = false; + return; + } + send = pending; + pending = null; + } + if (Thread.currentThread().isInterrupted()) { + // The interrupt ended this thread's send and is kept for it: the next one runs on a new + // thread, uninterrupted. Sends stays running meanwhile, so a request made now waits for it. + // start's factory and log calls run on this interrupted thread: keep them non-interruptible. + start(send, true); + return; + } + } + } + } + + /** + * send analyzes the classes and streams the table through sender, then the probe map, which waits in a + * temp file in java.io.tmpdir while the table streams ({@link Spill}). It logs what it sent, or why it + * stopped, and never throws. + */ + static void send(Supplier> classes, FrameSender sender) { + send(classes, sender, new File(System.getProperty("java.io.tmpdir", "/tmp"))); + } + + /** send with the probe map's temp file made in spillDir. */ + static void send(Supplier> classes, FrameSender sender, File spillDir) { + try { + if (Thread.currentThread().isInterrupted()) { + // An interrupt ends a send: read no class file on this thread (on JDK 8 that read fails). + throw Pacer.interrupted(); + } + long cpu0 = threadCpuNanos(); + long t0 = System.nanoTime(); + FrameWriter frames = new FrameWriter(sender, PART_TABLE, MAX_FRAME_BYTES); + FrameWriter maps = new FrameWriter(sender, PART_PROBES, MAX_FRAME_BYTES); + Summary s; + Pacer pacer = Pacer.jvm(); + ExitNotice exit = ExitNotice.add(t0, pacer, frames, maps); + try { + try (Spill spill = Spill.open(spillDir)) { + s = write(classes.get(), frames, spill, MAX_RECORD_BYTES, MAX_MAP_BYTES, MAX_MAP_STEPS, pacer); + frames.finish(s.build, s.records); + spill.sendTo(maps, s.build); + } + maps.finish(s.build, s.probeRecords); + } finally { + exit.remove(); + } + StringBuilder m = new StringBuilder("Keploy coverage: build table "); + m.append(s.build.isEmpty() ? "(no app classes)" : s.build).append(" sent (").append(s.records) + .append(" classes, ").append(s.notMeasured).append(" not measured"); + if (s.notMapped > 0) { + // Two copies of a class that differ, or JaCoCo's own steps disagreeing with its public result. + m.append(" (").append(s.notMapped).append(" of them as their probes could not be mapped: two" + + " different class files of one name, or a Keploy bug; the FINE log names each)"); + } + if (s.tooLarge > 0) { + m.append(", ").append(s.tooLarge).append(" not measured as too large (").append(RECORD_TOO_LARGE) + .append(": a record over ").append(MAX_RECORD_BYTES >> 20) + .append(" MiB; the FINE log names each)"); + } + if (s.tooComplex > 0) { + m.append(", ").append(s.tooComplex).append(" not measured as too complex to map (") + .append(RECORD_TOO_LARGE).append(": a probe map over ").append(MAX_MAP_BYTES >> 20) + .append(" MiB or ").append(MAX_MAP_STEPS >> 20) + .append(" Mi steps; the FINE log names each)"); + } + if (s.noLines > 0) { + m.append(", ").append(s.noLines) + .append(" class files without line numbers, so without lines: compile them with -g"); + } + if (s.unnamed > 0) { + m.append(", ").append(s.unnamed).append(" index entries without a class name left out"); + } + m.append("; probe map of ").append(s.probeRecords).append(" class files; ").append(frames.seq + maps.seq) + .append(" frames, ").append((System.nanoTime() - t0) / 1_000_000) + .append(" ms, cpu ").append((threadCpuNanos() - cpu0) / 1_000_000).append(" ms, paced ") + .append(pacer.waitedNanos() / 1_000_000).append(" ms)"); + log(Level.INFO, m.toString(), null); + } catch (IOException e) { + // An interrupt ends the send: the Pacer's, or a temp file's channel that it closed. Name it, and not + // the channel's error, which would point at java.io.tmpdir. + String why = Thread.currentThread().isInterrupted() ? "its thread was interrupted" : e.getMessage(); + log(Level.WARNING, "Keploy coverage: the build table was not sent: " + why + + "; it goes again when Keploy asks for it again", null); + } catch (Throwable t) { + // Never let the table take the app down: Keploy names the coverage it did not get. + log(Level.WARNING, "Keploy coverage: the build table was not sent", t); + } + } + + /** + * ExitNotice says, once, on System.err, that the JVM is exiting while a send runs: a native keploy test stops + * the app at each test set's end, and a send cut short there would otherwise leave no line in the app's log. + * It is a shutdown hook added for the send and removed when it ends. It writes to System.err, not + * java.util.logging: the JVM runs its shutdown hooks at once and in no order, and LogManager's own hook (and + * an app's logging framework's) may have closed the handlers already. A JVM that is killed (SIGKILL) or + * halted runs no hook and says nothing. The hook reads the send's progress, which the send's thread writes: + * the Pacer's class-file counts and wait, and each FrameWriter's frame count. What it says is that progress + * at the exit's start, not the send's outcome: the send's daemon thread runs on while the other hooks run, and + * may still send the rest. + */ + static final class ExitNotice implements Runnable { + static final String NAME = "keploy-build-table-exit"; + /** + * The hook the last add on this thread made, for the rows that check a send removes it. Per thread, so a + * row reads its own send's hook, not another class's, and the app keeps none past its table's thread: + * the value (and the send it holds) goes when that thread ends, with its last send. + */ + static final ThreadLocal LAST_HOOK = new ThreadLocal<>(); + private final long t0; + private final Pacer pacer; + private final FrameWriter table; + private final FrameWriter probes; + private final PrintStream out; + private volatile boolean ended; + private Thread hook; + + ExitNotice(long t0, Pacer pacer, FrameWriter table, FrameWriter probes, PrintStream out) { + this.t0 = t0; + this.pacer = pacer; + this.table = table; + this.probes = probes; + this.out = out; + } + + /** + * add makes the notice and adds its hook. A JVM that takes no hook (a security manager) has none; one that + * is exiting already has none either, and the notice says its line at once, as the exit may cut the send. + * A notice that fails never ends the send. + */ + static ExitNotice add(long t0, Pacer pacer, FrameWriter table, FrameWriter probes) { + ExitNotice n = new ExitNotice(t0, pacer, table, probes, System.err); + try { + Thread h = new Thread(n, NAME); + Runtime.getRuntime().addShutdownHook(h); + n.hook = h; + LAST_HOOK.set(h); + } catch (IllegalStateException exiting) { + n.run(); + } catch (Throwable t) { + log(Level.FINE, "Keploy coverage: no exit notice for the build table's send", t); + } + return n; + } + + /** remove ends the notice: the send is over, so its hook says nothing even when it runs (an exit now). */ + void remove() { + ended = true; + if (hook == null) { + return; + } + try { + Runtime.getRuntime().removeShutdownHook(hook); + } catch (Throwable t) { + // The JVM exits already (IllegalStateException): the hook runs and, as ended is set, says nothing. + } + } + + @Override + public void run() { + if (ended) { + return; + } + try { + out.println(line()); + } catch (Throwable ignored) { + // Nowhere left to say it. + } + } + + /** line is what the notice says: the send's progress, read now. */ + String line() { + long total = pacer.totalClassFiles(); + long done = pacer.doneClassFiles(); + String where; + if (total == 0) { + where = "reading the app's class files"; + } else if (done < total) { + where = done + " of " + total + " class files analyzed"; + } else { + where = "all " + total + " class files analyzed"; + } + return "Keploy coverage: the JVM is exiting while its build table is being sent. So far: " + where + "; " + + (table.seq + probes.seq) + " frames sent in " + (System.nanoTime() - t0) / 1_000_000 + + " ms (" + pacer.waitedNanos() / 1_000_000 + " ms of it paced). A table the exit cuts short" + + " does not reach Keploy: Keploy gets this build's table only from a JVM that runs until it is" + + " sent"; + } + } + + private static long threadCpuNanos() { + try { + return java.lang.management.ManagementFactory.getThreadMXBean().getCurrentThreadCpuTime(); + } catch (Throwable t) { + return 0; + } + } + + /** + * Pacer paces the table's thread by the JVM's collections, so its garbage does not make the collector grow + * the app's heap. The analysis allocates fast (about 80-90 KiB a class, over 1 GB/s at full speed, nearly + * all of it short-lived), and G1 and Parallel commit more heap when collecting takes more than their goal + * of the time (Parallel 1%; G1 a few percent, as low as 1% while it has committed a small part of its cap). + * So the thread owes {@link #WAIT_PER_GC} ms for each ms the collections take while it sends (the app's and + * its own), each ms it works or waits pays one off (never below 0), and after each unit it waits what it owes + * as far as its bounds let it. The time read is every collector bean's: a busy app's own collections (or + * ZGC's and Shenandoah's concurrent cycles, which their beans count) make the thread owe more than its bounds + * let it wait, and so do the send's own collections on every JVM measured (G1 and Parallel, JDK 8 to 21, + * quiet or busy): in practice the send waits about as long as it works. + * + *

The waits are bounded by the send's own work: each unit's wall time (the time since the unit before, + * its frames' socket writes included and its waits left out; the first unit's includes the index's read) is + * added to what the thread may wait, and at most {@link #MAX_BANK_NANOS} of it is saved up. So all the + * waits add up to at most the send's own work time plus one sleep's overrun (a table paced takes at most + * about twice as long as unpaced, whatever the app's collections), no one wait is over a second, and the + * waits over any run of units add up to at most a second plus those units' own time, so frames keep coming + * (Keploy's wait for the table is a bound on the time since its last frame). + * And no wait is taken that would end the table past {@link #deadlineNanos} (20 s at 50,000 class files), + * reckoned as the time since the send began (the index's read included) plus the rest of its class files at + * the rate the analysis went so far (from {@link #begin}, its waits left out): on a slow or busy CPU the + * pacing gives way first. A clock or collector bean that fails ends the pacing, and the rest is sent + * unpaced. An interrupt ends the send: unitDone throws, and the interrupt is kept for the thread. + */ + static final class Pacer { + /** Each millisecond of collection asks for this many of the send's: at most 0.5% of it collecting. */ + static final long WAIT_PER_GC = 200; + /** The most of the send's own work time kept for a later wait: the longest wait after a quiet stretch. */ + static final long MAX_BANK_NANOS = 1_000_000_000L; + /** Waits shorter than this are put off to the next unit, so the thread does not sleep per class. */ + static final long MIN_WAIT_NANOS = 10_000_000L; + /** The pacing's time bound up to {@link #DEADLINE_CLASS_FILES} class files: see deadlineNanos. */ + static final long DEADLINE_NANOS = 20_000_000_000L; + static final long DEADLINE_CLASS_FILES = 50_000; + // The most the thread may owe (an hour), so the sum cannot overflow; the work-time bound limits it anyway. + private static final long MAX_DEBT_NANOS = 3_600_000_000_000L; + + /** The clock, the collections' time and the sleep a Pacer reads; the JVM's, or a row's. */ + interface Clock { + long nanoTime(); + + /** The collectors' total time in ms since the JVM started. */ + long gcMillis(); + + void sleep(long nanos) throws InterruptedException; + } + + private final Clock clock; + private boolean off; + private long start; + /** + * When the analysis began, for the rate: set by begin, or at construction if begin is never reached (then + * total is 0 and the rate is never read). start is the send's, for the deadline. + */ + private long begun; + // total, done and waited are volatile for ExitNotice, which reads them on its own thread; only the + // send's thread writes them. + private volatile long total; + private volatile long done; + private long last; + private long gc; + /** The collections' time when the send began, for the line that says the send is paced. */ + private long gc0; + private boolean told; + private long debt; + private long bank; + private volatile long waited; + + Pacer(Clock clock) { + this.clock = clock; + try { + last = clock.nanoTime(); + start = last; + begun = last; + gc = clock.gcMillis(); + gc0 = gc; + } catch (RuntimeException e) { + stop("its clock failed", e); + } + } + + /** off is a Pacer that never waits: write's, outside a send. */ + static Pacer off() { + Pacer p = new Pacer(JvmClock.NONE); + p.off = true; + return p; + } + + /** jvm paces by this JVM's collectors; off when the runtime has no collector beans to read. */ + static Pacer jvm() { + List beans; + try { + beans = ManagementFactory.getGarbageCollectorMXBeans(); + } catch (RuntimeException | LinkageError e) { + log(Level.FINE, "Keploy coverage: the build table is sent unpaced: no collector beans", e); + return off(); + } + return new Pacer(new JvmClock(beans)); + } + + /** + * deadlineNanos is the most the pacing lets a table of classFiles take: 20 s, and 0.4 ms a class file past + * 50,000 (the table's bound, 60 s at 50,000, scales the same way). + */ + static long deadlineNanos(long classFiles) { + return Math.max(DEADLINE_NANOS, DEADLINE_NANOS / DEADLINE_CLASS_FILES * classFiles); + } + + /** + * begin is called as the analysis begins, after the index's read: it says how many class files the send + * analyzes, for its deadline (not called, the deadline is 20 s), and reads the clock the rate is taken + * from, so the index's read does not count as the analysis's work. + */ + void begin(long classFiles) { + total = Math.max(0, classFiles); + try { + begun = clock.nanoTime(); + } catch (RuntimeException e) { + stop("its clock failed", e); + } + } + + /** waitedNanos is how long the thread waited, in all. */ + long waitedNanos() { + return waited; + } + + /** totalClassFiles is the class files the send analyzes, from begin; 0 before it. */ + long totalClassFiles() { + return total; + } + + /** doneClassFiles is the class files analyzed so far, paced or not. */ + long doneClassFiles() { + return done; + } + + /** + * unitDone is called after each unit, whose classFiles were analyzed: it waits what is owed and may be. + * It throws on an interrupt, which it keeps for the thread: an interrupt ends the send. + */ + void unitDone(int classFiles) throws InterruptedIOException { + int files = Math.max(0, classFiles); + done += files; + if (Thread.currentThread().isInterrupted()) { + throw interrupted(); + } + if (off) { + return; + } + try { + long now = clock.nanoTime(); + long g = clock.gcMillis(); + if (g > gc) { + long perMs = WAIT_PER_GC * 1_000_000L; + debt = Math.min(MAX_DEBT_NANOS, debt + Math.min(g - gc, MAX_DEBT_NANOS / perMs) * perMs); + gc = g; + } + // The send's own time since the last unit pays off what it owes, and is what it may wait. + long worked = Math.max(0, now - last); + debt = Math.max(0, debt - worked); + bank = Math.min(MAX_BANK_NANOS, bank + worked); + last = now; + long wait = Math.min(Math.min(debt, bank), room(now)); + if (wait < MIN_WAIT_NANOS) { + return; + } + if (!told) { + // Deferred: said once a send, at its first wait, so a table that comes late names why. + told = true; + log(Level.INFO, "Keploy coverage: the build table's send is paced, so its garbage does not grow" + + " the app's heap: the JVM's collections took " + (g - gc0) + " ms in the " + + Math.max(0, now - start) / 1_000_000 + " ms since it began, so it owes " + + debt / 1_000_000 + " ms now (each ms of collection adds " + WAIT_PER_GC + ", and each" + + " ms the send works or waits takes one off, never below 0); its" + + " waits add up to at most its own work time (plus one sleep's overrun), so the table" + + " takes about twice as long as unpaced at most, and none ends the table past " + + deadlineNanos(total) / 1_000_000 + " ms from its start", null); + } + try { + clock.sleep(wait); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + throw interrupted(); + } + long after = clock.nanoTime(); + long took = Math.max(0, after - now); + // What the sleep overran comes off the bank too, so the bound holds for the time slept; and at + // least the wait asked for, so it holds on a clock that does not move across a sleep. + long charged = Math.max(took, wait); + bank -= charged; + debt = Math.max(0, debt - charged); + waited += took; + last = after; + } catch (RuntimeException e) { + stop("its clock failed", e); + } + } + + /** room is how long the thread may wait at now and still end the table by its deadline. */ + private long room(long now) { + long elapsed = Math.max(0, now - start); + double rest = 0; + if (done > 0 && total > done) { + rest = (double) Math.max(0, now - begun - waited) / done * (total - done); + } + return (long) Math.max(-1.0, deadlineNanos(total) - elapsed - rest); + } + + static InterruptedIOException interrupted() { + return new InterruptedIOException("the build table's thread was interrupted"); + } + + private void stop(String why, Throwable thrown) { + off = true; + log(Level.FINE, "Keploy coverage: the build table is sent unpaced: " + why, thrown); + } + + /** JvmClock reads System.nanoTime and the JVM's collector beans, and sleeps the thread. */ + static final class JvmClock implements Clock { + static final JvmClock NONE = new JvmClock(Collections.emptyList()); + private final List beans; + + JvmClock(List beans) { + this.beans = beans; + } + + @Override + public long nanoTime() { + return System.nanoTime(); + } + + @Override + public long gcMillis() { + long sum = 0; + for (GarbageCollectorMXBean b : beans) { + // -1: this collector does not report its time. + long t = b.getCollectionTime(); + if (t > 0) { + sum += t; + } + } + return sum; + } + + @Override + public void sleep(long nanos) throws InterruptedException { + Thread.sleep(nanos / 1_000_000L, (int) (nanos % 1_000_000L)); + } + } + } + + // ---------------------------------------------------------------- the table + + /** + * What write wrote: the build id, the record count, how many records are not measured, how many class + * files have code but no line numbers, and how many index entries were left out as no class has a name. + */ + static final class Summary { + final String build; + final int records; + final int notMeasured; + final int noLines; + final int unnamed; + /** How many of the not measured records JaCoCo analyzed but whose probes could not be mapped. */ + final int notMapped; + /** How many of the not measured records are record_too_large as the table record is too large. */ + final int tooLarge; + /** How many of the not measured records are record_too_large as their probe map is past its bound. */ + final int tooComplex; + /** How many probe-map records went to the probes sink: one per class name of each measured unit. */ + final int probeRecords; + + Summary(String build, int records, int notMeasured, int noLines, int unnamed, int notMapped, int tooLarge, + int tooComplex, int probeRecords) { + this.build = build; + this.records = records; + this.notMeasured = notMeasured; + this.noLines = noLines; + this.unnamed = unnamed; + this.notMapped = notMapped; + this.tooLarge = tooLarge; + this.tooComplex = tooComplex; + this.probeRecords = probeRecords; + } + } + + /** Where write puts each record (UTF-8 JSON, no newline), in the table's order. */ + interface RecordSink { + void record(String build, byte[] json) throws IOException; + } + + /** A sink that drops what it is given: write's probe map when only the table is wanted. */ + static final RecordSink DROP = (build, json) -> { }; + + /** + * write analyzes the classes and hands each table record to sink, in the contract's order, and each + * measured class's probe-map records to probes as they are made, in the same order: it keeps neither. With + * no class bytes the build id is "" and there is no record. + */ + static Summary write(List classes, RecordSink sink, RecordSink probes) throws IOException { + return write(classes, sink, probes, MAX_RECORD_BYTES); + } + + /** write with the probe map dropped. */ + static Summary write(List classes, RecordSink sink) throws IOException { + return write(classes, sink, DROP, MAX_RECORD_BYTES); + } + + static Summary write(List classes, RecordSink sink, int maxRecord) throws IOException { + return write(classes, sink, DROP, maxRecord); + } + + static Summary write(List classes, RecordSink sink, RecordSink probes, int maxRecord) + throws IOException { + return write(classes, sink, probes, maxRecord, MAX_MAP_BYTES, MAX_MAP_STEPS); + } + + /** write with the bounds of a unit's table record (maxRecord) and probe map (maxMap bytes, maxSteps). */ + static Summary write(List classes, RecordSink sink, RecordSink probes, int maxRecord, int maxMap, + long maxSteps) throws IOException { + return write(classes, sink, probes, maxRecord, maxMap, maxSteps, Pacer.off()); + } + + /** write with the bounds, its thread paced by pacer after each unit. */ + static Summary write(List classes, RecordSink sink, RecordSink probes, int maxRecord, int maxMap, + long maxSteps, Pacer pacer) throws IOException { + Grouped g = group(classes); + if (g.build.isEmpty()) { + return new Summary("", 0, 0, 0, g.unnamed, 0, 0, 0, 0); + } + String build = g.build; + List units = g.units; + int notMeasured = 0; + int noLines = 0; + int notMapped = 0; + int tooLarge = 0; + int tooComplex = 0; + int probeRecords = 0; + long classFiles = 0; + for (Unit u : units) { + classFiles += u.files.size(); + } + pacer.begin(classFiles); + for (int i = 0; i < units.size(); i++) { + // A unit written is dropped: what stays is the class list (names, ids), not the table. + Unit u = units.set(i, null); + byte[] rec = u.record(new MapBudget(maxMap, maxSteps)); + if (rec.length + 1 > maxRecord) { + rec = u.notMeasured(RECORD_TOO_LARGE); + u.failed = true; + u.tooLarge = true; + } + if (u.failed) { + notMeasured++; + if (u.notMapped) { + notMapped++; + } + if (u.tooLarge) { + tooLarge++; + } + if (u.tooComplex) { + tooComplex++; + } + } else { + noLines += u.noLines; + for (byte[] p : u.probes) { + probes.record(build, p); + probeRecords++; + } + } + sink.record(build, rec); + pacer.unitDone(u.files.size()); + } + return new Summary(build, units.size(), notMeasured, noLines, g.unnamed, notMapped, tooLarge, + tooComplex, probeRecords); + } + + /** What group makes of the index's classes: the build id ("" with none), the units in table order. */ + private static final class Grouped { + final String build; + final List units; + final int unnamed; + + Grouped(String build, List units, int unnamed) { + this.build = build; + this.units = units; + this.unnamed = unnamed; + } + } + + /** group folds the classes into units by name, in table order; its lookups end with it. */ + private static Grouped group(List classes) { + Map byId = new HashMap<>(); + Set seen = new HashSet<>(); + long[] ids = new long[classes.size()]; + int n = 0; + int unnamed = 0; + for (ClassFile in : classes) { + if (in == null) { + continue; + } + // Every indexed class is in the build id, named or not: the INDEX record's, by the same function. + long id = IndexReport.classId(in.bytes); + ids[n++] = id; + ClassFile f = new ClassFile(vmName(in), in.bytes); + Unit u = f.name == null ? null : Unit.of(f.name); + if (u == null || u.name.isEmpty()) { + // No unit has an empty name (Go refuses the whole table for one): a file named "a/.class". + unnamed++; + continue; + } + if (!seen.add(f.name + "\n" + id)) { + // The same class file again (the index found it under two paths): it is one class file, and + // the build id and the unit's hash count its id once either way. + continue; + } + Unit had = byId.get(u.id); + if (had == null) { + byId.put(u.id, u); + had = u; + } + had.files.add(new UnitFile(f, id)); + } + if (n == 0) { + return new Grouped("", Collections.emptyList(), unnamed); + } + List units = new ArrayList<>(byId.values()); + Collections.sort(units, UNIT_ORDER); + return new Grouped(IndexReport.buildId(Arrays.copyOf(ids, n)), units, unnamed); + } + + /** + * vmName is the class's own name, from its bytes: the name JaCoCo's runtime gives its probes, wherever + * the index found the file. When the bytes are not a class ASM reads, it is the index's name. + */ + static String vmName(ClassFile f) { + try { + String name = new ClassReader(f.bytes).getClassName(); + return name != null ? name : f.name; + } catch (RuntimeException e) { + return f.name; + } + } + + /** Units by package path, segment by segment ('/' sorts first), then by id: Go's comparePkg order. */ + static final Comparator UNIT_ORDER = (a, b) -> { + int c = comparePkg(a.pkg, b.pkg); + return c != 0 ? c : compareCodePoints(a.id, b.id); + }; + + /** comparePkg is codecoverage.comparePkg over code points (the order of their UTF-8 bytes). */ + static int comparePkg(String a, String b) { + int i = 0; + int j = 0; + while (i < a.length() && j < b.length()) { + int ca = a.codePointAt(i); + int cb = b.codePointAt(j); + if (ca != cb) { + if (ca == '/') { + return -1; + } + if (cb == '/') { + return 1; + } + return Integer.compare(ca, cb); + } + i += Character.charCount(ca); + j += Character.charCount(cb); + } + return Integer.compare(a.length() - i, b.length() - j); + } + + /** compareCodePoints orders strings as Go's strings.Compare orders their UTF-8 bytes. */ + static int compareCodePoints(String a, String b) { + int i = 0; + int j = 0; + while (i < a.length() && j < b.length()) { + int ca = a.codePointAt(i); + int cb = b.codePointAt(j); + if (ca != cb) { + return Integer.compare(ca, cb); + } + i += Character.charCount(ca); + j += Character.charCount(cb); + } + return Integer.compare(a.length() - i, b.length() - j); + } + + private static final class UnitFile { + final ClassFile file; + final long classId; + + UnitFile(ClassFile file, long classId) { + this.file = file; + this.classId = classId; + } + } + + /** A table record: a top-level class and the class files folded into it by name. */ + static final class Unit { + final String pkg; + final String name; + final String id; + final List files = new ArrayList<>(1); + boolean failed; + // Set when JaCoCo analyzed the unit but its probes could not be mapped (so failed is set too). + boolean notMapped; + // Set when the unit's record is past its bound: record_too_large (failed is set too). + boolean tooLarge; + // Set when the unit's probe map is past its bound: record_too_large (failed is set too). + boolean tooComplex; + // Class files with code but no line numbers: their methods have no leaf, so they are left out. + int noLines; + // The probe map's records for the unit's class files, in file order; set when record() succeeds. + List probes = Collections.emptyList(); + + private Unit(String pkg, String name) { + this.pkg = pkg; + this.name = name; + this.id = "c:" + (pkg.isEmpty() ? name : pkg + "/" + name); + } + + /** of finds a class file's unit by name: its simple name cut at the first $ after its first character. */ + static Unit of(String vmName) { + int slash = vmName.lastIndexOf('/'); + String pkg = slash < 0 ? "" : vmName.substring(0, slash); + String simple = vmName.substring(slash + 1); + int cut = simple.indexOf('$', 1); + return new Unit(pkg, cut < 0 ? simple : simple.substring(0, cut)); + } + + String path() { + return pkg.isEmpty() ? name : pkg + "/" + name; + } + + String hash() { + long[] ids = new long[files.size()]; + for (int i = 0; i < ids.length; i++) { + ids[i] = files.get(i).classId; + } + return IndexReport.classIdsSum(ids); + } + + byte[] notMeasured(String code) { + StringBuilder sb = head(sourcePath(null)); + sb.append(",\"state\":\"not_measured\",\"reason\":{\"code\":"); + Json.string(sb, code); + sb.append("},\"methods\":[]}"); + return sb.toString().getBytes(StandardCharsets.UTF_8); + } + + private StringBuilder head(String path) { + StringBuilder sb = new StringBuilder(256); + sb.append("{\"kind\":\"class\",\"id\":"); + Json.string(sb, id); + sb.append(",\"pkg\":"); + Json.string(sb, pkg); + sb.append(",\"name\":"); + Json.string(sb, name); + sb.append(",\"path\":"); + Json.string(sb, path); + sb.append(",\"hash\":"); + Json.string(sb, hash()); + return sb; + } + + /** + * sourcePath is the build path of the unit's source: the package path and the SourceFile of its + * first class file (by name, so the top-level class when it is there) whose SourceFile is a bare, + * non-empty file name; "" when none has one. + */ + private String sourcePath(Map infos) { + List sorted = sortedFiles(); + for (UnitFile f : sorted) { + ClassInfo info = infos != null ? infos.get(f.file.name) : ClassInfo.read(f.file.bytes); + String sf = info == null ? null : info.sourceFile; + if (sf != null && !sf.isEmpty() && sf.indexOf('/') < 0) { + return pkg.isEmpty() ? sf : pkg + "/" + sf; + } + } + return ""; + } + + private List sortedFiles() { + List sorted = new ArrayList<>(files); + Collections.sort(sorted, (a, b) -> compareCodePoints(a.file.name, b.file.name)); + return sorted; + } + + /** + * record analyzes each of the unit's class files once, all of them first, in name order, then makes + * the unit's records. The first analysis that fails decides: analyzer_failed when JaCoCo refuses the + * class file or its probes cannot be mapped, record_too_large when its probe map is past the budget. + * Only then are the probe-map records made and each charged in place of its text: a record past the + * budget makes the unit record_too_large, and a second class file of one name that maps its probes + * otherwise makes it analyzer_failed. A unit with two failures of different kinds gets the first one's + * reason in that order; either way it is not measured and has no probe record. + */ + byte[] record(MapBudget budget) { + Map infos = new HashMap<>(); + List methods = new ArrayList<>(); + Map byId = new HashMap<>(); + List sorted = sortedFiles(); + List ranges = new ArrayList<>(sorted.size()); + UnitFile at = null; + try { + String lastName = null; + for (UnitFile f : sorted) { + at = f; + ClassInfo info = ClassInfo.read(f.file.bytes); + infos.put(f.file.name, info); + Ranges r = ProbeMap.analyze(this, f, info, methods, byId, budget); + if (f.file.name.equals(lastName)) { + // A second class file of this name: its record is the one before it or none (below), + // so its text is not one the records take. + budget.forget(r.counted()); + } + ranges.add(r); + lastName = f.file.name; + } + } catch (TooComplex e) { + log(Level.FINE, "Keploy coverage: the probe map of " + path() + " is past its bound: " + e.getMessage(), + null); + failed = true; + tooComplex = true; + return notMeasured(RECORD_TOO_LARGE); + } catch (KeployBranchCoverage.NotMapped e) { + // A class whose probes cannot be mapped exactly is not measured: its hits could not be read. + log(Level.FINE, "Keploy coverage: the probes of " + path() + " could not be mapped", e); + failed = true; + notMapped = true; + return notMeasured("analyzer_failed"); + } catch (Exception | LinkageError e) { + log(Level.FINE, "Keploy coverage: JaCoCo could not analyze " + at.file.name, e); + failed = true; + return notMeasured("analyzer_failed"); + } + Collections.sort(methods, (a, b) -> { + int c = Integer.compare(a.line, b.line); + return c != 0 ? c : compareCodePoints(a.id, b.id); + }); + int base = 0; + for (Method m : methods) { + m.base = base; + base += m.leaves.length / 3; + } + List maps = new ArrayList<>(); + try { + String lastName = null; + byte[] last = null; + for (int i = 0; i < sorted.size(); i++) { + UnitFile f = sorted.get(i); + Ranges r = ranges.get(i); + // Only one class file's ranges stay alive while the records are made. + ranges.set(i, null); + byte[] p = ProbeMap.record(id, f, r); + if (f.file.name.equals(lastName)) { + // Two class files of one name whose bytes differ (the index found both). Keploy finds a + // class's map by its name, as a test's hits name it, so they share one record: exact + // only when a hit of either one covers the same leaves. + if (!ProbeMap.sameProbes(p, last)) { + throw new IllegalStateException(f.file.name + ": two class files of this name map" + + " their probes differently"); + } + continue; + } + // The unit's records, newlines included, within the budget, in place of their ranges' text + // counted as the analysis wrote it. + budget.take(p.length + 1, r.counted()); + maps.add(p); + lastName = f.file.name; + last = p; + } + } catch (TooComplex e) { + log(Level.FINE, "Keploy coverage: the probe map of " + path() + " is past its bound: " + e.getMessage(), + null); + failed = true; + tooComplex = true; + return notMeasured(RECORD_TOO_LARGE); + } catch (Exception | LinkageError e) { + log(Level.FINE, "Keploy coverage: the probes of " + path() + " could not be mapped", e); + failed = true; + notMapped = true; + return notMeasured("analyzer_failed"); + } + probes = maps; + StringBuilder sb = head(sourcePath(infos)); + sb.append(",\"state\":\"ok\",\"methods\":["); + for (int i = 0; i < methods.size(); i++) { + if (i > 0) { + sb.append(','); + } + methods.get(i).json(sb); + } + sb.append("]}"); + return sb.toString().getBytes(StandardCharsets.UTF_8); + } + } + + /** What a cheap ASM pass (no code) gives: the SourceFile and which methods take varargs. */ + static final class ClassInfo { + String sourceFile; + final Set varargs = new HashSet<>(); + + static ClassInfo read(byte[] bytes) { + final ClassInfo info = new ClassInfo(); + try { + new ClassReader(bytes).accept(new ClassVisitor(Opcodes.ASM9) { + @Override + public void visitSource(String source, String debug) { + info.sourceFile = source; + } + + @Override + public MethodVisitor visitMethod(int access, String name, String desc, String sig, String[] ex) { + if ((access & Opcodes.ACC_VARARGS) != 0) { + info.varargs.add(name + desc); + } + return null; + } + }, ClassReader.SKIP_CODE | ClassReader.SKIP_FRAMES); + } catch (RuntimeException e) { + return null; + } + return info; + } + } + + /** A method of a unit: its id, display name, first line and leaves in the contract's order. */ + static final class Method { + final String id; + final String display; + final int line; + final int[] leaves; // flat: line, count, kind per leaf + // The unit-relative index of the method's first leaf (the unit's leaves in record order). + int base; + // The lines of the method's code leaves, ascending, and each one's method-relative leaf index. + private final int[] codeLines; + private final int[] codeLeaves; + + private Method(String id, String display, int line, int[] leaves) { + this.id = id; + this.display = display; + this.line = line; + this.leaves = leaves; + int n = 0; + for (int i = 0; i < leaves.length; i += 3) { + if (leaves[i + 2] == 0) { + n++; + } + } + codeLines = new int[n]; + codeLeaves = new int[n]; + n = 0; + for (int i = 0; i < leaves.length; i += 3) { + if (leaves[i + 2] == 0) { + codeLines[n] = leaves[i]; + codeLeaves[n++] = i / 3; + } + } + } + + /** + * of makes the method's record from JaCoCo's per-method lines: one code leaf per line with + * instructions (n = their count), then one branch leaf per branch edge on that line. Lines below 1 + * have no leaf (javac never writes line 0; some bytecode tools do), and the method's line is its first + * line with a leaf. A method with no such line (a class built without line numbers) has no leaf to + * show and is left out, as the contract's method count leaves out a method with no code leaf. + */ + static Method of(String vmClass, String nested, String unitName, IMethodCoverage mc, ClassInfo info) { + int first = Math.max(1, mc.getFirstLine()); + int last = mc.getLastLine(); + if (last < first) { + return null; + } + int[] buf = new int[16]; + int k = 0; + for (int l = first; l <= last; l++) { + ILine line = mc.getLine(l); + int ic = line.getInstructionCounter().getTotalCount(); + if (ic == 0) { + continue; + } + int bc = line.getBranchCounter().getTotalCount(); + int need = k + 3 * (1 + bc); + if (need > buf.length) { + buf = Arrays.copyOf(buf, Math.max(need, buf.length * 2)); + } + buf[k++] = l; + buf[k++] = ic; + buf[k++] = 0; + for (int b = 0; b < bc; b++) { + buf[k++] = l; + buf[k++] = 1; + buf[k++] = 1; + } + } + if (k == 0) { + return null; + } + boolean varargs = info != null && info.varargs.contains(mc.getName() + mc.getDesc()); + return new Method("m:" + vmClass + "#" + mc.getName() + mc.getDesc(), + Names.method(unitName, nested, mc.getName(), mc.getDesc(), varargs), buf[0], + Arrays.copyOf(buf, k)); + } + + void json(StringBuilder sb) { + sb.append("{\"id\":"); + Json.string(sb, id); + sb.append(",\"name\":"); + Json.string(sb, display); + sb.append(",\"line\":").append(line).append(",\"leaves\":["); + for (int i = 0; i < leaves.length; i += 3) { + if (i > 0) { + sb.append(','); + } + int l = leaves[i]; + sb.append('[').append(l).append(",0,").append(l).append(",0,").append(leaves[i + 1]).append(',') + .append(leaves[i + 2]).append(']'); + } + sb.append("]}"); + } + + /** codeLeaf is the method-relative index of line l's code leaf, or -1 when the method has none there. */ + int codeLeaf(int l) { + int at = Arrays.binarySearch(codeLines, l); + return at < 0 ? -1 : codeLeaves[at]; + } + + /** + * check throws unless the analysis has the method's leaves: on each line, the code leaf's instruction + * count and the line's branch edges (lines below 1 have no leaf). + */ + void check(KeployBranchCoverage.Method a) { + Map lines = new HashMap<>(); // line -> instructions, branches + for (int k = 0; k < a.lines.length; k++) { + if (a.lines[k] < 1) { + continue; + } + int[] s = lines.get(a.lines[k]); + if (s == null) { + s = new int[2]; + lines.put(a.lines[k], s); + } + s[0]++; + s[1] += a.branches[k].length; + } + Map mine = new HashMap<>(); + for (int i = 0; i < leaves.length; i += 3) { + int[] s = mine.get(leaves[i]); + if (s == null) { + s = new int[2]; + mine.put(leaves[i], s); + } + if (leaves[i + 2] == 0) { + s[0] += leaves[i + 1]; + } else { + s[1]++; + } + } + if (lines.size() != mine.size()) { + throw new IllegalStateException(id + ": " + lines.size() + " lines in the analysis, " + mine.size() + + " in the table"); + } + for (Map.Entry e : lines.entrySet()) { + int[] t = mine.get(e.getKey()); + if (t == null || !Arrays.equals(t, e.getValue())) { + throw new IllegalStateException(id + ": line " + e.getKey() + " differs from the table"); + } + } + } + + // Per counted instruction of the class file's analysis of this method (set by prepare, until release): + // the method-relative index of its line's code leaf, and of its first branch edge (-1: a line below 1, + // no leaf). A probe's ranges add the method's base once the unit's methods are ordered. + private int[] codeAt; + private int[] edgeAt; + private KeployBranchCoverage.Method prepared; + + /** + * prepare checks the analysis has the method's leaves ({@link #check}) and notes, per counted + * instruction, its leaves: its line's code leaf, and its line's edges after the code leaf in the + * order of the instructions and of each instruction's branches. + */ + void prepare(KeployBranchCoverage.Method a) { + check(a); + codeAt = new int[a.lines.length]; + edgeAt = new int[a.lines.length]; + Map edges = new HashMap<>(); // line -> edges met so far + for (int k = 0; k < a.lines.length; k++) { + int l = a.lines[k]; + if (l < 1) { + codeAt[k] = -1; + edgeAt[k] = -1; + continue; + } + int leaf = codeLeaf(l); + if (leaf < 0) { + throw new IllegalStateException(id + ": line " + l + " has no code leaf"); + } + Integer met = edges.get(l); + int e = met == null ? 0 : met; + codeAt[k] = leaf; + edgeAt[k] = leaf + 1 + e; + edges.put(l, e + a.branches[k].length); + } + prepared = a; + } + + /** release forgets the analysis prepare noted, once its probes are mapped. */ + void release() { + codeAt = null; + edgeAt = null; + prepared = null; + } + + /** + * hits sets in out the method-relative leaf indices a probe covers: a line's code leaf when one of its + * instructions is covered, and the leaf of each covered branch edge. + */ + void hits(KeployBranchCoverage.Method a, KeployBranchCoverage.Hits h, BitSet out) { + if (prepared != a) { + throw new IllegalStateException(id + ": probes of an analysis not prepared"); + } + for (int j = 0; j < h.count; j++) { + int k = h.at[j]; + if (codeAt[k] < 0) { + continue; + } + out.set(codeAt[k]); + boolean[] b = h.branches[k]; + for (int t = 0; t < b.length; t++) { + if (b[t]) { + out.set(edgeAt[k] + t); + } + } + } + } + } + + /** + * ProbeMap maps each JaCoCo probe of a class file to the leaves a hit of that probe covers, from JaCoCo's + * own analysis (KeployBranchCoverage): a line's code leaf when an instruction of it is covered, and each + * covered branch edge's leaf. A probe covers more than its own line: JaCoCo marks every instruction on the + * path that ends at the probe, so a test's leaves are the union of its probes' leaves. + * + *

The class file is analyzed once, for both its table methods and its map: each probe's leaves are read + * off that analysis (KeployBranchCoverage walks back from the probe over the edges JaCoCo's builder wired), + * so the map costs the class's probes' walks and their ranges, with no object per probe. The + * walks and the text are quadratic in a chain of early returns, so a unit's map has a budget + * ({@link MapBudget}): past it the unit is record_too_large. + */ + static final class ProbeMap { + private ProbeMap() { + } + + /** + * record is the class file's probe-map record: {"class":vmName,"id":crc64,"unit":unitId,"probes":[...]}, + * with probes as {@link #analyze} made them, written once into a buffer of its size. + */ + static byte[] record(String unitId, UnitFile f, Ranges probes) { + StringBuilder sb = new StringBuilder(64 + f.file.name.length() + unitId.length()); + sb.append("{\"class\":"); + Json.string(sb, f.file.name); + sb.append(",\"id\":\"").append(String.format("%016x", f.classId)).append("\",\"unit\":"); + Json.string(sb, unitId); + sb.append(",\"probes\":"); + byte[] head = sb.toString().getBytes(StandardCharsets.UTF_8); + byte[] rec = new byte[head.length + probes.jsonLength() + 1]; + System.arraycopy(head, 0, rec, 0, head.length); + int at = probes.json(rec, head.length); + if (at != rec.length - 1) { + throw new IllegalStateException(f.file.name + ": its probes wrote " + (at - head.length) + + " bytes, not the " + probes.jsonLength() + " counted"); + } + rec[at] = '}'; + return rec; + } + + /** + * sameProbes says whether two records of one class name and unit map their probes alike: they are + * alike but for the id (16 hex digits), which ends at the same place in both. + */ + static boolean sameProbes(byte[] a, byte[] b) { + if (a.length != b.length) { + return false; + } + int id = indexOf(a, ID_KEY) + ID_KEY.length + 16; + for (int i = id; i < a.length; i++) { + if (a[i] != b[i]) { + return false; + } + } + return true; + } + + private static final byte[] ID_KEY = ",\"id\":\"".getBytes(StandardCharsets.US_ASCII); + + private static int indexOf(byte[] a, byte[] key) { + outer: + for (int i = 0; i + key.length <= a.length; i++) { + for (int j = 0; j < key.length; j++) { + if (a[i + j] != key[j]) { + continue outer; + } + } + return i; + } + throw new IllegalStateException("a probe record with no id"); + } + + /** + * analyze is the class file's one analysis (KeployBranchCoverage): it adds the file's methods to the + * unit's (methods and byId; the first class file of a name makes a method, a second one of that name + * is checked against it), counts a file with code but no line in u.noLines, and returns the file's + * probe map: one entry per probe (the class's probe count), each the probe's leaves as half-open ranges + * of its method's leaf indices, ascending, no two ranges touching; its record adds the method's base + * once the unit's methods are ordered ({@link Ranges#json}). It throws NotMapped when the class's lines + * or branches differ from the unit's methods (the same name under two roots), when a method with lines + * is in one and not in the other, or when a method's flow is not one KeployBranchCoverage maps; TooComplex + * as soon as the unit's map is past its budget's bytes or steps; and what JaCoCo throws for a class it + * does not read. + */ + static Ranges analyze(Unit u, UnitFile f, ClassInfo info, List methods, Map byId, + MapBudget budget) { + Map shape = new HashMap<>(); + String prefix = "m:" + f.file.name + "#"; + String nested = f.file.name.substring(u.path().length()); + Ranges ranges = new Ranges(budget); + boolean[] seen = new boolean[2]; // a method with code, and one with a line + int total = Math.max(0, KeployBranchCoverage.analyze(f.file.bytes, shape, + new KeployBranchCoverage.EachProbe() { + private Method method; + + @Override + public boolean wants(KeployBranchCoverage.Method s) { + try { + // As JaCoCo sums a class: code with no line when no method of it has one. + seen[0] = true; + if (s.coverage.getFirstLine() != ISourceNode.UNKNOWN_LINE) { + seen[1] = true; + } + method = byId.get(prefix + s.name + s.desc); + if (method == null) { + method = Method.of(f.file.name, nested, u.name, s.coverage, info); + if (method == null) { + // Checked below: a method with no line has no leaf for its probes. + return false; + } + byId.put(method.id, method); + methods.add(method); + } + method.prepare(s); + return true; + } catch (RuntimeException e) { + throw notMapped(e); + } + } + + @Override + public void probe(KeployBranchCoverage.Method s, int r, KeployBranchCoverage.Hits h) { + try { + budget.step(h.steps); + method.hits(s, h, ranges.bits); + ranges.put(s.probes[r], method); + } catch (RuntimeException e) { + throw notMapped(e); + } + } + })); + if (seen[0] && !seen[1]) { + u.noLines++; + } + try { + int inShape = 0; + for (Map.Entry e : shape.entrySet()) { + Method m = byId.get(prefix + e.getKey()); + if (m == null) { + for (int l : e.getValue().lines) { + if (l >= 1) { + throw new IllegalStateException(f.file.name + "#" + e.getKey() + + " has lines but no method in the table"); + } + } + continue; + } + m.release(); + inShape++; + } + // And the other way: each of the table's methods of this class file has its probes mapped here. + int inTable = 0; + for (String id : byId.keySet()) { + if (id.startsWith(prefix)) { + inTable++; + } + } + if (inTable != inShape) { + throw new IllegalStateException(f.file.name + ": " + inTable + " methods in the table, " + + inShape + " in the probe analysis"); + } + ranges.close(f.file.name, total); + } catch (RuntimeException e) { + throw notMapped(e); + } + return ranges; + } + + /** + * notMapped is e as a map that failed: TooComplex and NotMapped as they are, a check that failed as + * NotMapped. + */ + private static RuntimeException notMapped(RuntimeException e) { + if (e instanceof TooComplex || e instanceof KeployBranchCoverage.NotMapped) { + return e; + } + return new KeployBranchCoverage.NotMapped(String.valueOf(e.getMessage()), e); + } + } + + /** + * TooComplex ends a unit's probe map that is past its budget (MapBudget): the unit is sent not_measured + * record_too_large. It carries no stack trace: it is an answer, not a bug. + */ + static final class TooComplex extends RuntimeException { + private static final long serialVersionUID = 1L; + + TooComplex(String message) { + super(message, null, false, false); + } + } + + /** + * MapBudget is what one unit's probe map may still take: bytes (its records', newlines included; its + * ranges' text counts as it is written, since a record holds it) and walk steps. Past either it throws + * TooComplex, so a map is given up as soon as it is past, not once it is made. + */ + static final class MapBudget { + private final long maxBytes; + private final long maxSteps; + private long text; + private long bytes; + private long steps; + + MapBudget(long maxBytes, long maxSteps) { + this.maxBytes = maxBytes; + this.maxSteps = maxSteps; + } + + /** step counts a probe's walk. */ + void step(int n) { + steps += n; + if (steps > maxSteps) { + throw new TooComplex("its probes' walks take more than " + maxSteps + " steps"); + } + } + + /** + * grew counts ranges' text an analysis wrote for a record not taken yet: at most what the record writes + * for them, so the budget is past only when the records would be. + */ + void grew(int n) { + text += n; + if (bytes + text > maxBytes) { + throw new TooComplex("its probe map is more than " + maxBytes + " bytes"); + } + } + + /** forget takes back the text counted for a record not taken: it shares the record before it. */ + void forget(long counted) { + text -= counted; + } + + /** take counts a whole record (with its newline) in place of the text counted for it. */ + void take(int n, long counted) { + text -= counted; + bytes += n; + if (bytes + text > maxBytes) { + throw new TooComplex("its probe records are more than " + maxBytes + " bytes"); + } + } + } + + /** + * Ranges keeps a class file's probe map as its analysis makes it: each probe's leaf ranges relative to its + * method, as ints in one buffer, with where each probe's ranges start and end and the probe's method, so a + * probe costs its ranges and no object. Its record writes them once as ASCII JSON, each plus its method's + * base, after the unit's methods are ordered ({@link ProbeMap#record}). + */ + static final class Ranges { + /** The probe being written: its method-relative leaves, set by the caller and cleared by {@link #put}. */ + final BitSet bits = new BitSet(); + private final MapBudget budget; + private int[] bounds = new int[64]; + private int length; + private int[] from = new int[16]; + private int[] to = new int[16]; + private Method[] owner = new Method[16]; + private int total = -1; + // The text put counted in the budget: the ranges' text before the bases are added, so at most what + // the record writes for them. + private long counted; + + Ranges(MapBudget budget) { + this.budget = budget; + } + + /** + * put keeps bits as the given probe's ranges of m's leaves (start, end, ... ascending, none touching), + * then clears it. + */ + void put(int probe, Method m) { + if (probe >= from.length) { + int size = Math.max(probe + 1, from.length * 2); + from = Arrays.copyOf(from, size); + to = Arrays.copyOf(to, size); + owner = Arrays.copyOf(owner, size); + } + if (to[probe] != 0) { + throw new IllegalStateException("probe " + probe + " mapped twice"); + } + from[probe] = length; + owner[probe] = m; + int at = bits.nextSetBit(0); + while (at >= 0) { + int end = bits.nextClearBit(at); + int n = (length > from[probe] ? 1 : 0) + digits(at) + 1 + digits(end); + if (length + 2 > bounds.length) { + bounds = Arrays.copyOf(bounds, bounds.length * 2); + } + bounds[length++] = at; + bounds[length++] = end; + counted += n; + budget.grew(n); + at = bits.nextSetBit(end); + } + // to is one past the ranges' end, so 0 is "not mapped" and an empty range list is still mapped. + to[probe] = length + 1; + bits.clear(); + } + + /** counted is the text put counted in the budget for this map. */ + long counted() { + return counted; + } + + private static int digits(int v) { + if (v < 0) { + throw new IllegalStateException("leaf " + v); + } + int digits = 1; + for (int x = v; x >= 10; x /= 10) { + digits++; + } + return digits; + } + + /** close ends the map at the class's probe count (total): a probe at or past it is a bug. */ + void close(String className, int total) { + for (int p = total; p < from.length; p++) { + if (to[p] != 0) { + throw new IllegalStateException(className + ": probe " + p + " of " + total + " probes"); + } + } + this.total = total; + } + + /** + * jsonLength is the length of {@link #json}'s text: [ ], one [ ] a probe, the commas, and each probe's + * ranges plus its method's base. + */ + int jsonLength() { + long n = 2 + 3L * total - (total > 0 ? 1 : 0); + for (int p = 0; p < total && p < from.length; p++) { + if (to[p] == 0) { + continue; + } + int end = to[p] - 1; + n += Math.max(0, end - from[p] - 1); + int base = owner[p].base; + for (int i = from[p]; i < end; i++) { + n += digits(base + bounds[i]); + } + } + if (total < 0 || n > Integer.MAX_VALUE - 1024) { + throw new IllegalStateException("a probe map of " + n + " bytes"); + } + return (int) n; + } + + /** + * json writes the map into out at at and returns where it ends: one entry per probe of the class, + * [] for a probe no method mapped, each range's bounds plus its method's base (unit leaf indices). + */ + int json(byte[] out, int at) { + out[at++] = '['; + for (int p = 0; p < total; p++) { + if (p > 0) { + out[at++] = ','; + } + out[at++] = '['; + if (p < from.length && to[p] != 0) { + int base = owner[p].base; + for (int i = from[p]; i < to[p] - 1; i++) { + if (i > from[p]) { + out[at++] = ','; + } + int v = base + bounds[i]; + int d = digits(v); + for (int j = at + d - 1; j >= at; j--) { + out[j] = (byte) ('0' + v % 10); + v /= 10; + } + at += d; + } + } + out[at++] = ']'; + } + out[at++] = ']'; + return at; + } + } + + /** Display names of methods (API-CONTRACT 2.3). */ + static final class Names { + private Names() { + } + + /** + * method names a method for people: "bar(int)", a nested class's "Inner.bar(int)", a constructor as + * its class's nested name ("Foo(String)", "Inner(int)", "Inner.Deep(long)"), the static initializer as + * "static {}", a lambda lambda$bar$0 as "bar lambda #0"; a varargs method's last parameter as + * "Object...". + */ + static String method(String unitName, String nested, String name, String desc, boolean varargs) { + List parts = new ArrayList<>(); + for (String p : nested.split("\\$")) { + if (!p.isEmpty()) { + parts.add(p); + } + } + String prefix = parts.isEmpty() ? "" : join(parts, ".") + "."; + if (name.equals("")) { + String cls = parts.isEmpty() ? unitName : join(parts, "."); + return cls + "(" + params(desc, varargs) + ")"; + } + if (name.equals("")) { + return prefix + "static {}"; + } + if (name.startsWith("lambda$")) { + int d = name.lastIndexOf('$'); + String num = name.substring(d + 1); + if (d > "lambda$".length() && !num.isEmpty() && isDigits(num)) { + return prefix + name.substring("lambda$".length(), d) + " lambda #" + num; + } + } + return prefix + name + "(" + params(desc, varargs) + ")"; + } + + private static boolean isDigits(String s) { + for (int i = 0; i < s.length(); i++) { + if (s.charAt(i) < '0' || s.charAt(i) > '9') { + return false; + } + } + return true; + } + + private static String join(List parts, String sep) { + StringBuilder sb = new StringBuilder(); + for (String p : parts) { + if (sb.length() > 0) { + sb.append(sep); + } + sb.append(p); + } + return sb.toString(); + } + + /** params lists the descriptor's parameter types by simple name, ", "-joined. */ + static String params(String desc, boolean varargs) { + List out = new ArrayList<>(); + int i = desc.indexOf('(') + 1; + int end = desc.indexOf(')'); + if (i <= 0 || end < i) { + return ""; + } + while (i < end) { + int dims = 0; + while (i < end && desc.charAt(i) == '[') { + dims++; + i++; + } + if (i >= end) { + break; + } + String t; + char c = desc.charAt(i); + if (c == 'L') { + int semi = desc.indexOf(';', i); + if (semi < 0 || semi > end) { + break; + } + String cls = desc.substring(i + 1, semi); + t = cls.substring(cls.lastIndexOf('/') + 1).replace('$', '.'); + i = semi + 1; + } else { + t = primitive(c); + i++; + } + StringBuilder sb = new StringBuilder(t); + for (int d = 0; d < dims; d++) { + sb.append("[]"); + } + out.add(sb.toString()); + } + if (varargs && !out.isEmpty()) { + String lastParam = out.get(out.size() - 1); + if (lastParam.endsWith("[]")) { + out.set(out.size() - 1, lastParam.substring(0, lastParam.length() - 2) + "..."); + } + } + return join(out, ", "); + } + + private static String primitive(char c) { + switch (c) { + case 'Z': + return "boolean"; + case 'B': + return "byte"; + case 'C': + return "char"; + case 'S': + return "short"; + case 'I': + return "int"; + case 'J': + return "long"; + case 'F': + return "float"; + case 'D': + return "double"; + case 'V': + return "void"; + default: + return String.valueOf(c); + } + } + } + + /** JSON strings as Go's encoding/json writes them with HTML escaping off. */ + static final class Json { + private Json() { + } + + private static final char[] HEX = "0123456789abcdef".toCharArray(); + + static void string(StringBuilder sb, String s) { + sb.append('"'); + for (int i = 0; i < s.length(); i++) { + char c = s.charAt(i); + switch (c) { + case '"': + sb.append("\\\""); + continue; + case '\\': + sb.append("\\\\"); + continue; + case '\b': + sb.append("\\b"); + continue; + case '\f': + sb.append("\\f"); + continue; + case '\n': + sb.append("\\n"); + continue; + case '\r': + sb.append("\\r"); + continue; + case '\t': + sb.append("\\t"); + continue; + case '\u2028': + sb.append("\\u2028"); + continue; + case '\u2029': + sb.append("\\u2029"); + continue; + default: + break; + } + if (c < 0x20) { + sb.append("\\u00").append(HEX[c >> 4]).append(HEX[c & 0xf]); + } else if (Character.isHighSurrogate(c) && i + 1 < s.length() + && Character.isLowSurrogate(s.charAt(i + 1))) { + sb.append(c).append(s.charAt(++i)); + } else if (Character.isSurrogate(c)) { + // Not UTF-8: Go reads it as U+FFFD, which it writes as is. + sb.append('\ufffd'); + } else { + sb.append(c); + } + } + sb.append('"'); + } + } + + // ---------------------------------------------------------------- frames + + /** Sends one whole frame; it throws when the receiver did not take it. */ + interface FrameSender { + void send(byte[] frame) throws IOException; + + /** + * send sends the frame in b[off, off + len). The bytes are the writer's own and change once it returns, + * so a sender that keeps them copies them (as this default does); FrameWriter sends through this one. + */ + default void send(byte[] b, int off, int len) throws IOException { + send(Arrays.copyOfRange(b, off, off + len)); + } + } + + /** + * FrameWriter cuts the NDJSON stream of records into TABLE frames of at most MAX_FRAME_BYTES each and + * sends them in order, one at a time. + */ + static final class FrameWriter implements RecordSink { + private final FrameSender sender; + private final String part; + private final int bodyCap; + // The frame being filled, made at the first write and reused for every frame: its body from + // HEADER_ROOM on, its magic, length and header just before it at each flush. A frame allocated anew + // (twice: the body's copy and the frame) is a G1 humongous object at 1 MiB, for which G1 grows the + // heap when its free regions run short: 70-150 MiB of a 4 GiB pod's heap at 50,000 classes. + private byte[] frame; + private int bodyLen; + private final MessageDigest digest; + /** The frames sent; volatile for ExitNotice, which reads it on its own thread (only the send's writes it). */ + volatile int seq; + private String build; + + FrameWriter(FrameSender sender) { + this(sender, PART_TABLE, MAX_FRAME_BYTES); + } + + FrameWriter(FrameSender sender, int maxFrame) { + this(sender, PART_TABLE, maxFrame); + } + + FrameWriter(FrameSender sender, String part, int maxFrame) { + this.sender = sender; + this.part = part; + this.bodyCap = maxFrame - HEADER_ROOM; + if (bodyCap < 1) { + throw new IllegalArgumentException("frame cap " + maxFrame + " leaves no room for a body"); + } + try { + this.digest = MessageDigest.getInstance("SHA-256"); + } catch (NoSuchAlgorithmException e) { + throw new IllegalStateException(e); + } + } + + @Override + public void record(String build, byte[] json) throws IOException { + this.build = build; + digest.update(json); + digest.update((byte) '\n'); + write(json, 0, json.length); + write(new byte[] {'\n'}, 0, 1); + } + + private void write(byte[] b, int off, int len) throws IOException { + while (len > 0) { + int room = bodyCap - bodyLen; + if (room == 0) { + flush(false, -1); + continue; + } + int take = Math.min(room, len); + System.arraycopy(b, off, frame(), HEADER_ROOM + bodyLen, take); + bodyLen += take; + off += take; + len -= take; + } + } + + private byte[] frame() { + if (frame == null) { + frame = new byte[HEADER_ROOM + bodyCap]; + } + return frame; + } + + /** bytes adds whole records, each ended by its newline, to the stream as they are. */ + void bytes(String build, byte[] b, int off, int len) throws IOException { + this.build = build; + digest.update(b, off, len); + write(b, off, len); + } + + /** finish sends the last frame, with the record count and the stream's sha256. */ + void finish(String build, int records) throws IOException { + this.build = build; + flush(true, records); + } + + private void flush(boolean last, int records) throws IOException { + StringBuilder h = new StringBuilder(160); + h.append("{\"v\":1,\"part\":\"").append(part).append("\",\"build\":"); + Json.string(h, build); + h.append(",\"seq\":").append(seq).append(",\"last\":").append(last); + if (last) { + h.append(",\"records\":").append(records) + .append(",\"sha256\":\"").append(IndexReport.hex(digest.digest())).append('"'); + } + h.append("}\n"); + byte[] header = h.toString().getBytes(StandardCharsets.UTF_8); + int start = HEADER_ROOM - header.length - 8; + if (start < 0) { + // A build id or record count no header can carry: never one this class makes. + throw new IOException("TABLE frame " + seq + ": a header of " + header.length + " bytes"); + } + int n = header.length + bodyLen; + byte[] f = frame(); + System.arraycopy(MAGIC.getBytes(StandardCharsets.US_ASCII), 0, f, start, 4); + f[start + 4] = (byte) (n >>> 24); + f[start + 5] = (byte) (n >>> 16); + f[start + 6] = (byte) (n >>> 8); + f[start + 7] = (byte) n; + System.arraycopy(header, 0, f, start + 8, header.length); + try { + sender.send(f, start, 8 + n); + } catch (IOException e) { + throw new IOException("TABLE frame " + (PART_TABLE.equals(part) ? "" : part + " ") + seq + ": " + + e.getMessage(), e); + } + bodyLen = 0; + seq++; + } + } + + /** + * Spill keeps the probe map's records while the table streams, as the wire sends the map after the table's + * last frame: in a temp file made owner-only, which is unlinked as soon as it is open where the OS allows + * (so a JVM that dies mid-send leaves no file; one that dies between the file's making and its opening + * leaves it empty), and is removed on close otherwise. It is opened without following a link, so a link + * put in its place is never written through. A dir where no such file + * can be made, or a write that fails (a full disk), ends the send before the table's last frame: Keploy + * gets no table for that build and names it (table_incomplete), and the WARNING names the dir. + */ + static final class Spill implements RecordSink, AutoCloseable { + private final File dir; + private final Path path; + private final FileChannel channel; + private final OutputStream out; + + private Spill(File dir, Path path, FileChannel channel) { + this.dir = dir; + this.path = path; + this.channel = channel; + this.out = new BufferedOutputStream(Channels.newOutputStream(channel), 1 << 16); + } + + static Spill open(File dir) throws IOException { + Path path; + try { + path = Files.createTempFile(dir.toPath(), "keploy-probe-map-", ".ndjson"); + } catch (IOException | RuntimeException e) { + throw new IOException("no temp file for the probe map in " + dir + " (" + e + + "); set java.io.tmpdir to a dir the app can write", e); + } + try { + return new Spill(dir, path, openOwned(path)); + } catch (IOException | RuntimeException e) { + deleteQuietly(path); + throw new IOException("could not open the probe map's temp file " + path + " (" + e + ")", e); + } + } + + /** + * openOwned opens the file just made at path to write and read back, never through a link (a link put + * in its place fails the open), and deleted on close. + */ + static FileChannel openOwned(Path path) throws IOException { + return FileChannel.open(path, StandardOpenOption.READ, StandardOpenOption.WRITE, + StandardOpenOption.DELETE_ON_CLOSE, LinkOption.NOFOLLOW_LINKS); + } + + @Override + public void record(String build, byte[] json) throws IOException { + try { + out.write(json); + out.write('\n'); + } catch (IOException e) { + throw new IOException("could not keep the probe map in a temp file in " + dir + ": " + + e.getMessage(), e); + } + } + + /** sendTo streams what was kept into w, from its first byte. */ + void sendTo(FrameWriter w, String build) throws IOException { + out.flush(); + channel.position(0); + InputStream in = Channels.newInputStream(channel); + byte[] buf = new byte[1 << 16]; + for (int n = in.read(buf); n >= 0; n = in.read(buf)) { + w.bytes(build, buf, 0, n); + } + } + + @Override + public void close() { + try { + channel.close(); + } catch (IOException ignored) { + // the file is removed below either way + } + deleteQuietly(path); + } + + private static void deleteQuietly(Path path) { + try { + Files.deleteIfExists(path); + } catch (IOException | RuntimeException ignored) { + // best effort: the file is owner-only, and gone already where it was unlinked once open + } + } + } + + /** + * SocketSender sends each frame on its own connection to the data socket, shuts its output so a receiver + * reading to the end of the connection is never left waiting, and waits for OK. + */ + static final class SocketSender implements FrameSender { + private final File socket; + + SocketSender(File socket) { + this.socket = socket; + } + + @Override + public void send(byte[] frame) throws IOException { + send(frame, 0, frame.length); + } + + @Override + public void send(byte[] b, int off, int len) throws IOException { + AFUNIXSocket s = connect(); + try { + s.setSoTimeout(REPLY_TIMEOUT_MILLIS); + OutputStream out = s.getOutputStream(); + out.write(b, off, len); + out.flush(); + // Through the channel: junixsocket's Socket.shutdownOutput says "not connected" after connect. + s.getChannel().shutdownOutput(); + BufferedReader in = new BufferedReader(new InputStreamReader(s.getInputStream(), + StandardCharsets.UTF_8)); + String reply = in.readLine(); + if (!"OK".equals(reply)) { + throw new IOException(reply == null ? "Keploy closed the connection without OK" + : "Keploy answered " + reply + ", not OK"); + } + } finally { + closeQuietly(s); + } + } + + private AFUNIXSocket connect() throws IOException { + IOException lastErr = null; + for (int attempt = 0; attempt < CONNECT_ATTEMPTS; attempt++) { + if (attempt > 0) { + sleep(1000L * attempt); + } + AFUNIXSocket s = AFUNIXSocket.newInstance(); + try { + s.connect(AFUNIXSocketAddress.of(socket), CONNECT_TIMEOUT_MILLIS); + return s; + } catch (IOException e) { + lastErr = e; + closeQuietly(s); + } + } + throw new IOException("no Keploy listens on the data socket " + socket + " (" + lastErr + ")", + lastErr); + } + + private static void sleep(long millis) { + try { + Thread.sleep(millis); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + } + + private static void closeQuietly(AFUNIXSocket s) { + try { + s.close(); + } catch (IOException ignored) { + // nothing to do + } + } + } +} diff --git a/keploy-sdk/src/main/java/io/keploy/dedup/IndexReport.java b/keploy-sdk/src/main/java/io/keploy/dedup/IndexReport.java index 84095e49..8ae26b00 100644 --- a/keploy-sdk/src/main/java/io/keploy/dedup/IndexReport.java +++ b/keploy-sdk/src/main/java/io/keploy/dedup/IndexReport.java @@ -11,6 +11,8 @@ import java.util.Collection; import java.util.List; import java.util.TreeSet; +import java.util.logging.Level; +import java.util.logging.Logger; /** * IndexReport is the INDEX record: what the SDK took as the app's classes, @@ -53,7 +55,8 @@ final class IndexReport { // Serialized in this order, by name. final String kind = KIND; final int v = VERSION; - // "jv1-" + 32 hex (buildId), or "" when the index holds no class. + // "jv1-" + 32 hex (buildId: the class ids, the table format and the analyzer's JaCoCo), or "" when the + // index holds no class. final String build; final String source; // How the JVM was launched, or why the app's classes could not be told @@ -143,16 +146,87 @@ static String hexId(long id) { } /** - * buildId is the build's id from its classes' JaCoCo ids: "jv1-" and the - * first 32 hex of the sha256 over the ids sorted as unsigned numbers, - * de-duplicated, each as 16 lowercase hex, joined by newlines. Keploy's - * codecoverage.JavaBuildID computes the same from the same ids. "" when - * there is no id. + * buildId is the build's id: "jv1-" and the first 32 lowercase hex of the sha256 over the ASCII bytes + * {@code "keploy-java-table " F "\n" "jacoco " V "\n"} and the ids as {@link #classIdsSum} joins them, where + * F is the build table's format ({@link BuildTable#TABLE_FORMAT}) and V the JaCoCo release the SDK's analyzer + * runs ({@link Analyzer#JACOCO}); "" when there is no id. Keploy's codecoverage.JavaBuildID computes the same + * from the same bytes. It is the one build id: the INDEX record and the build table (BuildTable.write) both + * take it over the id of every class the index holds, read or not, so Keploy finds the table under the build + * the INDEX record named. An SDK whose table or probe map can differ for the same class bytes (another F or + * V) names another build, so Keploy never holds its table against one an older SDK saved for that build. */ static String buildId(long[] ids) { + return buildId(ids, BuildTable.TABLE_FORMAT, Analyzer.JACOCO); + } + + /** buildId is the build id of the ids for a table of format F and an analyzer of JaCoCo release V. */ + static String buildId(long[] ids, int format, String jacoco) { if (ids.length == 0) { return ""; } + return "jv1-" + sha256("keploy-java-table " + format + "\njacoco " + jacoco + "\n", joinedIds(ids)) + .substring(0, 32); + } + + /** + * Analyzer holds V of the build id apart from IndexReport, so reading it (JaCoCo's class init and its + * version resource) is paid by the first build id alone, never by the class-id helpers a testcase's END + * runs before its ACK, and a JaCoCo core that cannot load leaves those helpers working. + */ + static final class Analyzer { + /** + * JACOCO is the release of the JaCoCo core whose analyzer builds the table (the one keploy-sdk.jar + * carries renamed, read from its own JaCoCo.VERSION less the build qualifier, as "0.8.15"), not the + * measuring agent's: an attached JaCoCo agent of another version only records the probes. "" if that + * core cannot say (its version resource is missing), so the INDEX record and the table still name one + * build; the release smoke test fails a jar whose analyzer cannot say. + */ + static final String JACOCO = analyzerJacoco(); + + private Analyzer() { + } + + private static String analyzerJacoco() { + try { + return release(org.jacoco.core.JaCoCo.VERSION); + } catch (Throwable t) { + // JaCoCo reads its version from a resource bundle when the class loads (MissingResourceException, + // ExceptionInInitializerError): a jar without it still names one build in INDEX and TABLE. + // Logged here, not through BuildTable, whose link needs the JaCoCo core that may be missing. + try { + Logger.getLogger(IndexReport.class.getName()).log(Level.INFO, "Keploy coverage: keploy-sdk" + + " cannot read its JaCoCo core's version, so the build id names none", t); + } catch (Throwable ignored) { + // Nowhere left to say it: the log is what failed. + } + return ""; + } + } + } + + /** release is a JaCoCo version less its build qualifier: its first three dot-separated parts. */ + static String release(String version) { + int dots = 0; + for (int i = 0; i < version.length(); i++) { + if (version.charAt(i) == '.' && ++dots == 3) { + return version.substring(0, i); + } + } + return version; + } + + /** + * classIdsSum is the sha256, as 64 lowercase hex, over the ids sorted as + * unsigned numbers, de-duplicated, each as 16 lowercase hex, joined by + * newlines: a build table unit's hash, over that unit's class ids. + */ + static String classIdsSum(long[] ids) { + return sha256("", joinedIds(ids)); + } + + // joinedIds is the ids sorted as unsigned numbers, de-duplicated, each as 16 lowercase hex, joined by + // newlines, with none after the last. + private static String joinedIds(long[] ids) { long[] sorted = Arrays.copyOf(ids, ids.length); // Unsigned order: flipping the sign bit maps it onto signed order. for (int i = 0; i < sorted.length; i++) { @@ -169,18 +243,30 @@ static String buildId(long[] ids) { } joined.append(hexId(sorted[i] ^ Long.MIN_VALUE)); } - byte[] sum; + return joined.toString(); + } + + // sha256 is the sha256, as 64 lowercase hex, over the ASCII bytes of prefix and then of ids, taken in + // two parts so the ids (about 17 bytes a class) are not copied once more behind the prefix. + private static String sha256(String prefix, String ids) { + MessageDigest digest; try { - sum = MessageDigest.getInstance("SHA-256").digest(joined.toString().getBytes(StandardCharsets.US_ASCII)); + digest = MessageDigest.getInstance("SHA-256"); } catch (NoSuchAlgorithmException e) { // Every Java platform has SHA-256. throw new IllegalStateException(e); } - StringBuilder id = new StringBuilder("jv1-"); - for (int i = 0; i < 16; i++) { - id.append(Character.forDigit((sum[i] >> 4) & 0xf, 16)).append(Character.forDigit(sum[i] & 0xf, 16)); + digest.update(prefix.getBytes(StandardCharsets.US_ASCII)); + return hex(digest.digest(ids.getBytes(StandardCharsets.US_ASCII))); + } + + /** hex is b as lowercase hex, two digits a byte: the build id's and a TABLE frame's sha256. */ + static String hex(byte[] b) { + StringBuilder hex = new StringBuilder(b.length * 2); + for (byte x : b) { + hex.append(Character.forDigit((x >> 4) & 0xf, 16)).append(Character.forDigit(x & 0xf, 16)); } - return id.toString(); + return hex.toString(); } /** diff --git a/keploy-sdk/src/main/java/io/keploy/dedup/KeployDedupAgent.java b/keploy-sdk/src/main/java/io/keploy/dedup/KeployDedupAgent.java index 0e13ce83..078d99ba 100644 --- a/keploy-sdk/src/main/java/io/keploy/dedup/KeployDedupAgent.java +++ b/keploy-sdk/src/main/java/io/keploy/dedup/KeployDedupAgent.java @@ -807,6 +807,23 @@ private void handle(Socket socket) { return; } + if (BuildTable.COMMAND.equals(line.trim())) { // the build table: ACK, then sent on its own thread + String reply; // or IGNORED, with reason=low_heap when the heap is short + try { + CoverageIndex index = collector.coverageIndex; // its count only if read: no load here + reply = BuildTable.answer(() -> { + List read = index.entries; + return read == null ? -1 : read.size(); + }, + () -> BuildTable.files(index.entries(), e -> e.className, e -> e.bytes), + new File(dataSocketPath)); + } catch (Throwable t) { // thrown past here, it would end the accept loop: IGNORED instead + log(Level.WARNING, "Keploy coverage: the build table was not started", t); + reply = UNIX_REPLY_IGNORED; + } + writeReply(commandSocket.getOutputStream(), reply); + return; + } boolean taken; try { taken = run(line.trim()); diff --git a/keploy-sdk/src/main/java/org/jacoco/core/internal/analysis/KeployBranchCoverage.java b/keploy-sdk/src/main/java/org/jacoco/core/internal/analysis/KeployBranchCoverage.java new file mode 100644 index 00000000..e466f6cb --- /dev/null +++ b/keploy-sdk/src/main/java/org/jacoco/core/internal/analysis/KeployBranchCoverage.java @@ -0,0 +1,800 @@ +package org.jacoco.core.internal.analysis; + +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collection; +import java.util.HashMap; +import java.util.HashSet; +import java.util.IdentityHashMap; +import java.util.List; +import java.util.Map; +import java.util.Set; +import java.util.function.Consumer; + +import org.jacoco.core.analysis.ICounter; +import org.jacoco.core.analysis.ILine; +import org.jacoco.core.analysis.IMethodCoverage; +import org.jacoco.core.internal.analysis.filter.Filters; +import org.jacoco.core.internal.analysis.filter.IFilter; +import org.jacoco.core.internal.analysis.filter.IFilterOutput; +import org.jacoco.core.internal.analysis.filter.Replacements; +import org.jacoco.core.internal.flow.ClassProbesAdapter; +import org.jacoco.core.internal.flow.MethodProbesVisitor; +import org.jacoco.core.internal.instr.InstrSupport; +import org.objectweb.asm.ClassReader; +import org.objectweb.asm.Label; +import org.objectweb.asm.MethodVisitor; +import org.objectweb.asm.Opcodes; +import org.objectweb.asm.tree.AbstractInsnNode; +import org.objectweb.asm.tree.InsnList; +import org.objectweb.asm.tree.MethodNode; + +/** + * KeployBranchCoverage runs JaCoCo's own analysis of one class (ClassAnalyzer's steps for each method: the + * instruction graph, the filters, the merges and the branch replacements) and keeps what JaCoCo's public + * coverage API sums away: which counted instructions are covered and which branch of each one is. Keploy's + * build table makes each class's methods from the coverage this pass computes ({@link Method#coverage}) and maps + * each probe to the lines and branch edges that a hit of that probe covers, from the same pass: a class is + * analyzed once. + * + *

It lives in JaCoCo's package so that it runs JaCoCo's package-private analysis classes as they are. Each + * method's per-line sums are checked against the coverage JaCoCo computes for it in the same pass, so a + * mismatch throws instead of mapping a probe wrong. What one probe covers is read off that same pass (the + * edges JaCoCo's builder wired and the filters' output), as {@link #analyze(byte[], Map, EachProbe)} says. + */ +public final class KeployBranchCoverage { + + private KeployBranchCoverage() { + } + + /** One method JaCoCo counts (it has code), with its counted instructions in code order. */ + public static final class Method { + public final String name; + public final String desc; + /** The method's probe ids, in the order JaCoCo met them. */ + public int[] probes = new int[0]; + private int probeCount; + /** Per counted instruction: its line (-1 when it has none). */ + public int[] lines; + /** Per counted instruction: whether it is covered. */ + public boolean[] covered; + /** Per counted instruction: whether each of its branches is covered (empty below two branches). */ + public boolean[][] branches; + /** + * JaCoCo's coverage of the method from this pass, computed as ClassAnalyzer computes the coverage it + * adds to its class (a MethodCoverageCalculator over the same instructions and filter output), with + * the probes analyze was given: with none executed, what JaCoCo's Analyzer gives with no execution data. + */ + public IMethodCoverage coverage; + + Method(String name, String desc) { + this.name = name; + this.desc = desc; + } + + void addProbe(int id) { + if (probeCount == probes.length) { + int[] grown = new int[Math.max(4, probeCount * 2)]; + System.arraycopy(probes, 0, grown, 0, probeCount); + probes = grown; + } + probes[probeCount++] = id; + } + } + + /** + * analyze runs JaCoCo's analysis over the class bytes with the given probes (null: none executed) and puts + * each method that has code into out, keyed by name + descriptor. It returns + * the class's probe count, or -1 for a class JaCoCo's Analyzer does not analyze (module-info, + * package-info). It throws when the bytes are not a class JaCoCo reads, or when a method's per-line sums + * differ from JaCoCo's coverage of it. + */ + public static int analyze(byte[] bytes, boolean[] probes, Map out) { + return analyze(bytes, probes, out, null); + } + + /** + * NotMapped is a class file whose probes cannot be mapped exactly from this pass: a method's flow has a + * shape the walk does not map, its per-instruction coverage differs from JaCoCo's per-line sums, a check + * of the caller's fails, or Keploy's own steps of the pass fail (a bug of Keploy's, here as NotMapped with + * its cause). The caller marks the class not measured; it is not JaCoCo failing to read the class. + */ + public static final class NotMapped extends IllegalStateException { + private static final long serialVersionUID = 1L; + + public NotMapped(String message) { + super(message); + } + + public NotMapped(String message, Throwable cause) { + super(message, cause); + } + } + + /** Callers carries what a callback of EachProbe threw out of Keploy's steps, so it passes as it is. */ + private static final class Callers extends RuntimeException { + private static final long serialVersionUID = 1L; + final RuntimeException thrown; + + Callers(RuntimeException thrown) { + super(null, null, false, false); + this.thrown = thrown; + } + } + + /** + * Hits is what one probe covers in its method: the method's counted instructions that a class analysis + * with only that probe executed marks covered, and which of their branches. It is valid during + * {@link EachProbe#probe} only: the arrays are the method's, reused for each of its probes. + */ + public static final class Hits { + /** How many of the method's counted instructions the probe covers. */ + public int count; + /** The first count entries: those instructions' indexes into the method's lines, in no set order. */ + public final int[] at; + /** Per counted instruction, which of its branches the probe covers; all false for one not in at. */ + public final boolean[][] branches; + /** + * The steps this probe's walk took: the instructions it walked back over, plus each branch the + * filters' replacements fed from them. It is the probe's cost, which a caller can bound: in a method + * where each probe walks back over all the probes before it (a chain of early returns), the walks + * sum to the square of its length. + */ + public int steps; + + Hits(Method m) { + at = new int[m.lines.length]; + branches = new boolean[m.lines.length][]; + for (int k = 0; k < branches.length; k++) { + branches[k] = new boolean[m.branches[k].length]; + } + } + + void clear() { + for (int j = 0; j < count; j++) { + Arrays.fill(branches[at[j]], false); + } + count = 0; + } + } + + /** What {@link #analyze(byte[], Map, EachProbe)} hands over for each probe of each method with code. */ + public interface EachProbe { + /** + * probe gets method (as analyze with no probe executed puts it in out), the index r of one of its + * probes in method.probes, and what that probe alone covers (valid during this call only). + */ + void probe(Method method, int r, Hits hits); + + /** + * wants gets each method with code once, in the class's order, as soon as its coverage + * ({@link Method#coverage}) is computed and before any of its probes, and says whether its probes are + * walked at all: a method it declines costs no walk, and probe gets none of its probes. Every method + * with code is wanted unless this is overridden. + */ + default boolean wants(Method method) { + return true; + } + } + + /** + * analyze is {@link #analyze(byte[], boolean[], Map)} with no probe executed, and it also hands each + * probe's own coverage to each, method by method, as a class analysis with only that probe executed gives + * it, from that one pass: no method is analyzed again. JaCoCo's builder marks an executed probe's + * instruction and walks back through each instruction's predecessor, the one edge into it that carries no + * probe (JaCoCo puts a probe on every edge into an instruction that more than one edge reaches), stopping + * after an instruction it marked already (a loop); its coverage calculator then merges and replaces + * branches as the filters said, reading only which instructions and branches are marked. Each probe here + * takes that walk over the edges the pass's builder wired and applies the pass's filter output to what it + * marked, so a method costs its size plus its probes' walks ({@link Hits#steps}), with no allocation per + * probe. The walks are short when the method's statements join at labels many edges reach (JaCoCo puts a + * probe on each of those edges), and quadratic in a chain of early returns, where each probe walks back + * over every check before it. A method whose + * flow does not have that shape (an instruction two edges without a probe reach, a replacement that reads + * another replaced instruction) throws. What setting a method's counted instructions and the probe walk + * throw is NotMapped; JaCoCo's steps (its instruction graph, filters and coverage calculator, with the hooks + * Keploy's builder and filter output run inside them) and each's callbacks throw as they do. + */ + public static int analyze(byte[] bytes, Map out, EachProbe each) { + return analyze(bytes, null, out, each); + } + + private static int analyze(byte[] bytes, boolean[] probes, Map out, EachProbe each) { + ClassReader reader = InstrSupport.classReaderFor(bytes); + if ((reader.getAccess() & Opcodes.ACC_MODULE) != 0 || reader.getClassName().endsWith("/package-info")) { + return -1; + } + Visitor v = new Visitor(reader.getClassName(), probes, out, each); + reader.accept(new ClassProbesAdapter(v, false), 0); + return v.total; + } + + private static final class Visitor extends ClassAnalyzer { + private final boolean[] probes; + private final Map out; + private final EachProbe each; + private final IFilter filter = Filters.all(); + int total; + + Visitor(String name, boolean[] probes, Map out, EachProbe each) { + super(new ClassCoverageImpl(name, 0, false), probes, new StringPool()); + this.probes = probes; + this.out = out; + this.each = each; + } + + @Override + public void visitTotalProbeCount(int count) { + super.visitTotalProbeCount(count); + total = count; + } + + @Override + public MethodProbesVisitor visitMethod(int access, final String name, final String desc, + final String signature, String[] exceptions) { + InstrSupport.assertNotInstrumented(name, getClassName()); + final Method m = new Method(name, desc); + final Builder builder = new Builder(probes, m, each != null); + return new MethodAnalyzer(builder) { + @Override + public void accept(MethodNode node, MethodVisitor visitor) { + builder.start(node.instructions); + super.accept(node, visitor); + collect(m, builder, node, signature); + } + }; + } + + private void collect(Method m, Builder builder, MethodNode node, String signature) { + // JaCoCo's steps, as ClassAnalyzer takes them: the filters, then the coverage calculator. + Output output = new Output(); + filter.filter(node, this, output); + MethodCoverageCalculator calculator = new MethodCoverageCalculator(builder.getInstructions()); + output.applyTo(calculator); + MethodCoverageImpl coverage = new MethodCoverageImpl(m.name, m.desc, signature); + calculator.calculate(coverage); + if (!coverage.containsCode()) { + return; + } + // Keploy's steps, under one wrap: what fails in them is NotMapped, so a bug of Keploy's is not + // blamed on JaCoCo. What each's callbacks throw is carried out in a Callers and passes as it is. + try { + fill(m, builder, node, output, coverage); + out.put(m.name + m.desc, m); + if (each != null && wants(m)) { + map(m, builder, node, output); + } + } catch (Callers e) { + throw e.thrown; + } catch (RuntimeException e) { + throw keploys(m, e); + } + } + + /** wants is each.wants(m), with what it throws carried out in a Callers. */ + private boolean wants(Method m) { + try { + return each.wants(m); + } catch (RuntimeException e) { + throw new Callers(e); + } + } + + /** keploys is e, thrown by Keploy's own steps for m, as NotMapped naming the class and method. */ + private NotMapped keploys(Method m, RuntimeException e) { + if (e instanceof NotMapped) { + return (NotMapped) e; + } + return new NotMapped(getClassName() + "#" + m.name + m.desc + ": " + e, e); + } + + /** + * map hands each of m's probes to each with what that probe alone covers, from this pass: the edges + * builder wired (each instruction's predecessor) and the filters' output. + */ + private void map(Method m, Builder builder, MethodNode node, Output output) { + if (m.probes.length == 0) { + return; + } + InsnList insns = node.instructions; + int n = insns.size(); + Map instructions = builder.getInstructions(); + // Each instruction's predecessor and the branch of it that leads here: its fall-through, or the + // jump into it. JaCoCo's builder wires both and keeps the last; here none may have two. + int[] pred = Arrays.copyOf(builder.fall, n); + int[] predBranch = new int[n]; + for (int j = 0; j < builder.jumps; j++) { + Integer t = builder.labelAt.get(builder.jumpTarget.get(j)); + if (t == null) { + throw shape(m, "a jump to a label with no instruction"); + } + if (pred[t] >= 0) { + throw shape(m, "two edges without a probe into instruction " + t); + } + if (builder.jumpFrom[j] < 0) { + throw shape(m, "a jump from no instruction"); + } + pred[t] = builder.jumpFrom[j]; + predBranch[t] = builder.jumpBranch[j]; + } + // The counted instructions, in fill's order, and each instruction's merge representative. + int[] counted = new int[n]; + int[] root = new int[n]; + Arrays.fill(counted, -1); + int c = 0; + for (AbstractInsnNode i = insns.getFirst(); i != null; i = i.getNext()) { + if (instructions.get(i) == null) { + continue; + } + root[insns.indexOf(i)] = insns.indexOf(output.representative(i)); + if (!output.ignored.contains(i)) { + counted[insns.indexOf(i)] = c++; + } + } + if (c != m.lines.length) { + throw shape(m, c + " counted instructions, " + m.lines.length + " in its coverage"); + } + // A counted instruction's branch numbers in fill's order (its row of branches), and, for each new + // branch of a replaced one, the instruction branches it reads: key root << 32 | branch, sorted, the + // new branch (counted instruction << 32 | its index) beside it. + int[][] numbers = new int[c][]; + long[] reads = new long[8]; + long[] feeds = new long[8]; + int nReads = 0; + for (AbstractInsnNode i = insns.getFirst(); i != null; i = i.getNext()) { + int at = insns.indexOf(i); + int k = counted[at]; + if (k < 0) { + continue; + } + Replacements r = output.replacements.get(i); + if (r == null) { + // fill checked that a counted instruction's branch numbers are as many as its branches. + numbers[k] = new int[m.branches[k].length]; + for (int x = 0; x < numbers[k].length; x++) { + numbers[k][x] = builder.number(at, x); + } + continue; + } + int t = 0; + for (Collection branch : r.values()) { + for (Replacements.InstructionBranch from : branch) { + if (from.instruction != i && output.replacements.containsKey(from.instruction)) { + throw shape(m, "a replaced branch reads another replaced instruction"); + } + if (instructions.get(from.instruction) == null) { + throw shape(m, "a replaced branch reads no instruction"); + } + if (nReads == reads.length) { + reads = Arrays.copyOf(reads, nReads * 2); + feeds = Arrays.copyOf(feeds, nReads * 2); + } + reads[nReads] = key(root[insns.indexOf(from.instruction)], from.branch); + feeds[nReads++] = (long) k << 32 | t; + } + t++; + } + } + sortTogether(reads, feeds, nReads); + // Per probe: the walk, by stamp (the probe's round, so nothing is cleared between probes). + int[] stamp = new int[n]; + int[] walked = new int[16]; + int[] walkedBranch = new int[16]; + int[] newStamp = new int[c]; + Hits hits = new Hits(m); + for (int r = 0; r < m.probes.length; r++) { + int round = r + 1; + int w = 0; + int i = builder.probeFrom[r]; + int b = builder.probeBranch[r]; + if (i < 0) { + throw shape(m, "a probe on no instruction"); + } + while (i >= 0) { + if (w == walked.length) { + walked = Arrays.copyOf(walked, w * 2); + walkedBranch = Arrays.copyOf(walkedBranch, w * 2); + } + walked[w] = i; + walkedBranch[w++] = b; + if (stamp[i] == round) { + break; + } + stamp[i] = round; + b = predBranch[i]; + i = pred[i]; + } + int fed = 0; + for (int x = 0; x < w; x++) { + int rt = root[walked[x]]; + int br = walkedBranch[x]; + int k = counted[rt]; + if (k >= 0 && numbers[k] != null) { + // Merged: the representative carries its group's marks. + mark(hits, newStamp, round, k); + if (numbers[k].length > 0) { + int pos = Arrays.binarySearch(numbers[k], br); + if (pos < 0) { + throw shape(m, "branch " + br + " marked, not one of its instruction's"); + } + hits.branches[k][pos] = true; + } + } + // Replaced: each new branch that reads this instruction branch. + long key = key(rt, br); + for (int y = lowerBound(reads, nReads, key); y < nReads && reads[y] == key; y++) { + int kk = (int) (feeds[y] >>> 32); + int t = (int) feeds[y]; + fed++; + mark(hits, newStamp, round, kk); + if (t < hits.branches[kk].length) { + hits.branches[kk][t] = true; + } + } + } + hits.steps = w + fed; + try { + each.probe(m, r, hits); + } catch (RuntimeException e) { + throw new Callers(e); + } + hits.clear(); + } + } + + private static long key(int instruction, int branch) { + return (long) instruction << 32 | (branch & 0xffffffffL); + } + + /** lowerBound is the first index of the n sorted keys whose key is at least key. */ + private static int lowerBound(long[] keys, int n, long key) { + int lo = 0; + int hi = n; + while (lo < hi) { + int mid = (lo + hi) >>> 1; + if (keys[mid] < key) { + lo = mid + 1; + } else { + hi = mid; + } + } + return lo; + } + + /** sortTogether sorts the first n keys ascending, moving each value with its key (a merge sort). */ + private static void sortTogether(long[] keys, long[] values, int n) { + long[] k2 = new long[n]; + long[] v2 = new long[n]; + for (int width = 1; width < n; width *= 2) { + for (int lo = 0; lo < n; lo += 2 * width) { + int mid = Math.min(lo + width, n); + int hi = Math.min(lo + 2 * width, n); + int a = lo; + int b = mid; + for (int o = lo; o < hi; o++) { + if (a < mid && (b >= hi || keys[a] <= keys[b])) { + k2[o] = keys[a]; + v2[o] = values[a++]; + } else { + k2[o] = keys[b]; + v2[o] = values[b++]; + } + } + } + System.arraycopy(k2, 0, keys, 0, n); + System.arraycopy(v2, 0, values, 0, n); + } + } + + private static void mark(Hits hits, int[] newStamp, int round, int k) { + if (newStamp[k] != round) { + newStamp[k] = round; + hits.at[hits.count++] = k; + } + } + + private NotMapped shape(Method m, String what) { + return new NotMapped(getClassName() + "#" + m.name + m.desc + ": " + what); + } + + /** + * fill sets m's counted instructions from builder's graph and the filters' output, as ClassAnalyzer + * computes a method's coverage, and checks them against coverage, the method's coverage JaCoCo + * computed from them (it has code). + */ + private void fill(Method m, Builder builder, MethodNode node, Output output, MethodCoverageImpl coverage) { + Map instructions = builder.getInstructions(); + int[] p = new int[m.probeCount]; + System.arraycopy(m.probes, 0, p, 0, m.probeCount); + m.probes = p; + InsnList insns = node.instructions; + int[] lines = new int[insns.size()]; + boolean[] covered = new boolean[insns.size()]; + boolean[][] branches = new boolean[insns.size()][]; + int c = 0; + Map sums = new HashMap<>(); // line -> instructions, covered, branches, covered + for (AbstractInsnNode i = insns.getFirst(); i != null; i = i.getNext()) { + if (output.ignored.contains(i)) { + continue; + } + final Instruction insn = instructions.get(i); + if (insn == null) { + continue; + } + boolean hit = insn.getInstructionCounter().getCoveredCount() > 0; + int n = insn.getBranchCounter().getTotalCount(); + // The instruction's branch numbers: those its builder gave it (a switch's may skip numbers), + // or 0..k-1 for the k new branches a filter replaced them with. + Integer replaced = output.replaced.get(i); + int at = insns.indexOf(i); + int count = replaced != null ? replaced : builder.numbers(at); + if (n > 0 && count != n) { + throw new NotMapped("JaCoCo counts " + n + " branches at " + getClassName() + "#" + + m.name + m.desc + " line " + insn.getLine() + ", its builder gave " + count); + } + boolean[] b = new boolean[n]; + // An instruction not covered has no covered branch (JaCoCo covers an instruction when one of + // its branches is), so only a covered one's branches are read. + for (int k = 0; hit && k < n; k++) { + // replaceBranches with one new branch fed by this branch alone is covered exactly when the + // branch is: JaCoCo's own way to read one branch. + Replacements one = new Replacements(); + one.add(i, i, replaced != null ? k : builder.number(at, k)); + b[k] = insn.replaceBranches(one, x -> insn).getInstructionCounter().getCoveredCount() > 0; + } + lines[c] = insn.getLine(); + covered[c] = hit; + branches[c++] = b; + int[] s = sums.get(insn.getLine()); + if (s == null) { + s = new int[4]; + sums.put(insn.getLine(), s); + } + s[0]++; + s[1] += hit ? 1 : 0; + s[2] += n; + for (boolean x : b) { + s[3] += x ? 1 : 0; + } + } + for (Map.Entry e : sums.entrySet()) { + int l = e.getKey(); + if (l < 1) { + continue; + } + ILine line = coverage.getLine(l); + int[] s = e.getValue(); + if (!same(line.getInstructionCounter(), s[0], s[1]) || !same(line.getBranchCounter(), s[2], s[3])) { + throw new NotMapped("JaCoCo's coverage of " + getClassName() + "#" + m.name + m.desc + + " line " + l + " differs from its instructions: " + Arrays.toString(s) + " against " + + line.getInstructionCounter() + " " + line.getBranchCounter()); + } + } + m.lines = Arrays.copyOf(lines, c); + m.covered = Arrays.copyOf(covered, c); + m.branches = Arrays.copyOf(branches, c); + m.coverage = coverage; + } + + private static boolean same(ICounter c, int total, int covered) { + return c.getTotalCount() == total && c.getCoveredCount() == covered; + } + } + + /** + * Builder is JaCoCo's InstructionsBuilder that also notes the method's probe ids and each instruction's + * branch numbers, as JaCoCo's builder gives them: 0 to the instruction an instruction falls through to, + * and the number MethodAnalyzer passes with each jump and probe. + */ + private static final class Builder extends InstructionsBuilder { + private final Method method; + private AbstractInsnNode current; + private InsnList insns; + // Each branch number the builder gave an instruction, as index << 32 | number; sorted, de-duplicated + // and indexed (noteStart: where each instruction's numbers start; one more for the end) on first read. + private long[] notes = new long[16]; + private int noteCount; + private int[] noteStart; + private Map wired; + // With flow kept (for map), by index in the method's instruction list: each instruction's fall-through + // predecessor (-1: none), each jump (from, branch, target label, in the order the builder wires them), + // the instruction each label is at, and each probe's instruction and branch (in method.probes order). + private final boolean flow; + int[] fall; + int jumps; + int[] jumpFrom = new int[0]; + int[] jumpBranch = new int[0]; + final List

heap_after_gc_max_mb is the most heap any collection during the send left in use: an upper bound of what + * the analysis keeps live (a young collection also counts old garbage not collected yet). It is -1 when no + * collection ran. heap_delta_mb is that less the heap in use before the send (heap_before_mb, after a + * collection: the classes' bytes and the test's own): what the send kept live at most. heap_peak_mb sums each + * heap pool's peak during the send (garbage included, so it follows the collector and the cap, not the live + * set). heap_max_mb is the heap cap the JVM ran with (-Xmx, or its default from the machine's RAM). + * + *

With -Dkeploy.budget.live=true each frame, as it is sent, first runs a full collection and reads the heap + * in use: live_max_mb is the most of those (what the send holds live at a frame boundary, the table's last + * frame included), live_delta_mb that less heap_before_mb. Those collections slow the send: read its times + * from a run without it. Without it both are -1. + */ +public class BuildTableBudgetTest { + + static final int BRANCHES = 15; + + @Test(timeout = 1_800_000) + public void aGeneratedAppsTableIsMeasured() throws Exception { + int n = Integer.getInteger("keploy.budget.classes", 2000); + List app = new ArrayList<>(n); + for (int i = 0; i < n; i++) { + app.add(new BuildTable.ClassFile(name(i), gen(name(i), i, BRANCHES))); + } + Budget b = measure("app-" + n, app); + assertEquals(n, b.tableRecords); + assertEquals(n, b.probeRecords); + assertEquals("every class is measured", 0, b.notMeasured); + // What the send allocates per class: about 1.9 MiB when each probe re-analyzed its method, 123-141 KiB + // when JaCoCo analyzed each class twice (once for its table, once for its probe map), and 84-96 KiB + // since one analysis gives both (and the records' text). + long perClass = b.allocated / n; + assertTrue("allocated " + perClass + " bytes a class", perClass < ALLOCATED_PER_CLASS); + // Paced by the collections, the send grows the committed heap over what it was just after a full + // collection by no more than maxCommittedGrowth, within 60 s at 50,000 classes, and its frames keep + // coming. A live run collects at each frame, which the pacing waits out: read its numbers only. + if (Boolean.getBoolean("keploy.budget.live")) { + return; + } + long max = maxCommittedGrowth(gcNames().contains("G1"), Runtime.getRuntime().maxMemory(), + System.getProperty("java.specification.version")); + assertTrue("committed heap growth " + (b.committedGrowth >> 20) + " MiB, over " + (max >> 20), + b.committedGrowth <= max); + assertTrue("took " + b.wallNanos / 1_000_000 + " ms", b.wallNanos < maxWallNanos(n)); + assertTrue("a frame came " + b.maxFrameGapNanos / 1_000_000 + " ms after the one before", + b.maxFrameGapNanos < MAX_FRAME_GAP_NANOS); + } + + /** + * maxCommittedGrowth is the most the send may add to an app pod's committed heap over the heap just after a + * full collection: 128 MiB, but more on G1 with a heap cap of 512 MiB or more before JDK 21. There the + * collection has shrunk G1's heap, its pause raises G1's pause-time ratio, and G1 grows the heap back while + * the table is sent, paced or not: at 50,000 classes in a 4 GiB pod, 300 MiB on JDK 8 (unpaced 422-707) and + * 145 MiB on JDK 17, paced by the send's own work time (one run each; earlier pacers saw 420-421 and 13-166). + * Unpaced, JDK 21's G1 there grew 175-343 MiB (paced, 59), past the bound. + */ + static long maxCommittedGrowth(boolean g1, long maxHeap, String javaSpec) { + if (g1 && maxHeap >= 512L << 20) { + if (javaSpec.startsWith("1.")) { + return 448L << 20; + } + if (Integer.parseInt(javaSpec) < 21) { + return 192L << 20; + } + } + return 128L << 20; + } + + /** The longest a frame may come after the one before (or the send's start): Keploy waits 30 s for one. */ + static final long MAX_FRAME_GAP_NANOS = 10_000_000_000L; + + /** + * maxWallNanos is the table's time bound: 60 s at 50,000 classes, at least 60 s for fewer, and 1.2 ms a class + * past 50,000. At 50,000 that is 40 s past the pacing's own deadline (BuildTable.Pacer.deadlineNanos), which + * leaves a slow or busy CPU's own work that time; past 50,000 it leaves more. + */ + static long maxWallNanos(int classes) { + return Math.max(60_000_000_000L, 60_000_000_000L / 50_000 * classes); + } + + /** The most a generated class's table and probe map may allocate on the send's thread: below two analyses'. */ + static final long ALLOCATED_PER_CLASS = 112 << 10; + + /** The busy app's allocation, in MB/s over its threads: a JVM serving a test run's requests. */ + static final int BUSY_MBPS = 400; + static final int BUSY_THREADS = 2; + static final int BUSY_ARRAY = 64 << 10; + /** How long the app's load runs before Keploy asks for the table. */ + static final long BUSY_SETTLE_MS = 2_000; + /** What the waits may run past the send's own work: one sleep's overrun (a collection's pause in it). */ + static final long BUSY_OVERRUN_MS = 500; + /** The send's INFO line: its classes, those not measured, its time, its CPU and its waits. */ + static final Pattern SENT = Pattern.compile("sent \\((\\d+) classes, (\\d+) not measured.*; \\d+ frames," + + " (\\d+) ms, cpu -?\\d+ ms, paced (\\d+) ms\\)$"); + + /** + * A busy app's table, the native keploy test's case: app threads allocate BUSY_MBPS of short-lived garbage + * while the table is sent, so the JVM collects often for the app's own sake, and the pacing (which reads + * every collection, the app's too) is owed far more than it may wait. Its waits must stay within the send's + * own work time (one sleep's overrun past it at most), so a busy app's table takes at most about twice as + * long as unpaced: before, they took the whole allowance of 0.8 ms a class (40 s at 50,000), and the app's + * test set ended before its table was sent. The class count is -Dkeploy.budget.classes (5,000 by default). + */ + @Test(timeout = 900_000) + public void aBusyAppsTableIsPacedAtMostItsOwnWork() throws Exception { + int n = Integer.getInteger("keploy.budget.classes", 5000); + List app = new ArrayList<>(n); + for (int i = 0; i < n; i++) { + app.add(new BuildTable.ClassFile(name(i), gen(name(i), i, BRANCHES))); + } + AtomicBoolean stop = new AtomicBoolean(); + AtomicLong appBytes = new AtomicLong(); + // Each thread keeps a ring of its newest arrays, a churning retained set as request handling has: 64 MiB + // a thread, and at most 1/16 of the heap's cap in all. + int slots = (int) Math.min(1024, Runtime.getRuntime().maxMemory() / 16 / BUSY_THREADS / BUSY_ARRAY); + // The load's threads and the sampler: each ends at stop, and finally stops and joins them all. + List threads = new ArrayList<>(); + try (BuildTableTest.Logs logs = new BuildTableTest.Logs()) { + for (int t = 0; t < BUSY_THREADS; t++) { + Thread th = new Thread(() -> allocate(stop, appBytes, slots), "busy-app-" + t); + th.setDaemon(true); + th.start(); + threads.add(th); + } + Thread.sleep(BUSY_SETTLE_MS); + long committed0 = heapCommitted(); + AtomicLong committedMax = new AtomicLong(committed0); + AtomicBoolean sending = new AtomicBoolean(true); + Thread sampler = new Thread(() -> { + while (sending.get() && !stop.get()) { + committedMax.accumulateAndGet(heapCommitted(), Math::max); + try { + Thread.sleep(50); + } catch (InterruptedException e) { + return; + } + } + }, "busy-committed"); + sampler.setDaemon(true); + sampler.start(); + threads.add(sampler); + long gc0 = gcMillis(); + long app0 = appBytes.get(); + long t0 = System.nanoTime(); + long[] frames = {t0, 0}; // the last frame's time, the longest gap + BuildTable.send(() -> app, new BuildTable.FrameSender() { + @Override + public void send(byte[] frame) { + send(frame, 0, frame.length); + } + + @Override + public void send(byte[] frame, int off, int len) { + long now = System.nanoTime(); + frames[1] = Math.max(frames[1], now - frames[0]); + frames[0] = now; + } + }); + long wall = System.nanoTime() - t0; + long gc = gcMillis() - gc0; + sending.set(false); + sampler.join(); + String sent = null; + for (LogRecord r : logs.mine()) { + if (r.getMessage() != null && r.getMessage().contains(" sent (")) { + sent = r.getMessage(); + } + } + assertTrue("the send's INFO line", sent != null); + Matcher m = SENT.matcher(sent); + assertTrue(sent, m.find()); + long sendMs = Long.parseLong(m.group(3)); + long pacedMs = Long.parseLong(m.group(4)); + long workMs = sendMs - pacedMs; + System.out.printf(Locale.ROOT, "BUSY app-%d java=%s gc=%s heap_max_mb=%d app_mbps=%d wall_ms=%d" + + " send_ms=%d paced_ms=%d work_ms=%d gc_ms=%d gc_pct=%.2f max_frame_gap_ms=%d" + + " committed_start_mb=%d committed_max_mb=%d committed_growth_mb=%d app_mb=%d%n", + n, System.getProperty("java.version"), gcNames(), Runtime.getRuntime().maxMemory() >> 20, + BUSY_MBPS, wall / 1_000_000, sendMs, pacedMs, workMs, gc, + 100.0 * gc / Math.max(1, wall / 1_000_000), frames[1] / 1_000_000, committed0 >> 20, + committedMax.get() >> 20, (committedMax.get() - committed0) >> 20, + (appBytes.get() - app0) >> 20); + assertEquals(sent, n, Integer.parseInt(m.group(1))); + assertEquals(sent, 0, Integer.parseInt(m.group(2))); + assertTrue("paced " + pacedMs + " ms, past the send's own " + workMs + " ms of work", + pacedMs <= workMs + BUSY_OVERRUN_MS); + assertTrue("took " + wall / 1_000_000 + " ms", wall < maxWallNanos(n)); + assertTrue("a frame came " + frames[1] / 1_000_000 + " ms after the one before", + frames[1] < MAX_FRAME_GAP_NANOS); + } finally { + stop.set(true); + for (Thread th : threads) { + th.join(); + } + } + } + + /** allocate is a busy app's thread: BUSY_MBPS / BUSY_THREADS of BUSY_ARRAY arrays, a ring of slots kept. */ + private static void allocate(AtomicBoolean stop, AtomicLong appBytes, int slots) { + long perMs = (long) BUSY_MBPS * 1_000_000 / 1000 / BUSY_THREADS; + byte[][] ring = new byte[Math.max(1, slots)][]; + long t0 = System.nanoTime(); + long made = 0; + int k = 0; + while (!stop.get()) { + for (int j = 0; j < 16; j++) { + byte[] b = new byte[BUSY_ARRAY]; + b[0] = (byte) k; + ring[k] = b; + k = k + 1 == ring.length ? 0 : k + 1; + } + made += 16L * BUSY_ARRAY; + appBytes.addAndGet(16L * BUSY_ARRAY); + long ahead = made / perMs - (System.nanoTime() - t0) / 1_000_000; + if (ahead > 0) { + try { + Thread.sleep(ahead); + } catch (InterruptedException e) { + return; + } + } + } + } + + @Test(timeout = 600_000) + public void aClassWithOneLongMethodIsMeasured() throws Exception { + int branches = Integer.getInteger("keploy.budget.branches", 500); + List one = new ArrayList<>(); + one.add(new BuildTable.ClassFile("gen/big/Long", gen("gen/big/Long", 0, branches))); + Budget b = measure("one-method-" + branches + "-branches", one); + assertEquals(1, b.probeRecords); + assertEquals(0, b.notMeasured); + } + + @Test(timeout = 600_000) + public void aMethodOf3000BranchesMapsInUnderASecond() throws Exception { + // An if/else chain's statements join at labels two edges reach, so its probes' walks are short. When + // each probe re-analyzed its method, 3,500 took about a minute, past the CLI's 30 s wait for a frame, + // so one such class left every run's table incomplete. + List one = new ArrayList<>(); + one.add(new BuildTable.ClassFile("gen/big/Long3000", gen("gen/big/Long3000", 0, 3000))); + Budget b = measure("one-method-3000-branches", one); + assertEquals(1, b.probeRecords); + assertEquals(0, b.notMeasured); + assertTrue("the send took " + (b.cpuNanos / 1_000_000) + " ms of CPU", b.cpuNanos < 1_000_000_000L); + } + + @Test(timeout = 600_000) + public void aChainOfEarlyReturnsIsMappedOrTooLargeAndNeverUnbounded() throws Exception { + // Probe k of a chain of n early returns walks back over the k checks before it and covers each one's + // line and branch, so the map is about n * n ranges: 4.4 MB at 1,000 and 43 MB at 3,000; 4,500 ran a + // 256 MiB heap out of memory (and 3,000 a 128 MiB one) before the map had a bound. Past it the class + // is record_too_large, given up as its map grows past the bound. + for (int n : new int[] {1000, 3000, 4500}) { + String name = "gen/big/Returns" + n; + List one = new ArrayList<>(); + one.add(new BuildTable.ClassFile(name, returns(name, n))); + Budget b = measure("early-returns-" + n, one); + boolean mapped = n == 1000; + assertEquals(1, b.tableRecords); + assertEquals(name, mapped ? 1 : 0, b.probeRecords); + assertEquals(name, mapped ? 0 : 1, b.notMeasured); + assertEquals(name, mapped ? 0 : 1, b.tooLarge); + assertTrue(name + ": the send took " + (b.cpuNanos / 1_000_000) + " ms of CPU", + b.cpuNanos < 1_000_000_000L); + assertTrue(name + ": the send allocated " + (b.allocated >> 20) + " MiB", + b.allocated < MAP_ALLOCATED); + } + } + + /** The most one class's table and probe map may allocate, its map at the bound: 8 MiB of text. */ + static final long MAP_ALLOCATED = 64 << 20; + + @Test(timeout = 600_000) + public void aRealLibraryIsMeasured() throws Exception { + File jar = new File(Analyzer.class.getProtectionDomain().getCodeSource().getLocation().toURI()); + List lib = ProbeMapTest.jarClasses(jar); + Budget b = measure("jacoco-core-jar", lib); + assertTrue(b.tableRecords > 100); + assertEquals(0, b.notMeasured); + } + + static String name(int i) { + return String.format(Locale.ROOT, "gen/app/p%03d/C%05d", i / 100, i); + } + + /** gen is a class with a static run(int) of branches if/else pairs, a line each (about 2*branches+1 probes). */ + static byte[] gen(String name, int i, int branches) { + ClassWriter cw = new ClassWriter(ClassWriter.COMPUTE_FRAMES | ClassWriter.COMPUTE_MAXS); + cw.visit(Opcodes.V1_8, Opcodes.ACC_PUBLIC | Opcodes.ACC_SUPER, name, null, "java/lang/Object", null); + cw.visitSource(name.substring(name.lastIndexOf('/') + 1) + ".java", null); + MethodVisitor init = cw.visitMethod(Opcodes.ACC_PUBLIC, "", "()V", null, null); + init.visitCode(); + Label first = new Label(); + init.visitLabel(first); + init.visitLineNumber(5, first); + init.visitVarInsn(Opcodes.ALOAD, 0); + init.visitMethodInsn(Opcodes.INVOKESPECIAL, "java/lang/Object", "", "()V", false); + init.visitInsn(Opcodes.RETURN); + init.visitMaxs(0, 0); + init.visitEnd(); + MethodVisitor mv = cw.visitMethod(Opcodes.ACC_PUBLIC | Opcodes.ACC_STATIC, "run", "(I)I", null, null); + mv.visitCode(); + mv.visitInsn(Opcodes.ICONST_0); + mv.visitVarInsn(Opcodes.ISTORE, 1); + for (int b = 0; b < branches; b++) { + Label line = new Label(); + mv.visitLabel(line); + mv.visitLineNumber(10 + b, line); + Label orElse = new Label(); + Label next = new Label(); + mv.visitVarInsn(Opcodes.ILOAD, 0); + mv.visitIntInsn(Opcodes.SIPUSH, (b * 7 + i) % 64); + mv.visitJumpInsn(Opcodes.IF_ICMPLE, orElse); + mv.visitIincInsn(1, b % 100 + 1); + mv.visitJumpInsn(Opcodes.GOTO, next); + mv.visitLabel(orElse); + mv.visitIincInsn(1, -1); + mv.visitLabel(next); + } + mv.visitVarInsn(Opcodes.ILOAD, 1); + mv.visitInsn(Opcodes.IRETURN); + mv.visitMaxs(0, 0); + mv.visitEnd(); + cw.visitEnd(); + return cw.toByteArray(); + } + + /** Matches counts an ASCII text over a stream of byte chunks, a match split over chunks included. */ + static final class Matches { + private final byte[] what; + // The stream's last what.length - 1 bytes (fewer at its start), where a split match begins. + private final byte[] tail; + private int tailLen; + int n; + + Matches(String what) { + this.what = what.getBytes(StandardCharsets.US_ASCII); + this.tail = new byte[this.what.length - 1]; + } + + void add(byte[] b, int from, int to) { + // Matches that begin in the tail and end in this chunk. + for (int i = 0; i < tailLen; i++) { + int inTail = tailLen - i; + if (inTail + (to - from) >= what.length && same(tail, i, 0, inTail) + && same(b, from, inTail, what.length - inTail)) { + n++; + } + } + for (int i = from; i + what.length <= to; i++) { + if (same(b, i, 0, what.length)) { + n++; + } + } + int len = to - from; + if (len >= tail.length) { + System.arraycopy(b, to - tail.length, tail, 0, tail.length); + tailLen = tail.length; + } else { + int keep = Math.min(tailLen, tail.length - len); + System.arraycopy(tail, tailLen - keep, tail, 0, keep); + System.arraycopy(b, from, tail, keep, len); + tailLen = keep + len; + } + } + + // same says whether b[at, at + len) is what[whatAt, whatAt + len). + private boolean same(byte[] b, int at, int whatAt, int len) { + for (int k = 0; k < len; k++) { + if (b[at + k] != what[whatAt + k]) { + return false; + } + } + return true; + } + } + + /** returns is a class with a static run(int) of n lines "if (x == k) return k;", then "return -1;". */ + static byte[] returns(String name, int n) { + ClassWriter cw = new ClassWriter(ClassWriter.COMPUTE_FRAMES | ClassWriter.COMPUTE_MAXS); + cw.visit(Opcodes.V1_8, Opcodes.ACC_PUBLIC | Opcodes.ACC_SUPER, name, null, "java/lang/Object", null); + cw.visitSource(name.substring(name.lastIndexOf('/') + 1) + ".java", null); + MethodVisitor mv = cw.visitMethod(Opcodes.ACC_PUBLIC | Opcodes.ACC_STATIC, "run", "(I)I", null, null); + mv.visitCode(); + for (int k = 0; k < n; k++) { + Label line = new Label(); + mv.visitLabel(line); + mv.visitLineNumber(10 + k, line); + Label next = new Label(); + mv.visitVarInsn(Opcodes.ILOAD, 0); + mv.visitIntInsn(Opcodes.SIPUSH, k); + mv.visitJumpInsn(Opcodes.IF_ICMPNE, next); + mv.visitIntInsn(Opcodes.SIPUSH, k); + mv.visitInsn(Opcodes.IRETURN); + mv.visitLabel(next); + } + Label end = new Label(); + mv.visitLabel(end); + mv.visitLineNumber(10 + n, end); + mv.visitInsn(Opcodes.ICONST_M1); + mv.visitInsn(Opcodes.IRETURN); + mv.visitMaxs(0, 0); + mv.visitEnd(); + cw.visitEnd(); + return cw.toByteArray(); + } + + static final class Budget { + int tableRecords; + int probeRecords; + int notMeasured; + /** How many of the table's records are record_too_large. */ + int tooLarge; + /** Bytes the send allocated on its thread (the sink's own allocation left out). */ + long allocated; + /** The send thread's CPU time, in nanoseconds. */ + long cpuNanos; + /** The send's wall time, in nanoseconds. */ + long wallNanos; + /** The most the heap's committed size grew over what it was just before the send (after a collection). */ + long committedGrowth; + /** The longest time between two frames, or the send's start and its first frame. */ + long maxFrameGapNanos; + } + + /** measure sends the table of classes to a sink that counts, as the table's thread does, and reports. */ + static Budget measure(String label, List classes) throws Exception { + Set heapPools = new HashSet<>(); + for (MemoryPoolMXBean p : ManagementFactory.getMemoryPoolMXBeans()) { + if (p.getType() == MemoryType.HEAP) { + heapPools.add(p.getName()); + } + } + AtomicLong afterGcMax = new AtomicLong(-1); + AtomicLong committedGcMax = new AtomicLong(-1); + AtomicInteger collections = new AtomicInteger(); + NotificationListener listener = (n, handback) -> { + if (!GarbageCollectionNotificationInfo.GARBAGE_COLLECTION_NOTIFICATION.equals(n.getType())) { + return; + } + GarbageCollectionNotificationInfo info = GarbageCollectionNotificationInfo.from( + (CompositeData) n.getUserData()); + if ("System.gc()".equals(info.getGcCause())) { + // This method's own collection before the send, whose notification can come late. + return; + } + long used = 0; + for (Map.Entry e : info.getGcInfo().getMemoryUsageAfterGc().entrySet()) { + if (heapPools.contains(e.getKey())) { + used += e.getValue().getUsed(); + } + } + long before = 0; + long after = 0; + for (Map.Entry e : info.getGcInfo().getMemoryUsageBeforeGc().entrySet()) { + if (heapPools.contains(e.getKey())) { + before += e.getValue().getCommitted(); + } + } + for (Map.Entry e : info.getGcInfo().getMemoryUsageAfterGc().entrySet()) { + if (heapPools.contains(e.getKey())) { + after += e.getValue().getCommitted(); + } + } + committedGcMax.accumulateAndGet(Math.max(before, after), Math::max); + collections.incrementAndGet(); + afterGcMax.accumulateAndGet(used, Math::max); + }; + List emitters = new ArrayList<>(); + for (GarbageCollectorMXBean g : ManagementFactory.getGarbageCollectorMXBeans()) { + if (g instanceof NotificationEmitter) { + ((NotificationEmitter) g).addNotificationListener(listener, null, null); + emitters.add((NotificationEmitter) g); + } + } + try { + return measure(label, classes, heapPools, afterGcMax, committedGcMax, collections); + } finally { + for (NotificationEmitter e : emitters) { + e.removeNotificationListener(listener); + } + } + } + + private static Budget measure(String label, List classes, Set heapPools, + AtomicLong afterGcMax, AtomicLong committedGcMax, AtomicInteger collections) throws Exception { + long committedPre = heapCommitted(); + System.gc(); + long committedBase = heapCommitted(); + AtomicLong committedSampled = new AtomicLong(committedBase); + java.util.concurrent.atomic.AtomicBoolean sending = new java.util.concurrent.atomic.AtomicBoolean(true); + Thread sampler = new Thread(() -> { + while (sending.get()) { + committedSampled.accumulateAndGet(heapCommitted(), Math::max); + try { + Thread.sleep(2); + } catch (InterruptedException e) { + return; + } + } + }, "budget-committed"); + sampler.setDaemon(true); + long base = 0; + for (MemoryPoolMXBean p : ManagementFactory.getMemoryPoolMXBeans()) { + if (heapPools.contains(p.getName())) { + base += p.getUsage().getUsed(); + } + } + long classBytes = 0; + for (BuildTable.ClassFile f : classes) { + classBytes += f.bytes.length; + } + ThreadMXBean threads = ManagementFactory.getThreadMXBean(); + for (MemoryPoolMXBean p : ManagementFactory.getMemoryPoolMXBeans()) { + if (heapPools.contains(p.getName())) { + p.resetPeakUsage(); + } + } + long[] bytes = new long[2]; + boolean live = Boolean.getBoolean("keploy.budget.live"); + long[] liveMax = {-1}; + // A record may run on into the next frame: not_measured and record_too_large are counted over the + // table's stream, a match split over two frames included. The sink reads each frame where the writer + // keeps it, as the socket's does: a copy or a String of a 1 MiB frame is a G1 humongous object, which + // grows the heap this row measures. + Matches notMeasured = new Matches("\"state\":\"not_measured\""); + Matches tooLarge = new Matches("\"reason\":{\"code\":\"record_too_large\"}"); + Budget b = new Budget(); + long gc0 = gcMillis(); + long process0 = processCpuNanos(); + com.sun.management.ThreadMXBean allocation = (com.sun.management.ThreadMXBean) threads; + long self = Thread.currentThread().getId(); + long[] sinkAllocated = {0}; + long allocated0 = allocation.getThreadAllocatedBytes(self); + long cpu0 = threads.getCurrentThreadCpuTime(); + long t0 = System.nanoTime(); + long[] lastFrame = {t0, 0}; + sampler.start(); + BuildTable.send(() -> classes, new BuildTable.FrameSender() { + @Override + public void send(byte[] frame) throws java.io.IOException { + send(frame, 0, frame.length); + } + + @Override + public void send(byte[] frame, int off, int len) { + long sink0 = allocation.getThreadAllocatedBytes(self); + long at0 = System.nanoTime(); + lastFrame[1] = Math.max(lastFrame[1], at0 - lastFrame[0]); + if (live) { + System.gc(); + long used = 0; + for (MemoryPoolMXBean p : ManagementFactory.getMemoryPoolMXBeans()) { + if (heapPools.contains(p.getName())) { + used += p.getUsage().getUsed(); + } + } + liveMax[0] = Math.max(liveMax[0], used); + } + // The header is the line after the 8 bytes of magic and length. + int nl = off + 8; + while (frame[nl] != '\n') { + nl++; + } + String header = new String(frame, off + 8, nl - off - 8, StandardCharsets.UTF_8); + boolean probes = header.contains("\"part\":\"probes\""); + bytes[probes ? 1 : 0] += len; + if (header.contains("\"last\":true")) { + int records = Integer.parseInt(header.replaceAll(".*\"records\":(\\d+).*", "$1")); + if (probes) { + b.probeRecords = records; + } else { + b.tableRecords = records; + } + } + if (!probes) { + notMeasured.add(frame, nl + 1, off + len); + tooLarge.add(frame, nl + 1, off + len); + } + sinkAllocated[0] += allocation.getThreadAllocatedBytes(self) - sink0; + // The sink's own time (a live run's collection) is not the send's gap. + lastFrame[0] = System.nanoTime(); + } + }); + b.notMeasured = notMeasured.n; + b.tooLarge = tooLarge.n; + long wall = System.nanoTime() - t0; + long allocated = allocation.getThreadAllocatedBytes(self) - allocated0 - sinkAllocated[0]; + sending.set(false); + sampler.join(); + long committedEnd = heapCommitted(); + long cpu = threads.getCurrentThreadCpuTime() - cpu0; + long process = processCpuNanos() - process0; + long gc = gcMillis() - gc0; + long peak = 0; + for (MemoryPoolMXBean p : ManagementFactory.getMemoryPoolMXBeans()) { + if (heapPools.contains(p.getName())) { + peak += p.getPeakUsage().getUsed(); + } + } + // The JVM sends a collection's notification after it ends, from a thread of its own. + Thread.sleep(200); + long afterGc = afterGcMax.get(); + System.out.printf(Locale.ROOT, "BUDGET %s java=%s heap_max_mb=%d classes=%d class_bytes=%d wall_ms=%d" + + " thread_cpu_ms=%d process_cpu_ms=%d gc_ms=%d collections=%d heap_before_mb=%d" + + " heap_after_gc_max_mb=%d heap_delta_mb=%d heap_peak_mb=%d live_max_mb=%d live_delta_mb=%d" + + " table_bytes=%d probes_bytes=%d not_measured=%d allocated_mb=%d allocated_kb_per_class=%d%n", + label, System.getProperty("java.version"), Runtime.getRuntime().maxMemory() >> 20, classes.size(), + classBytes, wall / 1_000_000, cpu / 1_000_000, process / 1_000_000, gc, collections.get(), + base >> 20, afterGc < 0 ? -1 : afterGc >> 20, afterGc < 0 ? -1 : (afterGc - base) >> 20, peak >> 20, + liveMax[0] < 0 ? -1 : liveMax[0] >> 20, liveMax[0] < 0 ? -1 : (liveMax[0] - base) >> 20, + bytes[0], bytes[1], b.notMeasured, allocated >> 20, (allocated / Math.max(1, classes.size())) >> 10); + b.allocated = allocated; + b.cpuNanos = cpu; + b.wallNanos = wall; + b.maxFrameGapNanos = lastFrame[1]; + long committedPeak = Math.max(committedSampled.get(), committedGcMax.get()); + b.committedGrowth = committedPeak - committedBase; + System.out.printf(Locale.ROOT, "BUDGET-COMMITTED %s java=%s gc=%s heap_max_mb=%d committed_pre_gc_mb=%d" + + " committed_before_mb=%d committed_peak_mb=%d committed_end_mb=%d committed_growth_mb=%d" + + " committed_growth_from_pre_gc_mb=%d max_frame_gap_ms=%d%n", + label, System.getProperty("java.version"), gcNames(), Runtime.getRuntime().maxMemory() >> 20, + committedPre >> 20, committedBase >> 20, committedPeak >> 20, committedEnd >> 20, + (committedPeak - committedBase) >> 20, (committedPeak - committedPre) >> 20, lastFrame[1] / 1_000_000); + return b; + } + + private static long heapCommitted() { + return ManagementFactory.getMemoryMXBean().getHeapMemoryUsage().getCommitted(); + } + + private static String gcNames() { + StringBuilder names = new StringBuilder(); + for (GarbageCollectorMXBean g : ManagementFactory.getGarbageCollectorMXBeans()) { + names.append(names.length() == 0 ? "" : "+").append(g.getName().replace(' ', '_')); + } + return names.toString(); + } + + private static long gcMillis() { + long t = 0; + for (GarbageCollectorMXBean g : ManagementFactory.getGarbageCollectorMXBeans()) { + t += Math.max(0, g.getCollectionTime()); + } + return t; + } + + private static long processCpuNanos() { + OperatingSystemMXBean os = ManagementFactory.getOperatingSystemMXBean(); + if (os instanceof com.sun.management.OperatingSystemMXBean) { + return ((com.sun.management.OperatingSystemMXBean) os).getProcessCpuTime(); + } + return -1; + } +} diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableNoticeTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableNoticeTest.java new file mode 100644 index 00000000..35927b71 --- /dev/null +++ b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableNoticeTest.java @@ -0,0 +1,246 @@ +package io.keploy.dedup; + +import org.junit.Test; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.io.InputStream; +import java.nio.charset.StandardCharsets; +import java.util.ArrayList; +import java.util.List; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.logging.Level; +import java.util.logging.LogRecord; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + +/** + * The lines the SDK logs when it defers the build table's send or the JVM cuts it short: a native keploy test + * stops the app at each test set's end, and a table that comes late, or never, must say why in the app's log. + */ +public class BuildTableNoticeTest { + + private static final String EXIT_LINE = "Keploy coverage: the JVM is exiting while its build table is being sent." + + " So far: "; + private static final String DEFERRED = "Keploy asked for the build table while this JVM sends one"; + private static final String PACED = "Keploy coverage: the build table's send is paced"; + + @Test(timeout = 30_000) + public void anAskWhileASendRunsSaysItIsDeferredOnceEach() throws Exception { + CountDownLatch inFirst = new CountDownLatch(1); + CountDownLatch release = new CountDownLatch(1); + CountDownLatch ended = new CountDownLatch(2); + List ran = java.util.Collections.synchronizedList(new ArrayList<>()); + BuildTable.Sends sends = new BuildTable.Sends(BuildTable::thread); + try (BuildTableTest.Logs logs = new BuildTableTest.Logs()) { + assertTrue(sends.request(() -> { + inFirst.countDown(); + try { + release.await(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + ran.add("first"); + ended.countDown(); + })); + assertTrue(inFirst.await(10, TimeUnit.SECONDS)); + assertEquals(0, lines(logs, DEFERRED).size()); + assertTrue(sends.request(() -> ran.add("second"))); + List one = lines(logs, DEFERRED); + assertEquals(one.toString(), 1, one.size()); + assertTrue(one.get(0), !one.get(0).contains("dropped")); + assertTrue(sends.request(() -> { + ran.add("third"); + ended.countDown(); + })); + List two = lines(logs, DEFERRED); + assertEquals(two.toString(), 2, two.size()); + assertTrue(two.get(1), two.get(1).contains("(an earlier ask that waited for it is dropped")); + assertEquals("[INFO, INFO]", levels(logs, DEFERRED)); + release.countDown(); + assertTrue(ended.await(10, TimeUnit.SECONDS)); + // The first send and the newest ask ran; the ask it replaced did not, as its line said. + assertEquals("[first, third]", ran.toString()); + } + } + + @Test(timeout = 30_000) + public void aPacedSendSaysSoOnceAtItsFirstWait() throws Exception { + BuildTablePacerTest.Clock c = new BuildTablePacerTest.Clock(); + try (BuildTableTest.Logs logs = new BuildTableTest.Logs()) { + BuildTable.Pacer p = new BuildTable.Pacer(c); + c.now += 400_000_000L; // the index's read: 400 ms, longer than the 1st unit owes, so the floor is hit + p.begin(1_000); + for (int i = 0; i < 1_000; i++) { + c.now += 1_000_000L; + c.gc += 1; // 1 ms of collections a unit: over the 0.5% goal, so it waits + c.units++; + p.unitDone(1); + } + assertTrue("the row's pacer waited " + c.sleeps.size() + " times", c.sleeps.size() > 1); + List got = lines(logs, PACED); + assertEquals(got.toString(), 1, got.size()); + // The 1st unit owes 200 ms, less its 401 ms: 0, never below. The 2nd owes 200 ms less its 1 ms: 199 ms, + // its first wait (200 times the 2 ms of collections less the 402 ms since the start is below 0). + assertTrue(got.get(0), got.get(0).contains("the JVM's collections took 2 ms in the 402 ms since it began," + + " so it owes 199 ms now (each ms of collection adds 200, and each ms the send works or waits" + + " takes one off, never below 0)")); + assertTrue(got.get(0), got.get(0).contains("its waits add up to at most its own work time (plus one" + + " sleep's overrun), so the table takes about twice as long as unpaced at most, and none ends the" + + " table past 20000 ms from its start")); + } + } + + @Test(timeout = 30_000) + public void aSendThatIsNotPacedSaysNothingOfIt() throws Exception { + BuildTablePacerTest.Clock c = new BuildTablePacerTest.Clock(); + try (BuildTableTest.Logs logs = new BuildTableTest.Logs()) { + BuildTable.Pacer p = new BuildTable.Pacer(c); + p.begin(1_000); + for (int i = 0; i < 1_000; i++) { + c.now += 1_000_000L; + c.units++; + p.unitDone(1); + } + assertEquals(0, c.sleeps.size()); + assertEquals(0, lines(logs, PACED).size()); + } + } + + @Test(timeout = 120_000) + public void aJvmThatExitsMidSendSaysSoOnce() throws Exception { + String out = runChild("mid"); + assertEquals(out, 1, count(out, EXIT_LINE)); + assertTrue(out, out.contains(" class files analyzed; 0 frames sent in ")); + assertTrue(out, out.contains("CHILD exiting")); + } + + @Test(timeout = 120_000) + public void aSendThatEndsBeforeTheJvmExitsSaysNothing() throws Exception { + String out = runChild("after"); + assertEquals(out, 0, count(out, EXIT_LINE)); + assertTrue(out, out.contains("CHILD sent")); + } + + @Test(timeout = 60_000) + public void aSendRemovesItsExitHookWhenItEnds() throws Exception { + List app = new ArrayList<>(); + for (int i = 0; i < 20; i++) { + String n = BuildTableBudgetTest.name(i); + app.add(new BuildTable.ClassFile(n, BuildTableBudgetTest.gen(n, i, 3))); + } + // A send that ends whole, and one whose socket fails at its first frame. + BuildTable.FrameSender whole = frame -> { }; + BuildTable.FrameSender failing = frame -> { + throw new IOException("row: the data socket is closed"); + }; + try { + for (BuildTable.FrameSender sender : new BuildTable.FrameSender[] {whole, failing}) { + BuildTable.ExitNotice.LAST_HOOK.remove(); + BuildTable.send(() -> app, sender); // on this thread, so LAST_HOOK is this send's + Thread hook = BuildTable.ExitNotice.LAST_HOOK.get(); + assertTrue("the send added an exit hook", hook != null); + // Removing it again finds none: the send removed it (else this removes it, for the rows after). + assertFalse("the send left its exit hook registered", + Runtime.getRuntime().removeShutdownHook(hook)); + } + } finally { + BuildTable.ExitNotice.LAST_HOOK.remove(); + } + } + + /** runChild runs ExitMain in a JVM of its own (an exit ends it), and returns its stdout and stderr. */ + private static String runChild(String mode) throws Exception { + String javaBin = new java.io.File(System.getProperty("java.home"), "bin/java").getPath(); + Process process = new ProcessBuilder(javaBin, "-Xmx512m", "-cp", System.getProperty("java.class.path"), + ExitMain.class.getName(), mode).redirectErrorStream(true).start(); + String out = new String(readAll(process.getInputStream()), StandardCharsets.UTF_8); + assertTrue(out, process.waitFor(60, TimeUnit.SECONDS)); + assertEquals(out, 0, process.exitValue()); + return out; + } + + /** + * ExitMain sends a table of generated classes on the SDK's own thread. "mid": its first frame never returns, + * and the JVM exits mid-send. "after": the send ends, then the JVM exits. + */ + public static final class ExitMain { + public static void main(String[] args) throws Exception { + boolean mid = "mid".equals(args[0]); + List app = new ArrayList<>(); + for (int i = 0; i < 300; i++) { + String n = BuildTableBudgetTest.name(i); + app.add(new BuildTable.ClassFile(n, BuildTableBudgetTest.gen(n, i, BuildTableBudgetTest.BRANCHES))); + } + CountDownLatch inFrame = new CountDownLatch(1); + CountDownLatch sent = new CountDownLatch(1); + BuildTable.thread(() -> { + BuildTable.send(() -> app, frame -> { + inFrame.countDown(); + if (mid) { + try { + new CountDownLatch(1).await(); // the frame never returns: the exit cuts the send + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + } + }); + sent.countDown(); + }).start(); + if (mid) { + if (!inFrame.await(30, TimeUnit.SECONDS)) { + throw new IllegalStateException("no frame"); + } + System.out.println("CHILD exiting"); + } else { + if (!sent.await(30, TimeUnit.SECONDS)) { + throw new IllegalStateException("not sent"); + } + System.out.println("CHILD sent"); + } + System.out.flush(); + System.exit(0); + } + } + + /** lines are the messages with part that this row's thread logged (a send of another class may log too). */ + private static List lines(BuildTableTest.Logs logs, String part) { + List out = new ArrayList<>(); + for (LogRecord r : logs.mine()) { + if (r.getMessage() != null && r.getMessage().contains(part)) { + out.add(r.getMessage()); + } + } + return out; + } + + private static String levels(BuildTableTest.Logs logs, String part) { + List out = new ArrayList<>(); + for (LogRecord r : logs.mine()) { + if (r.getMessage() != null && r.getMessage().contains(part)) { + out.add(r.getLevel()); + } + } + return out.toString(); + } + + private static int count(String s, String part) { + int n = 0; + for (int i = s.indexOf(part); i >= 0; i = s.indexOf(part, i + 1)) { + n++; + } + return n; + } + + private static byte[] readAll(InputStream in) throws IOException { + ByteArrayOutputStream out = new ByteArrayOutputStream(); + byte[] b = new byte[8192]; + for (int n; (n = in.read(b)) > 0; ) { + out.write(b, 0, n); + } + return out.toByteArray(); + } +} diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTablePacerTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTablePacerTest.java new file mode 100644 index 00000000..5a52dfae --- /dev/null +++ b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTablePacerTest.java @@ -0,0 +1,441 @@ +package io.keploy.dedup; + +import org.junit.Test; + +import java.io.InterruptedIOException; +import java.util.ArrayList; +import java.util.List; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +/** + * The table's pacing (BuildTable.Pacer) on a clock the rows drive: it waits out 200 times the collections' + * time, only after a collection, never more in all than the send's own work time, saved up to a second, and + * never past the table's deadline. + */ +public class BuildTablePacerTest { + + private static final long MS = 1_000_000L; + private static final long MIN = BuildTable.Pacer.MIN_WAIT_NANOS; + // The units of 1 ms work whose time first reaches the smallest wait, after a wait used it all. + private static final long K = MIN / MS; + + /** A clock that moves only as a row says and by what the pacer sleeps (plus overrun a sleep; or not). */ + static final class Clock implements BuildTable.Pacer.Clock { + long now = 1_000 * MS; + long gc = 7; + long overrun; + /** A sleep does not move the clock (one too coarse to see it). */ + boolean frozen; + /** The time the last sleep ended. */ + long lastWake; + int gcReads; + RuntimeException failGc; + boolean interrupt; + final List sleeps = new ArrayList<>(); + /** The unit after which each sleep came (the count of unitDone calls made by then). */ + final List sleptAt = new ArrayList<>(); + int units; + + @Override + public long nanoTime() { + return now; + } + + @Override + public long gcMillis() { + gcReads++; + if (failGc != null) { + throw failGc; + } + return gc; + } + + @Override + public void sleep(long nanos) throws InterruptedException { + if (interrupt) { + throw new InterruptedException("row"); + } + sleeps.add(nanos); + sleptAt.add(units); + if (!frozen) { + now += nanos + overrun; + } + lastWake = now; + } + + long slept() { + long s = 0; + for (long n : sleeps) { + s += n; + } + return s; + } + } + + /** run makes n units of one class file each, workNanos apiece, collecting gcPerUnit ms in each. */ + private static void run(BuildTable.Pacer p, Clock c, int n, long workNanos, long gcPerUnit) + throws InterruptedIOException { + for (int i = 0; i < n; i++) { + c.now += workNanos; + c.gc += gcPerUnit; + c.units++; + p.unitDone(1); + } + } + + @Test + public void aJvmThatDoesNotCollectIsNotPaced() throws Exception { + Clock c = new Clock(); + BuildTable.Pacer p = new BuildTable.Pacer(c); + run(p, c, 50_000, 50_000, 0); + assertEquals(0, c.sleeps.size()); + assertEquals(0, p.waitedNanos()); + } + + @Test + public void aCollectionIsWaitedOutTo200TimesItsTime() throws Exception { + Clock c = new Clock(); + BuildTable.Pacer p = new BuildTable.Pacer(c); + run(p, c, 1_000, MS, 0); + assertEquals("no wait before a collection", 0, c.sleeps.size()); + // A 4 ms collection during the next unit: 800 ms of the send's time, less that unit's 1 ms (a second of + // the send's own work came before, so its bound lets it all be waited). + c.now += MS; + c.gc += 4; + c.units++; + p.unitDone(1); + assertEquals(1, c.sleeps.size()); + assertEquals(799 * MS, (long) c.sleeps.get(0)); + run(p, c, 1_000, MS, 0); + assertEquals("paid off: no wait after it", 1, c.sleeps.size()); + assertEquals(799 * MS, p.waitedNanos()); + } + + @Test + public void workSinceACollectionCountsTowardItsWait() throws Exception { + Clock c = new Clock(); + BuildTable.Pacer p = new BuildTable.Pacer(c); + run(p, c, 2_000, 50_000, 0); + // A 1 ms collection (200 ms owed) during a unit that took 150 ms: 50 ms are left to wait. + c.now += 150 * MS; + c.gc += 1; + c.units++; + p.unitDone(1); + assertEquals(1, c.sleeps.size()); + assertEquals(50 * MS, (long) c.sleeps.get(0)); + // Owed less than the smallest wait is never slept. + c.now += 195 * MS + 1; + c.gc += 1; + c.units++; + p.unitDone(1); + assertEquals(1, c.sleeps.size()); + } + + @Test + public void theWaitsAreBoundedByTheSendsOwnWork() throws Exception { + Clock c = new Clock(); + BuildTable.Pacer p = new BuildTable.Pacer(c); + long start = c.now; + // A JVM that collects 10 ms in every unit (a busy app's own collections) owes 2 s a unit: only the send's + // own work, 0.1 ms a unit (5 s in all), bounds the waits. + long w = MS / 10; + run(p, c, 50_000, w, 10); + long slept = c.slept(); + assertTrue("slept " + slept, slept <= 50_000 * w); + assertTrue("slept " + slept, slept > 50_000 * w - MIN); + assertEquals(slept, p.waitedNanos()); + assertTrue("the table takes at most twice its work", c.now - start <= 2 * 50_000 * w); + // Over any run of units, the waits add up to at most a second plus those units' own time. + long[] sum = new long[c.units + 1]; + for (int i = 0; i < c.sleeps.size(); i++) { + sum[c.sleptAt.get(i)] += c.sleeps.get(i); + } + long low = 0; + long prefix = 0; + for (int j = 1; j <= c.units; j++) { + prefix += sum[j]; + long at = prefix - j * w; + assertTrue("waits over a run of units, up to unit " + j, at - low <= BuildTable.Pacer.MAX_BANK_NANOS); + low = Math.min(low, at); + } + } + + @Test + public void aQuietStretchSavesUpAtMostASecond() throws Exception { + Clock c = new Clock(); + BuildTable.Pacer p = new BuildTable.Pacer(c); + run(p, c, 5_000, MS, 0); + // 100 ms of collection owes 20 s; 5 s of the send's work went by, but a second of it was kept. + c.now += 20_000; + c.gc += 100; + c.units++; + p.unitDone(1); + assertEquals(1, c.sleeps.size()); + assertEquals(BuildTable.Pacer.MAX_BANK_NANOS, (long) c.sleeps.get(0)); + // Then a wait each time the send's work since reaches the smallest wait: K units' time every K of them. + run(p, c, 100, MS, 0); + assertEquals(1 + 100 / K, c.sleeps.size()); + for (int i = 1; i < c.sleeps.size(); i++) { + assertEquals(K * MS, (long) c.sleeps.get(i)); + } + } + + @Test + public void aUnitMayWaitItsOwnTimeNotItsClassFiles() throws Exception { + Clock c = new Clock(); + BuildTable.Pacer p = new BuildTable.Pacer(c); + // A unit of 300 class files that took 20 us: a second of collection owes 200 s, and it may wait 20 us. + c.now += 20_000; + c.gc += 1_000; + p.unitDone(300); + assertEquals(0, c.sleeps.size()); + // One class file that took 300 ms: it may wait that, and the 20 us before it. + c.now += 300 * MS; + c.gc += 1_000; + p.unitDone(1); + assertEquals(1, c.sleeps.size()); + assertEquals(300 * MS + 20_000, (long) c.sleeps.get(0)); + } + + @Test + public void aSleepsOverrunComesOffWhatItMayWait() throws Exception { + Clock c = new Clock(); + c.overrun = 5 * MS; + BuildTable.Pacer p = new BuildTable.Pacer(c); + long start = c.now; + long w = MS / 10; + run(p, c, 10_000, w, 10); + long waited = c.now - start - 10_000 * w; + assertEquals(waited, p.waitedNanos()); + // The time asleep, overruns included, stays within the send's own work (and one overrun past it). + assertTrue("waited " + waited, waited <= 10_000 * w + c.overrun); + assertTrue("waited " + waited, waited > 10_000 * w - MIN - c.overrun); + } + + @Test + public void anInterruptEndsTheSendAndIsKept() throws Exception { + try { + // During a wait. + Clock c = new Clock(); + BuildTable.Pacer p = new BuildTable.Pacer(c); + c.interrupt = true; + try { + run(p, c, 100, MS, 10); + fail("an interrupt during a wait ends the send"); + } catch (InterruptedIOException e) { + assertEquals("the build table's thread was interrupted", e.getMessage()); + assertTrue("the interrupt is kept for the thread", Thread.interrupted()); + } + assertEquals("it came at the first wait", K, c.units); + // Between waits, and on a Pacer that never waits. + for (BuildTable.Pacer q : new BuildTable.Pacer[] {new BuildTable.Pacer(new Clock()), + BuildTable.Pacer.off()}) { + q.unitDone(1); + Thread.currentThread().interrupt(); + try { + q.unitDone(1); + fail("an interrupt ends the send at the next unit"); + } catch (InterruptedIOException e) { + assertTrue("the interrupt is kept for the thread", Thread.interrupted()); + } + } + } finally { + // A failing row must not leave the runner's thread interrupted for the rows after it. + Thread.interrupted(); + } + } + + @Test + public void theWaitsGiveWayToTheDeadline() throws Exception { + // 50,000 class files of 0.3 ms work each (15 s) in a JVM that collects 10 ms in each: the work bound + // alone would wait 15 s, a table of 30 s. The deadline holds it to 20 s, waiting all the room left. + long w = 3 * MS / 10; + Clock c = new Clock(); + BuildTable.Pacer p = new BuildTable.Pacer(c); + p.begin(50_000); + long start = c.now; + run(p, c, 50_000, w, 10); + long took = c.now - start; + assertTrue("took " + took, took <= BuildTable.Pacer.deadlineNanos(50_000)); + assertTrue("waited " + p.waitedNanos(), p.waitedNanos() > 5_000 * MS - 2 * MIN); + // Not told its class files, a Pacer still waits no later than 20 s into the table. + Clock d = new Clock(); + BuildTable.Pacer q = new BuildTable.Pacer(d); + start = d.now; + run(q, d, 50_000, w, 10); + assertTrue("woke at " + (d.lastWake - start), d.lastWake - start <= BuildTable.Pacer.deadlineNanos(0)); + assertTrue("waited " + q.waitedNanos(), q.waitedNanos() > 5_000 * MS); + assertEquals(20_000 * MS, BuildTable.Pacer.deadlineNanos(50_000)); + assertEquals("0.4 ms a class file past 50,000", 40_000 * MS, BuildTable.Pacer.deadlineNanos(100_000)); + assertEquals(20_000 * MS, BuildTable.Pacer.deadlineNanos(10)); + } + + @Test + public void theRateLeavesOutTheTimeBeforeTheAnalysis() throws Exception { + // The index's read takes 4 s before the analysis begins; then 50,000 class files of 0.2 ms work each + // (10 s), 10 ms of collection in each. The rate is the analysis's own (0.2 ms a class file), so the waits + // begin at once (the read is the send's own work too); the deadline still counts the 4 s, so the waits + // fill the 6 s left and no more. + Clock c = new Clock(); + BuildTable.Pacer p = new BuildTable.Pacer(c); + long start = c.now; + c.now += 4_000 * MS; + p.begin(50_000); + run(p, c, 50_000, MS / 5, 10); + // The read's 4 s paid off the first unit's 2 s owed, and saved a second up: the first wait is at the + // second unit, a second long. Had the rate counted the read, the deadline would have left no room. + assertEquals("the first wait at the second unit", 2, (long) c.sleptAt.get(0)); + assertEquals(BuildTable.Pacer.MAX_BANK_NANOS, (long) c.sleeps.get(0)); + assertTrue("took " + (c.now - start), c.now - start <= BuildTable.Pacer.deadlineNanos(50_000)); + assertTrue("waited " + p.waitedNanos(), p.waitedNanos() > 6_000 * MS - 2 * MIN); + } + + @Test + public void aTableWhoseWorkFillsItsDeadlineIsNotPaced() throws Exception { + Clock c = new Clock(); + BuildTable.Pacer p = new BuildTable.Pacer(c); + p.begin(50_000); + run(p, c, 50_000, MS, 10); + assertEquals(0, c.sleeps.size()); + } + + @Test + public void aClockThatDoesNotMoveAcrossASleepIsChargedTheWait() throws Exception { + Clock c = new Clock(); + c.frozen = true; + BuildTable.Pacer p = new BuildTable.Pacer(c); + run(p, c, 10_000, 20_000, 10); + long slept = c.slept(); + assertTrue("slept " + slept, slept <= 10_000 * 20_000L); + assertTrue("slept " + slept, slept > 10_000 * 20_000L - MIN); + } + + @Test + public void aCollectorBeanThatFailsEndsThePacing() throws Exception { + Clock c = new Clock(); + BuildTable.Pacer p = new BuildTable.Pacer(c); + run(p, c, 100, MS, 10); + int before = c.sleeps.size(); + assertTrue(before > 0); + c.failGc = new UnsupportedOperationException("row"); + run(p, c, 1, MS, 10); + c.failGc = null; + run(p, c, 1_000, MS, 10); + assertEquals(before, c.sleeps.size()); + } + + @Test + public void aPacerOffNeverReadsOrWaits() throws Exception { + BuildTable.Pacer p = BuildTable.Pacer.off(); + for (int i = 0; i < 1_000; i++) { + p.unitDone(1_000); + } + assertEquals(0, p.waitedNanos()); + } + + @Test + public void aPacerCountsItsClassFilesPacedOrNot() throws Exception { + for (BuildTable.Pacer p : new BuildTable.Pacer[] {BuildTable.Pacer.off(), new BuildTable.Pacer(new Clock())}) { + assertEquals(0, p.totalClassFiles()); + p.begin(5); + p.unitDone(2); + p.unitDone(3); + assertEquals(5, p.doneClassFiles()); + assertEquals(5, p.totalClassFiles()); + } + } + + @Test + public void theJvmsPacerReadsItsCollectors() throws Exception { + BuildTable.Pacer.JvmClock jvm = new BuildTable.Pacer.JvmClock( + java.lang.management.ManagementFactory.getGarbageCollectorMXBeans()); + long before = jvm.gcMillis(); + assertTrue("a collection's time is read: " + before, before >= 0); + System.gc(); + assertTrue(jvm.gcMillis() >= before); + assertEquals(0, BuildTable.Pacer.JvmClock.NONE.gcMillis()); + } + + @Test + public void writePacesAfterEachUnitByItsTime() throws Exception { + List app = new ArrayList<>(); + for (int i = 0; i < 100; i++) { + app.add(new BuildTable.ClassFile(BuildTableBudgetTest.name(i), + BuildTableBudgetTest.gen(BuildTableBudgetTest.name(i), i, 3))); + } + Clock c = new Clock(); + int[] records = {0}; + BuildTable.Pacer.Clock collecting = new BuildTable.Pacer.Clock() { + @Override + public long nanoTime() { + return c.nanoTime(); + } + + @Override + public long gcMillis() { + // Each read finds a second more of collection: the send's own work alone bounds the waits. + c.gc += 1_000; + return c.gcMillis(); + } + + @Override + public void sleep(long nanos) throws InterruptedException { + c.units = records[0]; + c.sleep(nanos); + } + }; + // Each unit's record takes 1 ms of the send's work. + BuildTable.Summary s = BuildTable.write(app, (build, json) -> { + records[0]++; + c.now += MS; + }, BuildTable.DROP, BuildTable.MAX_RECORD_BYTES, BuildTable.MAX_MAP_BYTES, BuildTable.MAX_MAP_STEPS, + new BuildTable.Pacer(collecting)); + assertEquals(100, s.records); + assertEquals("one read at the start, one after each unit", 101, c.gcReads); + // The work of K units at a time: after every Kth unit's record. + assertEquals(100 / K, c.sleeps.size()); + for (int i = 0; i < c.sleeps.size(); i++) { + assertEquals(K * MS, (long) c.sleeps.get(i)); + assertEquals((i + 1) * K, (int) c.sleptAt.get(i)); + } + } + + @Test + public void writeTellsThePacerItsClassFilesForTheDeadline() throws Exception { + // 100 classes of 0.9 s work each: 90 s, past the 20 s deadline from the first, so never a wait. A Pacer + // not told how many class files come would wait through its first 20 s. + List app = new ArrayList<>(); + for (int i = 0; i < 100; i++) { + app.add(new BuildTable.ClassFile(BuildTableBudgetTest.name(i), + BuildTableBudgetTest.gen(BuildTableBudgetTest.name(i), i, 3))); + } + Clock c = new Clock(); + BuildTable.Pacer.Clock slow = new BuildTable.Pacer.Clock() { + @Override + public long nanoTime() { + return c.nanoTime(); + } + + @Override + public long gcMillis() { + // Read once a unit: a second more of collection. + c.gc += 1_000; + return c.gcMillis(); + } + + @Override + public void sleep(long nanos) throws InterruptedException { + c.sleep(nanos); + } + }; + // Each unit's record takes 0.9 s of the send's work. + BuildTable.Summary s = BuildTable.write(app, (build, json) -> c.now += 900 * MS, BuildTable.DROP, + BuildTable.MAX_RECORD_BYTES, BuildTable.MAX_MAP_BYTES, BuildTable.MAX_MAP_STEPS, + new BuildTable.Pacer(slow)); + assertEquals(100, s.records); + assertEquals(0, c.sleeps.size()); + } +} diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java new file mode 100644 index 00000000..61bc7cb0 --- /dev/null +++ b/keploy-sdk/src/test/java/io/keploy/dedup/BuildTableTest.java @@ -0,0 +1,1598 @@ +package io.keploy.dedup; + +import com.google.gson.JsonArray; +import com.google.gson.JsonElement; +import com.google.gson.JsonObject; +import com.google.gson.JsonParser; +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; +import org.jacoco.core.internal.analysis.KeployBranchCoverage; +import org.jacoco.core.internal.data.CRC64; +import org.objectweb.asm.ClassWriter; +import org.objectweb.asm.Label; +import org.objectweb.asm.MethodVisitor; +import org.objectweb.asm.Opcodes; +import org.newsclub.net.unix.AFUNIXServerSocket; +import org.newsclub.net.unix.AFUNIXSocketAddress; + +import java.io.ByteArrayOutputStream; +import java.io.DataInputStream; +import java.io.File; +import java.io.IOException; +import java.io.InputStream; +import java.net.Socket; +import java.net.URISyntaxException; +import java.net.URL; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.security.MessageDigest; +import java.security.NoSuchAlgorithmException; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.Collections; +import java.util.HashMap; +import java.util.HashSet; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.function.Supplier; +import java.util.stream.Collectors; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; +import java.util.logging.Handler; +import java.util.logging.Level; +import java.util.logging.LogRecord; +import java.util.logging.Logger; +import java.util.stream.Stream; + +import static org.junit.Assert.assertArrayEquals; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertNotSame; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertSame; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +/** + * Rows for the build table: the java-dedup sample's classes (compiled once with javac --release 8 -g from + * keploy/samples-java java-dedup, kept as .class.bin) encode to the golden table byte for byte. The golden + * table (java-dedup.table.ndjson) is the Go encoder's output (enterprise pkg/service/codecoverage NewTable + + * EncodeTable) over these records, with the unit hashes computed in Go from the class bytes, so the order, the + * JSON form, the fold and the unit hashes are Go's, not this class's. So is edge.table.ndjson, over a set of + * classes generated here ({@link #edge}) for the names, packages and escapes the sample lacks. + * + *

The .build goldens (java-dedup.build, edge.build) are the build id of IndexReport.buildId's bytes: the + * table format F ({@link BuildTable#TABLE_FORMAT}), the JaCoCo release V of the analyzer keploy-sdk.jar + * carries, then the class ids (JaCoCo's CRC64 of each class file), computed by an independent CRC64 and + * sha256 oracle until codecoverage.JavaBuildID takes F and V. A JaCoCo upgrade or a raised TABLE_FORMAT + * changes both .build files on purpose. + * + *

To regenerate a golden table: run these rows with -Dbuildtable.dump=DIR (each golden row writes + * DIR/NAME.ndjson and the class files as DIR/NAME/VM-NAME.class.bin), then in an enterprise checkout read + * each record with codecoverage's Unit JSON, recompute its hash from the class files with JaCoCo's CRC64, + * NewTable(LanguageJava, build, units) and EncodeTable, and write the result back over the golden here. Then + * update {@link #aTableThatChangesForTheSameClassesNamesAnotherBuild}. To regenerate a .build golden, take + * the sha256 of IndexReport.buildId's bytes over the dumped class files' ids. + */ +public class BuildTableTest { + + private static final String RES = "/io/keploy/dedup/buildtable/"; + + @Rule + public TemporaryFolder tmp = new TemporaryFolder(); + + static List javaDedup() throws IOException, URISyntaxException { + URL root = BuildTableTest.class.getResource(RES + "java-dedup"); + assertNotNull("java-dedup class files", root); + Path dir = Paths.get(root.toURI()); + List out = new ArrayList<>(); + try (Stream walk = Files.walk(dir)) { + for (Path p : walk.filter(p -> p.toString().endsWith(".class.bin")).sorted().collect(Collectors.toList())) { + String rel = dir.relativize(p).toString().replace(File.separatorChar, '/'); + out.add(new BuildTable.ClassFile(rel.substring(0, rel.length() - ".class.bin".length()), + Files.readAllBytes(p))); + } + } + assertEquals(5, out.size()); + return out; + } + + static byte[] resource(String name) throws IOException { + try (InputStream in = BuildTableTest.class.getResourceAsStream(RES + name)) { + assertNotNull(name, in); + ByteArrayOutputStream b = new ByteArrayOutputStream(); + byte[] buf = new byte[8192]; + for (int n; (n = in.read(buf)) > 0; ) { + b.write(buf, 0, n); + } + return b.toByteArray(); + } + } + + /** sha256Hex is the sha256 of b as 64 lowercase hex, as a frame's "sha256" says it. */ + static String sha256Hex(byte[] b) { + try { + StringBuilder hex = new StringBuilder(); + for (byte x : MessageDigest.getInstance("SHA-256").digest(b)) { + hex.append(String.format("%02x", x)); + } + return hex.toString(); + } catch (NoSuchAlgorithmException e) { + throw new IllegalStateException(e); + } + } + + /** Collects the records write hands out, as the NDJSON stream. */ + static final class Collect implements BuildTable.RecordSink { + final ByteArrayOutputStream ndjson = new ByteArrayOutputStream(); + final List records = new ArrayList<>(); + String build; + + @Override + public void record(String build, byte[] json) { + this.build = build; + records.add(new String(json, StandardCharsets.UTF_8)); + ndjson.write(json, 0, json.length); + ndjson.write('\n'); + } + } + + static Collect table(List files) throws IOException { + Collect c = new Collect(); + assertNotNull(BuildTable.write(files, c)); + return c; + } + + /** dump writes a golden row's output and class files under -Dbuildtable.dump, to regenerate its golden. */ + static void dump(String name, List files, Collect c) throws IOException { + String dir = System.getProperty("buildtable.dump"); + if (dir == null) { + return; + } + Files.createDirectories(Paths.get(dir)); + Files.write(Paths.get(dir, name + ".ndjson"), c.ndjson.toByteArray()); + for (BuildTable.ClassFile f : files) { + Path p = Paths.get(dir, name, f.name + ".class.bin"); + Files.createDirectories(p.getParent()); + Files.write(p, f.bytes); + } + } + + private static final String BUILD_GOLDEN = "the build id of IndexReport.buildId's bytes (F, the analyzer's JaCoCo" + + " release, the class ids): a JaCoCo upgrade or a raised BuildTable.TABLE_FORMAT changes it on purpose"; + + /** + * A tripwire for the build id's F: TABLE_FORMAT 1 is these bounds, Hits' step count, the record each cap + * writes (with its reason) and the records these golden tables hold. A change to any of them changes the + * table for the same class bytes, and an SDK that kept F would name the build an older SDK saved its table + * under, so Keploy would refuse each later save of it. In a release, raise BuildTable.TABLE_FORMAT, then + * update this row; before the first release that writes a table, update the pinned values alone. + */ + @Test + public void aTableThatChangesForTheSameClassesNamesAnotherBuild() throws Exception { + String why = "the table for the same class bytes changed: in a release, raise BuildTable.TABLE_FORMAT," + + " then update this row"; + assertEquals(why, 1, BuildTable.TABLE_FORMAT); + assertEquals(why, 8 << 20, BuildTable.MAX_RECORD_BYTES); + assertEquals(why, 8 << 20, BuildTable.MAX_MAP_BYTES); + assertEquals(why, 64L << 20, BuildTable.MAX_MAP_STEPS); + // The record each cap writes, not the constants: a unit past the probe map's steps (a chain of 200 + // early returns, whose walks Hits counts) and a record past a 1600-byte chunk (java-dedup's two). + String name = "gen/big/Returns"; + byte[] bytes = BuildTableBudgetTest.returns(name, 200); + List one = Collections.singletonList(new BuildTable.ClassFile(name, bytes)); + long[] steps = {0}; + KeployBranchCoverage.analyze(bytes, new HashMap<>(), (m, r, h) -> steps[0] += h.steps); + assertEquals(why, 61302L, steps[0]); + Collect probes = new Collect(); + BuildTable.write(one, new Collect(), probes, BuildTable.MAX_RECORD_BYTES); + assertEquals(why, 1, probes.records.size()); + assertEquals(why, "c4271098e6e5103773be528dee9ca19edd1fded1409e2cba5a4aad84232e61c9", + sha256Hex(probes.ndjson.toByteArray())); + int mapBytes = probes.records.get(0).length() + 1; + assertEquals(why, 149124, mapBytes); + Collect mapped = new Collect(); + Collect map = new Collect(); + BuildTable.write(one, mapped, map, BuildTable.MAX_RECORD_BYTES, mapBytes, steps[0]); + assertEquals(why, probes.records, map.records); + assertTrue(why, mapped.records.get(0).contains("\"state\":\"ok\"")); + Collect capped = new Collect(); + BuildTable.write(one, capped, new Collect(), BuildTable.MAX_RECORD_BYTES, mapBytes, steps[0] - 1); + assertTrue(why, capped.records.get(0) + .contains("\"state\":\"not_measured\",\"reason\":{\"code\":\"record_too_large\"}")); + assertEquals(why, "3ee7977c74cf5939ab7af3957bfc1bbd4e495614d4dcb7bd066c707618f7ec1c", + sha256Hex(capped.ndjson.toByteArray())); + Collect chunked = new Collect(); + BuildTable.write(javaDedup(), chunked, 1600); + assertEquals(why, "4c5ebe7cc05a4f0e4c34faa45c7521216df376409cac5cc4f8e6447b4359d584", + sha256Hex(chunked.ndjson.toByteArray())); + assertEquals(why, "d9d2f53a4e36fb2a63b63a307187373a1ee87ec4e44bcbcdec1a2c4fb47caf1c", + sha256Hex(resource("java-dedup.table.ndjson"))); + assertEquals(why, "ddb2d5ac7990ed620e370825fc84aec6782ea89ccda029716cdfb1b27519a7dc", + sha256Hex(resource("java-dedup.probes.ndjson"))); + assertEquals(why, "e0e30865ebc060b2a576f961df79bd002191791daa6336f47f210e666feadd20", + sha256Hex(resource("edge.table.ndjson"))); + } + + @Test + public void javaDedupTableIsTheGoEncodersByteForByte() throws Exception { + List files = javaDedup(); + Collect c = table(files); + dump("java-dedup", files, c); + assertEquals(BUILD_GOLDEN, new String(resource("java-dedup.build"), StandardCharsets.UTF_8).trim(), c.build); + assertEquals(new String(resource("java-dedup.table.ndjson"), StandardCharsets.UTF_8), + new String(c.ndjson.toByteArray(), StandardCharsets.UTF_8)); + // The same classes in another order give the same bytes. + List reversed = new ArrayList<>(files); + java.util.Collections.reverse(reversed); + assertArrayEquals(c.ndjson.toByteArray(), table(reversed).ndjson.toByteArray()); + } + + static Map byId(List lines) { + Map out = new LinkedHashMap<>(); + for (String l : lines) { + if (l.trim().isEmpty()) { + continue; + } + JsonObject o = JsonParser.parseString(l).getAsJsonObject(); + out.put(o.get("id").getAsString(), o); + } + return out; + } + + static Map methods(JsonObject unit) { + Map out = new LinkedHashMap<>(); + for (JsonElement m : unit.getAsJsonArray("methods")) { + out.put(m.getAsJsonObject().get("id").getAsString(), m.getAsJsonObject()); + } + return out; + } + + // per line: [code leaves, branch edges] + static Map> lineShape(JsonObject method) { + Map> out = new LinkedHashMap<>(); + for (JsonElement e : method.getAsJsonArray("leaves")) { + JsonArray l = e.getAsJsonArray(); + List s = out.computeIfAbsent(l.get(0).getAsInt(), k -> new ArrayList<>(Arrays.asList(0, 0))); + s.set(l.get(5).getAsInt(), s.get(l.get(5).getAsInt()) + 1); + } + return out; + } + + /** + * The contract fixture (fixtures/java-dedup) is hand-made: its leaf counts (n = 3), hashes and the model + * classes' lines are illustrative, it lists only some of DedupController's methods and not Product, and + * its simpleMessage takes jakarta's HttpServletRequest where the sample imports javax's. What it holds as + * real must agree: every fixture class is a record with its id, pkg, name and path; every fixture method + * of DedupController is a method with the same display name and first line; and the two methods with + * branches have the fixture's lines and branch edges. + */ + @Test + public void javaDedupTableAgreesWithTheContractFixture() throws Exception { + Map ours = byId(table(javaDedup()).records); + Map fixture = byId(Arrays.asList( + new String(resource("fixture-java-dedup-table.ndjson"), StandardCharsets.UTF_8).split("\n"))); + assertEquals(4, fixture.size()); + for (JsonObject f : fixture.values()) { + String id = f.get("id").getAsString(); + JsonObject o = ours.get(id); + assertNotNull(id, o); + for (String k : new String[] {"kind", "pkg", "name", "path", "state"}) { + assertEquals(id + " " + k, f.get(k), o.get(k)); + } + } + String ctl = "c:io/keploy/samples/javadedup/DedupController"; + Map fm = methods(fixture.get(ctl)); + Map om = methods(ours.get(ctl)); + for (Map.Entry e : fm.entrySet()) { + String mid = e.getKey().replace("jakarta/servlet", "javax/servlet"); + JsonObject o = om.get(mid); + assertNotNull(mid, o); + assertEquals(mid, e.getValue().get("name"), o.get("name")); + assertEquals(mid, e.getValue().get("line"), o.get("line")); + } + for (String m : new String[] {"messageForPath(Ljava/lang/String;)Ljava/lang/String;", + "response([Ljava/lang/Object;)Ljava/util/Map;"}) { + String mid = "m:io/keploy/samples/javadedup/DedupController#" + m; + assertEquals(mid, lineShape(fm.get(mid)), lineShape(om.get(mid))); + } + } + + static byte[] patchedMajor(byte[] bytes, int major) { + byte[] b = bytes.clone(); + b[6] = (byte) (major >> 8); + b[7] = (byte) major; + return b; + } + + @Test + public void aClassJacocoCannotAnalyzeIsNotMeasuredAndSoIsItsUnit() throws Exception { + List files = new ArrayList<>(javaDedup()); + BuildTable.ClassFile item = null; + for (BuildTable.ClassFile f : files) { + if (f.name.endsWith("/model/Item")) { + item = f; + } + } + assertNotNull(item); + // A nested class of Item no ASM in this SDK reads (class file version 99), and a class that is + // not a class file at all. + files.add(new BuildTable.ClassFile(item.name + "$Later", patchedMajor(item.bytes, 99))); + files.add(new BuildTable.ClassFile("io/keploy/samples/javadedup/Junk", + "not a class".getBytes(StandardCharsets.UTF_8))); + // A class JaCoCo's offline instrumenter wrote: JaCoCo refuses to analyze it. + String instrumented = "io/keploy/samples/javadedup/Instrumented"; + files.add(new BuildTable.ClassFile(instrumented, ProbeMapTest.instrumented(instrumented, + edgeClass(instrumented, "Instrumented.java", true, 10)))); + Collect c = new Collect(); + BuildTable.Summary s = BuildTable.write(files, c); + Map ours = byId(c.records); + Map golden = byId(Arrays.asList( + new String(resource("java-dedup.table.ndjson"), StandardCharsets.UTF_8).split("\n"))); + + JsonObject it = ours.get("c:io/keploy/samples/javadedup/model/Item"); + assertEquals("not_measured", it.get("state").getAsString()); + assertEquals("{\"code\":\"analyzer_failed\"}", it.get("reason").toString()); + assertEquals(0, it.getAsJsonArray("methods").size()); + assertEquals("its own class file still names the path", "io/keploy/samples/javadedup/model/Item.java", + it.get("path").getAsString()); + assertFalse("the hash takes the new class file's id", golden.get(it.get("id").getAsString()) + .get("hash").equals(it.get("hash"))); + + JsonObject junk = ours.get("c:io/keploy/samples/javadedup/Junk"); + assertEquals("not_measured", junk.get("state").getAsString()); + assertEquals("", junk.get("path").getAsString()); + assertEquals("[]", junk.get("methods").toString()); + + JsonObject in = ours.get("c:" + instrumented); + assertEquals("not_measured", in.get("state").getAsString()); + assertEquals("{\"code\":\"analyzer_failed\"}", in.get("reason").toString()); + assertEquals("[]", in.get("methods").toString()); + + // Each of the three is JaCoCo refusing a class file, none a probe map that failed. + assertEquals(3, s.notMeasured); + assertEquals("JaCoCo's refusals are not counted as maps that failed", 0, s.notMapped); + assertEquals(0, s.tooLarge); + assertEquals(0, s.tooComplex); + + // The other classes are as they were. + for (String id : golden.keySet()) { + if (!id.endsWith("/model/Item")) { + assertEquals(id, golden.get(id), ours.get(id)); + } + } + } + + @Test + public void aRecordLargerThanAChunkInflatesToIsRecordTooLarge() throws Exception { + Collect c = new Collect(); + BuildTable.Summary s = BuildTable.write(javaDedup(), c, 1600); + assertEquals(5, s.records); + int tooLarge = 0; + for (String r : c.records) { + assertTrue(r, r.length() + 1 <= 1600); + if (r.contains("\"record_too_large\"")) { + tooLarge++; + assertTrue(r, r.contains("\"state\":\"not_measured\"") && r.endsWith("\"methods\":[]}")); + } + } + // DedupController and Product are the two records past 1600 bytes. + assertEquals(2, tooLarge); + assertEquals(2, s.notMeasured); + assertEquals(2, s.tooLarge); + assertEquals("a record too large is not a unit too complex", 0, s.tooComplex); + } + + @Test + public void aProbeMapPastItsBytesOrStepsIsRecordTooLargeWithNoProbeRecord() throws Exception { + String name = "gen/big/Returns"; + byte[] bytes = BuildTableBudgetTest.returns(name, 200); + List one = Collections.singletonList(new BuildTable.ClassFile(name, bytes)); + Collect probes = new Collect(); + BuildTable.write(one, new Collect(), probes, BuildTable.MAX_RECORD_BYTES); + assertEquals(1, probes.records.size()); + // What the map takes: its record and newline (ASCII), and its probes' walks (the method has a line, so + // the table's walks are all of them). + int mapBytes = probes.records.get(0).length() + 1; + long[] steps = {0}; + KeployBranchCoverage.analyze(bytes, new HashMap<>(), (m, r, h) -> steps[0] += h.steps); + // A chain of 200 early returns: each probe walks back over the checks before it. + assertTrue(steps[0] + " steps", steps[0] > 200L * 200); + assertTrue(mapBytes + " bytes", mapBytes > 100_000); + + long[][] bounds = {{mapBytes, steps[0]}, {mapBytes - 1, steps[0]}, {mapBytes, steps[0] - 1}}; + for (int i = 0; i < bounds.length; i++) { + boolean at = i == 0; + Collect table = new Collect(); + Collect map = new Collect(); + BuildTable.Summary s = BuildTable.write(one, table, map, BuildTable.MAX_RECORD_BYTES, + (int) bounds[i][0], bounds[i][1]); + String why = "bytes " + bounds[i][0] + ", steps " + bounds[i][1]; + assertEquals(why, 1, s.records); + assertEquals(why, at ? 0 : 1, s.notMeasured); + // Sent as record_too_large (the code Keploy's readers accept), counted apart in the SDK's own log. + assertEquals(why, at ? 0 : 1, s.tooComplex); + assertEquals(why + ": a map past its bound is not counted as a record too large", 0, s.tooLarge); + assertEquals(why, 0, s.notMapped); + assertEquals(why, at ? 1 : 0, s.probeRecords); + assertEquals(why, at ? 1 : 0, map.records.size()); + String rec = table.records.get(0); + if (at) { + assertTrue(rec, rec.contains("\"state\":\"ok\"")); + assertEquals(probes.records.get(0), map.records.get(0)); + } else { + assertTrue(rec, rec.contains("\"state\":\"not_measured\",\"reason\":{\"code\":\"record_too_large\"}") + && rec.endsWith("\"methods\":[]}")); + } + } + } + + @Test + public void theSendsLineCountsUnitsTooComplexApartFromRecordsTooLarge() throws Exception { + // A chain of 3,000 early returns: its probe map (about 43 MB) is past the 8 MiB bound. + String name = "gen/big/Returns3000"; + List one = Collections.singletonList( + new BuildTable.ClassFile(name, BuildTableBudgetTest.returns(name, 3000))); + try (Logs logs = new Logs()) { + BuildTable.send(() -> one, frame -> { }); + String line = null; + for (LogRecord r : logs.mine()) { + if (r.getMessage().contains(" sent (")) { + line = r.getMessage(); + } + } + assertNotNull(line); + assertTrue(line, line.contains("(1 classes, 1 not measured, 1 not measured as too complex to map" + + " (record_too_large: a probe map over 8 MiB or 64 Mi steps; the FINE log names each);")); + assertFalse(line, line.contains("not measured as too large")); + assertFalse(line, line.contains("unit_too_complex")); + } + } + + @Test + public void theTableAnswerBoundsTheHeapByTheIndexCountHeldAndReadsNoClassFile() { + long mb = 1 << 20; + String property = System.getProperty(BuildTable.MIN_HEAP_PROPERTY); + System.clearProperty(BuildTable.MIN_HEAP_PROPERTY); + try (Logs logs = new Logs()) { + // 63 MiB free. The classes are never read here: the count is the index's when it is read already. + BuildTable.Heap heap = new BuildTable.Heap(512 * mb, 449 * mb, true); + Supplier> never = () -> { + throw new AssertionError("the answer read the classes"); + }; + File none = new File("/nonexistent/keploy-data.sock"); + assertEquals(KeployDedupAgent.UNIX_REPLY_IGNORED + " reason=low_heap heap_free_mb=63 heap_needed_mb=66", + BuildTable.answer(heap, 1025, never, none)); + assertEquals("50,000 classes", KeployDedupAgent.UNIX_REPLY_IGNORED + + " reason=low_heap heap_free_mb=63 heap_needed_mb=113", BuildTable.answer(heap, 50_000, never, + none)); + assertEquals("an index not read yet: the base alone", KeployDedupAgent.UNIX_REPLY_IGNORED + + " reason=low_heap heap_free_mb=63 heap_needed_mb=64", BuildTable.answer(heap, -1, never, none)); + assertEquals(3, logs.records.size()); + } finally { + if (property != null) { + System.setProperty(BuildTable.MIN_HEAP_PROPERTY, property); + } + } + } + + /** Reads frames back: each is KTB1, a length, a header line and the next bytes of the stream. */ + static final class Frames implements BuildTable.FrameSender { + final List frames = new ArrayList<>(); + + @Override + public void send(byte[] frame) { + frames.add(frame); + } + } + + @Test + public void framesCarryTheStreamInOrderAndNeverPassTheCap() throws Exception { + Collect c = table(javaDedup()); + for (int cap : new int[] {1100, 1500, BuildTable.MAX_FRAME_BYTES}) { + Frames sent = new Frames(); + BuildTable.FrameWriter w = new BuildTable.FrameWriter(sent, cap); + BuildTable.Summary s = BuildTable.write(javaDedup(), w); + w.finish(s.build, s.records); + ByteArrayOutputStream stream = new ByteArrayOutputStream(); + for (int i = 0; i < sent.frames.size(); i++) { + byte[] f = sent.frames.get(i); + assertTrue("frame " + i + " is " + f.length + " bytes", f.length <= cap); + assertEquals(BuildTable.MAGIC, new String(f, 0, 4, StandardCharsets.US_ASCII)); + DataInputStream in = new DataInputStream(new java.io.ByteArrayInputStream(f, 4, f.length - 4)); + int n = in.readInt(); + assertEquals(f.length - 8, n); + int nl = 8; + while (f[nl] != '\n') { + nl++; + } + JsonObject h = JsonParser.parseString(new String(f, 8, nl - 8, StandardCharsets.UTF_8)).getAsJsonObject(); + assertEquals(1, h.get("v").getAsInt()); + assertEquals("table", h.get("part").getAsString()); + assertEquals(c.build, h.get("build").getAsString()); + assertEquals(i, h.get("seq").getAsInt()); + boolean last = i == sent.frames.size() - 1; + assertEquals(last, h.get("last").getAsBoolean()); + if (last) { + assertEquals(5, h.get("records").getAsInt()); + assertEquals(sha256Hex(c.ndjson.toByteArray()), h.get("sha256").getAsString()); + } else { + assertFalse(h.has("records")); + } + stream.write(f, nl + 1, f.length - nl - 1); + } + if (cap < 2000) { + assertTrue(sent.frames.size() > 5); + } else { + assertEquals(1, sent.frames.size()); + } + assertArrayEquals(c.ndjson.toByteArray(), stream.toByteArray()); + } + } + + @Test + public void theGoJsonFormIsKept() { + StringBuilder sb = new StringBuilder(); + BuildTable.Json.string(sb, "#&\"\\\b\f\n\r\t\u0001\u001f\u007f

é😀\uD800x"); + assertEquals("\"#&\\\"\\\\\\b\\f\\n\\r\\t\\u0001\\u001f\u007f\\u2028\\u2029é😀�x\"", + sb.toString()); + assertTrue(BuildTable.comparePkg("a", "a/b") < 0); + assertTrue(BuildTable.comparePkg("a/b", "a/b/c") < 0); + assertTrue(BuildTable.comparePkg("a/b/c", "a-x") < 0); + assertTrue(BuildTable.comparePkg("", "a") < 0); + // Code point order, as Go orders UTF-8 bytes: U+FF5E sorts before U+1F600. + assertTrue(BuildTable.compareCodePoints("~", "😀") < 0); + } + + @Test + public void namesFollowTheContract() { + assertEquals("Foo(String, int[])", BuildTable.Names.method("Foo", "", "", "(Ljava/lang/String;[I)V", false)); + assertEquals("Inner.Deep(long)", BuildTable.Names.method("Foo", "$Inner$Deep", "", "(J)V", false)); + assertEquals("Inner.bar(int)", BuildTable.Names.method("Foo", "$Inner", "bar", "(I)V", false)); + assertEquals("static {}", BuildTable.Names.method("Foo", "", "", "()V", false)); + assertEquals("bar lambda #0", BuildTable.Names.method("Foo", "", "lambda$bar$0", "(Ljava/lang/Object;)V", false)); + assertEquals("response(Object...)", BuildTable.Names.method("Foo", "", "response", "([Ljava/lang/Object;)V", true)); + assertEquals("of(Map.Entry, byte[][])", BuildTable.Names.method("Foo", "", "of", "(Ljava/util/Map$Entry;[[B)V", false)); + assertEquals("apply()", BuildTable.Names.method("Foo", "$", "apply", "()V", false)); + } + + static String header(byte[] frame) { + int nl = 8; + while (frame[nl] != '\n') { + nl++; + } + return new String(frame, 8, nl - 8, StandardCharsets.UTF_8); + } + + @Test + public void noClassesIsOneEmptyLastFrameOfEachPart() throws Exception { + BuildTable.Summary s = BuildTable.write(new ArrayList(), new Collect()); + assertEquals("", s.build); + assertEquals(0, s.records); + Frames f = new Frames(); + BuildTable.send(ArrayList::new, f); + assertEquals(2, f.frames.size()); + assertEquals("{\"v\":1,\"part\":\"table\",\"build\":\"\",\"seq\":0,\"last\":true,\"records\":0,\"sha256\":" + + "\"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\"}", header(f.frames.get(0))); + assertEquals("{\"v\":1,\"part\":\"probes\",\"build\":\"\",\"seq\":0,\"last\":true,\"records\":0,\"sha256\":" + + "\"e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855\"}", header(f.frames.get(1))); + } + + // ---------------------------------------------------------------- an edge set, against Go + + /** + * edgeClass generates a class: a constructor, run (varargs, an if and a switch), a lambda, a method on + * run's first line, two methods with odd names (a quote, #, $ and U+2028) and a static initializer on line 1; with lines false, none + * of them has line numbers. + */ + static byte[] edgeClass(String name, String source, boolean lines, int base) { + ClassWriter cw = new ClassWriter(ClassWriter.COMPUTE_FRAMES | ClassWriter.COMPUTE_MAXS); + cw.visit(Opcodes.V1_8, Opcodes.ACC_PUBLIC | Opcodes.ACC_SUPER, name, null, "java/lang/Object", null); + if (source != null) { + cw.visitSource(source, null); + } + MethodVisitor mv = cw.visitMethod(Opcodes.ACC_PUBLIC, "", "()V", null, null); + mv.visitCode(); + Label l0 = new Label(); + mv.visitLabel(l0); + if (lines) { + mv.visitLineNumber(base, l0); + } + mv.visitVarInsn(Opcodes.ALOAD, 0); + mv.visitMethodInsn(Opcodes.INVOKESPECIAL, "java/lang/Object", "", "()V", false); + mv.visitInsn(Opcodes.RETURN); + mv.visitMaxs(0, 0); + mv.visitEnd(); + int line = base + 1; + for (String m : new String[] {"run", "lambda$run$0", "same", "we\"ird#na$me", "a\u2028b"}) { + boolean run = m.equals("run"); + mv = cw.visitMethod(Opcodes.ACC_PUBLIC | Opcodes.ACC_STATIC | (run ? Opcodes.ACC_VARARGS : 0), m, + run ? "(I[Ljava/lang/String;)I" : "(I)I", null, null); + mv.visitCode(); + int first = m.equals("same") ? base + 1 : line; + Label a = new Label(); + mv.visitLabel(a); + if (lines) { + mv.visitLineNumber(first, a); + } + mv.visitVarInsn(Opcodes.ILOAD, 0); + Label els = new Label(); + mv.visitJumpInsn(Opcodes.IFEQ, els); + Label b = new Label(); + mv.visitLabel(b); + if (lines) { + mv.visitLineNumber(first + 1, b); + } + mv.visitInsn(Opcodes.ICONST_1); + mv.visitInsn(Opcodes.IRETURN); + mv.visitLabel(els); + Label c = new Label(); + mv.visitLabel(c); + if (lines) { + mv.visitLineNumber(first + 2, c); + } + mv.visitVarInsn(Opcodes.ILOAD, 0); + Label d1 = new Label(); + Label d2 = new Label(); + Label d3 = new Label(); + Label dflt = new Label(); + mv.visitTableSwitchInsn(1, 3, dflt, d1, d2, d3); + mv.visitLabel(d1); + mv.visitInsn(Opcodes.ICONST_2); + mv.visitInsn(Opcodes.IRETURN); + mv.visitLabel(d2); + mv.visitInsn(Opcodes.ICONST_3); + mv.visitInsn(Opcodes.IRETURN); + mv.visitLabel(d3); + mv.visitInsn(Opcodes.ICONST_4); + mv.visitInsn(Opcodes.IRETURN); + mv.visitLabel(dflt); + mv.visitInsn(Opcodes.ICONST_0); + mv.visitInsn(Opcodes.IRETURN); + mv.visitMaxs(0, 0); + mv.visitEnd(); + line += 10; + } + mv = cw.visitMethod(Opcodes.ACC_STATIC, "", "()V", null, null); + mv.visitCode(); + Label z = new Label(); + mv.visitLabel(z); + if (lines) { + mv.visitLineNumber(1, z); + } + mv.visitInsn(Opcodes.RETURN); + mv.visitMaxs(0, 0); + mv.visitEnd(); + cw.visitEnd(); + return cw.toByteArray(); + } + + /** The edge set: VM name and SourceFile (null: none). */ + static final String[][] EDGE = { + {"a/b/Foo", "Foo.java"}, {"a/b/Foo$Bar", "Foo.java"}, {"a/b/Foo$1", "Foo.java"}, {"a/b/Foo$", "Foo.scala"}, + {"a/b/Foo$Bar$Deep", null}, {"a-x/Baz", "Baz.java"}, {"a/Qux", "sub/Qux.java"}, {"a/b/c/Deep", ""}, + {"Default", "Default.java"}, {"$Lead", "Lead.java"}, {"$Lead$In", "Lead.java"}, + {"a/ü/Ünï", "Ünï.java"}, {"a/b/Z~", "Z.java"}, {"a/b/Z😀", "Z.java"}, + {"a/b/Q\"uo\\te<&>", "Q\"uo\\te.java"}, {"a/b/Ctl\u0001\u001f\u007f", "C
.java"}, + {"a/b/L
S
", "L.java"}, {"a/b/NoLines", "NoLines.java"}, {"a/b/Foo$Bar$", null}, + {"a/b/Foo$$anon$1", "Foo.scala"}, {"a/b/module-info", null}, {"a/b/package-info", "package-info.java"}, + }; + + static List edge() { + List out = new ArrayList<>(); + int base = 10; + for (String[] e : EDGE) { + out.add(new BuildTable.ClassFile(e[0], edgeClass(e[0], e[1], !e[0].endsWith("NoLines"), base))); + base += 7; + } + return out; + } + + @Test + public void anEdgeSetIsTheGoEncodersByteForByte() throws Exception { + List files = edge(); + Collect c = table(files); + dump("edge", files, c); + assertEquals(BUILD_GOLDEN, new String(resource("edge.build"), StandardCharsets.UTF_8).trim(), c.build); + assertEquals(new String(resource("edge.table.ndjson"), StandardCharsets.UTF_8), + new String(c.ndjson.toByteArray(), StandardCharsets.UTF_8)); + BuildTable.Summary s = BuildTable.write(files, new Collect()); + assertEquals(15, s.records); + assertEquals("NoLines has code and no line numbers", 1, s.noLines); + assertEquals(0, s.notMeasured); + } + + @Test + public void aClassIsNamedByItsBytesNotByWhereTheIndexFoundIt() throws Exception { + byte[] foo = edgeClass("com/acme/Foo", "Foo.java", true, 10); + byte[] in = edgeClass("com/acme/Foo$In", "Foo.java", true, 20); + byte[] junk = "not a class".getBytes(StandardCharsets.UTF_8); + Collect c = new Collect(); + BuildTable.Summary s = BuildTable.write(Arrays.asList(new BuildTable.ClassFile("classes/com/acme/Foo", foo), + new BuildTable.ClassFile("a/", in), new BuildTable.ClassFile("a/", junk)), c); + assertEquals(1, s.records); + assertEquals("not a class, and its index name names no class", 1, s.unnamed); + JsonObject u = JsonParser.parseString(c.records.get(0)).getAsJsonObject(); + assertEquals("c:com/acme/Foo", u.get("id").getAsString()); + assertEquals("com/acme", u.get("pkg").getAsString()); + assertEquals("com/acme/Foo.java", u.get("path").getAsString()); + assertEquals(IndexReport.classIdsSum(new long[] {CRC64.classId(foo), CRC64.classId(in)}), + u.get("hash").getAsString()); + // Every indexed class is in the build id, as in the index record's, named or not. + assertEquals(IndexReport.buildId(new long[] {CRC64.classId(foo), CRC64.classId(in), CRC64.classId(junk)}), + s.build); + assertTrue(c.records.get(0), c.records.get(0).contains("\"id\":\"m:com/acme/Foo$In#run(I[Ljava/lang/String;)I\"")); + } + + // ---------------------------------------------------------------- the data socket + + enum Mode { + /** reads the whole frame and answers OK */ + OK, + /** reads at most 512 bytes and closes, as the JSON decoder of an older Keploy does */ + READ_512_AND_CLOSE, + /** reads to the end of the connection and closes without an answer */ + READ_TO_EOF + } + + /** A receiver on a unix socket: what it read on each connection, and its answer per mode. */ + static final class Receiver implements AutoCloseable { + final AFUNIXServerSocket server; + final List got = new ArrayList<>(); + + Receiver(File path, Mode mode) throws IOException { + server = AFUNIXServerSocket.newInstance(); + server.bind(AFUNIXSocketAddress.of(path)); + Thread thread = new Thread(() -> { + while (!server.isClosed()) { + try (Socket s = server.accept()) { + s.setSoTimeout(15_000); + InputStream in = s.getInputStream(); + byte[] read; + if (mode == Mode.OK) { + DataInputStream d = new DataInputStream(in); + byte[] magic = new byte[4]; + d.readFully(magic); + byte[] body = new byte[d.readInt()]; + d.readFully(body); + read = body; + // Kept before the OK, as Keploy keeps a frame before it answers: once the sender has + // its OK, count() holds the frame. + synchronized (got) { + got.add(read); + } + s.getOutputStream().write("OK\n".getBytes(StandardCharsets.UTF_8)); + s.getOutputStream().flush(); + continue; + } else if (mode == Mode.READ_512_AND_CLOSE) { + byte[] b = new byte[512]; + int n = in.read(b); + read = Arrays.copyOf(b, Math.max(n, 0)); + } else { + ByteArrayOutputStream all = new ByteArrayOutputStream(); + byte[] b = new byte[8192]; + for (int n; (n = in.read(b)) > 0; ) { + all.write(b, 0, n); + } + read = all.toByteArray(); + } + synchronized (got) { + got.add(read); + } + } catch (IOException ignored) { + // closed, or the sender went away + } + } + }, "test-table-receiver"); + thread.setDaemon(true); + thread.start(); + } + + int count() { + synchronized (got) { + return got.size(); + } + } + + @Override + public void close() throws IOException { + server.close(); + } + } + + /** Logs collects what BuildTable logs. */ + static final class Logs extends Handler implements AutoCloseable { + final List records = new java.util.concurrent.CopyOnWriteArrayList<>(); + private final Logger logger = Logger.getLogger(BuildTable.class.getName()); + + Logs() { + logger.addHandler(this); + } + + @Override + public void publish(LogRecord record) { + records.add(record); + } + + /** mine is the records the calling thread logged: a send of another row's class may log meanwhile. */ + List mine() { + return mine(records); + } + + /** mine is those of records the calling thread logged. */ + @SuppressWarnings("deprecation") // getLongThreadID is JDK 16+; this SDK builds for Java 8 + static List mine(List records) { + int me = (int) Thread.currentThread().getId(); + List out = new ArrayList<>(); + for (LogRecord r : records) { + if (r.getThreadID() == me) { + out.add(r); + } + } + return out; + } + + @Override + public void flush() { + } + + @Override + public void close() { + logger.removeHandler(this); + } + } + + static List javaDedupUnchecked() { + try { + return javaDedup(); + } catch (Exception e) { + throw new IllegalStateException(e); + } + } + + @Test(timeout = 30_000) + public void theSocketSenderSendsEveryFrameEachAnsweredOk() throws Exception { + File sock = new File(tmp.getRoot(), "data.sock"); + try (Receiver r = new Receiver(sock, Mode.OK)) { + BuildTable.FrameWriter w = new BuildTable.FrameWriter(new BuildTable.SocketSender(sock), 1500); + BuildTable.Summary s = BuildTable.write(javaDedup(), w); + w.finish(s.build, s.records); + assertTrue(w.seq > 1); + assertEquals(w.seq, r.count()); + } + } + + @Test(timeout = 30_000) + public void aReceiverThatClosesWithoutOkEndsTheStreamAtOnceAndNothingThrows() throws Exception { + for (Mode mode : new Mode[] {Mode.READ_512_AND_CLOSE, Mode.READ_TO_EOF}) { + File sock = new File(tmp.getRoot(), mode + ".sock"); + try (Receiver r = new Receiver(sock, mode); Logs logs = new Logs()) { + long t0 = System.nanoTime(); + BuildTable.FrameWriter w = new BuildTable.FrameWriter(new BuildTable.SocketSender(sock), 1500); + try { + BuildTable.Summary s = BuildTable.write(javaDedup(), w); + w.finish(s.build, s.records); + fail(mode + ": no OK, so the stream stops"); + } catch (IOException e) { + assertTrue(mode + ": " + e.getMessage(), e.getMessage().startsWith("TABLE frame 0: ")); + } + assertEquals(mode.toString(), 0, w.seq); + assertTrue(mode + ": the sender shut its output, so no read waited for the reply timeout", + System.nanoTime() - t0 < TimeUnit.SECONDS.toNanos(10)); + if (mode == Mode.READ_TO_EOF) { + while (r.count() == 0) { + Thread.sleep(10); + } + synchronized (r.got) { + assertEquals(1, r.got.size()); + byte[] f = r.got.get(0); + assertEquals(BuildTable.MAGIC, new String(f, 0, 4, StandardCharsets.US_ASCII)); + int n = ((f[4] & 0xff) << 24) | ((f[5] & 0xff) << 16) | ((f[6] & 0xff) << 8) | (f[7] & 0xff); + assertEquals("the whole frame, then the end of the connection", 8 + n, f.length); + } + } + // send never throws: it says why at WARNING, with no stack trace. + BuildTable.send(BuildTableTest::javaDedupUnchecked, new BuildTable.SocketSender(sock)); + LogRecord last = logs.records.get(logs.records.size() - 1); + assertEquals(Level.WARNING, last.getLevel()); + assertTrue(last.getMessage(), last.getMessage().contains("the build table was not sent: TABLE frame 0: ")); + assertNull(last.getThrown()); + } + } + } + + @Test(timeout = 30_000) + public void noReceiverIsSaidSo() throws Exception { + try (Logs logs = new Logs()) { + BuildTable.send(BuildTableTest::javaDedupUnchecked, + new BuildTable.SocketSender(new File(tmp.getRoot(), "none.sock"))); + LogRecord last = logs.records.get(logs.records.size() - 1); + assertTrue(last.getMessage(), last.getMessage().contains("TABLE frame 0: no Keploy listens on the data socket")); + } + } + + @Test(timeout = 30_000) + public void anInterruptEndsTheSendAndIsSaidSo() throws Exception { + // Before the send, it reads no class (on JDK 8 a class file's read fails on an interrupted thread, and the + // index would keep what it got), so no frame goes. After the table's last frame, the interrupt closes the + // probe map's temp file (an interruptible channel) as it is read back. Either way the WARNING names the + // interrupt, not the temp file, the interrupt is kept, and no temp file is left. + List app = javaDedup(); + for (boolean afterTable : new boolean[] {false, true}) { + AtomicInteger read = new AtomicInteger(); + File dir = tmp.newFolder("spill-" + afterTable); + List heads = new ArrayList<>(); + BuildTable.FrameSender sender = frame -> { + int nl = 8; + while (frame[nl] != '\n') { + nl++; + } + String h = new String(frame, 8, nl - 8, StandardCharsets.UTF_8); + heads.add(h); + if (afterTable && h.contains("\"part\":\"table\"") && h.contains("\"last\":true")) { + Thread.currentThread().interrupt(); + } + }; + try (Logs logs = new Logs()) { + if (!afterTable) { + Thread.currentThread().interrupt(); + } + try { + BuildTable.send(() -> { + read.incrementAndGet(); + return app; + }, sender, dir); + assertTrue(afterTable + ": the interrupt is kept", Thread.currentThread().isInterrupted()); + } finally { + Thread.interrupted(); + } + LogRecord last = logs.records.get(logs.records.size() - 1); + assertEquals(Level.WARNING, last.getLevel()); + assertTrue(last.getMessage(), last.getMessage().endsWith("the build table was not sent: its thread was" + + " interrupted; it goes again when Keploy asks for it again")); + assertNull(last.getThrown()); + } + if (afterTable) { + assertFalse(heads.isEmpty()); + assertTrue(heads.toString(), heads.get(heads.size() - 1).contains("\"last\":true")); + for (String h : heads) { + assertTrue("no probe map frame: " + h, h.contains("\"part\":\"table\"")); + } + } else { + assertEquals(heads.toString(), 0, heads.size()); + } + assertEquals(afterTable + ": the classes read", afterTable ? 1 : 0, read.get()); + assertArrayEquals("no temp file left", new String[0], dir.list()); + } + } + + @Test(timeout = 30_000) + public void aRequestedTableArrivesWithTheBuildId() throws Exception { + File sock = new File(tmp.getRoot(), "table.sock"); + try (Receiver r = new Receiver(sock, Mode.OK)) { + BuildTable.Sends sends = new BuildTable.Sends(BuildTable::thread); + assertTrue(sends.request(() -> BuildTable.send(BuildTableTest::javaDedupUnchecked, + new BuildTable.SocketSender(sock)))); + String last = null; + String lastProbes = null; + while (last == null || lastProbes == null) { + synchronized (r.got) { + for (byte[] b : r.got) { + String h = new String(b, StandardCharsets.UTF_8); + h = h.substring(0, h.indexOf('\n')); + if (h.contains("\"last\":true")) { + if (h.contains("\"part\":\"table\"")) { + last = h; + } else { + lastProbes = h; + } + } + } + } + Thread.sleep(20); + } + String build = new String(resource("java-dedup.build"), StandardCharsets.UTF_8).trim(); + assertTrue(last, last.startsWith("{\"v\":1,\"part\":\"table\",\"build\":\"" + build + "\",")); + assertTrue(last, last.contains("\"records\":5,")); + assertTrue(lastProbes, lastProbes.startsWith("{\"v\":1,\"part\":\"probes\",\"build\":\"" + build + "\",")); + assertTrue(lastProbes, lastProbes.contains("\"records\":5,")); + } + } + + @Test(timeout = 30_000) + public void theRunAfterASendIsTheNewestRequests() throws Exception { + // A request names its index and data socket: after KeployDedupAgent.stop() and start() during a send, the + // new server's TABLE must be served from the new index to the new socket, not the first request's again. + CountDownLatch inFirst = new CountDownLatch(1); + CountDownLatch release = new CountDownLatch(1); + List ran = Collections.synchronizedList(new ArrayList<>()); + BuildTable.Sends sends = new BuildTable.Sends(BuildTable::thread); + assertTrue(sends.request(() -> { + ran.add("old"); + inFirst.countDown(); + try { + release.await(10, TimeUnit.SECONDS); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + })); + assertTrue(inFirst.await(10, TimeUnit.SECONDS)); + assertTrue(sends.request(() -> ran.add("middle"))); + assertTrue(sends.request(() -> ran.add("new"))); + release.countDown(); + while (ran.size() < 2) { + Thread.sleep(10); + } + Thread.sleep(300); + assertEquals(Arrays.asList("old", "new"), new ArrayList<>(ran)); + } + + @Test(timeout = 30_000) + public void aRequestMadeWhileASendFailsStillRuns() throws Exception { + // A send that throws is logged; the request made while it ran names the agent's newest index and data + // socket, so it must still run, and Sends must take requests again after it. + CountDownLatch inFirst = new CountDownLatch(1); + CountDownLatch release = new CountDownLatch(1); + CountDownLatch next = new CountDownLatch(1); + CountDownLatch after = new CountDownLatch(1); + List ran = Collections.synchronizedList(new ArrayList<>()); + BuildTable.Sends sends = new BuildTable.Sends(BuildTable::thread); + assertTrue(sends.request(() -> { + ran.add("failing"); + inFirst.countDown(); + try { + release.await(10, TimeUnit.SECONDS); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + throw new IllegalStateException("a send that fails (this row's own)"); + })); + assertTrue(inFirst.await(10, TimeUnit.SECONDS)); + assertTrue(sends.request(() -> { + ran.add("next"); + next.countDown(); + })); + release.countDown(); + assertTrue("the request made while the send failed ran", next.await(10, TimeUnit.SECONDS)); + assertTrue(sends.request(() -> { + ran.add("after"); + after.countDown(); + })); + assertTrue("a request after them runs", after.await(10, TimeUnit.SECONDS)); + assertEquals(Arrays.asList("failing", "next", "after"), new ArrayList<>(ran)); + } + + @Test(timeout = 30_000) + public void aRequestMadeWhileASendIsInterruptedRunsOnANewThread() throws Exception { + // An interrupt ends a send and is kept for its thread. The request made while it ran names the agent's + // newest index, not read yet: it must not run on that thread, where on JDK 8 the index's read of a class + // file fails. It runs on a new thread, uninterrupted, and Sends takes requests after it. + CountDownLatch inFirst = new CountDownLatch(1); + CountDownLatch release = new CountDownLatch(1); + CountDownLatch next = new CountDownLatch(1); + CountDownLatch after = new CountDownLatch(1); + AtomicReference first = new AtomicReference<>(); + AtomicReference second = new AtomicReference<>(); + AtomicReference secondInterrupted = new AtomicReference<>(); + AtomicInteger made = new AtomicInteger(); + BuildTable.Sends sends = new BuildTable.Sends(r -> { + made.incrementAndGet(); + return BuildTable.thread(r); + }); + assertTrue(sends.request(() -> { + first.set(Thread.currentThread()); + inFirst.countDown(); + try { + release.await(10, TimeUnit.SECONDS); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + Thread.currentThread().interrupt(); + })); + assertTrue(inFirst.await(10, TimeUnit.SECONDS)); + assertTrue(sends.request(() -> { + second.set(Thread.currentThread()); + secondInterrupted.set(Thread.currentThread().isInterrupted()); + next.countDown(); + })); + release.countDown(); + assertTrue("the request made while it ran ran", next.await(10, TimeUnit.SECONDS)); + assertNotSame("on a new thread", first.get(), second.get()); + assertEquals("not interrupted", Boolean.FALSE, secondInterrupted.get()); + assertEquals(2, made.get()); + assertTrue(sends.request(after::countDown)); + assertTrue("a request after them runs", after.await(10, TimeUnit.SECONDS)); + } + + @Test(timeout = 30_000) + public void aRequestMadeWhileTheNextThreadStartsRunsAfterItOnThatThread() throws Exception { + // Sends stays running while the interrupted thread starts the next one, so sends stay one at a time: a + // request made in that window starts no thread of its own and runs after the one being started, on the + // new thread. The factory holds the window open; it runs on the interrupted thread, so it spins rather + // than waits. + CountDownLatch inFirst = new CountDownLatch(1); + CountDownLatch release = new CountDownLatch(1); + CountDownLatch inHandover = new CountDownLatch(1); + CountDownLatch done = new CountDownLatch(1); + AtomicBoolean go = new AtomicBoolean(); + AtomicInteger made = new AtomicInteger(); + List ran = Collections.synchronizedList(new ArrayList<>()); + List on = Collections.synchronizedList(new ArrayList<>()); + BuildTable.Sends sends = new BuildTable.Sends(r -> { + if (made.incrementAndGet() == 2) { + inHandover.countDown(); + spin(go); + } + return BuildTable.thread(r); + }); + assertTrue(sends.request(() -> { + on.add(Thread.currentThread()); + inFirst.countDown(); + try { + release.await(10, TimeUnit.SECONDS); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + Thread.currentThread().interrupt(); + })); + assertTrue(inFirst.await(10, TimeUnit.SECONDS)); + assertTrue(sends.request(() -> { + on.add(Thread.currentThread()); + ran.add("next"); + })); + release.countDown(); + assertTrue("the interrupted thread starts the next one", inHandover.await(10, TimeUnit.SECONDS)); + try { + assertTrue("a request made meanwhile is taken", sends.request(() -> { + on.add(Thread.currentThread()); + ran.add("meanwhile"); + done.countDown(); + })); + assertEquals("and starts no thread of its own", 2, made.get()); + } finally { + go.set(true); + } + assertTrue("it runs", done.await(10, TimeUnit.SECONDS)); + assertEquals("after the one being started", Arrays.asList("next", "meanwhile"), new ArrayList<>(ran)); + assertEquals(2, made.get()); + assertNotSame("on the new thread", on.get(0), on.get(1)); + assertSame("both", on.get(1), on.get(2)); + } + + /** spin waits, without being interruptible, until go is set or 10 s pass. */ + private static void spin(AtomicBoolean go) { + long end = System.nanoTime() + 10_000_000_000L; + while (!go.get() && System.nanoTime() - end < 0) { + Thread.yield(); + } + } + + @Test(timeout = 30_000) + public void anInterruptedSendWhoseNextThreadCannotStartLetsSendsTakeRequests() throws Exception { + // The new thread for the request made during an interrupted send cannot start: that request is dropped + // with a WARNING (Keploy asks again), and so is one made while the thread was being started (ACKed, never + // sent). The next request starts a thread. The factory holds that window open by spinning: it runs on the + // interrupted thread. + CountDownLatch inFirst = new CountDownLatch(1); + CountDownLatch release = new CountDownLatch(1); + CountDownLatch inHandover = new CountDownLatch(1); + CountDownLatch after = new CountDownLatch(1); + AtomicBoolean go = new AtomicBoolean(); + AtomicInteger made = new AtomicInteger(); + AtomicInteger dropped = new AtomicInteger(); + BuildTable.Sends sends = new BuildTable.Sends(r -> { + if (made.incrementAndGet() == 2) { + inHandover.countDown(); + spin(go); + throw new OutOfMemoryError("unable to create native thread"); + } + return BuildTable.thread(r); + }); + try (Logs logs = new Logs()) { + assertTrue(sends.request(() -> { + inFirst.countDown(); + try { + release.await(10, TimeUnit.SECONDS); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + Thread.currentThread().interrupt(); + })); + assertTrue(inFirst.await(10, TimeUnit.SECONDS)); + assertTrue(sends.request(dropped::incrementAndGet)); + release.countDown(); + assertTrue("the interrupted thread tries to start one", inHandover.await(10, TimeUnit.SECONDS)); + try { + assertTrue("a request made meanwhile is ACKed", sends.request(dropped::incrementAndGet)); + assertEquals(2, made.get()); + } finally { + go.set(true); + } + // The WARNING is logged once Sends has let go of the dropped requests. + long end = System.nanoTime() + 10_000_000_000L; + boolean warned = false; + while (!warned && System.nanoTime() < end) { + for (LogRecord rec : logs.records) { + warned |= rec.getLevel() == Level.WARNING + && rec.getMessage().contains("the build table's thread did not start"); + } + Thread.sleep(10); + } + assertTrue("a WARNING says so", warned); + assertEquals(2, made.get()); + assertTrue("the next request starts a thread", sends.request(after::countDown)); + assertTrue(after.await(10, TimeUnit.SECONDS)); + assertEquals("the dropped requests never ran", 0, dropped.get()); + assertEquals(3, made.get()); + } + } + + /** Throwing is a JUL handler on BuildTable's logger that throws on every record, as an app's bridge can. */ + static final class Throwing extends Handler implements AutoCloseable { + final List published = new java.util.concurrent.CopyOnWriteArrayList<>(); + private final Logger logger = Logger.getLogger(BuildTable.class.getName()); + + Throwing() { + logger.addHandler(this); + } + + @Override + public void publish(LogRecord record) { + published.add(record); + throw new IllegalStateException("a log handler that throws (this row's own)"); + } + + /** + * mine is the messages of the records the calling thread logged (another row's send may log meanwhile), + * each with its exception's message after " | ". + */ + List mine() { + List out = new ArrayList<>(); + for (LogRecord r : Logs.mine(published)) { + Throwable t = r.getThrown(); + out.add(r.getMessage() + (t == null ? "" : " | " + t.getMessage())); + } + return out; + } + + /** count is how many of mine hold each of parts. */ + int count(String... parts) { + int n = 0; + next: + for (String m : mine()) { + for (String p : parts) { + if (!m.contains(p)) { + continue next; + } + } + n++; + } + return n; + } + + @Override + public void flush() { + } + + @Override + public void close() { + logger.removeHandler(this); + } + } + + @Test(timeout = 60_000) + public void aLogHandlerThatThrowsNeitherFailsASendNorWedgesSends() throws Exception { + List classes = javaDedup(); + try (Throwing h = new Throwing()) { + // send logs what it sent (INFO) or why it stopped (WARNING), and never throws. + List frames = Collections.synchronizedList(new ArrayList<>()); + BuildTable.send(() -> classes, frames::add); + assertFalse("the table was sent", frames.isEmpty()); + BuildTable.send(() -> { + throw new IllegalStateException("classes that cannot be read (this row's own)"); + }, f -> fail("nothing to send")); + // Each send's outcome went through the handler. Only those lines are counted: a collection during + // the first send adds its paced line. + assertEquals("the send logged through the handler: " + h.mine(), 1, + h.count("Keploy coverage: build table ", " sent (5 classes")); + assertEquals("the failed send logged through the handler: " + h.mine(), 1, + h.count("Keploy coverage: the build table was not sent | classes that cannot be read")); + // Sends logs a send that throws through the same handler: the request made after it still runs. + BuildTable.Sends sends = new BuildTable.Sends(BuildTable::thread); + CountDownLatch failing = new CountDownLatch(1); + assertTrue(sends.request(() -> { + failing.countDown(); + throw new IllegalStateException("a send that fails (this row's own)"); + })); + assertTrue(failing.await(10, TimeUnit.SECONDS)); + CountDownLatch next = new CountDownLatch(1); + assertTrue(sends.request(next::countDown)); + assertTrue("a request after a send that failed runs", next.await(10, TimeUnit.SECONDS)); + // The production send through Sends, then one more request. + CountDownLatch sent = new CountDownLatch(1); + assertTrue(sends.request(() -> BuildTable.send(() -> classes, frame -> sent.countDown()))); + assertTrue(sent.await(20, TimeUnit.SECONDS)); + CountDownLatch last = new CountDownLatch(1); + assertTrue(sends.request(last::countDown)); + assertTrue("a request after the production send runs", last.await(20, TimeUnit.SECONDS)); + } + } + + @Test(timeout = 30_000) + public void aRequestDuringASendRunsItOnceMore() throws Exception { + CountDownLatch inFirst = new CountDownLatch(1); + CountDownLatch release = new CountDownLatch(1); + AtomicInteger runs = new AtomicInteger(); + Runnable send = () -> { + if (runs.incrementAndGet() == 1) { + inFirst.countDown(); + try { + release.await(); + } catch (InterruptedException e) { + Thread.currentThread().interrupt(); + } + } + }; + BuildTable.Sends sends = new BuildTable.Sends(BuildTable::thread); + assertTrue(sends.request(send)); + assertTrue(inFirst.await(10, TimeUnit.SECONDS)); + assertTrue(sends.request(send)); + assertTrue(sends.request(send)); + release.countDown(); + while (runs.get() < 2) { + Thread.sleep(10); + } + Thread.sleep(300); + assertEquals("two requests during a send: one more run", 2, runs.get()); + assertTrue(sends.request(send)); + while (runs.get() < 3) { + Thread.sleep(10); + } + } + + @Test + public void aThreadThatCannotStartIsIgnoredNeverThrown() throws Exception { + AtomicInteger made = new AtomicInteger(); + BuildTable.Sends sends = new BuildTable.Sends(r -> { + if (made.incrementAndGet() == 1) { + throw new OutOfMemoryError("unable to create native thread"); + } + return BuildTable.thread(r); + }); + CountDownLatch ran = new CountDownLatch(1); + try (Logs logs = new Logs()) { + assertFalse(sends.request(ran::countDown)); + assertEquals(Level.WARNING, logs.records.get(logs.records.size() - 1).getLevel()); + } + assertTrue("nothing is left running: the next request starts", sends.request(ran::countDown)); + assertTrue(ran.await(10, TimeUnit.SECONDS)); + } + + // ---------------------------------------------------------------- what a send holds, and the heap it needs + + /** body is a frame's bytes after its header line. */ + static String body(byte[] frame) { + int nl = 8; + while (frame[nl] != '\n') { + nl++; + } + return new String(frame, nl + 1, frame.length - nl - 1, StandardCharsets.UTF_8); + } + + @Test + public void eachClassesProbeRecordsGoOutAsItIsWrittenNotKeptForTheEnd() throws Exception { + // Nothing of the probe map waits for the table's end in the heap: a measured unit's probe records reach + // their sink just before its own table record, so before the next unit is analyzed. + List events = new ArrayList<>(); + BuildTable.Summary s = BuildTable.write(javaDedup(), + (build, json) -> events.add("T" + new String(json, StandardCharsets.UTF_8)), + (build, json) -> events.add("P" + new String(json, StandardCharsets.UTF_8))); + assertEquals(5, s.records); + assertEquals(5, s.probeRecords); + List since = new ArrayList<>(); + int units = 0; + for (String e : events) { + JsonObject o = JsonParser.parseString(e.substring(1)).getAsJsonObject(); + if (e.charAt(0) == 'P') { + since.add(o.get("unit").getAsString()); + continue; + } + units++; + boolean measured = "ok".equals(o.get("state").getAsString()); + assertEquals(e, measured, !since.isEmpty()); + for (String unit : since) { + assertEquals("a probe record goes just before its own unit's table record", o.get("id").getAsString(), + unit); + } + since.clear(); + } + assertEquals(5, units); + assertTrue("no probe record after the last table record", since.isEmpty()); + } + + @Test + public void theProbeMapWaitsInATempFileThatIsGoneOnceTheSendEnds() throws Exception { + File dir = tmp.newFolder("spill"); + List frames = new ArrayList<>(); + List during = new ArrayList<>(); + BuildTable.send(BuildTableTest::javaDedupUnchecked, frame -> { + frames.add(frame); + during.add(Arrays.toString(dir.list())); + }, dir); + assertEquals(2, frames.size()); + String build = new String(resource("java-dedup.build"), StandardCharsets.UTF_8).trim(); + byte[] map = resource("java-dedup.probes.ndjson"); + assertEquals("{\"v\":1,\"part\":\"probes\",\"build\":\"" + build + "\",\"seq\":0,\"last\":true,\"records\":5," + + "\"sha256\":\"" + sha256Hex(map) + "\"}", header(frames.get(1))); + assertEquals(new String(map, StandardCharsets.UTF_8), body(frames.get(1))); + assertEquals(new String(resource("java-dedup.table.ndjson"), StandardCharsets.UTF_8), body(frames.get(0))); + assertArrayEquals("the temp file is removed", new String[0], dir.list()); + if (File.separatorChar == '/') { + // Unlinked as soon as it is open: a JVM that dies mid-send leaves no file behind. + assertEquals(Arrays.asList("[]", "[]"), during); + } + } + + @Test + public void aTempDirTheAppCannotWriteSendsNoFrameAndSaysWhich() throws Exception { + File notADir = tmp.newFile("not-a-dir"); + List frames = new ArrayList<>(); + AtomicInteger read = new AtomicInteger(); + try (Logs logs = new Logs()) { + BuildTable.send(() -> { + read.incrementAndGet(); + return javaDedupUnchecked(); + }, frames::add, notADir); + assertTrue("no frame: Keploy gets no table for the build, and asks again next run", frames.isEmpty()); + assertEquals("the classes are not read", 0, read.get()); + LogRecord last = logs.records.get(logs.records.size() - 1); + assertEquals(Level.WARNING, last.getLevel()); + assertTrue(last.getMessage(), last.getMessage().contains(notADir.getPath()) + && last.getMessage().contains("java.io.tmpdir")); + } + } + + @Test + public void aShortHeapDeclinesTheTableWithLowHeap() { + long mb = 1 << 20; + try (Logs logs = new Logs()) { + assertEquals("reason=low_heap heap_free_mb=112 heap_needed_mb=256", + BuildTable.lowHeap(new BuildTable.Heap(512 * mb, 400 * mb, true), 256)); + assertEquals(1, logs.records.size()); + LogRecord r = logs.records.get(0); + assertEquals(Level.WARNING, r.getLevel()); + assertTrue(r.getMessage(), r.getMessage().contains("112 MiB free of the 512 MiB heap cap the JVM reports" + + " (in use 400 MiB after the last collection), and it needs 256 MiB") + && r.getMessage().contains("low_heap") + && r.getMessage().contains("KEPLOY_COVERAGE_TABLE_MIN_HEAP_MB")); + } + assertNull("exactly what it needs is enough", BuildTable.lowHeap(new BuildTable.Heap(512 * mb, 256 * mb, + true), 256)); + assertEquals("reason=low_heap heap_free_mb=255 heap_needed_mb=256", + BuildTable.lowHeap(new BuildTable.Heap(512 * mb, 256 * mb + 1, true), 256)); + assertNull("0 turns the check off", BuildTable.lowHeap(new BuildTable.Heap(512 * mb, 512 * mb, false), 0)); + assertEquals("in use past the cap is no free heap", "reason=low_heap heap_free_mb=0 heap_needed_mb=1", + BuildTable.lowHeap(new BuildTable.Heap(512 * mb, 600 * mb, false), 1)); + assertNull("a JVM with no heap cap", BuildTable.lowHeap(new BuildTable.Heap(Long.MAX_VALUE, 400 * mb, + true), 256)); + } + + @Test + public void theFreeHeapATableNeedsIsTheEnvElseThePropertyElse64MiBAnd1KiBAClass() { + assertEquals(64, BuildTable.minHeapMb(null, null, 0)); + assertEquals(65, BuildTable.minHeapMb(" ", "", 1)); + assertEquals(65, BuildTable.minHeapMb(null, null, 1024)); + assertEquals(66, BuildTable.minHeapMb(null, null, 1025)); + assertEquals("50,000 classes", 113, BuildTable.minHeapMb(null, null, 50_000)); + assertEquals(100, BuildTable.minHeapMb("100", "50", 50_000)); + assertEquals(50, BuildTable.minHeapMb(" ", "50", 50_000)); + assertEquals(7, BuildTable.minHeapMb(" 7 ", null, 50_000)); + assertEquals(0, BuildTable.minHeapMb(null, "0", 50_000)); + try (Logs logs = new Logs()) { + assertEquals(113, BuildTable.minHeapMb("-1", "50", 50_000)); + assertEquals(64, BuildTable.minHeapMb(null, "1g", 0)); + assertEquals(2, logs.records.size()); + assertEquals(Level.WARNING, logs.records.get(1).getLevel()); + assertTrue(logs.records.get(1).getMessage(), logs.records.get(1).getMessage().contains("'1g'") + && logs.records.get(1).getMessage().contains("needs 64 MiB")); + } + } + + @Test + public void aTableOf50000ClassesIsBuiltWith113MiBFreeAndDeclinedWith112() { + // Both edges of the default bound, as the answer reads it: the count of the index's class list. + long mb = 1 << 20; + long need = BuildTable.minHeapMb(null, null, 50_000); + assertNull("113 MiB free is enough", BuildTable.lowHeap(new BuildTable.Heap(1024 * mb, (1024 - 113) * mb, + true), need)); + try (Logs logs = new Logs()) { + assertEquals("reason=low_heap heap_free_mb=112 heap_needed_mb=113", BuildTable.lowHeap( + new BuildTable.Heap(1024 * mb, (1024 - 113) * mb + 1, true), need)); + } + // A 1 GiB pod with no -Xmx caps the heap at about 256 MiB: with its app's live set under 143 MiB, a + // 50,000-class app's table is built (the 256 MiB bound before declined every such pod). + assertNull(BuildTable.lowHeap(new BuildTable.Heap(256 * mb, 143 * mb, true), need)); + } + + @Test + public void theHeapInUseIsWhatTheNewestCollectionLeftInTheHeapPools() { + long mb = 1 << 20; + java.util.Set heap = new HashSet<>(Arrays.asList("Eden", "Old")); + // The newest is neither the first nor the last; the older ones left more in use; Metaspace and the code + // cache are not the heap. + BuildTable.GcEnd older = gcEnd(100, "Eden", 0, "Old", 300 * mb, "Metaspace", 40 * mb); + BuildTable.GcEnd newest = gcEnd(300, "Eden", 2 * mb, "Old", 100 * mb, "Metaspace", 50 * mb, + "CodeHeap", 20 * mb); + BuildTable.GcEnd middle = gcEnd(200, "Eden", 0, "Old", 250 * mb); + assertEquals(102 * mb, BuildTable.Heap.usedAfterNewest(Arrays.asList(older, newest, middle), heap)); + assertEquals(102 * mb, BuildTable.Heap.usedAfterNewest(Arrays.asList(middle, older, newest), heap)); + assertEquals("no collection seen", -1, + BuildTable.Heap.usedAfterNewest(Collections.emptyList(), heap)); + } + + @Test + public void theHeapInUseIsTheSmallerOfNowAndTheNewestCollection() { + BuildTable.Heap h = BuildTable.Heap.of(512, 400, 300); + assertEquals(512, h.max); + assertEquals("garbage made since the collection does not count", 300, h.used); + assertTrue(h.afterGc); + h = BuildTable.Heap.of(512, 400, 400); + assertEquals(400, h.used); + assertTrue(h.afterGc); + h = BuildTable.Heap.of(512, 400, 500); + assertEquals("less in use now than the collection left: now is newer", 400, h.used); + assertFalse("the figure is the heap in use now", h.afterGc); + h = BuildTable.Heap.of(512, 400, -1); + assertEquals("no collection seen", 400, h.used); + assertFalse(h.afterGc); + } + + @Test + public void thisJvmsCollectionsAreRead() { + // This JVM reports its collections; the check itself never runs one. + System.gc(); + long after = BuildTable.LastCollection.heapUsedAfter(); + assertTrue("a collection ran: " + after, after > 0); + BuildTable.Heap h = BuildTable.Heap.read(); + assertEquals(Runtime.getRuntime().maxMemory(), h.max); + assertTrue(h.used + " of " + h.max, h.used > 0 && h.used <= h.max); + } + + @Test + public void theProbeMapsTempFileIsNeverOpenedThroughALink() throws IOException { + // Another user who swaps the just-made temp file for a link to a file the app can write gets nothing + // written through it. + Path target = tmp.newFile("target").toPath(); + Files.write(target, "keep".getBytes(StandardCharsets.UTF_8)); + Path link = tmp.getRoot().toPath().resolve("keploy-probe-map-1.ndjson"); + try { + Files.createSymbolicLink(link, target); + } catch (IOException | UnsupportedOperationException e) { + org.junit.Assume.assumeNoException(e); + } + try { + BuildTable.Spill.openOwned(link).close(); + fail("opened through a link"); + } catch (IOException expected) { + // ELOOP + } + assertEquals("keep", new String(Files.readAllBytes(target), StandardCharsets.UTF_8)); + assertTrue("the target is not deleted", Files.exists(target)); + } + + private static BuildTable.GcEnd gcEnd(long endTime, Object... poolUsed) { + Map used = new HashMap<>(); + for (int i = 0; i < poolUsed.length; i += 2) { + used.put((String) poolUsed[i], ((Number) poolUsed[i + 1]).longValue()); + } + return new BuildTable.GcEnd(endTime, used); + } +} diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/ControlSocketTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/ControlSocketTest.java index c365d74b..4524f0a6 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/ControlSocketTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/ControlSocketTest.java @@ -19,7 +19,11 @@ import java.nio.file.Files; import java.nio.file.attribute.FileTime; import java.nio.file.attribute.PosixFilePermission; +import java.security.MessageDigest; +import java.security.NoSuchAlgorithmException; +import java.util.ArrayList; import java.util.List; +import java.util.TreeSet; import java.util.concurrent.BlockingQueue; import java.util.concurrent.CopyOnWriteArrayList; import java.util.concurrent.LinkedBlockingQueue; @@ -28,6 +32,7 @@ import java.util.logging.Level; import java.util.logging.LogRecord; import java.util.logging.Logger; +import org.jacoco.core.internal.data.CRC64; import org.junit.After; import org.junit.Before; import org.junit.Test; @@ -52,6 +57,7 @@ public class ControlSocketTest { private static final String JACOCO_PORT_PROPERTY = "keploy.jacoco.port"; private static final String WARMUP_DISABLED_PROPERTY = "keploy.java.dedup.warmup.disabled"; + private static final String CLASS_DIRS_PROPERTY = "keploy.java.class.dirs"; private static final String TAKEN = "ACK proto=2\n"; private static final String IGNORED = "IGNORED proto=2\n"; @@ -92,6 +98,15 @@ public void setUp() throws IOException { KeployDedupAgent.dataSocketPath = new File(dir, "data.sock").getAbsolutePath(); // The warmup loads every class on the test classpath; nothing here needs it. System.setProperty(WARMUP_DISABLED_PROPERTY, "true"); + // A TABLE row answers ACK whatever heap this JVM was given (-DargLine=-Xmx...): the heap check is off + // unless a row sets its own bound. + System.setProperty(BuildTable.MIN_HEAP_PROPERTY, "0"); + } + + /** A TABLE row needs the heap check's env var unset: it would win over the property setUp sets. */ + private static void assumeNoMinHeapEnv() { + String v = System.getenv(BuildTable.MIN_HEAP_ENV); + org.junit.Assume.assumeTrue(BuildTable.MIN_HEAP_ENV + " is set", v == null || v.trim().isEmpty()); } @After @@ -107,19 +122,25 @@ public void tearDown() throws Exception { KeployDedupAgent.dataSocketPath = savedDataPath; System.clearProperty(JACOCO_PORT_PROPERTY); System.clearProperty(WARMUP_DISABLED_PROPERTY); + System.clearProperty(CLASS_DIRS_PROPERTY); + System.clearProperty(BuildTable.MIN_HEAP_PROPERTY); if (jacoco != null) { jacoco.close(); } if (data != null) { data.close(); } - File[] left = dir.listFiles(); + delete(dir); + } + + private static void delete(File f) { + File[] left = f.listFiles(); if (left != null) { - for (File f : left) { - f.delete(); + for (File c : left) { + delete(c); } } - dir.delete(); + f.delete(); } @Test @@ -170,6 +191,232 @@ public void aTestStartedAndEndedIsAnsweredTakenAndItsCoverageIsSent() throws Exc assertTrue(payload, payload.contains("\"id\":\"test-set-0/test-1\"")); } + @Test(timeout = 60000) + public void theBuildTableGoesOnlyWhenKeployAsksForIt() throws Exception { + assumeNoMinHeapEnv(); + // The index is the java-dedup sample's classes, so the table is the Go encoder's golden for them. + File classes = new File(dir, "classes"); + for (BuildTable.ClassFile f : BuildTableTest.javaDedup()) { + File out = new File(classes, f.name + ".class"); + assertTrue(out.getParentFile().isDirectory() || out.getParentFile().mkdirs()); + Files.write(out.toPath(), f.bytes); + } + System.setProperty(CLASS_DIRS_PROPERTY, classes.getPath()); + startWithJacoco(); + data = new DataReceiver(KeployDedupAgent.dataSocketPath); + int asked = BuildTable.requests(); + assertEquals(TAKEN, send("START test-set-0/test-1")); + assertEquals(TAKEN, send("END test-set-0/test-1")); + String payload = data.next(); + assertNotNull("the END's coverage reached the data socket", payload); + assertTrue(payload, payload.contains("\"id\":\"test-set-0/test-1\"")); + // Commands are handled one at a time, so once END is answered all START and END asked for is asked. + // An older Keploy never sends TABLE: it never gets a frame. + assertEquals("START and END never ask for the table", asked, BuildTable.requests()); + assertNull("no table frame without TABLE", data.poll()); + + assertEquals("answered at once; the table goes on its own thread", TAKEN, send("TABLE")); + assertEquals(asked + 1, BuildTable.requests()); + String frame = data.next(); + assertNotNull("the TABLE's frame reached the data socket", frame); + assertTrue(frame, frame.startsWith("KTB1")); + int header = frame.indexOf("{\"v\":1,"); + int body = frame.indexOf('\n', header) + 1; + assertTrue(frame, header > 0 && body > header); + byte[] golden = BuildTableTest.resource("java-dedup.table.ndjson"); + String build = new String(BuildTableTest.resource("java-dedup.build"), StandardCharsets.UTF_8).trim(); + assertEquals("{\"v\":1,\"part\":\"table\",\"build\":\"" + build + "\",\"seq\":0,\"last\":true," + + "\"records\":5,\"sha256\":\"" + sha256(golden) + "\"}\n", frame.substring(header, body)); + assertEquals(new String(golden, StandardCharsets.UTF_8), frame.substring(body)); + // The probe map follows the table, as its own stream with the same build. + String probes = data.next(); + assertNotNull("the probe map followed the table", probes); + byte[] map = BuildTableTest.resource("java-dedup.probes.ndjson"); + header = probes.indexOf("{\"v\":1,"); + body = probes.indexOf('\n', header) + 1; + assertEquals("{\"v\":1,\"part\":\"probes\",\"build\":\"" + build + "\",\"seq\":0,\"last\":true," + + "\"records\":5,\"sha256\":\"" + sha256(map) + "\"}\n", probes.substring(header, body)); + assertEquals(new String(map, StandardCharsets.UTF_8), probes.substring(body)); + assertNull("nothing more", data.payloads.poll(1, TimeUnit.SECONDS)); + } + + @Test(timeout = 60000) + public void aTableAskedWhenTheHeapIsShortIsDeclinedWithLowHeapAndSendsNothing() throws Exception { + assumeNoMinHeapEnv(); + File classes = new File(dir, "classes"); + for (BuildTable.ClassFile f : BuildTableTest.javaDedup()) { + File out = new File(classes, f.name + ".class"); + assertTrue(out.getParentFile().isDirectory() || out.getParentFile().mkdirs()); + Files.write(out.toPath(), f.bytes); + } + System.setProperty(CLASS_DIRS_PROPERTY, classes.getPath()); + long needed = Long.MAX_VALUE >> 20; + System.setProperty(BuildTable.MIN_HEAP_PROPERTY, String.valueOf(needed)); + try { + startWithJacoco(); + data = new DataReceiver(KeployDedupAgent.dataSocketPath); + int asked = BuildTable.requests(); + assertEquals(TAKEN, send("START test-set-0/test-1")); + String reply = send("TABLE"); + assertTrue(reply, reply.matches("IGNORED proto=2 reason=low_heap heap_free_mb=\\d+ heap_needed_mb=" + + needed + "\n")); + assertEquals("no send started", asked, BuildTable.requests()); + assertNull("no table frame", data.payloads.poll(2, TimeUnit.SECONDS)); + assertEquals("the control server goes on", TAKEN, send("END test-set-0/test-1")); + } finally { + System.clearProperty(BuildTable.MIN_HEAP_PROPERTY); + } + } + + @Test(timeout = 60000) + public void theIndexRecordAndTheTableNameOneBuild() throws Exception { + assumeNoMinHeapEnv(); + // Keploy finds the table under the build the INDEX record named. The index is the java-dedup sample's + // classes and one file no class reader reads: both builds take every class the index holds. + File classes = new File(dir, "classes"); + List files = new ArrayList<>(BuildTableTest.javaDedup()); + files.add(new BuildTable.ClassFile("com/acme/Junk", "not a class".getBytes(StandardCharsets.UTF_8))); + TreeSet ids = new TreeSet<>(); + for (BuildTable.ClassFile f : files) { + File out = new File(classes, f.name + ".class"); + assertTrue(out.getParentFile().isDirectory() || out.getParentFile().mkdirs()); + Files.write(out.toPath(), f.bytes); + ids.add(String.format("%016x", CRC64.classId(f.bytes))); + } + // Keploy's codecoverage.JavaBuildID: the table format, the analyzer's JaCoCo release, then + // the ids as 16 hex, sorted (as hex: unsigned), joined by newlines. + String build = "jv1-" + sha256(("keploy-java-table 1\njacoco " + IndexReportTest.jacocoCore() + "\n" + + String.join("\n", ids)).getBytes(StandardCharsets.UTF_8)).substring(0, 32); + assertNotEquals("the file no class reader reads is in the build", + new String(BuildTableTest.resource("java-dedup.build"), StandardCharsets.UTF_8).trim(), build); + System.setProperty(CLASS_DIRS_PROPERTY, classes.getPath()); + data = new DataReceiver(KeployDedupAgent.dataSocketPath); + startWithJacoco(); + String index = data.indexes.poll(30, TimeUnit.SECONDS); + assertNotNull("the INDEX record came once the data socket was bound", index); + assertTrue(index, index.contains("\"classes\":" + files.size() + ",")); + assertEquals(index, build, field(index, "build")); + + assertEquals(TAKEN, send("TABLE")); + boolean tableDone = false; + boolean probesDone = false; + while (!tableDone || !probesDone) { + String frame = data.next(); + assertNotNull("the table and the probe map each reached their last frame", frame); + int at = frame.indexOf("{\"v\":1,"); + assertTrue(frame, frame.startsWith("KTB1") && at > 0); + String header = frame.substring(at, frame.indexOf('\n', at)); + assertEquals(header, build, field(header, "build")); + boolean last = header.contains("\"last\":true"); + if (header.contains("\"part\":\"table\"")) { + tableDone |= last; + } else { + assertTrue(header, header.contains("\"part\":\"probes\"")); + probesDone |= last; + } + } + assertNull("nothing more", data.payloads.poll(1, TimeUnit.SECONDS)); + } + + @Test(timeout = 300000) + public void startAndEndAreAnsweredWhileTheTableIsAnalyzed() throws Exception { + assumeNoMinHeapEnv(); + // Keploy sends TABLE after the first ACK and goes on with its tests: the table's analysis runs on its own + // thread, so no START or END waits for it. The index is a generated app large enough that its analysis + // outlasts many START/END pairs. + File classes = new File(dir, "classes"); + int n = 5000; + for (int i = 0; i < n; i++) { + String name = BuildTableBudgetTest.name(i); + File out = new File(classes, name + ".class"); + assertTrue(out.getParentFile().isDirectory() || out.getParentFile().mkdirs()); + Files.write(out.toPath(), BuildTableBudgetTest.gen(name, i, BuildTableBudgetTest.BRANCHES)); + } + System.setProperty(CLASS_DIRS_PROPERTY, classes.getPath()); + data = new DataReceiver(KeployDedupAgent.dataSocketPath); + startWithJacoco(); + assertNotNull("the INDEX record (the index is read)", data.indexes.poll(60, TimeUnit.SECONDS)); + assertEquals(TAKEN, send("START test-set-0/test-0")); + assertEquals(TAKEN, send("END test-set-0/test-0")); + assertNotNull(data.next()); + + long t0 = System.nanoTime(); + assertEquals(TAKEN, send("TABLE")); + long tableAck = System.nanoTime() - t0; + boolean tableDone = false; + boolean probesDone = false; + int during = 0; + long worst = 0; + for (int k = 1; !tableDone || !probesDone; k++) { + long s = System.nanoTime(); + assertEquals(TAKEN, send("START test-set-0/test-" + k)); + assertEquals(TAKEN, send("END test-set-0/test-" + k)); + worst = Math.max(worst, System.nanoTime() - s); + boolean ended = false; + for (String p; (p = data.payloads.poll(ended ? 0 : 30, TimeUnit.SECONDS)) != null; ) { + if (!p.startsWith("KTB1")) { + assertTrue(p, p.contains("\"id\":\"test-set-0/test-" + k + "\"")); + ended = true; + continue; + } + int at = p.indexOf("{\"v\":1,"); + String header = p.substring(at, p.indexOf('\n', at)); + boolean last = header.contains("\"last\":true"); + tableDone |= last && header.contains("\"part\":\"table\""); + probesDone |= last && header.contains("\"part\":\"probes\""); + } + assertTrue("the END's coverage reached the data socket", ended); + if (!tableDone || !probesDone) { + during++; + } + } + long table = System.nanoTime() - t0; + System.out.printf("START/END pairs answered while the table of %d classes was analyzed and sent: %d in %d ms;" + + " slowest pair %d ms; TABLE answered in %d ms%n", n, during, table / 1_000_000, worst / 1_000_000, + tableAck / 1_000_000); + assertTrue("START and END were answered while the table was analyzed: " + during + " pairs in " + + table / 1_000_000 + " ms", during >= 3); + // The bounds are generous (a pair takes about 0.1 s): they fail when a START/END or TABLE waits on a + // slow analysis. A fast runner can finish the analysis inside them; during >= 3 above is what fails + // when the control thread does the analysis itself. + assertTrue("TABLE itself was answered at once: " + tableAck / 1_000_000 + " ms", + tableAck < TimeUnit.SECONDS.toNanos(5)); + assertTrue("no START/END pair waited on the analysis: the slowest took " + worst / 1_000_000 + " ms", + worst < TimeUnit.SECONDS.toNanos(5)); + } + + // field is the string value of name in one JSON object's text. + private static String field(String json, String name) { + String key = "\"" + name + "\":\""; + int at = json.indexOf(key); + assertTrue(json, at >= 0); + return json.substring(at + key.length(), json.indexOf('"', at + key.length())); + } + + @Test(timeout = 60000) + public void aTableThatFailsToStartIsIgnoredAndTheServerGoesOn() throws Exception { + assumeNoMinHeapEnv(); + startWithJacoco(); + String dataPath = KeployDedupAgent.dataSocketPath; + data = new DataReceiver(dataPath); + KeployDedupAgent.dataSocketPath = null; // new File(null) throws on the control thread + assertEquals(IGNORED, send("TABLE")); + KeployDedupAgent.dataSocketPath = dataPath; + assertEquals(TAKEN, send("START test-set-0/test-1")); + assertEquals(TAKEN, send("END test-set-0/test-1")); + String payload = data.next(); + assertNotNull("the END's coverage reached the data socket", payload); + assertTrue(payload, payload.contains("\"id\":\"test-set-0/test-1\"")); + } + + private static String sha256(byte[] b) throws NoSuchAlgorithmException { + StringBuilder hex = new StringBuilder(); + for (byte x : MessageDigest.getInstance("SHA-256").digest(b)) { + hex.append(String.format("%02x", x)); + } + return hex.toString(); + } + @Test public void anEndForAnotherTestIsIgnoredAndTheTestRunningGoesOn() throws Exception { startWithJacoco(); @@ -603,6 +850,10 @@ private void serve() { try (Socket socket = server.accept()) { socket.setSoTimeout(30000); String payload = readAll(socket.getInputStream()); + if (payload.startsWith("KTB1")) { + // A TABLE frame waits for OK before the next one goes. + socket.getOutputStream().write("OK\n".getBytes(StandardCharsets.UTF_8)); + } all.add(payload); (payload.startsWith(INDEX_START) ? indexes : payloads).add(payload); } catch (IOException e) { diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/IndexReportTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/IndexReportTest.java index 2a7464fb..b3cabc30 100644 --- a/keploy-sdk/src/test/java/io/keploy/dedup/IndexReportTest.java +++ b/keploy-sdk/src/test/java/io/keploy/dedup/IndexReportTest.java @@ -8,6 +8,7 @@ import java.io.File; import java.io.FileOutputStream; +import java.io.IOException; import java.io.InputStream; import java.net.URL; import java.net.URLClassLoader; @@ -16,6 +17,7 @@ import java.util.Arrays; import java.util.Collections; import java.util.List; +import java.util.Properties; import java.util.jar.Attributes; import java.util.jar.JarOutputStream; import java.util.jar.Manifest; @@ -23,6 +25,7 @@ import static org.junit.Assert.assertEquals; import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; import static org.junit.Assert.assertNull; import static org.junit.Assert.assertTrue; @@ -31,17 +34,125 @@ public class IndexReportTest { @Rule public final TemporaryFolder tmp = new TemporaryFolder(); - // Keploy's codecoverage.JavaBuildID of the ids 1, 0xff and - // 0xffffffffffffffff (its TestBuildIDs): sha256 over - // "0000000000000001\n00000000000000ff\nffffffffffffffff". - private static final String GO_BUILD_ID = "jv1-73ab094cbaa75db7d27e1cc885ca2260"; + // The build id's golden vectors (shared with Keploy's codecoverage.JavaBuildID): the sha256 over + // "keploy-java-table 1\njacoco 0.8.15\n0000000000000001\n00000000000000ff\nffffffffffffffff". + private static final long[] GOLDEN_IDS = {1, 0xff, 0xffffffffffffffffL}; + private static final String GOLDEN_BUILD_ID = "jv1-ac2499a6a1d337817534c44eb82f6ab3"; @Test public void theBuildIdIsKeploysForTheSameIdsInAnyOrderAndRepeated() { - assertEquals(GO_BUILD_ID, IndexReport.buildId(new long[] {1, 0xff, 0xffffffffffffffffL})); + assertEquals(GOLDEN_BUILD_ID, IndexReport.buildId(GOLDEN_IDS, 1, "0.8.15")); // A signed sort puts 0xffffffffffffffff (-1) first, a different id. - assertEquals(GO_BUILD_ID, IndexReport.buildId(new long[] {0xffffffffffffffffL, 1, 0xff, 1})); - assertEquals(GO_BUILD_ID, IndexReport.buildId(new long[] {0xff, 0xffffffffffffffffL, 0xff, 1})); + assertEquals(GOLDEN_BUILD_ID, + IndexReport.buildId(new long[] {0xffffffffffffffffL, 1, 0xff, 1}, 1, "0.8.15")); + assertEquals(GOLDEN_BUILD_ID, + IndexReport.buildId(new long[] {0xff, 0xffffffffffffffffL, 0xff, 1}, 1, "0.8.15")); + } + + @Test + public void anotherTableFormatOrAnalyzerNamesAnotherBuildOfTheSameClasses() { + // An SDK whose table can differ for the same class bytes must not name the build an older SDK saved a + // table under: Keploy would refuse every later save of it (409 build_table_mismatch). + assertEquals("jv1-32f6f460d59e38d89bac87928b9421ae", IndexReport.buildId(GOLDEN_IDS, 2, "0.8.15")); + assertEquals("jv1-776b07757cb4a25493b652ff31a8ca86", IndexReport.buildId(GOLDEN_IDS, 1, "0.8.14")); + // Over the ids alone (the build id before the format and the analyzer were in it): none of the above. + assertEquals("jv1-73ab094cbaa75db7d27e1cc885ca2260", + "jv1-" + IndexReport.classIdsSum(GOLDEN_IDS).substring(0, 32)); + } + + @Test + public void theSdksBuildIdIsOfThisTablesFormatAndItsAnalyzersJacoco() { + assertEquals(1, BuildTable.TABLE_FORMAT); + assertEquals(jacocoCore(), IndexReport.Analyzer.JACOCO); + assertEquals(IndexReport.buildId(GOLDEN_IDS, 1, jacocoCore()), IndexReport.buildId(GOLDEN_IDS)); + } + + /** + * An END reads class ids (hexId, classId) before its ACK; only the build id needs V, so those helpers must + * not run JaCoCo's class init and its version lookup. Run in a loader of its own over the SDK's classes, + * JaCoCo's core and Gson, so the classes it loads are its own to count. + */ + @Test + public void theClassIdHelpersNeverLoadTheAnalyzersJacoco() throws Exception { + URL[] urls = { + IndexReport.class.getProtectionDomain().getCodeSource().getLocation(), + org.jacoco.core.JaCoCo.class.getProtectionDomain().getCodeSource().getLocation(), + com.google.gson.Gson.class.getProtectionDomain().getCodeSource().getLocation(), + }; + byte[] classBytes = Files.readAllBytes(new File(urls[0].toURI()).toPath() + .resolve(IndexReport.class.getName().replace('.', '/') + ".class")); + try (Counting loader = new Counting(urls)) { + Class report = Class.forName(IndexReport.class.getName(), true, loader); + assertTrue(report != IndexReport.class); + assertEquals("0000000000000001", call(report, "hexId", long.class, 1L)); + assertEquals(IndexReport.classId(classBytes), call(report, "classId", byte[].class, classBytes)); + assertFalse("V was read by a class-id helper", loader.loaded(IndexReport.Analyzer.class.getName())); + assertFalse("JaCoCo's version was loaded by a class-id helper", + loader.loaded(org.jacoco.core.JaCoCo.class.getName())); + assertEquals(IndexReport.buildId(new long[] {1}), call(report, "buildId", long[].class, new long[] {1})); + assertTrue(loader.loaded(IndexReport.Analyzer.class.getName())); + assertTrue(loader.loaded(org.jacoco.core.JaCoCo.class.getName())); + } + } + + private static final class Counting extends URLClassLoader { + Counting(URL[] urls) { + super(urls, null); + } + + boolean loaded(String name) { + return findLoadedClass(name) != null; + } + } + + private static Object call(Class c, String name, Class parameter, Object argument) throws Exception { + java.lang.reflect.Method m = c.getDeclaredMethod(name, parameter); + m.setAccessible(true); + return m.invoke(null, argument); + } + + @Test + public void aJacocoReleaseIsItsVersionLessTheBuildQualifier() { + assertEquals("0.8.15", IndexReport.release("0.8.15.202606040825")); + assertEquals("0.8.15", IndexReport.release("0.8.15")); + assertEquals("1.0.0", IndexReport.release("1.0.0.SNAPSHOT.x")); + assertEquals("0.8", IndexReport.release("0.8")); + assertEquals("", IndexReport.release("")); + } + + /** + * jacocoCore is the release of the org.jacoco.core jar the analyzer runs, as its Maven build named it (its + * pom.properties, read from the jar JaCoCo's class was loaded from, so another jar's pom.properties on the + * class path cannot answer), not as JaCoCo.VERSION says: what IndexReport.Analyzer.JACOCO must be. + */ + static String jacocoCore() { + try { + File jar = new File(org.jacoco.core.JaCoCo.class.getProtectionDomain().getCodeSource().getLocation() + .toURI()); + assertTrue(jar + " is org.jacoco.core's jar", jar.isFile()); + try (java.util.jar.JarFile file = new java.util.jar.JarFile(jar)) { + ZipEntry entry = file.getEntry("META-INF/maven/org.jacoco/org.jacoco.core/pom.properties"); + assertNotNull("org.jacoco.core's pom.properties in " + jar, entry); + Properties p = new Properties(); + try (InputStream in = file.getInputStream(entry)) { + p.load(in); + } + String version = p.getProperty("version"); + assertTrue(version, version.matches("\\d+\\.\\d+\\.\\d+")); + return version; + } + } catch (IOException | java.net.URISyntaxException e) { + throw new AssertionError(e); + } + } + + // The whole sha256 over the same three ids: what a build table unit's + // hash is for a unit of those class ids. + private static final String IDS_SUM = "73ab094cbaa75db7d27e1cc885ca22604fb30a4555ebac8aae6e264612a027d0"; + + @Test + public void theUnitHashIsTheSumOverTheUnitsIdsAlone() { + assertEquals(IDS_SUM, IndexReport.classIdsSum(new long[] {0xffffffffffffffffL, 1, 0xff, 1})); } @Test @@ -82,13 +193,13 @@ public void aRootIsTheLocationWithoutTheClassPath() { @Test public void theReportIsOneSmallJsonObjectWithNoClassList() { - IndexReport report = new IndexReport(GO_BUILD_ID, IndexReport.SOURCE_DERIVED, "java -jar /app/app.jar", + IndexReport report = new IndexReport(GOLDEN_BUILD_ID, IndexReport.SOURCE_DERIVED, "java -jar /app/app.jar", Arrays.asList("/app/libs/b.jar", "/app/classes", "/app/libs/b.jar"), Collections.singletonList("org.acme"), 3, "org.acme.*", IndexReport.JACOCO_EMBEDDED, "2.2.0", "0.8.15"); JsonObject json = JsonParser.parseString(report.toJson()).getAsJsonObject(); assertEquals("index", json.get("kind").getAsString()); assertEquals(1, json.get("v").getAsInt()); - assertEquals(GO_BUILD_ID, json.get("build").getAsString()); + assertEquals(GOLDEN_BUILD_ID, json.get("build").getAsString()); assertEquals("derived", json.get("source").getAsString()); assertEquals("java -jar /app/app.jar", json.get("launch").getAsString()); assertEquals("[\"/app/classes\",\"/app/libs/b.jar\"]", json.get("roots").toString()); diff --git a/keploy-sdk/src/test/java/io/keploy/dedup/ProbeMapTest.java b/keploy-sdk/src/test/java/io/keploy/dedup/ProbeMapTest.java new file mode 100644 index 00000000..785fd5c4 --- /dev/null +++ b/keploy-sdk/src/test/java/io/keploy/dedup/ProbeMapTest.java @@ -0,0 +1,782 @@ +package io.keploy.dedup; + +import com.google.gson.JsonArray; +import com.google.gson.JsonElement; +import com.google.gson.JsonObject; +import com.google.gson.JsonParser; +import org.jacoco.core.analysis.Analyzer; +import org.jacoco.core.analysis.IClassCoverage; +import org.jacoco.core.analysis.ICounter; +import org.jacoco.core.analysis.ICoverageNode; +import org.jacoco.core.analysis.ILine; +import org.jacoco.core.analysis.IMethodCoverage; +import org.jacoco.core.analysis.ISourceNode; +import org.jacoco.core.data.ExecutionData; +import org.jacoco.core.data.ExecutionDataStore; +import org.jacoco.core.instr.Instrumenter; +import org.jacoco.core.internal.analysis.KeployBranchCoverage; +import org.jacoco.core.internal.data.CRC64; +import org.jacoco.core.runtime.IExecutionDataAccessorGenerator; +import org.junit.Test; +import org.objectweb.asm.ClassWriter; +import org.objectweb.asm.Label; +import org.objectweb.asm.MethodVisitor; +import org.objectweb.asm.Opcodes; + +import java.io.ByteArrayOutputStream; +import java.io.File; +import java.io.IOException; +import java.io.InputStream; +import java.net.URL; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; +import java.nio.file.Paths; +import java.util.ArrayList; +import java.util.Arrays; +import java.util.BitSet; +import java.util.Collections; +import java.util.HashMap; +import java.util.LinkedHashMap; +import java.util.List; +import java.util.Map; +import java.util.TreeMap; +import java.util.regex.Pattern; +import java.util.stream.Collectors; +import java.util.stream.Stream; +import java.util.zip.ZipEntry; +import java.util.zip.ZipFile; + +import static org.junit.Assert.assertArrayEquals; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotEquals; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertSame; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +/** + * Rows for the probe map: for every class file of the java-dedup sample and of a class of code shapes (if and + * &&, loops, an int, a String and an enum switch, try/finally, try-with-resources, synchronized, a lambda, + * an assert), the leaves the map gives a set of probes are what JaCoCo's own public Analyzer counts for that + * set, line by line: a line's code leaf is hit when JaCoCo covers an instruction of it, and as many of its + * branch leaves as JaCoCo covers branches. The sets are no probe, each probe alone, every pair of probes and + * all of them, so the branch leaves of a line keep their identity across probes (a union of two probes counts + * two edges only when JaCoCo does). The Shapes source is shapes/Shapes.java.txt (javac --release 8 -g); the + * Modern source (records, sealed types, pattern and enum switches, a String switch with yield) is + * modern/Modern.java.txt (javac --release 21 -g). + */ +public class ProbeMapTest { + + private static final String RES = "/io/keploy/dedup/buildtable/"; + + static List load(String dir) throws Exception { + URL root = ProbeMapTest.class.getResource(RES + dir); + assertNotNull(dir, root); + Path base = Paths.get(root.toURI()); + List out = new ArrayList<>(); + try (Stream walk = Files.walk(base)) { + for (Path p : walk.filter(p -> p.toString().endsWith(".class.bin")).sorted().collect(Collectors.toList())) { + String rel = base.relativize(p).toString().replace(File.separatorChar, '/'); + out.add(new BuildTable.ClassFile(rel.substring(0, rel.length() - ".class.bin".length()), + Files.readAllBytes(p))); + } + } + return out; + } + + /** probes writes the table and returns the probe map's NDJSON stream. */ + static String probes(List files, BuildTableTest.Collect table) throws IOException { + ByteArrayOutputStream b = new ByteArrayOutputStream(); + BuildTable.Summary s = BuildTable.write(files, table, (build, r) -> { + b.write(r, 0, r.length); + b.write('\n'); + }); + assertEquals("the summary counts the probe records", s.probeRecords, + new String(b.toByteArray(), StandardCharsets.UTF_8).split("\n", -1).length - 1); + return new String(b.toByteArray(), StandardCharsets.UTF_8); + } + + @Test + public void javaDedupProbeMapIsTheGolden() throws Exception { + List files = BuildTableTest.javaDedup(); + String got = probes(files, new BuildTableTest.Collect()); + String dir = System.getProperty("buildtable.dump"); + if (dir != null) { + Files.write(Paths.get(dir, "java-dedup.probes.ndjson"), got.getBytes(StandardCharsets.UTF_8)); + } + assertEquals(new String(BuildTableTest.resource("java-dedup.probes.ndjson"), StandardCharsets.UTF_8), got); + // One record per class file, each with its class id and as many entries as JaCoCo has probes in it. + String[] lines = got.split("\n"); + assertEquals(files.size(), lines.length); + for (int i = 0; i < lines.length; i++) { + JsonObject r = JsonParser.parseString(lines[i]).getAsJsonObject(); + assertEquals(files.get(i).name, r.get("class").getAsString()); + assertEquals(String.format("%016x", CRC64.classId(files.get(i).bytes)), r.get("id").getAsString()); + assertEquals(files.get(i).name, instrumentedProbes(files.get(i)), r.getAsJsonArray("probes").size()); + } + } + + @Test + public void aMethodTheCallerDeclinesIsNotWalked() { + // The map declines the methods the table has no line of (a class built without -g): their probes cover + // no leaf, so their walks would be thrown away. The class pass still analyzes them. + byte[] bytes = BuildTableBudgetTest.gen("gen/app/Declined", 0, 40); + Map shape = new HashMap<>(); + List walked = new ArrayList<>(); + int total = KeployBranchCoverage.analyze(bytes, shape, new KeployBranchCoverage.EachProbe() { + @Override + public boolean wants(KeployBranchCoverage.Method m) { + return !m.name.equals("run"); + } + + @Override + public void probe(KeployBranchCoverage.Method m, int r, KeployBranchCoverage.Hits h) { + walked.add(m.name); + } + }); + assertTrue(shape.containsKey("run(I)I")); + assertTrue("run has probes to walk", shape.get("run(I)I").probes.length > 40); + assertEquals(total - shape.get("run(I)I").probes.length, walked.size()); + assertFalse("a declined method is not walked", walked.contains("run")); + assertTrue(walked.contains("")); + } + + /** What one probe covers, copied out of the analysis's reused Hits: per counted instruction. */ + static final class Got { + final boolean[] covered; + final boolean[][] branches; + + Got(KeployBranchCoverage.Method m, KeployBranchCoverage.Hits h) { + covered = new boolean[m.lines.length]; + branches = new boolean[m.lines.length][]; + for (int k = 0; k < branches.length; k++) { + branches[k] = h.branches[k].clone(); + } + for (int j = 0; j < h.count; j++) { + assertFalse("an instruction once in hits", covered[h.at[j]]); + covered[h.at[j]] = true; + } + for (int k = 0; k < branches.length; k++) { + for (boolean b : branches[k]) { + assertTrue("a covered branch of a covered instruction", !b || covered[k]); + } + } + } + } + + @Test(timeout = 300000) + public void eachProbesMapIsTheClassAnalyzedWithThatProbeExecuted() throws Exception { + // The oracle is the class analyzed whole by JaCoCo's builder and calculator, once per round r, with the + // r-th probe of every method executed (a method's coverage reads only its own probes). Against it, what + // the one pass maps each probe to, on real libraries' classes (JaCoCo core's and Gson's, the jars on + // this test's classpath) and on every fixture (java-dedup, the code shapes, the Java 21 shapes). + int classes = 0; + int probes = 0; + List all = new ArrayList<>(); + for (Class c : new Class[] {Analyzer.class, JsonParser.class}) { + File jar = new File(c.getProtectionDomain().getCodeSource().getLocation().toURI()); + assertTrue(jar + " is a jar", jar.isFile()); + all.addAll(jarClasses(jar)); + } + all.addAll(BuildTableTest.javaDedup()); + all.addAll(load("shapes")); + all.addAll(load("modern")); + all.add(new BuildTable.ClassFile("gen/app/Long", BuildTableBudgetTest.gen("gen/app/Long", 3, 300))); + // Each probe of a chain of early returns walks back over every check before it. + all.add(new BuildTable.ClassFile("gen/app/Returns", BuildTableBudgetTest.returns("gen/app/Returns", 80))); + for (BuildTable.ClassFile f : all) { + Map shape = new HashMap<>(); + Map> replayed = new HashMap<>(); + int total = KeployBranchCoverage.analyze(f.bytes, shape, (m, r, h) -> { + List l = replayed.computeIfAbsent(m.name + m.desc, k -> new ArrayList<>()); + assertEquals(f.name + "#" + m.name + m.desc, r, l.size()); + l.add(new Got(m, h)); + }); + if (total < 0) { + continue; + } + classes++; + int rounds = 0; + int replays = 0; + for (Map.Entry e : shape.entrySet()) { + int n = e.getValue().probes.length; + rounds = Math.max(rounds, n); + replays += n; + List l = replayed.get(e.getKey()); + assertEquals(f.name + "#" + e.getKey(), n, l == null ? 0 : l.size()); + } + // A method whose code the filters leave out entirely (an enum's values()) keeps its probes: no leaf. + assertTrue(f.name + ": " + replays + " probes replayed of " + total, replays <= total); + for (int r = 0; r < rounds; r++) { + boolean[] set = new boolean[total]; + for (KeployBranchCoverage.Method m : shape.values()) { + if (m.probes.length > r) { + set[m.probes[r]] = true; + } + } + Map run = new HashMap<>(); + KeployBranchCoverage.analyze(f.bytes, set, run); + for (Map.Entry e : shape.entrySet()) { + if (e.getValue().probes.length <= r) { + continue; + } + KeployBranchCoverage.Method want = run.get(e.getKey()); + Got got = replayed.get(e.getKey()).get(r); + String at = f.name + "#" + e.getKey() + " probe " + r; + assertArrayEquals(at, want.lines, e.getValue().lines); + assertArrayEquals(at, want.covered, got.covered); + assertEquals(at, want.branches.length, got.branches.length); + for (int k = 0; k < want.branches.length; k++) { + assertArrayEquals(at, want.branches[k], got.branches[k]); + } + probes++; + } + } + } + assertTrue("classes: " + classes, classes > 300); + assertTrue("probes: " + probes, probes > 5000); + } + + @Test(timeout = 300000) + public void eachTableMethodIsWhatJacocosAnalyzerCounts() throws Exception { + // The table's methods come from the probe map's pass (KeployBranchCoverage.Method#coverage), not from + // JaCoCo's public Analyzer. Against the Analyzer, on real libraries' classes (JaCoCo core's and Gson's) + // and every fixture, with no probe executed and with every probe: the same methods in the same order, + // and each one's counters, whole and line by line. + List all = new ArrayList<>(); + for (Class c : new Class[] {Analyzer.class, JsonParser.class}) { + all.addAll(jarClasses(new File(c.getProtectionDomain().getCodeSource().getLocation().toURI()))); + } + all.addAll(BuildTableTest.javaDedup()); + all.addAll(load("shapes")); + all.addAll(load("modern")); + int classes = 0; + int methods = 0; + for (BuildTable.ClassFile f : all) { + int total = KeployBranchCoverage.analyze(f.bytes, null, new HashMap<>()); + for (boolean hit : new boolean[] {false, true}) { + boolean[] probes = null; + ExecutionDataStore store = new ExecutionDataStore(); + if (hit && total > 0) { + probes = new boolean[total]; + Arrays.fill(probes, true); + store.put(new ExecutionData(CRC64.classId(f.bytes), f.name, probes)); + } + Map ours = new LinkedHashMap<>(); + assertEquals(f.name, total, KeployBranchCoverage.analyze(f.bytes, probes, ours)); + List got = new ArrayList<>(); + for (KeployBranchCoverage.Method m : ours.values()) { + got.add(counts(m.coverage)); + } + List classCoverage = new ArrayList<>(); + new Analyzer(store, classCoverage::add).analyzeClass(f.bytes, f.name); + List want = new ArrayList<>(); + for (IClassCoverage cc : classCoverage) { + for (IMethodCoverage mc : cc.getMethods()) { + want.add(counts(mc)); + } + } + assertEquals(f.name + (hit ? ", every probe executed" : ", none executed"), want, got); + if (!hit && total >= 0) { + classes++; + methods += got.size(); + } + } + } + assertTrue("classes: " + classes, classes > 300); + assertTrue("methods: " + methods, methods > 1500); + } + + /** counts is a method's coverage as JaCoCo's API gives it: its counters, then each line's, with its lines. */ + static String counts(IMethodCoverage m) { + StringBuilder sb = new StringBuilder(); + sb.append(m.getName()).append(m.getDesc()).append(' ').append(m.getSignature()).append(" lines ") + .append(m.getFirstLine()).append('-').append(m.getLastLine()); + for (ICoverageNode.CounterEntity e : ICoverageNode.CounterEntity.values()) { + ICounter c = m.getCounter(e); + sb.append(' ').append(e).append(' ').append(c.getMissedCount()).append('/').append(c.getCoveredCount()); + } + if (m.getFirstLine() != ISourceNode.UNKNOWN_LINE) { + for (int l = m.getFirstLine(); l <= m.getLastLine(); l++) { + ILine line = m.getLine(l); + sb.append(" @").append(l).append(' ').append(line.getStatus()) + .append(" i ").append(line.getInstructionCounter().getMissedCount()).append('/') + .append(line.getInstructionCounter().getCoveredCount()) + .append(" b ").append(line.getBranchCounter().getMissedCount()).append('/') + .append(line.getBranchCounter().getCoveredCount()); + } + } + return sb.toString(); + } + + @Test + public void theTableNeedsNoSecondAnalysisOfAClass() throws Exception { + // One JaCoCo analysis of each class file gives both its table methods and its probe map. JaCoCo's public + // Analyzer would be a second one: no class of BuildTable's names it (the per-test CoverageReporter + // still does, for the coverage of a test's hits). + File dir = new File(BuildTable.class.getProtectionDomain().getCodeSource().getLocation().toURI()); + assertTrue(dir + " is the SDK's classes directory", dir.isDirectory()); + File[] own = new File(dir, "io/keploy/dedup").listFiles( + (d, n) -> n.equals("BuildTable.class") || n.startsWith("BuildTable$")); + assertNotNull(own); + assertTrue(Arrays.toString(own), own.length > 10); + Pattern analyzer = Pattern.compile("org/jacoco/core/analysis/Analyzer(?![\\w$])"); + for (File f : own) { + String text = new String(Files.readAllBytes(f.toPath()), StandardCharsets.ISO_8859_1); + assertFalse(f.getName() + " names JaCoCo's Analyzer", analyzer.matcher(text).find()); + } + } + + /** instrumented is the class file as JaCoCo's offline instrumenter writes it, its probes read from null. */ + static byte[] instrumented(String name, byte[] bytes) throws IOException { + return new Instrumenter(new IExecutionDataAccessorGenerator() { + @Override + public int generateDataAccessor(long classid, String classname, int probecount, MethodVisitor mv) { + mv.visitInsn(Opcodes.ACONST_NULL); + mv.visitTypeInsn(Opcodes.CHECKCAST, "[Z"); + return 1; + } + }).instrument(bytes, name); + } + + static List jarClasses(File jar) throws IOException { + List out = new ArrayList<>(); + try (ZipFile z = new ZipFile(jar)) { + for (ZipEntry e : Collections.list(z.entries())) { + String name = e.getName(); + if (!name.endsWith(".class") || name.startsWith("META-INF/")) { + continue; + } + ByteArrayOutputStream b = new ByteArrayOutputStream(); + try (InputStream in = z.getInputStream(e)) { + byte[] buf = new byte[8192]; + for (int k; (k = in.read(buf)) > 0; ) { + b.write(buf, 0, k); + } + } + out.add(new BuildTable.ClassFile(name.substring(0, name.length() - 6), b.toByteArray())); + } + } + return out; + } + + @Test + public void javaDedupProbesCoverWhatJacocoCounts() throws Exception { + agree(BuildTableTest.javaDedup()); + } + + @Test + public void everyCodeShapeCoversWhatJacocoCounts() throws Exception { + List files = load("shapes"); + assertEquals(3, files.size()); + int edges = agree(files); + assertTrue("the shapes have branch edges to tell apart: " + edges, edges >= 40); + } + + @Test + public void java21CodeShapesCoverWhatJacocoCounts() throws Exception { + // Filters Java 8 code never meets: records, pattern switches (typeSwitch, MatchException), record + // patterns, exhaustive switches. ASM reads the class version on every JDK the SDK runs on. + List files = load("modern"); + assertEquals(7, files.size()); + int edges = agree(files); + assertTrue("the shapes have branch edges to tell apart: " + edges, edges >= 40); + } + + /** + * instrumentedProbes is the length of the probe array JaCoCo's instrumenter gives the class: the array the + * agent fills, read without the analysis the map is built from. + */ + static int instrumentedProbes(BuildTable.ClassFile f) throws IOException { + int[] count = {0}; + new Instrumenter(new IExecutionDataAccessorGenerator() { + @Override + public int generateDataAccessor(long classid, String classname, int probecount, MethodVisitor mv) { + count[0] = probecount; + mv.visitInsn(Opcodes.ACONST_NULL); + mv.visitTypeInsn(Opcodes.CHECKCAST, "[Z"); + return 1; + } + }).instrument(f.bytes, f.name); + return count[0]; + } + + /** + * agree checks every class file's map against JaCoCo for no probe, each probe, each pair and all probes, + * and returns how many branch leaves the classes have. + */ + static int agree(List files) throws Exception { + BuildTableTest.Collect table = new BuildTableTest.Collect(); + String[] maps = probes(files, table).split("\n"); + Map units = new HashMap<>(); + for (String r : table.records) { + JsonObject o = JsonParser.parseString(r).getAsJsonObject(); + assertEquals(r, "ok", o.get("state").getAsString()); + units.put(o.get("id").getAsString(), o); + } + Map byName = new HashMap<>(); + for (BuildTable.ClassFile f : files) { + byName.put(f.name, f); + } + assertEquals(files.size(), maps.length); + int edges = 0; + for (String line : maps) { + JsonObject m = JsonParser.parseString(line).getAsJsonObject(); + BuildTable.ClassFile f = byName.get(m.get("class").getAsString()); + JsonObject unit = units.get(m.get("unit").getAsString()); + assertNotNull(line, f); + assertNotNull(line, unit); + List probes = new ArrayList<>(); + for (JsonElement p : m.getAsJsonArray("probes")) { + JsonArray a = p.getAsJsonArray(); + int[] r = new int[a.size()]; + for (int i = 0; i < r.length; i++) { + r[i] = a.get(i).getAsInt(); + } + assertEquals("ranges come in pairs", 0, r.length % 2); + for (int i = 0; i + 1 < r.length; i++) { + assertTrue(f.name + ": ranges ascend and never touch " + Arrays.toString(r), r[i] < r[i + 1]); + } + probes.add(r); + } + int n = probes.size(); + assertEquals(f.name, instrumentedProbes(f), n); + check(f, unit, probes, new boolean[n]); + boolean[] all = new boolean[n]; + Arrays.fill(all, true); + check(f, unit, probes, all); + for (int p = 0; p < n; p++) { + for (int q = p; q < n; q++) { + boolean[] set = new boolean[n]; + set[p] = true; + set[q] = true; + check(f, unit, probes, set); + } + } + for (List v : jacoco(f, new boolean[n]).values()) { + edges += v.get(2); + } + } + return edges; + } + + static void check(BuildTable.ClassFile f, JsonObject unit, List probes, boolean[] set) throws IOException { + Map> want = jacoco(f, set); + Map> got = mapped(f.name, unit, probes, set); + List on = new ArrayList<>(); + for (int i = 0; i < set.length; i++) { + if (set[i]) { + on.add(i); + } + } + assertEquals(f.name + " probes " + on, want, got); + } + + /** jacoco is what JaCoCo's Analyzer counts with these probes: per method and line, code hit, branches hit, branches. */ + static Map> jacoco(BuildTable.ClassFile f, boolean[] probes) throws IOException { + ExecutionDataStore store = new ExecutionDataStore(); + store.put(new ExecutionData(CRC64.classId(f.bytes), f.name, probes)); + List got = new ArrayList<>(); + new Analyzer(store, got::add).analyzeClass(f.bytes, f.name); + Map> out = new TreeMap<>(); + for (IClassCoverage cc : got) { + for (IMethodCoverage mc : cc.getMethods()) { + for (int l = Math.max(1, mc.getFirstLine()); l <= mc.getLastLine(); l++) { + ILine line = mc.getLine(l); + if (line.getInstructionCounter().getTotalCount() == 0) { + continue; + } + out.put("m:" + f.name + "#" + mc.getName() + mc.getDesc() + "@" + l, Arrays.asList( + line.getInstructionCounter().getCoveredCount() > 0 ? 1 : 0, + line.getBranchCounter().getCoveredCount(), line.getBranchCounter().getTotalCount())); + } + } + } + return out; + } + + /** mapped is the same count from the map: the union of the set's leaves, read back through the table. */ + static Map> mapped(String name, JsonObject unit, List probes, boolean[] set) { + List keys = new ArrayList<>(); + List kinds = new ArrayList<>(); + Map> out = new TreeMap<>(); + for (JsonElement me : unit.getAsJsonArray("methods")) { + JsonObject m = me.getAsJsonObject(); + String id = m.get("id").getAsString(); + for (JsonElement le : m.getAsJsonArray("leaves")) { + JsonArray leaf = le.getAsJsonArray(); + String key = id + "@" + leaf.get(0).getAsInt(); + keys.add(key); + kinds.add(leaf.get(5).getAsInt()); + if (id.startsWith("m:" + name + "#")) { + List v = out.get(key); + if (v == null) { + v = new ArrayList<>(Arrays.asList(0, 0, 0)); + out.put(key, v); + } + if (leaf.get(5).getAsInt() == 1) { + v.set(2, v.get(2) + 1); + } + } + } + } + BitSet hit = new BitSet(); + for (int p = 0; p < set.length; p++) { + if (set[p]) { + int[] r = probes.get(p); + for (int i = 0; i < r.length; i += 2) { + hit.set(r[i], r[i + 1]); + } + } + } + assertTrue(name + ": a leaf past the unit's", hit.length() <= keys.size()); + for (int i = hit.nextSetBit(0); i >= 0; i = hit.nextSetBit(i + 1)) { + List v = out.get(keys.get(i)); + assertNotNull(name + ": a probe of it covers another class's leaf " + keys.get(i), v); + if (kinds.get(i) == 0) { + v.set(0, 1); + } else { + v.set(1, v.get(1) + 1); + } + } + return out; + } + + @Test + public void aUnitThatIsNotMeasuredHasNoProbeRecord() throws Exception { + List files = new ArrayList<>(BuildTableTest.javaDedup()); + files.add(new BuildTable.ClassFile("io/keploy/samples/javadedup/model/Item$Later", + "not a class".getBytes(StandardCharsets.UTF_8))); + String got = probes(files, new BuildTableTest.Collect()); + assertFalse(got, got.contains("\"unit\":\"c:io/keploy/samples/javadedup/model/Item\"")); + assertEquals(4, got.split("\n").length); + } + + /** copy is a class d/C with one method per line given (m0 at the first, ...), and an int field when field. */ + static byte[] copy(boolean field, int... lines) { + int[][] methods = new int[lines.length][]; + for (int i = 0; i < lines.length; i++) { + methods[i] = new int[] {lines[i]}; + } + return cls(field, false, methods); + } + + /** + * cls is a class d/C with a method mi per entry of methods, one instruction on each of its lines in order + * (NOPs, then a RETURN), an int field when field, and the methods written last to first when reversed. + */ + static byte[] cls(boolean field, boolean reversed, int[]... methods) { + ClassWriter cw = new ClassWriter(ClassWriter.COMPUTE_MAXS); + cw.visit(Opcodes.V1_8, Opcodes.ACC_PUBLIC | Opcodes.ACC_SUPER, "d/C", null, "java/lang/Object", null); + cw.visitSource("C.java", null); + if (field) { + cw.visitField(Opcodes.ACC_STATIC, "x", "I", null, null).visitEnd(); + } + for (int j = 0; j < methods.length; j++) { + int i = reversed ? methods.length - 1 - j : j; + MethodVisitor mv = cw.visitMethod(Opcodes.ACC_STATIC, "m" + i, "()V", null, null); + mv.visitCode(); + for (int k = 0; k < methods[i].length; k++) { + Label l = new Label(); + mv.visitLabel(l); + mv.visitLineNumber(methods[i][k], l); + mv.visitInsn(k + 1 < methods[i].length ? Opcodes.NOP : Opcodes.RETURN); + } + mv.visitMaxs(0, 0); + mv.visitEnd(); + } + cw.visitEnd(); + return cw.toByteArray(); + } + + @Test + public void twoCopiesOfAClassThatDifferAreNotMeasured() throws Exception { + // The same class under two roots, as the index can find it. The table's methods come from one copy, + // so the other's probes cannot be mapped onto them: the unit fails closed in the table and the map. + byte[][][] cases = { + {copy(false, 10), copy(false, 20)}, // a line moved + {copy(false, 10, 11), copy(false, 10)}, // a method the other copy lacks + {copy(false, 10), copy(false, 10, 11)}, + // The same methods and lines, but their probes in another order: a hit of probe 0 of one copy is + // not a hit of probe 0 of the other, and a test's hits name only the class. + {cls(false, false, new int[] {10}, new int[] {12}), cls(false, true, new int[] {10}, new int[] {12})}, + }; + for (byte[][] c : cases) { + List files = new ArrayList<>(); + files.add(new BuildTable.ClassFile("d/C", c[0])); + files.add(new BuildTable.ClassFile("d/C", c[1])); + BuildTableTest.Collect table = new BuildTableTest.Collect(); + BuildTableTest.Collect map = new BuildTableTest.Collect(); + BuildTable.Summary s = BuildTable.write(files, table, map); + assertEquals(1, table.records.size()); + String rec = table.records.get(0); + assertTrue(rec, rec.contains("\"id\":\"c:d/C\"")); + assertTrue(rec, rec.contains("\"state\":\"not_measured\",\"reason\":{\"code\":\"analyzer_failed\"}")); + assertEquals(1, s.notMeasured); + assertEquals("counted as a probe map failure", 1, s.notMapped); + assertTrue("no probe record for a unit that is not measured", map.records.isEmpty()); + assertEquals(0, s.probeRecords); + } + } + + @Test + public void aFailureInKeploysOwnStepsIsNotMappedAndTheCallersOwnPassesAsItIs() throws Exception { + byte[] bytes = copy(false, 10, 12); + // An index error in the probe walk, Keploy's step after JaCoCo's: here a caller that empties the + // method's branch rows before its probes are walked. It is NotMapped (the class is not measured and + // counted as a map that failed), not an error that blames JaCoCo for the class. + try { + KeployBranchCoverage.analyze(bytes, new HashMap<>(), new KeployBranchCoverage.EachProbe() { + @Override + public boolean wants(KeployBranchCoverage.Method m) { + m.branches = new boolean[0][]; + return true; + } + + @Override + public void probe(KeployBranchCoverage.Method m, int r, KeployBranchCoverage.Hits h) { + } + }); + fail("an index error in the walk passed"); + } catch (KeployBranchCoverage.NotMapped e) { + assertTrue(e.getMessage(), e.getMessage().startsWith("d/C#m")); + assertTrue(String.valueOf(e.getCause()), e.getCause() instanceof ArrayIndexOutOfBoundsException); + } + // What the caller's own callback throws passes as it is: BuildTable ends a map past its bound so. + IllegalArgumentException mine = new IllegalArgumentException("the caller's"); + try { + KeployBranchCoverage.analyze(bytes, new HashMap<>(), (m, r, h) -> { + throw mine; + }); + fail("the callback's exception was lost"); + } catch (IllegalArgumentException e) { + assertSame(mine, e); + } + // And so does what its wants throws, though it runs under the same wrap as the walk. + IllegalStateException declined = new IllegalStateException("the caller's wants"); + try { + KeployBranchCoverage.analyze(bytes, new HashMap<>(), new KeployBranchCoverage.EachProbe() { + @Override + public boolean wants(KeployBranchCoverage.Method m) { + throw declined; + } + + @Override + public void probe(KeployBranchCoverage.Method m, int r, KeployBranchCoverage.Hits h) { + } + }); + fail("the wants exception was lost"); + } catch (IllegalStateException e) { + assertSame(declined, e); + } + } + + @Test + public void twoCopiesOfAClassThatMapAlikeShareOneRecord() throws Exception { + // Two class files of one name whose bytes differ but whose probes map alike: a test's hits name the + // class only, and a hit of either copy covers the same leaves, so the map has one record per name + // (the Go decoder refuses a name listed twice). Its id is the first copy the index listed. + byte[] a = copy(false, 10, 12); + byte[] b = copy(true, 10, 12); + assertNotEquals(CRC64.classId(a), CRC64.classId(b)); + for (byte[][] c : new byte[][][] {{a, b}, {b, a}}) { + List files = new ArrayList<>(); + files.add(new BuildTable.ClassFile("d/C", c[0])); + files.add(new BuildTable.ClassFile("d/C", c[1])); + BuildTableTest.Collect table = new BuildTableTest.Collect(); + String[] got = probes(files, table).split("\n"); + assertTrue(table.records.get(0), table.records.get(0).contains("\"state\":\"ok\"")); + assertEquals(1, got.length); + JsonObject r = JsonParser.parseString(got[0]).getAsJsonObject(); + assertEquals("d/C", r.get("class").getAsString()); + assertEquals(String.format("%016x", CRC64.classId(c[0])), r.get("id").getAsString()); + assertEquals("[[0,1],[1,2]]", r.getAsJsonArray("probes").toString()); + } + } + + @Test + public void theSameClassFileTwiceIsOneClassFile() throws Exception { + // The index can find one class file under two paths: the table, its build id and the map are those of + // one copy. + byte[] a = copy(false, 10, 12); + BuildTableTest.Collect once = new BuildTableTest.Collect(); + String onceMap = probes(Collections.singletonList(new BuildTable.ClassFile("d/C", a)), once); + BuildTableTest.Collect twice = new BuildTableTest.Collect(); + String twiceMap = probes(Arrays.asList(new BuildTable.ClassFile("d/C", a), + new BuildTable.ClassFile("x/d/C", a)), twice); + assertEquals(once.build, twice.build); + assertEquals(once.records, twice.records); + assertEquals(onceMap, twiceMap); + assertEquals(1, twiceMap.split("\n").length); + } + + @Test + public void aLineBelowOneHasNoLeafAndItsMethodStays() throws Exception { + // javac never writes line 0, but bytecode tools do. The table leaves such a line out and keeps the + // method's other lines: the map must agree, not fail the class as a method missing from the table. + int[][][] cases = { + {{0, 5}, {7}}, + {{5, 0}, {7}}, + {{0}, {7}}, // a method with no other line has no leaf: left out of both + }; + for (int[][] c : cases) { + List files = Collections.singletonList(new BuildTable.ClassFile("d/C", + cls(false, false, c))); + BuildTableTest.Collect table = new BuildTableTest.Collect(); + BuildTable.Summary s = BuildTable.write(files, table); + assertEquals(0, s.notMeasured); + assertEquals(0, s.notMapped); + JsonObject unit = JsonParser.parseString(table.records.get(0)).getAsJsonObject(); + JsonArray methods = unit.getAsJsonArray("methods"); + assertEquals(Arrays.deepToString(c), c[0].length == 1 ? 1 : 2, methods.size()); + if (c[0].length > 1) { + JsonObject m0 = methods.get(0).getAsJsonObject(); + assertEquals("m:d/C#m0()V", m0.get("id").getAsString()); + assertEquals(5, m0.get("line").getAsInt()); + assertEquals(1, m0.getAsJsonArray("leaves").size()); + } + agree(files); + } + } + + @Test + public void theProbeMapFollowsTheTableAsItsOwnStream() throws Exception { + BuildTableTest.Collect table = new BuildTableTest.Collect(); + String golden = probes(BuildTableTest.javaDedup(), table); + { + BuildTableTest.Frames sent = new BuildTableTest.Frames(); + List files = BuildTableTest.javaDedup(); + BuildTable.send(() -> files, frame -> { + assertTrue(frame.length <= BuildTable.MAX_FRAME_BYTES); + sent.send(frame); + }); + int tables = 0; + ByteArrayOutputStream stream = new ByteArrayOutputStream(); + int seq = 0; + for (byte[] f : sent.frames) { + String h = BuildTableTest.header(f); + JsonObject o = JsonParser.parseString(h).getAsJsonObject(); + if (o.get("part").getAsString().equals("table")) { + assertEquals("the table comes first", 0, stream.size()); + tables++; + continue; + } + assertEquals("probes", o.get("part").getAsString()); + assertEquals(table.build, o.get("build").getAsString()); + assertEquals(seq++, o.get("seq").getAsInt()); + stream.write(f, 8 + h.length() + 1, f.length - 8 - h.length() - 1); + if (o.get("last").getAsBoolean()) { + assertEquals(5, o.get("records").getAsInt()); + assertEquals(BuildTableTest.sha256Hex(golden.getBytes(StandardCharsets.UTF_8)), + o.get("sha256").getAsString()); + assertEquals("the last frame ends the answer", f, sent.frames.get(sent.frames.size() - 1)); + } + } + assertEquals(1, tables); + assertTrue(seq >= 1); + assertArrayEquals(golden.getBytes(StandardCharsets.UTF_8), stream.toByteArray()); + } + } +} diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/edge.build b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/edge.build new file mode 100644 index 00000000..876c4c16 --- /dev/null +++ b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/edge.build @@ -0,0 +1 @@ +jv1-91aa3fda68544563b2a8cca5ad1dde54 diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/edge.table.ndjson b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/edge.table.ndjson new file mode 100644 index 00000000..8b23c5c9 --- /dev/null +++ b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/edge.table.ndjson @@ -0,0 +1,15 @@ +{"kind":"class","id":"c:$Lead","pkg":"","name":"$Lead","path":"Lead.java","hash":"dc3c222023b867f1112019d50ceba20066ece22df39651cddf4058ec3d3c8f25","state":"ok","methods":[{"id":"m:$Lead#()V","name":"static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:$Lead$In#()V","name":"In.static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:$Lead#()V","name":"$Lead()","line":73,"leaves":[[73,0,73,0,3,0]]},{"id":"m:$Lead#run(I[Ljava/lang/String;)I","name":"run(int, String...)","line":74,"leaves":[[74,0,74,0,2,0],[74,0,74,0,1,1],[74,0,74,0,1,1],[75,0,75,0,2,0],[76,0,76,0,10,0],[76,0,76,0,1,1],[76,0,76,0,1,1],[76,0,76,0,1,1],[76,0,76,0,1,1]]},{"id":"m:$Lead#same(I)I","name":"same(int)","line":74,"leaves":[[74,0,74,0,2,0],[74,0,74,0,1,1],[74,0,74,0,1,1],[75,0,75,0,2,0],[76,0,76,0,10,0],[76,0,76,0,1,1],[76,0,76,0,1,1],[76,0,76,0,1,1],[76,0,76,0,1,1]]},{"id":"m:$Lead$In#()V","name":"In()","line":80,"leaves":[[80,0,80,0,3,0]]},{"id":"m:$Lead$In#run(I[Ljava/lang/String;)I","name":"In.run(int, String...)","line":81,"leaves":[[81,0,81,0,2,0],[81,0,81,0,1,1],[81,0,81,0,1,1],[82,0,82,0,2,0],[83,0,83,0,10,0],[83,0,83,0,1,1],[83,0,83,0,1,1],[83,0,83,0,1,1],[83,0,83,0,1,1]]},{"id":"m:$Lead$In#same(I)I","name":"In.same(int)","line":81,"leaves":[[81,0,81,0,2,0],[81,0,81,0,1,1],[81,0,81,0,1,1],[82,0,82,0,2,0],[83,0,83,0,10,0],[83,0,83,0,1,1],[83,0,83,0,1,1],[83,0,83,0,1,1],[83,0,83,0,1,1]]},{"id":"m:$Lead#lambda$run$0(I)I","name":"run lambda #0","line":84,"leaves":[[84,0,84,0,2,0],[84,0,84,0,1,1],[84,0,84,0,1,1],[85,0,85,0,2,0],[86,0,86,0,10,0],[86,0,86,0,1,1],[86,0,86,0,1,1],[86,0,86,0,1,1],[86,0,86,0,1,1]]},{"id":"m:$Lead$In#lambda$run$0(I)I","name":"In.run lambda #0","line":91,"leaves":[[91,0,91,0,2,0],[91,0,91,0,1,1],[91,0,91,0,1,1],[92,0,92,0,2,0],[93,0,93,0,10,0],[93,0,93,0,1,1],[93,0,93,0,1,1],[93,0,93,0,1,1],[93,0,93,0,1,1]]},{"id":"m:$Lead#we\"ird#na$me(I)I","name":"we\"ird#na$me(int)","line":104,"leaves":[[104,0,104,0,2,0],[104,0,104,0,1,1],[104,0,104,0,1,1],[105,0,105,0,2,0],[106,0,106,0,10,0],[106,0,106,0,1,1],[106,0,106,0,1,1],[106,0,106,0,1,1],[106,0,106,0,1,1]]},{"id":"m:$Lead$In#we\"ird#na$me(I)I","name":"In.we\"ird#na$me(int)","line":111,"leaves":[[111,0,111,0,2,0],[111,0,111,0,1,1],[111,0,111,0,1,1],[112,0,112,0,2,0],[113,0,113,0,10,0],[113,0,113,0,1,1],[113,0,113,0,1,1],[113,0,113,0,1,1],[113,0,113,0,1,1]]},{"id":"m:$Lead#a\u2028b(I)I","name":"a\u2028b(int)","line":114,"leaves":[[114,0,114,0,2,0],[114,0,114,0,1,1],[114,0,114,0,1,1],[115,0,115,0,2,0],[116,0,116,0,10,0],[116,0,116,0,1,1],[116,0,116,0,1,1],[116,0,116,0,1,1],[116,0,116,0,1,1]]},{"id":"m:$Lead$In#a\u2028b(I)I","name":"In.a\u2028b(int)","line":121,"leaves":[[121,0,121,0,2,0],[121,0,121,0,1,1],[121,0,121,0,1,1],[122,0,122,0,2,0],[123,0,123,0,10,0],[123,0,123,0,1,1],[123,0,123,0,1,1],[123,0,123,0,1,1],[123,0,123,0,1,1]]}]} +{"kind":"class","id":"c:Default","pkg":"","name":"Default","path":"Default.java","hash":"2608d1b58304af9071c418e3653d575c90f4ea734b06f55997566719ab39a2ed","state":"ok","methods":[{"id":"m:Default#()V","name":"static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:Default#()V","name":"Default()","line":66,"leaves":[[66,0,66,0,3,0]]},{"id":"m:Default#run(I[Ljava/lang/String;)I","name":"run(int, String...)","line":67,"leaves":[[67,0,67,0,2,0],[67,0,67,0,1,1],[67,0,67,0,1,1],[68,0,68,0,2,0],[69,0,69,0,10,0],[69,0,69,0,1,1],[69,0,69,0,1,1],[69,0,69,0,1,1],[69,0,69,0,1,1]]},{"id":"m:Default#same(I)I","name":"same(int)","line":67,"leaves":[[67,0,67,0,2,0],[67,0,67,0,1,1],[67,0,67,0,1,1],[68,0,68,0,2,0],[69,0,69,0,10,0],[69,0,69,0,1,1],[69,0,69,0,1,1],[69,0,69,0,1,1],[69,0,69,0,1,1]]},{"id":"m:Default#lambda$run$0(I)I","name":"run lambda #0","line":77,"leaves":[[77,0,77,0,2,0],[77,0,77,0,1,1],[77,0,77,0,1,1],[78,0,78,0,2,0],[79,0,79,0,10,0],[79,0,79,0,1,1],[79,0,79,0,1,1],[79,0,79,0,1,1],[79,0,79,0,1,1]]},{"id":"m:Default#we\"ird#na$me(I)I","name":"we\"ird#na$me(int)","line":97,"leaves":[[97,0,97,0,2,0],[97,0,97,0,1,1],[97,0,97,0,1,1],[98,0,98,0,2,0],[99,0,99,0,10,0],[99,0,99,0,1,1],[99,0,99,0,1,1],[99,0,99,0,1,1],[99,0,99,0,1,1]]},{"id":"m:Default#a\u2028b(I)I","name":"a\u2028b(int)","line":107,"leaves":[[107,0,107,0,2,0],[107,0,107,0,1,1],[107,0,107,0,1,1],[108,0,108,0,2,0],[109,0,109,0,10,0],[109,0,109,0,1,1],[109,0,109,0,1,1],[109,0,109,0,1,1],[109,0,109,0,1,1]]}]} +{"kind":"class","id":"c:a/Qux","pkg":"a","name":"Qux","path":"","hash":"623ebc9845cb84b69f7686e6f8406e26165945d737cb511f1c9926548a48b9a0","state":"ok","methods":[{"id":"m:a/Qux#()V","name":"static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:a/Qux#()V","name":"Qux()","line":52,"leaves":[[52,0,52,0,3,0]]},{"id":"m:a/Qux#run(I[Ljava/lang/String;)I","name":"run(int, String...)","line":53,"leaves":[[53,0,53,0,2,0],[53,0,53,0,1,1],[53,0,53,0,1,1],[54,0,54,0,2,0],[55,0,55,0,10,0],[55,0,55,0,1,1],[55,0,55,0,1,1],[55,0,55,0,1,1],[55,0,55,0,1,1]]},{"id":"m:a/Qux#same(I)I","name":"same(int)","line":53,"leaves":[[53,0,53,0,2,0],[53,0,53,0,1,1],[53,0,53,0,1,1],[54,0,54,0,2,0],[55,0,55,0,10,0],[55,0,55,0,1,1],[55,0,55,0,1,1],[55,0,55,0,1,1],[55,0,55,0,1,1]]},{"id":"m:a/Qux#lambda$run$0(I)I","name":"run lambda #0","line":63,"leaves":[[63,0,63,0,2,0],[63,0,63,0,1,1],[63,0,63,0,1,1],[64,0,64,0,2,0],[65,0,65,0,10,0],[65,0,65,0,1,1],[65,0,65,0,1,1],[65,0,65,0,1,1],[65,0,65,0,1,1]]},{"id":"m:a/Qux#we\"ird#na$me(I)I","name":"we\"ird#na$me(int)","line":83,"leaves":[[83,0,83,0,2,0],[83,0,83,0,1,1],[83,0,83,0,1,1],[84,0,84,0,2,0],[85,0,85,0,10,0],[85,0,85,0,1,1],[85,0,85,0,1,1],[85,0,85,0,1,1],[85,0,85,0,1,1]]},{"id":"m:a/Qux#a\u2028b(I)I","name":"a\u2028b(int)","line":93,"leaves":[[93,0,93,0,2,0],[93,0,93,0,1,1],[93,0,93,0,1,1],[94,0,94,0,2,0],[95,0,95,0,10,0],[95,0,95,0,1,1],[95,0,95,0,1,1],[95,0,95,0,1,1],[95,0,95,0,1,1]]}]} +{"kind":"class","id":"c:a/b/Ctl\u0001\u001f","pkg":"a/b","name":"Ctl\u0001\u001f","path":"a/b/C\u2028.java","hash":"5b96177df3b932214ab696a4b7eda4f0cb0fc447721ad05bdfbdba5f4c932fe7","state":"ok","methods":[{"id":"m:a/b/Ctl\u0001\u001f#()V","name":"static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:a/b/Ctl\u0001\u001f#()V","name":"Ctl\u0001\u001f()","line":115,"leaves":[[115,0,115,0,3,0]]},{"id":"m:a/b/Ctl\u0001\u001f#run(I[Ljava/lang/String;)I","name":"run(int, String...)","line":116,"leaves":[[116,0,116,0,2,0],[116,0,116,0,1,1],[116,0,116,0,1,1],[117,0,117,0,2,0],[118,0,118,0,10,0],[118,0,118,0,1,1],[118,0,118,0,1,1],[118,0,118,0,1,1],[118,0,118,0,1,1]]},{"id":"m:a/b/Ctl\u0001\u001f#same(I)I","name":"same(int)","line":116,"leaves":[[116,0,116,0,2,0],[116,0,116,0,1,1],[116,0,116,0,1,1],[117,0,117,0,2,0],[118,0,118,0,10,0],[118,0,118,0,1,1],[118,0,118,0,1,1],[118,0,118,0,1,1],[118,0,118,0,1,1]]},{"id":"m:a/b/Ctl\u0001\u001f#lambda$run$0(I)I","name":"run lambda #0","line":126,"leaves":[[126,0,126,0,2,0],[126,0,126,0,1,1],[126,0,126,0,1,1],[127,0,127,0,2,0],[128,0,128,0,10,0],[128,0,128,0,1,1],[128,0,128,0,1,1],[128,0,128,0,1,1],[128,0,128,0,1,1]]},{"id":"m:a/b/Ctl\u0001\u001f#we\"ird#na$me(I)I","name":"we\"ird#na$me(int)","line":146,"leaves":[[146,0,146,0,2,0],[146,0,146,0,1,1],[146,0,146,0,1,1],[147,0,147,0,2,0],[148,0,148,0,10,0],[148,0,148,0,1,1],[148,0,148,0,1,1],[148,0,148,0,1,1],[148,0,148,0,1,1]]},{"id":"m:a/b/Ctl\u0001\u001f#a\u2028b(I)I","name":"a\u2028b(int)","line":156,"leaves":[[156,0,156,0,2,0],[156,0,156,0,1,1],[156,0,156,0,1,1],[157,0,157,0,2,0],[158,0,158,0,10,0],[158,0,158,0,1,1],[158,0,158,0,1,1],[158,0,158,0,1,1],[158,0,158,0,1,1]]}]} +{"kind":"class","id":"c:a/b/Foo","pkg":"a/b","name":"Foo","path":"a/b/Foo.java","hash":"313aeac8b1344fd6831d47fff4a1ad5fc283f46fd9b1157d8e340858ac49b26e","state":"ok","methods":[{"id":"m:a/b/Foo#()V","name":"static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:a/b/Foo$#()V","name":"static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:a/b/Foo$$anon$1#()V","name":"anon.1.static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:a/b/Foo$1#()V","name":"1.static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:a/b/Foo$Bar#()V","name":"Bar.static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:a/b/Foo$Bar$#()V","name":"Bar.static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:a/b/Foo$Bar$Deep#()V","name":"Bar.Deep.static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:a/b/Foo#()V","name":"Foo()","line":10,"leaves":[[10,0,10,0,3,0]]},{"id":"m:a/b/Foo#run(I[Ljava/lang/String;)I","name":"run(int, String...)","line":11,"leaves":[[11,0,11,0,2,0],[11,0,11,0,1,1],[11,0,11,0,1,1],[12,0,12,0,2,0],[13,0,13,0,10,0],[13,0,13,0,1,1],[13,0,13,0,1,1],[13,0,13,0,1,1],[13,0,13,0,1,1]]},{"id":"m:a/b/Foo#same(I)I","name":"same(int)","line":11,"leaves":[[11,0,11,0,2,0],[11,0,11,0,1,1],[11,0,11,0,1,1],[12,0,12,0,2,0],[13,0,13,0,10,0],[13,0,13,0,1,1],[13,0,13,0,1,1],[13,0,13,0,1,1],[13,0,13,0,1,1]]},{"id":"m:a/b/Foo$Bar#()V","name":"Bar()","line":17,"leaves":[[17,0,17,0,3,0]]},{"id":"m:a/b/Foo$Bar#run(I[Ljava/lang/String;)I","name":"Bar.run(int, String...)","line":18,"leaves":[[18,0,18,0,2,0],[18,0,18,0,1,1],[18,0,18,0,1,1],[19,0,19,0,2,0],[20,0,20,0,10,0],[20,0,20,0,1,1],[20,0,20,0,1,1],[20,0,20,0,1,1],[20,0,20,0,1,1]]},{"id":"m:a/b/Foo$Bar#same(I)I","name":"Bar.same(int)","line":18,"leaves":[[18,0,18,0,2,0],[18,0,18,0,1,1],[18,0,18,0,1,1],[19,0,19,0,2,0],[20,0,20,0,10,0],[20,0,20,0,1,1],[20,0,20,0,1,1],[20,0,20,0,1,1],[20,0,20,0,1,1]]},{"id":"m:a/b/Foo#lambda$run$0(I)I","name":"run lambda #0","line":21,"leaves":[[21,0,21,0,2,0],[21,0,21,0,1,1],[21,0,21,0,1,1],[22,0,22,0,2,0],[23,0,23,0,10,0],[23,0,23,0,1,1],[23,0,23,0,1,1],[23,0,23,0,1,1],[23,0,23,0,1,1]]},{"id":"m:a/b/Foo$1#()V","name":"1()","line":24,"leaves":[[24,0,24,0,3,0]]},{"id":"m:a/b/Foo$1#run(I[Ljava/lang/String;)I","name":"1.run(int, String...)","line":25,"leaves":[[25,0,25,0,2,0],[25,0,25,0,1,1],[25,0,25,0,1,1],[26,0,26,0,2,0],[27,0,27,0,10,0],[27,0,27,0,1,1],[27,0,27,0,1,1],[27,0,27,0,1,1],[27,0,27,0,1,1]]},{"id":"m:a/b/Foo$1#same(I)I","name":"1.same(int)","line":25,"leaves":[[25,0,25,0,2,0],[25,0,25,0,1,1],[25,0,25,0,1,1],[26,0,26,0,2,0],[27,0,27,0,10,0],[27,0,27,0,1,1],[27,0,27,0,1,1],[27,0,27,0,1,1],[27,0,27,0,1,1]]},{"id":"m:a/b/Foo$Bar#lambda$run$0(I)I","name":"Bar.run lambda #0","line":28,"leaves":[[28,0,28,0,2,0],[28,0,28,0,1,1],[28,0,28,0,1,1],[29,0,29,0,2,0],[30,0,30,0,10,0],[30,0,30,0,1,1],[30,0,30,0,1,1],[30,0,30,0,1,1],[30,0,30,0,1,1]]},{"id":"m:a/b/Foo$#()V","name":"Foo()","line":31,"leaves":[[31,0,31,0,3,0]]},{"id":"m:a/b/Foo$#run(I[Ljava/lang/String;)I","name":"run(int, String...)","line":32,"leaves":[[32,0,32,0,2,0],[32,0,32,0,1,1],[32,0,32,0,1,1],[33,0,33,0,2,0],[34,0,34,0,10,0],[34,0,34,0,1,1],[34,0,34,0,1,1],[34,0,34,0,1,1],[34,0,34,0,1,1]]},{"id":"m:a/b/Foo$#same(I)I","name":"same(int)","line":32,"leaves":[[32,0,32,0,2,0],[32,0,32,0,1,1],[32,0,32,0,1,1],[33,0,33,0,2,0],[34,0,34,0,10,0],[34,0,34,0,1,1],[34,0,34,0,1,1],[34,0,34,0,1,1],[34,0,34,0,1,1]]},{"id":"m:a/b/Foo$1#lambda$run$0(I)I","name":"1.run lambda #0","line":35,"leaves":[[35,0,35,0,2,0],[35,0,35,0,1,1],[35,0,35,0,1,1],[36,0,36,0,2,0],[37,0,37,0,10,0],[37,0,37,0,1,1],[37,0,37,0,1,1],[37,0,37,0,1,1],[37,0,37,0,1,1]]},{"id":"m:a/b/Foo$Bar$Deep#()V","name":"Bar.Deep()","line":38,"leaves":[[38,0,38,0,3,0]]},{"id":"m:a/b/Foo$Bar$Deep#run(I[Ljava/lang/String;)I","name":"Bar.Deep.run(int, String...)","line":39,"leaves":[[39,0,39,0,2,0],[39,0,39,0,1,1],[39,0,39,0,1,1],[40,0,40,0,2,0],[41,0,41,0,10,0],[41,0,41,0,1,1],[41,0,41,0,1,1],[41,0,41,0,1,1],[41,0,41,0,1,1]]},{"id":"m:a/b/Foo$Bar$Deep#same(I)I","name":"Bar.Deep.same(int)","line":39,"leaves":[[39,0,39,0,2,0],[39,0,39,0,1,1],[39,0,39,0,1,1],[40,0,40,0,2,0],[41,0,41,0,10,0],[41,0,41,0,1,1],[41,0,41,0,1,1],[41,0,41,0,1,1],[41,0,41,0,1,1]]},{"id":"m:a/b/Foo#we\"ird#na$me(I)I","name":"we\"ird#na$me(int)","line":41,"leaves":[[41,0,41,0,2,0],[41,0,41,0,1,1],[41,0,41,0,1,1],[42,0,42,0,2,0],[43,0,43,0,10,0],[43,0,43,0,1,1],[43,0,43,0,1,1],[43,0,43,0,1,1],[43,0,43,0,1,1]]},{"id":"m:a/b/Foo$#lambda$run$0(I)I","name":"run lambda #0","line":42,"leaves":[[42,0,42,0,2,0],[42,0,42,0,1,1],[42,0,42,0,1,1],[43,0,43,0,2,0],[44,0,44,0,10,0],[44,0,44,0,1,1],[44,0,44,0,1,1],[44,0,44,0,1,1],[44,0,44,0,1,1]]},{"id":"m:a/b/Foo$Bar#we\"ird#na$me(I)I","name":"Bar.we\"ird#na$me(int)","line":48,"leaves":[[48,0,48,0,2,0],[48,0,48,0,1,1],[48,0,48,0,1,1],[49,0,49,0,2,0],[50,0,50,0,10,0],[50,0,50,0,1,1],[50,0,50,0,1,1],[50,0,50,0,1,1],[50,0,50,0,1,1]]},{"id":"m:a/b/Foo$Bar$Deep#lambda$run$0(I)I","name":"Bar.Deep.run lambda #0","line":49,"leaves":[[49,0,49,0,2,0],[49,0,49,0,1,1],[49,0,49,0,1,1],[50,0,50,0,2,0],[51,0,51,0,10,0],[51,0,51,0,1,1],[51,0,51,0,1,1],[51,0,51,0,1,1],[51,0,51,0,1,1]]},{"id":"m:a/b/Foo#a\u2028b(I)I","name":"a\u2028b(int)","line":51,"leaves":[[51,0,51,0,2,0],[51,0,51,0,1,1],[51,0,51,0,1,1],[52,0,52,0,2,0],[53,0,53,0,10,0],[53,0,53,0,1,1],[53,0,53,0,1,1],[53,0,53,0,1,1],[53,0,53,0,1,1]]},{"id":"m:a/b/Foo$1#we\"ird#na$me(I)I","name":"1.we\"ird#na$me(int)","line":55,"leaves":[[55,0,55,0,2,0],[55,0,55,0,1,1],[55,0,55,0,1,1],[56,0,56,0,2,0],[57,0,57,0,10,0],[57,0,57,0,1,1],[57,0,57,0,1,1],[57,0,57,0,1,1],[57,0,57,0,1,1]]},{"id":"m:a/b/Foo$Bar#a\u2028b(I)I","name":"Bar.a\u2028b(int)","line":58,"leaves":[[58,0,58,0,2,0],[58,0,58,0,1,1],[58,0,58,0,1,1],[59,0,59,0,2,0],[60,0,60,0,10,0],[60,0,60,0,1,1],[60,0,60,0,1,1],[60,0,60,0,1,1],[60,0,60,0,1,1]]},{"id":"m:a/b/Foo$#we\"ird#na$me(I)I","name":"we\"ird#na$me(int)","line":62,"leaves":[[62,0,62,0,2,0],[62,0,62,0,1,1],[62,0,62,0,1,1],[63,0,63,0,2,0],[64,0,64,0,10,0],[64,0,64,0,1,1],[64,0,64,0,1,1],[64,0,64,0,1,1],[64,0,64,0,1,1]]},{"id":"m:a/b/Foo$1#a\u2028b(I)I","name":"1.a\u2028b(int)","line":65,"leaves":[[65,0,65,0,2,0],[65,0,65,0,1,1],[65,0,65,0,1,1],[66,0,66,0,2,0],[67,0,67,0,10,0],[67,0,67,0,1,1],[67,0,67,0,1,1],[67,0,67,0,1,1],[67,0,67,0,1,1]]},{"id":"m:a/b/Foo$Bar$Deep#we\"ird#na$me(I)I","name":"Bar.Deep.we\"ird#na$me(int)","line":69,"leaves":[[69,0,69,0,2,0],[69,0,69,0,1,1],[69,0,69,0,1,1],[70,0,70,0,2,0],[71,0,71,0,10,0],[71,0,71,0,1,1],[71,0,71,0,1,1],[71,0,71,0,1,1],[71,0,71,0,1,1]]},{"id":"m:a/b/Foo$#a\u2028b(I)I","name":"a\u2028b(int)","line":72,"leaves":[[72,0,72,0,2,0],[72,0,72,0,1,1],[72,0,72,0,1,1],[73,0,73,0,2,0],[74,0,74,0,10,0],[74,0,74,0,1,1],[74,0,74,0,1,1],[74,0,74,0,1,1],[74,0,74,0,1,1]]},{"id":"m:a/b/Foo$Bar$Deep#a\u2028b(I)I","name":"Bar.Deep.a\u2028b(int)","line":79,"leaves":[[79,0,79,0,2,0],[79,0,79,0,1,1],[79,0,79,0,1,1],[80,0,80,0,2,0],[81,0,81,0,10,0],[81,0,81,0,1,1],[81,0,81,0,1,1],[81,0,81,0,1,1],[81,0,81,0,1,1]]},{"id":"m:a/b/Foo$Bar$#()V","name":"Bar()","line":136,"leaves":[[136,0,136,0,3,0]]},{"id":"m:a/b/Foo$Bar$#run(I[Ljava/lang/String;)I","name":"Bar.run(int, String...)","line":137,"leaves":[[137,0,137,0,2,0],[137,0,137,0,1,1],[137,0,137,0,1,1],[138,0,138,0,2,0],[139,0,139,0,10,0],[139,0,139,0,1,1],[139,0,139,0,1,1],[139,0,139,0,1,1],[139,0,139,0,1,1]]},{"id":"m:a/b/Foo$Bar$#same(I)I","name":"Bar.same(int)","line":137,"leaves":[[137,0,137,0,2,0],[137,0,137,0,1,1],[137,0,137,0,1,1],[138,0,138,0,2,0],[139,0,139,0,10,0],[139,0,139,0,1,1],[139,0,139,0,1,1],[139,0,139,0,1,1],[139,0,139,0,1,1]]},{"id":"m:a/b/Foo$$anon$1#()V","name":"anon.1()","line":143,"leaves":[[143,0,143,0,3,0]]},{"id":"m:a/b/Foo$$anon$1#run(I[Ljava/lang/String;)I","name":"anon.1.run(int, String...)","line":144,"leaves":[[144,0,144,0,2,0],[144,0,144,0,1,1],[144,0,144,0,1,1],[145,0,145,0,2,0],[146,0,146,0,10,0],[146,0,146,0,1,1],[146,0,146,0,1,1],[146,0,146,0,1,1],[146,0,146,0,1,1]]},{"id":"m:a/b/Foo$$anon$1#same(I)I","name":"anon.1.same(int)","line":144,"leaves":[[144,0,144,0,2,0],[144,0,144,0,1,1],[144,0,144,0,1,1],[145,0,145,0,2,0],[146,0,146,0,10,0],[146,0,146,0,1,1],[146,0,146,0,1,1],[146,0,146,0,1,1],[146,0,146,0,1,1]]},{"id":"m:a/b/Foo$Bar$#lambda$run$0(I)I","name":"Bar.run lambda #0","line":147,"leaves":[[147,0,147,0,2,0],[147,0,147,0,1,1],[147,0,147,0,1,1],[148,0,148,0,2,0],[149,0,149,0,10,0],[149,0,149,0,1,1],[149,0,149,0,1,1],[149,0,149,0,1,1],[149,0,149,0,1,1]]},{"id":"m:a/b/Foo$$anon$1#lambda$run$0(I)I","name":"anon.1.run lambda #0","line":154,"leaves":[[154,0,154,0,2,0],[154,0,154,0,1,1],[154,0,154,0,1,1],[155,0,155,0,2,0],[156,0,156,0,10,0],[156,0,156,0,1,1],[156,0,156,0,1,1],[156,0,156,0,1,1],[156,0,156,0,1,1]]},{"id":"m:a/b/Foo$Bar$#we\"ird#na$me(I)I","name":"Bar.we\"ird#na$me(int)","line":167,"leaves":[[167,0,167,0,2,0],[167,0,167,0,1,1],[167,0,167,0,1,1],[168,0,168,0,2,0],[169,0,169,0,10,0],[169,0,169,0,1,1],[169,0,169,0,1,1],[169,0,169,0,1,1],[169,0,169,0,1,1]]},{"id":"m:a/b/Foo$$anon$1#we\"ird#na$me(I)I","name":"anon.1.we\"ird#na$me(int)","line":174,"leaves":[[174,0,174,0,2,0],[174,0,174,0,1,1],[174,0,174,0,1,1],[175,0,175,0,2,0],[176,0,176,0,10,0],[176,0,176,0,1,1],[176,0,176,0,1,1],[176,0,176,0,1,1],[176,0,176,0,1,1]]},{"id":"m:a/b/Foo$Bar$#a\u2028b(I)I","name":"Bar.a\u2028b(int)","line":177,"leaves":[[177,0,177,0,2,0],[177,0,177,0,1,1],[177,0,177,0,1,1],[178,0,178,0,2,0],[179,0,179,0,10,0],[179,0,179,0,1,1],[179,0,179,0,1,1],[179,0,179,0,1,1],[179,0,179,0,1,1]]},{"id":"m:a/b/Foo$$anon$1#a\u2028b(I)I","name":"anon.1.a\u2028b(int)","line":184,"leaves":[[184,0,184,0,2,0],[184,0,184,0,1,1],[184,0,184,0,1,1],[185,0,185,0,2,0],[186,0,186,0,10,0],[186,0,186,0,1,1],[186,0,186,0,1,1],[186,0,186,0,1,1],[186,0,186,0,1,1]]}]} +{"kind":"class","id":"c:a/b/L\u2028S\u2029","pkg":"a/b","name":"L\u2028S\u2029","path":"a/b/L.java","hash":"251d9cff82aeb2b359aeaccf0a1e7016751ab5c4cb0844602bd8a4887b453f8a","state":"ok","methods":[{"id":"m:a/b/L\u2028S\u2029#()V","name":"static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:a/b/L\u2028S\u2029#()V","name":"L\u2028S\u2029()","line":122,"leaves":[[122,0,122,0,3,0]]},{"id":"m:a/b/L\u2028S\u2029#run(I[Ljava/lang/String;)I","name":"run(int, String...)","line":123,"leaves":[[123,0,123,0,2,0],[123,0,123,0,1,1],[123,0,123,0,1,1],[124,0,124,0,2,0],[125,0,125,0,10,0],[125,0,125,0,1,1],[125,0,125,0,1,1],[125,0,125,0,1,1],[125,0,125,0,1,1]]},{"id":"m:a/b/L\u2028S\u2029#same(I)I","name":"same(int)","line":123,"leaves":[[123,0,123,0,2,0],[123,0,123,0,1,1],[123,0,123,0,1,1],[124,0,124,0,2,0],[125,0,125,0,10,0],[125,0,125,0,1,1],[125,0,125,0,1,1],[125,0,125,0,1,1],[125,0,125,0,1,1]]},{"id":"m:a/b/L\u2028S\u2029#lambda$run$0(I)I","name":"run lambda #0","line":133,"leaves":[[133,0,133,0,2,0],[133,0,133,0,1,1],[133,0,133,0,1,1],[134,0,134,0,2,0],[135,0,135,0,10,0],[135,0,135,0,1,1],[135,0,135,0,1,1],[135,0,135,0,1,1],[135,0,135,0,1,1]]},{"id":"m:a/b/L\u2028S\u2029#we\"ird#na$me(I)I","name":"we\"ird#na$me(int)","line":153,"leaves":[[153,0,153,0,2,0],[153,0,153,0,1,1],[153,0,153,0,1,1],[154,0,154,0,2,0],[155,0,155,0,10,0],[155,0,155,0,1,1],[155,0,155,0,1,1],[155,0,155,0,1,1],[155,0,155,0,1,1]]},{"id":"m:a/b/L\u2028S\u2029#a\u2028b(I)I","name":"a\u2028b(int)","line":163,"leaves":[[163,0,163,0,2,0],[163,0,163,0,1,1],[163,0,163,0,1,1],[164,0,164,0,2,0],[165,0,165,0,10,0],[165,0,165,0,1,1],[165,0,165,0,1,1],[165,0,165,0,1,1],[165,0,165,0,1,1]]}]} +{"kind":"class","id":"c:a/b/NoLines","pkg":"a/b","name":"NoLines","path":"a/b/NoLines.java","hash":"25d1deee75fbd65a4d742d7e65f7c93ab9d0b39859f709a53df3e98111f1a70e","state":"ok","methods":[]} +{"kind":"class","id":"c:a/b/Q\"uo\\te<&>","pkg":"a/b","name":"Q\"uo\\te<&>","path":"a/b/Q\"uo\\te.java","hash":"ae57d97bcbf4241c5a0e02ea8a1a9287528b3ac8bfe539141d10878fb35e4794","state":"ok","methods":[{"id":"m:a/b/Q\"uo\\te<&>#()V","name":"static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:a/b/Q\"uo\\te<&>#()V","name":"Q\"uo\\te<&>()","line":108,"leaves":[[108,0,108,0,3,0]]},{"id":"m:a/b/Q\"uo\\te<&>#run(I[Ljava/lang/String;)I","name":"run(int, String...)","line":109,"leaves":[[109,0,109,0,2,0],[109,0,109,0,1,1],[109,0,109,0,1,1],[110,0,110,0,2,0],[111,0,111,0,10,0],[111,0,111,0,1,1],[111,0,111,0,1,1],[111,0,111,0,1,1],[111,0,111,0,1,1]]},{"id":"m:a/b/Q\"uo\\te<&>#same(I)I","name":"same(int)","line":109,"leaves":[[109,0,109,0,2,0],[109,0,109,0,1,1],[109,0,109,0,1,1],[110,0,110,0,2,0],[111,0,111,0,10,0],[111,0,111,0,1,1],[111,0,111,0,1,1],[111,0,111,0,1,1],[111,0,111,0,1,1]]},{"id":"m:a/b/Q\"uo\\te<&>#lambda$run$0(I)I","name":"run lambda #0","line":119,"leaves":[[119,0,119,0,2,0],[119,0,119,0,1,1],[119,0,119,0,1,1],[120,0,120,0,2,0],[121,0,121,0,10,0],[121,0,121,0,1,1],[121,0,121,0,1,1],[121,0,121,0,1,1],[121,0,121,0,1,1]]},{"id":"m:a/b/Q\"uo\\te<&>#we\"ird#na$me(I)I","name":"we\"ird#na$me(int)","line":139,"leaves":[[139,0,139,0,2,0],[139,0,139,0,1,1],[139,0,139,0,1,1],[140,0,140,0,2,0],[141,0,141,0,10,0],[141,0,141,0,1,1],[141,0,141,0,1,1],[141,0,141,0,1,1],[141,0,141,0,1,1]]},{"id":"m:a/b/Q\"uo\\te<&>#a\u2028b(I)I","name":"a\u2028b(int)","line":149,"leaves":[[149,0,149,0,2,0],[149,0,149,0,1,1],[149,0,149,0,1,1],[150,0,150,0,2,0],[151,0,151,0,10,0],[151,0,151,0,1,1],[151,0,151,0,1,1],[151,0,151,0,1,1],[151,0,151,0,1,1]]}]} +{"kind":"class","id":"c:a/b/Z~","pkg":"a/b","name":"Z~","path":"a/b/Z.java","hash":"9f34a97e2d2c5f4d2d321437cba3c738d2f85b785929d0e2a2f4e8b561413dda","state":"ok","methods":[{"id":"m:a/b/Z~#()V","name":"static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:a/b/Z~#()V","name":"Z~()","line":94,"leaves":[[94,0,94,0,3,0]]},{"id":"m:a/b/Z~#run(I[Ljava/lang/String;)I","name":"run(int, String...)","line":95,"leaves":[[95,0,95,0,2,0],[95,0,95,0,1,1],[95,0,95,0,1,1],[96,0,96,0,2,0],[97,0,97,0,10,0],[97,0,97,0,1,1],[97,0,97,0,1,1],[97,0,97,0,1,1],[97,0,97,0,1,1]]},{"id":"m:a/b/Z~#same(I)I","name":"same(int)","line":95,"leaves":[[95,0,95,0,2,0],[95,0,95,0,1,1],[95,0,95,0,1,1],[96,0,96,0,2,0],[97,0,97,0,10,0],[97,0,97,0,1,1],[97,0,97,0,1,1],[97,0,97,0,1,1],[97,0,97,0,1,1]]},{"id":"m:a/b/Z~#lambda$run$0(I)I","name":"run lambda #0","line":105,"leaves":[[105,0,105,0,2,0],[105,0,105,0,1,1],[105,0,105,0,1,1],[106,0,106,0,2,0],[107,0,107,0,10,0],[107,0,107,0,1,1],[107,0,107,0,1,1],[107,0,107,0,1,1],[107,0,107,0,1,1]]},{"id":"m:a/b/Z~#we\"ird#na$me(I)I","name":"we\"ird#na$me(int)","line":125,"leaves":[[125,0,125,0,2,0],[125,0,125,0,1,1],[125,0,125,0,1,1],[126,0,126,0,2,0],[127,0,127,0,10,0],[127,0,127,0,1,1],[127,0,127,0,1,1],[127,0,127,0,1,1],[127,0,127,0,1,1]]},{"id":"m:a/b/Z~#a\u2028b(I)I","name":"a\u2028b(int)","line":135,"leaves":[[135,0,135,0,2,0],[135,0,135,0,1,1],[135,0,135,0,1,1],[136,0,136,0,2,0],[137,0,137,0,10,0],[137,0,137,0,1,1],[137,0,137,0,1,1],[137,0,137,0,1,1],[137,0,137,0,1,1]]}]} +{"kind":"class","id":"c:a/b/Z😀","pkg":"a/b","name":"Z😀","path":"a/b/Z.java","hash":"52292e7aaf05aeb62ddf6484e25618bb828638ad0ab9fcb89bc1811a5d502b34","state":"ok","methods":[{"id":"m:a/b/Z😀#()V","name":"static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:a/b/Z😀#()V","name":"Z😀()","line":101,"leaves":[[101,0,101,0,3,0]]},{"id":"m:a/b/Z😀#run(I[Ljava/lang/String;)I","name":"run(int, String...)","line":102,"leaves":[[102,0,102,0,2,0],[102,0,102,0,1,1],[102,0,102,0,1,1],[103,0,103,0,2,0],[104,0,104,0,10,0],[104,0,104,0,1,1],[104,0,104,0,1,1],[104,0,104,0,1,1],[104,0,104,0,1,1]]},{"id":"m:a/b/Z😀#same(I)I","name":"same(int)","line":102,"leaves":[[102,0,102,0,2,0],[102,0,102,0,1,1],[102,0,102,0,1,1],[103,0,103,0,2,0],[104,0,104,0,10,0],[104,0,104,0,1,1],[104,0,104,0,1,1],[104,0,104,0,1,1],[104,0,104,0,1,1]]},{"id":"m:a/b/Z😀#lambda$run$0(I)I","name":"run lambda #0","line":112,"leaves":[[112,0,112,0,2,0],[112,0,112,0,1,1],[112,0,112,0,1,1],[113,0,113,0,2,0],[114,0,114,0,10,0],[114,0,114,0,1,1],[114,0,114,0,1,1],[114,0,114,0,1,1],[114,0,114,0,1,1]]},{"id":"m:a/b/Z😀#we\"ird#na$me(I)I","name":"we\"ird#na$me(int)","line":132,"leaves":[[132,0,132,0,2,0],[132,0,132,0,1,1],[132,0,132,0,1,1],[133,0,133,0,2,0],[134,0,134,0,10,0],[134,0,134,0,1,1],[134,0,134,0,1,1],[134,0,134,0,1,1],[134,0,134,0,1,1]]},{"id":"m:a/b/Z😀#a\u2028b(I)I","name":"a\u2028b(int)","line":142,"leaves":[[142,0,142,0,2,0],[142,0,142,0,1,1],[142,0,142,0,1,1],[143,0,143,0,2,0],[144,0,144,0,10,0],[144,0,144,0,1,1],[144,0,144,0,1,1],[144,0,144,0,1,1],[144,0,144,0,1,1]]}]} +{"kind":"class","id":"c:a/b/module-info","pkg":"a/b","name":"module-info","path":"","hash":"f2d4c49e23f2b4b46332871c40cded4fb31cd369d48f2de0563fa3a15fe1247b","state":"ok","methods":[{"id":"m:a/b/module-info#()V","name":"static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:a/b/module-info#()V","name":"module-info()","line":150,"leaves":[[150,0,150,0,3,0]]},{"id":"m:a/b/module-info#run(I[Ljava/lang/String;)I","name":"run(int, String...)","line":151,"leaves":[[151,0,151,0,2,0],[151,0,151,0,1,1],[151,0,151,0,1,1],[152,0,152,0,2,0],[153,0,153,0,10,0],[153,0,153,0,1,1],[153,0,153,0,1,1],[153,0,153,0,1,1],[153,0,153,0,1,1]]},{"id":"m:a/b/module-info#same(I)I","name":"same(int)","line":151,"leaves":[[151,0,151,0,2,0],[151,0,151,0,1,1],[151,0,151,0,1,1],[152,0,152,0,2,0],[153,0,153,0,10,0],[153,0,153,0,1,1],[153,0,153,0,1,1],[153,0,153,0,1,1],[153,0,153,0,1,1]]},{"id":"m:a/b/module-info#lambda$run$0(I)I","name":"run lambda #0","line":161,"leaves":[[161,0,161,0,2,0],[161,0,161,0,1,1],[161,0,161,0,1,1],[162,0,162,0,2,0],[163,0,163,0,10,0],[163,0,163,0,1,1],[163,0,163,0,1,1],[163,0,163,0,1,1],[163,0,163,0,1,1]]},{"id":"m:a/b/module-info#we\"ird#na$me(I)I","name":"we\"ird#na$me(int)","line":181,"leaves":[[181,0,181,0,2,0],[181,0,181,0,1,1],[181,0,181,0,1,1],[182,0,182,0,2,0],[183,0,183,0,10,0],[183,0,183,0,1,1],[183,0,183,0,1,1],[183,0,183,0,1,1],[183,0,183,0,1,1]]},{"id":"m:a/b/module-info#a\u2028b(I)I","name":"a\u2028b(int)","line":191,"leaves":[[191,0,191,0,2,0],[191,0,191,0,1,1],[191,0,191,0,1,1],[192,0,192,0,2,0],[193,0,193,0,10,0],[193,0,193,0,1,1],[193,0,193,0,1,1],[193,0,193,0,1,1],[193,0,193,0,1,1]]}]} +{"kind":"class","id":"c:a/b/package-info","pkg":"a/b","name":"package-info","path":"a/b/package-info.java","hash":"ed7f74fba3fe4572cda03001178ca3138337e83f015b90c7a54e8282d80b7651","state":"ok","methods":[]} +{"kind":"class","id":"c:a/b/c/Deep","pkg":"a/b/c","name":"Deep","path":"","hash":"13d1610117f52711d1289994ce274a66e9a44ee4fe5d1501d8a69ff58d3a29ea","state":"ok","methods":[{"id":"m:a/b/c/Deep#()V","name":"static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:a/b/c/Deep#()V","name":"Deep()","line":59,"leaves":[[59,0,59,0,3,0]]},{"id":"m:a/b/c/Deep#run(I[Ljava/lang/String;)I","name":"run(int, String...)","line":60,"leaves":[[60,0,60,0,2,0],[60,0,60,0,1,1],[60,0,60,0,1,1],[61,0,61,0,2,0],[62,0,62,0,10,0],[62,0,62,0,1,1],[62,0,62,0,1,1],[62,0,62,0,1,1],[62,0,62,0,1,1]]},{"id":"m:a/b/c/Deep#same(I)I","name":"same(int)","line":60,"leaves":[[60,0,60,0,2,0],[60,0,60,0,1,1],[60,0,60,0,1,1],[61,0,61,0,2,0],[62,0,62,0,10,0],[62,0,62,0,1,1],[62,0,62,0,1,1],[62,0,62,0,1,1],[62,0,62,0,1,1]]},{"id":"m:a/b/c/Deep#lambda$run$0(I)I","name":"run lambda #0","line":70,"leaves":[[70,0,70,0,2,0],[70,0,70,0,1,1],[70,0,70,0,1,1],[71,0,71,0,2,0],[72,0,72,0,10,0],[72,0,72,0,1,1],[72,0,72,0,1,1],[72,0,72,0,1,1],[72,0,72,0,1,1]]},{"id":"m:a/b/c/Deep#we\"ird#na$me(I)I","name":"we\"ird#na$me(int)","line":90,"leaves":[[90,0,90,0,2,0],[90,0,90,0,1,1],[90,0,90,0,1,1],[91,0,91,0,2,0],[92,0,92,0,10,0],[92,0,92,0,1,1],[92,0,92,0,1,1],[92,0,92,0,1,1],[92,0,92,0,1,1]]},{"id":"m:a/b/c/Deep#a\u2028b(I)I","name":"a\u2028b(int)","line":100,"leaves":[[100,0,100,0,2,0],[100,0,100,0,1,1],[100,0,100,0,1,1],[101,0,101,0,2,0],[102,0,102,0,10,0],[102,0,102,0,1,1],[102,0,102,0,1,1],[102,0,102,0,1,1],[102,0,102,0,1,1]]}]} +{"kind":"class","id":"c:a/ü/Ünï","pkg":"a/ü","name":"Ünï","path":"a/ü/Ünï.java","hash":"2c745e3098b938c18e3c19b7adeca112a933d1e527fed86837be10080991f137","state":"ok","methods":[{"id":"m:a/ü/Ünï#()V","name":"static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:a/ü/Ünï#()V","name":"Ünï()","line":87,"leaves":[[87,0,87,0,3,0]]},{"id":"m:a/ü/Ünï#run(I[Ljava/lang/String;)I","name":"run(int, String...)","line":88,"leaves":[[88,0,88,0,2,0],[88,0,88,0,1,1],[88,0,88,0,1,1],[89,0,89,0,2,0],[90,0,90,0,10,0],[90,0,90,0,1,1],[90,0,90,0,1,1],[90,0,90,0,1,1],[90,0,90,0,1,1]]},{"id":"m:a/ü/Ünï#same(I)I","name":"same(int)","line":88,"leaves":[[88,0,88,0,2,0],[88,0,88,0,1,1],[88,0,88,0,1,1],[89,0,89,0,2,0],[90,0,90,0,10,0],[90,0,90,0,1,1],[90,0,90,0,1,1],[90,0,90,0,1,1],[90,0,90,0,1,1]]},{"id":"m:a/ü/Ünï#lambda$run$0(I)I","name":"run lambda #0","line":98,"leaves":[[98,0,98,0,2,0],[98,0,98,0,1,1],[98,0,98,0,1,1],[99,0,99,0,2,0],[100,0,100,0,10,0],[100,0,100,0,1,1],[100,0,100,0,1,1],[100,0,100,0,1,1],[100,0,100,0,1,1]]},{"id":"m:a/ü/Ünï#we\"ird#na$me(I)I","name":"we\"ird#na$me(int)","line":118,"leaves":[[118,0,118,0,2,0],[118,0,118,0,1,1],[118,0,118,0,1,1],[119,0,119,0,2,0],[120,0,120,0,10,0],[120,0,120,0,1,1],[120,0,120,0,1,1],[120,0,120,0,1,1],[120,0,120,0,1,1]]},{"id":"m:a/ü/Ünï#a\u2028b(I)I","name":"a\u2028b(int)","line":128,"leaves":[[128,0,128,0,2,0],[128,0,128,0,1,1],[128,0,128,0,1,1],[129,0,129,0,2,0],[130,0,130,0,10,0],[130,0,130,0,1,1],[130,0,130,0,1,1],[130,0,130,0,1,1],[130,0,130,0,1,1]]}]} +{"kind":"class","id":"c:a-x/Baz","pkg":"a-x","name":"Baz","path":"a-x/Baz.java","hash":"9ac2b9e989ea27c93330ec7f1b7d7ba81c253b4b7a3da606e2866e2c3693c72d","state":"ok","methods":[{"id":"m:a-x/Baz#()V","name":"static {}","line":1,"leaves":[[1,0,1,0,1,0]]},{"id":"m:a-x/Baz#()V","name":"Baz()","line":45,"leaves":[[45,0,45,0,3,0]]},{"id":"m:a-x/Baz#run(I[Ljava/lang/String;)I","name":"run(int, String...)","line":46,"leaves":[[46,0,46,0,2,0],[46,0,46,0,1,1],[46,0,46,0,1,1],[47,0,47,0,2,0],[48,0,48,0,10,0],[48,0,48,0,1,1],[48,0,48,0,1,1],[48,0,48,0,1,1],[48,0,48,0,1,1]]},{"id":"m:a-x/Baz#same(I)I","name":"same(int)","line":46,"leaves":[[46,0,46,0,2,0],[46,0,46,0,1,1],[46,0,46,0,1,1],[47,0,47,0,2,0],[48,0,48,0,10,0],[48,0,48,0,1,1],[48,0,48,0,1,1],[48,0,48,0,1,1],[48,0,48,0,1,1]]},{"id":"m:a-x/Baz#lambda$run$0(I)I","name":"run lambda #0","line":56,"leaves":[[56,0,56,0,2,0],[56,0,56,0,1,1],[56,0,56,0,1,1],[57,0,57,0,2,0],[58,0,58,0,10,0],[58,0,58,0,1,1],[58,0,58,0,1,1],[58,0,58,0,1,1],[58,0,58,0,1,1]]},{"id":"m:a-x/Baz#we\"ird#na$me(I)I","name":"we\"ird#na$me(int)","line":76,"leaves":[[76,0,76,0,2,0],[76,0,76,0,1,1],[76,0,76,0,1,1],[77,0,77,0,2,0],[78,0,78,0,10,0],[78,0,78,0,1,1],[78,0,78,0,1,1],[78,0,78,0,1,1],[78,0,78,0,1,1]]},{"id":"m:a-x/Baz#a\u2028b(I)I","name":"a\u2028b(int)","line":86,"leaves":[[86,0,86,0,2,0],[86,0,86,0,1,1],[86,0,86,0,1,1],[87,0,87,0,2,0],[88,0,88,0,10,0],[88,0,88,0,1,1],[88,0,88,0,1,1],[88,0,88,0,1,1],[88,0,88,0,1,1]]}]} diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/fixture-java-dedup-table.ndjson b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/fixture-java-dedup-table.ndjson new file mode 100644 index 00000000..3f68ee15 --- /dev/null +++ b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/fixture-java-dedup-table.ndjson @@ -0,0 +1,4 @@ +{"kind":"class","id":"c:io/keploy/samples/javadedup/DedupController","pkg":"io/keploy/samples/javadedup","name":"DedupController","path":"io/keploy/samples/javadedup/DedupController.java","hash":"7c03b81f9f20db7531ed647244af5fa640a6faa6c1b804e039b0d67782aa7371","state":"ok","methods":[{"id":"m:io/keploy/samples/javadedup/DedupController#()V","name":"DedupController()","line":33,"leaves":[[33,0,33,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#root()Ljava/util/Map;","name":"root()","line":37,"leaves":[[37,0,37,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#hello(Ljava/lang/String;)Ljava/lang/String;","name":"hello(String)","line":42,"leaves":[[42,0,42,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#random()Ljava/util/Map;","name":"random()","line":47,"leaves":[[47,0,47,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#welcome(Ljava/lang/String;)Ljava/util/Map;","name":"welcome(String)","line":52,"leaves":[[52,0,52,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#createUser(Lio/keploy/samples/javadedup/model/User;)Ljava/util/Map;","name":"createUser(User)","line":57,"leaves":[[57,0,57,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#items()Ljava/util/List;","name":"items()","line":62,"leaves":[[62,0,62,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#updateItem(Ljava/lang/String;Lio/keploy/samples/javadedup/model/Item;)Ljava/util/Map;","name":"updateItem(String, Item)","line":67,"leaves":[[67,0,67,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#deleteItem(Ljava/lang/String;)Ljava/util/Map;","name":"deleteItem(String)","line":72,"leaves":[[72,0,72,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#simpleMessage(Ljakarta/servlet/http/HttpServletRequest;)Ljava/util/Map;","name":"simpleMessage(HttpServletRequest)","line":81,"leaves":[[81,0,81,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#ping()Ljava/lang/String;","name":"ping()","line":86,"leaves":[[86,0,86,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#status()Ljava/util/Map;","name":"status()","line":91,"leaves":[[91,0,91,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#greeting(Ljava/lang/String;)Ljava/lang/String;","name":"greeting(String)","line":285,"leaves":[[285,0,285,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#itemList()Ljava/util/List;","name":"itemList()","line":289,"leaves":[[289,0,289,0,3,0],[290,0,290,0,3,0],[291,0,291,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#messageForPath(Ljava/lang/String;)Ljava/lang/String;","name":"messageForPath(String)","line":306,"leaves":[[306,0,306,0,3,0],[307,0,307,0,3,0],[307,0,307,0,1,1],[307,0,307,0,1,1],[308,0,308,0,3,0],[310,0,310,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#response([Ljava/lang/Object;)Ljava/util/Map;","name":"response(Object...)","line":314,"leaves":[[314,0,314,0,3,0],[315,0,315,0,3,0],[315,0,315,0,1,1],[315,0,315,0,1,1],[316,0,316,0,3,0],[318,0,318,0,3,0]]}]} +{"kind":"class","id":"c:io/keploy/samples/javadedup/JavaDedupApplication","pkg":"io/keploy/samples/javadedup","name":"JavaDedupApplication","path":"io/keploy/samples/javadedup/JavaDedupApplication.java","hash":"85cfab44164d4f24ca778bfac41594bb52fd1bf2d42c75532c048d25ebb61372","state":"ok","methods":[{"id":"m:io/keploy/samples/javadedup/JavaDedupApplication#()V","name":"JavaDedupApplication()","line":8,"leaves":[[8,0,8,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/JavaDedupApplication#main([Ljava/lang/String;)V","name":"main(String[])","line":10,"leaves":[[10,0,10,0,3,0]]}]} +{"kind":"class","id":"c:io/keploy/samples/javadedup/model/Item","pkg":"io/keploy/samples/javadedup/model","name":"Item","path":"io/keploy/samples/javadedup/model/Item.java","hash":"2e4af0ca3be34b61f048b8ad0440406cb107142896d18cb80d9557f496173ba4","state":"ok","methods":[{"id":"m:io/keploy/samples/javadedup/model/Item#()V","name":"Item()","line":5,"leaves":[[5,0,5,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/model/Item#getId()Ljava/lang/String;","name":"getId()","line":10,"leaves":[[10,0,10,0,3,0]]}]} +{"kind":"class","id":"c:io/keploy/samples/javadedup/model/User","pkg":"io/keploy/samples/javadedup/model","name":"User","path":"io/keploy/samples/javadedup/model/User.java","hash":"2fd7360ebf831da22c293cfaf9f29408521d03343cf5fb6ef06608d39e521188","state":"ok","methods":[{"id":"m:io/keploy/samples/javadedup/model/User#()V","name":"User()","line":5,"leaves":[[5,0,5,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/model/User#getName()Ljava/lang/String;","name":"getName()","line":9,"leaves":[[9,0,9,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/model/User#setName(Ljava/lang/String;)V","name":"setName(String)","line":13,"leaves":[[13,0,13,0,3,0]]}]} diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup.build b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup.build new file mode 100644 index 00000000..1179d0ee --- /dev/null +++ b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup.build @@ -0,0 +1 @@ +jv1-d2b0c051a964d63b882ad20b37ebda64 diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup.probes.ndjson b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup.probes.ndjson new file mode 100644 index 00000000..c06bf3b5 --- /dev/null +++ b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup.probes.ndjson @@ -0,0 +1,5 @@ +{"class":"io/keploy/samples/javadedup/DedupController","id":"45a3f53a4c63794a","unit":"c:io/keploy/samples/javadedup/DedupController","probes":[[0,1],[1,2],[2,3],[3,4],[4,5],[5,6],[6,7],[7,8],[8,9],[9,10],[10,11],[11,12],[12,13],[13,14],[14,15],[15,16],[16,17],[17,18],[18,19],[19,20],[20,21],[21,22],[22,23],[23,24],[24,25],[25,26],[26,27],[27,28],[28,29],[29,30],[30,31],[31,32],[32,33],[33,34],[34,35],[35,36],[36,37],[37,38],[38,39],[39,40],[40,41],[41,42],[42,43],[43,44],[44,45],[45,46],[46,47],[47,48],[48,49],[49,50],[50,51],[51,52],[52,53],[53,54],[54,55],[55,56],[56,57],[57,58],[58,59],[59,60],[60,61],[61,62],[59,60],[62,63],[63,65,66,67],[63,64,65,66,67,68],[68,70],[69,71],[69,70,72,73],[69,70,71,72,73,74]]} +{"class":"io/keploy/samples/javadedup/JavaDedupApplication","id":"2fc08e98e5791eee","unit":"c:io/keploy/samples/javadedup/JavaDedupApplication","probes":[[0,1],[1,3]]} +{"class":"io/keploy/samples/javadedup/model/Item","id":"4b73722ef52560fa","unit":"c:io/keploy/samples/javadedup/model/Item","probes":[[0,2],[2,7],[7,8],[8,10],[10,11],[11,13],[13,14],[14,16]]} +{"class":"io/keploy/samples/javadedup/model/Product","id":"99d2d88ee67c142a","unit":"c:io/keploy/samples/javadedup/model/Product","probes":[[0,2],[2,8],[8,9],[9,11],[11,12],[12,14],[14,15],[15,17],[17,18],[18,20]]} +{"class":"io/keploy/samples/javadedup/model/User","id":"91a01d80e208b6dc","unit":"c:io/keploy/samples/javadedup/model/User","probes":[[0,2],[2,6],[6,7],[7,9],[9,10],[10,12]]} diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup.table.ndjson b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup.table.ndjson new file mode 100644 index 00000000..9991c36c --- /dev/null +++ b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup.table.ndjson @@ -0,0 +1,5 @@ +{"kind":"class","id":"c:io/keploy/samples/javadedup/DedupController","pkg":"io/keploy/samples/javadedup","name":"DedupController","path":"io/keploy/samples/javadedup/DedupController.java","hash":"33497ea96394e86dd8e0ed484fcf8441bd4205353303837f097938abd377c03a","state":"ok","methods":[{"id":"m:io/keploy/samples/javadedup/DedupController#()V","name":"DedupController()","line":33,"leaves":[[33,0,33,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#root()Ljava/util/Map;","name":"root()","line":37,"leaves":[[37,0,37,0,13,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#hello(Ljava/lang/String;)Ljava/lang/String;","name":"hello(String)","line":42,"leaves":[[42,0,42,0,4,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#random()Ljava/util/Map;","name":"random()","line":47,"leaves":[[47,0,47,0,19,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#welcome(Ljava/lang/String;)Ljava/util/Map;","name":"welcome(String)","line":52,"leaves":[[52,0,52,0,20,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#createUser(Lio/keploy/samples/javadedup/model/User;)Ljava/util/Map;","name":"createUser(User)","line":57,"leaves":[[57,0,57,0,21,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#items()Ljava/util/List;","name":"items()","line":62,"leaves":[[62,0,62,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#updateItem(Ljava/lang/String;Lio/keploy/samples/javadedup/model/Item;)Ljava/util/Map;","name":"updateItem(String, Item)","line":67,"leaves":[[67,0,67,0,29,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#deleteItem(Ljava/lang/String;)Ljava/util/Map;","name":"deleteItem(String)","line":72,"leaves":[[72,0,72,0,21,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#simpleMessage(Ljavax/servlet/http/HttpServletRequest;)Ljava/util/Map;","name":"simpleMessage(HttpServletRequest)","line":81,"leaves":[[81,0,81,0,16,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#ping()Ljava/lang/String;","name":"ping()","line":86,"leaves":[[86,0,86,0,2,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#status()Ljava/util/Map;","name":"status()","line":91,"leaves":[[91,0,91,0,21,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#namedStatus(Ljava/lang/String;)Ljava/util/Map;","name":"namedStatus(String)","line":96,"leaves":[[96,0,96,0,23,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#healthz()Ljava/util/Map;","name":"healthz()","line":101,"leaves":[[101,0,101,0,14,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#info()Ljava/util/Map;","name":"info()","line":106,"leaves":[[106,0,106,0,21,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#timestamp()Ljava/util/Map;","name":"timestamp()","line":111,"leaves":[[111,0,111,0,14,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#products()Ljava/util/List;","name":"products()","line":116,"leaves":[[116,0,116,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#createProduct(Lio/keploy/samples/javadedup/model/Product;)Lorg/springframework/http/ResponseEntity;","name":"createProduct(Product)","line":121,"leaves":[[121,0,121,0,25,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#product(Ljava/lang/String;)Ljava/util/Map;","name":"product(String)","line":126,"leaves":[[126,0,126,0,32,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#updateProduct(Ljava/lang/String;Lio/keploy/samples/javadedup/model/Product;)Ljava/util/Map;","name":"updateProduct(String, Product)","line":131,"leaves":[[131,0,131,0,30,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#deleteProduct(Ljava/lang/String;)Ljava/util/Map;","name":"deleteProduct(String)","line":136,"leaves":[[136,0,136,0,22,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#patchProduct(Ljava/lang/String;Ljava/util/Map;)Ljava/util/Map;","name":"patchProduct(String, Map)","line":141,"leaves":[[141,0,141,0,30,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#post(Ljava/lang/String;Ljava/lang/String;)Ljava/util/Map;","name":"post(String, String)","line":146,"leaves":[[146,0,146,0,29,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#search(Ljava/lang/String;Ljava/lang/String;)Ljava/util/Map;","name":"search(String, String)","line":152,"leaves":[[152,0,152,0,21,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#file(Ljavax/servlet/http/HttpServletRequest;)Ljava/util/Map;","name":"file(HttpServletRequest)","line":157,"leaves":[[157,0,157,0,10,0],[158,0,158,0,6,0],[159,0,159,0,13,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#html()Ljava/lang/String;","name":"html()","line":164,"leaves":[[164,0,164,0,2,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#xml()Ljava/lang/String;","name":"xml()","line":169,"leaves":[[169,0,169,0,2,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#redirect()Lorg/springframework/http/ResponseEntity;","name":"redirect()","line":174,"leaves":[[174,0,174,0,4,0],[175,0,175,0,4,0],[176,0,176,0,6,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#config(Ljava/util/Map;)Ljava/util/Map;","name":"config(Map)","line":181,"leaves":[[181,0,181,0,21,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#resourceOptions()Lorg/springframework/http/ResponseEntity;","name":"resourceOptions()","line":186,"leaves":[[186,0,186,0,4,0],[187,0,187,0,4,0],[188,0,188,0,6,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#apiV1Data()Ljava/util/Map;","name":"apiV1Data()","line":193,"leaves":[[193,0,193,0,22,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#apiV1Users()Ljava/util/Map;","name":"apiV1Users()","line":198,"leaves":[[198,0,198,0,32,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#createApiV1User()Lorg/springframework/http/ResponseEntity;","name":"createApiV1User()","line":203,"leaves":[[203,0,203,0,26,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#apiV2Data()Ljava/util/Map;","name":"apiV2Data()","line":208,"leaves":[[208,0,208,0,22,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#apiV2Users()Ljava/util/Map;","name":"apiV2Users()","line":213,"leaves":[[213,0,213,0,13,0],[214,0,214,0,13,0],[215,0,215,0,32,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#createApiV2User()Lorg/springframework/http/ResponseEntity;","name":"createApiV2User()","line":220,"leaves":[[220,0,220,0,26,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#logs()Ljava/util/Map;","name":"logs()","line":225,"leaves":[[225,0,225,0,22,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#metrics()Ljava/util/Map;","name":"metrics()","line":230,"leaves":[[230,0,230,0,21,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#reboot()Lorg/springframework/http/ResponseEntity;","name":"reboot()","line":235,"leaves":[[235,0,235,0,17,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#proxy()Ljava/util/Map;","name":"proxy()","line":240,"leaves":[[240,0,240,0,13,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#legacy()Lorg/springframework/http/ResponseEntity;","name":"legacy()","line":245,"leaves":[[245,0,245,0,17,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#secureData()Lorg/springframework/http/ResponseEntity;","name":"secureData()","line":250,"leaves":[[250,0,250,0,17,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#adminPanel()Lorg/springframework/http/ResponseEntity;","name":"adminPanel()","line":255,"leaves":[[255,0,255,0,17,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#longPoll()Ljava/util/Map;","name":"longPoll()","line":260,"leaves":[[260,0,260,0,2,0],[261,0,261,0,13,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#password(Ljava/lang/String;)Ljava/util/Map;","name":"password(String)","line":266,"leaves":[[266,0,266,0,22,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#profile(Ljava/lang/String;)Ljava/util/Map;","name":"profile(String)","line":271,"leaves":[[271,0,271,0,21,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#events()Lorg/springframework/http/ResponseEntity;","name":"events()","line":276,"leaves":[[276,0,276,0,17,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#sessionInfo()Ljava/util/Map;","name":"sessionInfo()","line":281,"leaves":[[281,0,281,0,22,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#greeting(Ljava/lang/String;)Ljava/lang/String;","name":"greeting(String)","line":285,"leaves":[[285,0,285,0,11,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#itemList()Ljava/util/List;","name":"itemList()","line":289,"leaves":[[289,0,289,0,40,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#productList()Ljava/util/List;","name":"productList()","line":297,"leaves":[[297,0,297,0,21,0],[299,0,299,0,20,0],[301,0,301,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#messageForPath(Ljava/lang/String;)Ljava/lang/String;","name":"messageForPath(String)","line":306,"leaves":[[306,0,306,0,8,0],[307,0,307,0,4,0],[307,0,307,0,1,1],[307,0,307,0,1,1],[308,0,308,0,2,0],[310,0,310,0,15,0]]},{"id":"m:io/keploy/samples/javadedup/DedupController#response([Ljava/lang/Object;)Ljava/util/Map;","name":"response(Object...)","line":314,"leaves":[[314,0,314,0,4,0],[315,0,315,0,8,0],[315,0,315,0,1,1],[315,0,315,0,1,1],[316,0,316,0,12,0],[318,0,318,0,2,0]]}]} +{"kind":"class","id":"c:io/keploy/samples/javadedup/JavaDedupApplication","pkg":"io/keploy/samples/javadedup","name":"JavaDedupApplication","path":"io/keploy/samples/javadedup/JavaDedupApplication.java","hash":"59ddb82bf4da76309565d9805e05fd73a16d60f2672511efa4d793141be708ad","state":"ok","methods":[{"id":"m:io/keploy/samples/javadedup/JavaDedupApplication#()V","name":"JavaDedupApplication()","line":7,"leaves":[[7,0,7,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/JavaDedupApplication#main([Ljava/lang/String;)V","name":"main(String[])","line":10,"leaves":[[10,0,10,0,4,0],[11,0,11,0,1,0]]}]} +{"kind":"class","id":"c:io/keploy/samples/javadedup/model/Item","pkg":"io/keploy/samples/javadedup/model","name":"Item","path":"io/keploy/samples/javadedup/model/Item.java","hash":"29bf729dbb3da977244066732cac7307006f81cd719613004b6449768d357706","state":"ok","methods":[{"id":"m:io/keploy/samples/javadedup/model/Item#()V","name":"Item()","line":11,"leaves":[[11,0,11,0,2,0],[12,0,12,0,1,0]]},{"id":"m:io/keploy/samples/javadedup/model/Item#(Ljava/lang/String;Ljava/lang/String;Ljava/math/BigDecimal;)V","name":"Item(String, String, BigDecimal)","line":14,"leaves":[[14,0,14,0,2,0],[15,0,15,0,3,0],[16,0,16,0,3,0],[17,0,17,0,3,0],[18,0,18,0,1,0]]},{"id":"m:io/keploy/samples/javadedup/model/Item#getId()Ljava/lang/String;","name":"getId()","line":21,"leaves":[[21,0,21,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/model/Item#setId(Ljava/lang/String;)V","name":"setId(String)","line":25,"leaves":[[25,0,25,0,3,0],[26,0,26,0,1,0]]},{"id":"m:io/keploy/samples/javadedup/model/Item#getName()Ljava/lang/String;","name":"getName()","line":29,"leaves":[[29,0,29,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/model/Item#setName(Ljava/lang/String;)V","name":"setName(String)","line":33,"leaves":[[33,0,33,0,3,0],[34,0,34,0,1,0]]},{"id":"m:io/keploy/samples/javadedup/model/Item#getPrice()Ljava/math/BigDecimal;","name":"getPrice()","line":37,"leaves":[[37,0,37,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/model/Item#setPrice(Ljava/math/BigDecimal;)V","name":"setPrice(BigDecimal)","line":41,"leaves":[[41,0,41,0,3,0],[42,0,42,0,1,0]]}]} +{"kind":"class","id":"c:io/keploy/samples/javadedup/model/Product","pkg":"io/keploy/samples/javadedup/model","name":"Product","path":"io/keploy/samples/javadedup/model/Product.java","hash":"6195ac4af3e554fa4ed4374e1e8124d6b194beecd32412c5424988b4c56f0310","state":"ok","methods":[{"id":"m:io/keploy/samples/javadedup/model/Product#()V","name":"Product()","line":15,"leaves":[[15,0,15,0,2,0],[16,0,16,0,1,0]]},{"id":"m:io/keploy/samples/javadedup/model/Product#(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/util/List;)V","name":"Product(String, String, String, List)","line":18,"leaves":[[18,0,18,0,2,0],[19,0,19,0,3,0],[20,0,20,0,3,0],[21,0,21,0,3,0],[22,0,22,0,3,0],[23,0,23,0,1,0]]},{"id":"m:io/keploy/samples/javadedup/model/Product#getProductId()Ljava/lang/String;","name":"getProductId()","line":26,"leaves":[[26,0,26,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/model/Product#setProductId(Ljava/lang/String;)V","name":"setProductId(String)","line":30,"leaves":[[30,0,30,0,3,0],[31,0,31,0,1,0]]},{"id":"m:io/keploy/samples/javadedup/model/Product#getName()Ljava/lang/String;","name":"getName()","line":34,"leaves":[[34,0,34,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/model/Product#setName(Ljava/lang/String;)V","name":"setName(String)","line":38,"leaves":[[38,0,38,0,3,0],[39,0,39,0,1,0]]},{"id":"m:io/keploy/samples/javadedup/model/Product#getDescription()Ljava/lang/String;","name":"getDescription()","line":42,"leaves":[[42,0,42,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/model/Product#setDescription(Ljava/lang/String;)V","name":"setDescription(String)","line":46,"leaves":[[46,0,46,0,3,0],[47,0,47,0,1,0]]},{"id":"m:io/keploy/samples/javadedup/model/Product#getTags()Ljava/util/List;","name":"getTags()","line":50,"leaves":[[50,0,50,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/model/Product#setTags(Ljava/util/List;)V","name":"setTags(List)","line":54,"leaves":[[54,0,54,0,3,0],[55,0,55,0,1,0]]}]} +{"kind":"class","id":"c:io/keploy/samples/javadedup/model/User","pkg":"io/keploy/samples/javadedup/model","name":"User","path":"io/keploy/samples/javadedup/model/User.java","hash":"b591fb83b5aa4cfc6e71b286daeecc93fc81ab930644a2087d0bee3a3a03d2df","state":"ok","methods":[{"id":"m:io/keploy/samples/javadedup/model/User#()V","name":"User()","line":8,"leaves":[[8,0,8,0,2,0],[9,0,9,0,1,0]]},{"id":"m:io/keploy/samples/javadedup/model/User#(Ljava/lang/String;Ljava/lang/String;)V","name":"User(String, String)","line":11,"leaves":[[11,0,11,0,2,0],[12,0,12,0,3,0],[13,0,13,0,3,0],[14,0,14,0,1,0]]},{"id":"m:io/keploy/samples/javadedup/model/User#getName()Ljava/lang/String;","name":"getName()","line":17,"leaves":[[17,0,17,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/model/User#setName(Ljava/lang/String;)V","name":"setName(String)","line":21,"leaves":[[21,0,21,0,3,0],[22,0,22,0,1,0]]},{"id":"m:io/keploy/samples/javadedup/model/User#getEmail()Ljava/lang/String;","name":"getEmail()","line":25,"leaves":[[25,0,25,0,3,0]]},{"id":"m:io/keploy/samples/javadedup/model/User#setEmail(Ljava/lang/String;)V","name":"setEmail(String)","line":29,"leaves":[[29,0,29,0,3,0],[30,0,30,0,1,0]]}]} diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup/io/keploy/samples/javadedup/DedupController.class.bin b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup/io/keploy/samples/javadedup/DedupController.class.bin new file mode 100644 index 00000000..80e47e73 Binary files /dev/null and b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup/io/keploy/samples/javadedup/DedupController.class.bin differ diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup/io/keploy/samples/javadedup/JavaDedupApplication.class.bin b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup/io/keploy/samples/javadedup/JavaDedupApplication.class.bin new file mode 100644 index 00000000..337c0c2e Binary files /dev/null and b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup/io/keploy/samples/javadedup/JavaDedupApplication.class.bin differ diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup/io/keploy/samples/javadedup/model/Item.class.bin b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup/io/keploy/samples/javadedup/model/Item.class.bin new file mode 100644 index 00000000..0a64ec40 Binary files /dev/null and b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup/io/keploy/samples/javadedup/model/Item.class.bin differ diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup/io/keploy/samples/javadedup/model/Product.class.bin b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup/io/keploy/samples/javadedup/model/Product.class.bin new file mode 100644 index 00000000..d94a11c0 Binary files /dev/null and b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup/io/keploy/samples/javadedup/model/Product.class.bin differ diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup/io/keploy/samples/javadedup/model/User.class.bin b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup/io/keploy/samples/javadedup/model/User.class.bin new file mode 100644 index 00000000..4c294bf5 Binary files /dev/null and b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/java-dedup/io/keploy/samples/javadedup/model/User.class.bin differ diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/Modern.java.txt b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/Modern.java.txt new file mode 100644 index 00000000..c2a60b66 --- /dev/null +++ b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/Modern.java.txt @@ -0,0 +1,101 @@ +package io.keploy.modern; + +import java.util.List; +import java.util.function.IntUnaryOperator; + +/** Java 17 and 21 code shapes for the probe map's rows (javac --release 21 -g). */ +public class Modern { + + sealed interface Shape permits Circle, Square, Rect { + } + + record Circle(int r) implements Shape { + } + + record Square(int side) implements Shape { + } + + record Rect(int w, int h) implements Shape { + Rect { + if (w < 0 || h < 0) { + throw new IllegalArgumentException("negative"); + } + } + } + + enum Color { RED, GREEN, BLUE } + + record Pair(Object left, Object right) { + } + + static int area(Shape s) { + return switch (s) { + case Circle c when c.r() > 10 -> 3 * c.r() * c.r() + 1; + case Circle c -> 3 * c.r() * c.r(); + case Square q -> q.side() * q.side(); + case Rect(int w, int h) when w == h -> w * w; + case Rect(int w, int h) -> w * h; + }; + } + + static int weight(Color c) { + return switch (c) { + case RED -> 1; + case GREEN -> 2; + case BLUE -> 3; + }; + } + + static int code(String s) { + return switch (s) { + case "a", "b" -> 1; + case "c" -> { + int n = s.length(); + yield n + 2; + } + default -> { + if (s.isEmpty()) { + yield 0; + } + yield -1; + } + }; + } + + static String describe(Object o) { + if (o instanceof Integer i && i > 0) { + return "positive " + i; + } + if (!(o instanceof String s)) { + return "other"; + } + return s.isBlank() ? "blank" : s; + } + + static int nested(Pair p) { + if (p instanceof Pair(Integer a, Integer b)) { + return a + b; + } + return switch (p.left()) { + case null -> -1; + case String s when s.length() > 3 -> s.length(); + case String s -> 0; + default -> 1; + }; + } + + static int guarded(List xs) { + int total = 0; + try { + for (String x : xs) { + total += code(x); + } + } catch (IllegalStateException e) { + total = -2; + } finally { + total++; + } + IntUnaryOperator twice = v -> v > 100 ? v : v * 2; + return twice.applyAsInt(total); + } +} diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern$Circle.class.bin b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern$Circle.class.bin new file mode 100644 index 00000000..95ba1f4e Binary files /dev/null and b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern$Circle.class.bin differ diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern$Color.class.bin b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern$Color.class.bin new file mode 100644 index 00000000..a7075f62 Binary files /dev/null and b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern$Color.class.bin differ diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern$Pair.class.bin b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern$Pair.class.bin new file mode 100644 index 00000000..64f098b8 Binary files /dev/null and b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern$Pair.class.bin differ diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern$Rect.class.bin b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern$Rect.class.bin new file mode 100644 index 00000000..443abf42 Binary files /dev/null and b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern$Rect.class.bin differ diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern$Shape.class.bin b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern$Shape.class.bin new file mode 100644 index 00000000..c1e5243f Binary files /dev/null and b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern$Shape.class.bin differ diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern$Square.class.bin b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern$Square.class.bin new file mode 100644 index 00000000..794bffb3 Binary files /dev/null and b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern$Square.class.bin differ diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern.class.bin b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern.class.bin new file mode 100644 index 00000000..4f573e94 Binary files /dev/null and b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/modern/io/keploy/modern/Modern.class.bin differ diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/shapes/Shapes.java.txt b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/shapes/Shapes.java.txt new file mode 100644 index 00000000..85057650 --- /dev/null +++ b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/shapes/Shapes.java.txt @@ -0,0 +1,113 @@ +package io.keploy.shapes; + +import java.io.Closeable; +import java.io.IOException; +import java.util.function.IntUnaryOperator; + +/** Code shapes for the probe map's rows: each one is a JaCoCo probe, branch or filter case. */ +public class Shapes { + enum Color { RED, GREEN, BLUE } + + static final class Res implements Closeable { + @Override + public void close() throws IOException { + } + } + + private final Object lock = new Object(); + private int total; + + int both(int a, int b) { + if (a > 0 && b > 0) { return 1; } + return a > b ? 2 : 3; + } + + int loop(int n) { + int s = 0; + for (int i = 0; i < n; i++) { + if (i % 3 == 0) { + continue; + } + s += i; + } + while (s > 100) { + s -= 7; + if (s == 50) { + break; + } + } + return s; + } + + String table(int k) { + switch (k) { + case 1: return "one"; + case 2: + case 3: return "few"; + case 7: total++; + case 8: return "many"; + default: return "other"; + } + } + + int strings(String s) { + switch (s) { + case "Aa": return 1; + case "BB": return 2; + case "c": return 3; + default: return 0; + } + } + + int colors(Color c) { + switch (c) { + case RED: return 1; + case BLUE: return 3; + default: return 0; + } + } + + int guarded(int x) { + try { + if (x < 0) { + throw new IllegalArgumentException("negative"); + } + return x * 2; + } finally { + if (x == 3) { total++; } + } + } + + int resources(int x) throws IOException { + try (Res r = new Res()) { + return x + 1; + } + } + + int locked(int x) { + synchronized (lock) { + total += x; + return total; + } + } + + int lambda(int x) { + IntUnaryOperator f = v -> v > 10 ? v - 10 : v + 10; + return f.applyAsInt(x); + } + + int checked(int x) { + assert x != 13 : "thirteen"; + int y = x; y++; if (y > 2) y--; return y; + } + + int nested(boolean a, boolean b, boolean c) { + if (a) { + if (b || c) { + return 1; + } + return 2; + } + return c ? 3 : 4; + } +} diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/shapes/io/keploy/shapes/Shapes$Color.class.bin b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/shapes/io/keploy/shapes/Shapes$Color.class.bin new file mode 100644 index 00000000..c6a66211 Binary files /dev/null and b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/shapes/io/keploy/shapes/Shapes$Color.class.bin differ diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/shapes/io/keploy/shapes/Shapes$Res.class.bin b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/shapes/io/keploy/shapes/Shapes$Res.class.bin new file mode 100644 index 00000000..320b6e1b Binary files /dev/null and b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/shapes/io/keploy/shapes/Shapes$Res.class.bin differ diff --git a/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/shapes/io/keploy/shapes/Shapes.class.bin b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/shapes/io/keploy/shapes/Shapes.class.bin new file mode 100644 index 00000000..5ee86b30 Binary files /dev/null and b/keploy-sdk/src/test/resources/io/keploy/dedup/buildtable/shapes/io/keploy/shapes/Shapes.class.bin differ diff --git a/scripts/smoke-javaagent.sh b/scripts/smoke-javaagent.sh index 08d499f0..974df3a0 100755 --- a/scripts/smoke-javaagent.sh +++ b/scripts/smoke-javaagent.sh @@ -259,6 +259,26 @@ public final class SmokeHarness { throw new IllegalStateException("the INDEX report did not name keploy-sdk " + wantSdkVersion + " and JaCoCo " + wantJacocoVersion + " in " + mode + ": " + report); } + // The build id's V: the release of the JaCoCo core keploy-sdk.jar carries renamed, as the SDK reads it + // from that core's own version resource, must be the one the core's Maven build named. A jar whose + // renamed core cannot say gives V "", and a JaCoCo upgrade would then keep the build id. It is read as + // the SDK loaded here (from the class path, with this mode's agents and libraries) and from the jar alone + // in a loader of its own with no parent, so V is the jar's own core's with nothing else visible. + File sdkJar = new File(System.getProperty("smoke.sdkJar")); + String core = corePomVersion(sdkJar); + String analyzer = analyzerJacoco(ClassLoader.getSystemClassLoader()); + URLClassLoader alone = new URLClassLoader(new URL[] {sdkJar.toURI().toURL()}, null); + String analyzerAlone; + try { + analyzerAlone = analyzerJacoco(alone); + } finally { + alone.close(); + } + if (core.isEmpty() || !core.equals(analyzer) || !core.equals(analyzerAlone)) { + throw new IllegalStateException("keploy-sdk's analyzer names JaCoCo '" + analyzer + "' ('" + + analyzerAlone + "' in a loader of its own) and its org.jacoco.core is '" + core + "' in " + mode); + } + System.out.println(mode + ": analyzer JaCoCo " + analyzer); String ownLibraries = System.getProperty("smoke.ownLibraries", ""); if (!ownLibraries.isEmpty()) { loadsItsOwn(ownLibraries); @@ -385,6 +405,28 @@ public final class SmokeHarness { } } + // analyzerJacoco is IndexReport.Analyzer.JACOCO, V of the build id, of keploy-sdk as loader loads it. + private static String analyzerJacoco(ClassLoader loader) throws Exception { + java.lang.reflect.Field field = Class.forName("io.keploy.dedup.IndexReport$Analyzer", true, loader) + .getDeclaredField("JACOCO"); + field.setAccessible(true); + return (String) field.get(null); + } + + private static String corePomVersion(File jar) throws Exception { + try (JarFile file = new JarFile(jar)) { + java.util.zip.ZipEntry entry = file.getEntry("META-INF/maven/org.jacoco/org.jacoco.core/pom.properties"); + if (entry == null) { + return ""; + } + java.util.Properties p = new java.util.Properties(); + try (InputStream in = file.getInputStream(entry)) { + p.load(in); + } + return p.getProperty("version", ""); + } + } + private static void waitFor(File file) throws InterruptedException { long deadline = System.currentTimeMillis() + 10000L; while (System.currentTimeMillis() < deadline) {