From b42125da13641fdbe2299a18bda1e4e205b7286b Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Mon, 2 Nov 2026 04:00:41 +0900 Subject: [PATCH 1/2] 0.2.0 --- openkal 0.20: a measurement of an image (okprof), the name of an address, and an okb's admission remembered - The runner answers openkal.prof and openkal.debug, a loader's interfaces (SPEC 12.3), and states them in kal_interfaces beside the implementation's. OKRUN_PROF, or `okrun --prof=[:dir]' (okrun_profile), measures an image: `calls' binds each name through a counting entry (an unmeasured image is bound as before), `counters' keeps the counters and phases the program registers, `perfmap' writes /tmp/perf-.map on Linux; the report, okprof-.json, is written where the program ends (its entry returns, kal_exit, kal_abort). - kal_debug_symbolize names an address from the symbol table of the image that holds it, read from the image's file again when first asked (libokld: okld_symbols). - okprof (openkal-tools): `record' runs a program with OKRUN_PROF set, `report' groups the calls by interface and by name with the counters and phases, `diff' sets two reports side by side by how their time moved. - An okb's admission is remembered per user (okrun/admitted in the cache directory), by the manifest's SHA-256 and the member, with the file's identity, length and modification time: a later start of the same file admits a member by its shape alone. - okld_interface_of knows openkal.map (missing since 0.18), region, prof and debug; okld and okrun name them. - openkal 0.20.0; openkal-linux 0.21.0, openkal-macos 0.18.0, openkal-windows 0.16.0. Co-authored-by: speak-agent <248744407+speak-agent@users.noreply.github.com> --- .github/workflows/ci.yml | 22 ++ README.md | 33 ++- bind/mcpp.toml | 2 +- docs/bind.md | 5 +- examples/prof/app.c | 41 +++ ld/include/okld.h | 11 + ld/mcpp.toml | 2 +- ld/src/read.cpp | 54 ++++ run/build.mcpp | 18 +- run/include/okrun.h | 10 + run/mcpp.toml | 12 +- run/src/bindings.cpp | 63 ++++- run/src/bindings_space.cpp | 6 +- run/src/okrun.cppm | 1 + run/src/prof.cpp | 526 +++++++++++++++++++++++++++++++++++++ run/src/run.cpp | 207 ++++++++++++++- run/src/runner.h | 27 ++ tools/mcpp.toml | 7 +- tools/src/okld.cpp | 5 +- tools/src/okprof.cpp | 339 ++++++++++++++++++++++++ tools/src/okrun.cpp | 5 + 21 files changed, 1362 insertions(+), 34 deletions(-) create mode 100644 examples/prof/app.c create mode 100644 run/src/prof.cpp create mode 100644 tools/src/okprof.cpp diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 5ade148..74122b8 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -73,6 +73,7 @@ jobs: bash examples/build-image.sh "$arch" "out/$arch/libmathlib.okso" examples/bind/mathlib/image/mathlib.c # 8 MiB of the file's bytes, and a sleep: what stays resident once it is loaded. bash examples/build-image.sh "$arch" "out/$arch/resident.oke" examples/resident/app.c + bash examples/build-image.sh "$arch" "out/$arch/prof.oke" examples/prof/app.c done "$OKLD" keygen "$RUNNER_TEMP/seed" > out/key.hex for arch in x86_64 aarch64; do @@ -177,6 +178,9 @@ jobs: echo "OKRUN=$okrun" >> "$GITHUB_ENV" okld=$(find "$PWD/target" -path '*bin*' \( -name okld -o -name okld.exe \) -type f | head -1) echo "OKLD_HOST=$okld" >> "$GITHUB_ENV" + okprof=$(find "$PWD/target" -path '*bin*' \( -name okprof -o -name okprof.exe \) -type f | head -1) + [ -n "$okprof" ] || { echo "::error::okprof did not build"; exit 1; } + echo "OKPROF=$okprof" >> "$GITHUB_ENV" - name: The same bytes run: | a='${{ matrix.arch }}' @@ -193,6 +197,24 @@ jobs: w=$("$OKRUN" "out/$a/weak.oke" | tr -d '\r') echo "$w" grep -q "weak: openkal.net bound" <<<"$w" || { echo "::error::a weak import the implementation provides was not bound"; exit 1; } + # A MEASUREMENT AND THE NAME OF AN ADDRESS (0.2). Unmeasured, the + # image is told so and the loader names its function; measured, the + # report holds its calls, its counter and its phase. + p=$("$OKRUN" "out/$a/prof.oke" | tr -d '\r') + echo "$p" + grep -q "prof: not measured" <<<"$p" || { echo "::error::an unmeasured image was told it is measured"; exit 1; } + grep -q "debug: prof_named_function" <<<"$p" || { echo "::error::the loader did not name the image's function"; exit 1; } + rm -rf prof && mkdir prof + "$OKRUN" "--prof=calls,counters:$(pwd)/prof" "out/$a/prof.oke" > prof.out + grep -q "prof: measured" prof.out || { echo "::error::a measured image was told it is not"; exit 1; } + report=$(ls prof/okprof-*.json | head -1) + [ -n "$report" ] || { echo "::error::a measured image wrote no report"; exit 1; } + cat "$report" + grep -q '"example.widgets", "value": 42' "$report" || { echo "::error::the counter was not read"; exit 1; } + grep -q '"example.phase", "count": 2' "$report" || { echo "::error::the phase was not counted"; exit 1; } + grep -q '"kal_stream_write", "count": ' "$report" || { echo "::error::the calls were not counted"; exit 1; } + "$OKPROF" report prof | tee okprof.out + grep -q "openkal.stream" okprof.out || { echo "::error::okprof report did not group the calls by interface"; exit 1; } - name: The refusals run: | a='${{ matrix.arch }}' diff --git a/README.md b/README.md index f41f92f..1b670b3 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ architecture that has a runner. |---|---|---| | `openkal-ld` | `libokld` | The format's definition. A reader of bytes, a checker, a signer (Ed25519) and a loader. Freestanding: no C library, no exceptions. Its users are libokrun, `okld`, and a C library's `dlopen`. | | `openkal-run` | `libokrun` | The runner's core. It binds an image's names to this process's one openkal implementation, in the image's calling convention, then enters the image. | -| `openkal-tools` | `okrun`, `okld` | `okrun [--require-signature] [--trust ] [--debugger] image\|okb args...` runs an image or an okb. `okld check \| note \| sign \| keygen \| inspect \| note-source \| bundle \| bundle check` works on images and okbs on a build machine. | +| `openkal-tools` | `okrun`, `okld`, `okprof` | `okrun [--require-signature] [--trust ] [--debugger] [--prof=[:dir]] image\|okb args...` runs an image or an okb. `okld check \| note \| sign \| keygen \| inspect \| note-source \| bundle \| bundle check` works on images and okbs on a build machine. `okprof record \| report \| diff` measures what an image did across openkal. | | `openkal-bind` | `mcpp.mcpplibs.openkal_bind` | A build plugin. A library published as an image (`.okso`) and a C header is called from a program of Linux, macOS or Windows through a binding generated from the header ([docs/bind.md](docs/bind.md)). | `libokld` is the loader at run time and `okld` is the tool on the build machine. They share a @@ -44,6 +44,37 @@ The images an image needs are found in the okb, then beside it, in `lib//` in `OKRUN_PATH`. A program that carries its images gives their archive to `okrun_run_okb_bytes`, which runs it from memory and writes nothing to disk. +## A measurement of an image + +The runner binds every name an image calls across openkal and read the image's file, so it +measures what the image did there and names its addresses (openkal SPEC 12.3, 0.20): + +```sh +okprof record -o prof -- okrun app.okb arguments... # or a program that starts an okrun +okprof report prof # calls by interface and by name; counters; phases +okprof diff before after # the same names, ordered by how their time moved +``` + +`okprof record` sets `OKRUN_PROF` (`okrun --prof=` says the same), so every image an okrun +beneath the program runs writes `okprof-.json` when it ends: + +| what | measured | +|---|---| +| `calls` | each name the image imports, bound through an entry that counts its calls and their time; an image that is not measured is bound as before and pays nothing | +| `counters` | the counters and phases the program states through `openkal.prof` (`kal_prof_counter`, `kal_prof_mark`); `kal_prof_active` answers zero when nothing measures | +| `perfmap` | `/tmp/perf-.map` on Linux, where `perf` reads the names of an image's addresses; the runner does not sample, the system's profiler does | + +`openkal.debug` (`kal_debug_symbolize`) names an address from the symbol table of the image +that holds it, which the runner reads from the image's file again when it is first asked; the +table is never placed in memory by loading. + +**An okb's admission is remembered.** The first start of an okb compares the images it runs +with the digests its manifest states; the comparison is then recorded per user (`okrun/admitted` +in the user's cache directory), keyed by the manifest's SHA-256 and the member, with the file's +identity, length and modification time. A later start of the same file admits the member by its +shape alone; a file replaced or changed is compared again. A language server that starts a +worker per module no longer reads its 146 MB okb at each start. + ## A library from a program of any system `okrun_open_library`, `okrun_symbol` and `okrun_close` load an image into a program of this diff --git a/bind/mcpp.toml b/bind/mcpp.toml index 1ed97a2..88a6e3f 100644 --- a/bind/mcpp.toml +++ b/bind/mcpp.toml @@ -1,7 +1,7 @@ [package] namespace = "mcpplibs" name = "openkal-bind" -version = "0.1.9" +version = "0.2.0" # `mcpp::xpkg_request', `mcpp::decision' (protocol 15). mcpp = ">=2026.10.1.3" description = "A build plugin: a library published as an ok image (.okso) and a C header is called from a program of any system, through a binding generated from the header and libokrun." diff --git a/docs/bind.md b/docs/bind.md index 649c2ec..d1e6e61 100644 --- a/docs/bind.md +++ b/docs/bind.md @@ -24,10 +24,10 @@ sources = [] include_dirs = ["include"] [target.'cfg(not(os = "openkal"))'.dependencies] -openkal-run = "0.1.9" +openkal-run = "0.2.0" [build-dependencies] -openkal-bind = "0.1.9" +openkal-bind = "0.2.0" ``` ```cpp @@ -104,4 +104,5 @@ llvm toolchain's clang converts the call. | 0.1.6 | No change of its own; released with the archive, whose runner releases the file it ran once the image is loaded, and whose loader digests on the processor's SHA-256 instructions and writes each segment once. | | 0.1.7 | No change of its own; released with the archive, whose loader also digests on ARMv8's SHA-256 instructions on Linux where the processor states them. | | 0.1.8 | No change of its own; released with the archive, built against openkal 0.19.0 (the runner binds `kal_task_cores` and `kal_task_set_priority`), whose okb carries chunk digests the runner verifies on several contexts at once. | +| 0.2.0 | No change of its own; released with the archive, whose runner measures an image (okprof) and remembers an okb's admission. | | 0.1.9 | No change of its own; released with the archive, whose runner on macOS is built over openkal-macos 0.17.1 (`kal_task_current` without a system call). | diff --git a/examples/prof/app.c b/examples/prof/app.c new file mode 100644 index 0000000..5738f30 --- /dev/null +++ b/examples/prof/app.c @@ -0,0 +1,41 @@ +/* What a program states to a measurement, and the name of one of its own + * addresses (openkal.prof and openkal.debug, a loader's interfaces, SPEC 12.3; + * openkal-image 0.2). The image registers a counter, counts into it, marks a + * phase twice, and asks the loader for the name of a function of its own, which + * the loader reads from the image's symbol table --- a table loading never + * places in memory. The entry is `int ok_main(void)' (SPEC 12.2). */ +typedef unsigned long kal_uintptr; +typedef long kal_intptr; +typedef unsigned long long kal_u64; +struct kal_stream { kal_uintptr h; }; + +struct kal_stream kal_stdout(void); +kal_intptr kal_stream_write(struct kal_stream, const void*, kal_uintptr); +int kal_prof_active(void); +int kal_prof_counter(const char* name, kal_uintptr len, const volatile kal_u64* cell); +void kal_prof_mark(const char* name, kal_uintptr len, int begin); +kal_intptr kal_debug_symbolize(kal_uintptr address, char* out, kal_uintptr cap, kal_uintptr* start); + +static void put(const char* s, kal_uintptr n) { kal_stream_write(kal_stdout(), s, n); } +static void say(const char* s) { kal_uintptr n = 0; while (s[n]) ++n; put(s, n); } + +static volatile kal_u64 widgets; + +__attribute__((noinline, visibility("default"))) int prof_named_function(int x) { return x * 7 + 3; } + +__attribute__((visibility("default"))) int ok_main(void) { + say(kal_prof_active() ? "prof: measured\n" : "prof: not measured\n"); + kal_prof_counter("example.widgets", 15, &widgets); + for (int round = 0; round < 2; ++round) { + kal_prof_mark("example.phase", 13, 1); + for (int i = 0; i < 21; ++i) widgets = widgets + 1; + kal_prof_mark("example.phase", 13, 0); + } + char name[128]; + kal_uintptr start = 0; + int (*volatile fn)(int) = &prof_named_function; + const kal_intptr n = kal_debug_symbolize((kal_uintptr)fn + 1, name, sizeof name, &start); + if (n > 0 && n < (kal_intptr)sizeof name) { say("debug: "); put(name, (kal_uintptr)n); say("\n"); } + else say("debug: no name\n"); + return prof_named_function(0) == 3 ? 0 : 1; +} diff --git a/ld/include/okld.h b/ld/include/okld.h index ddece3b..192216f 100644 --- a/ld/include/okld.h +++ b/ld/include/okld.h @@ -63,6 +63,17 @@ void okld_imports(const void* bytes, okld_size size, void (*each)(const char* na void okld_required_imports(const void* bytes, okld_size size, void (*each)(const char* name, void* ctx), void* ctx); void okld_exports(const void* bytes, okld_size size, void (*each)(const char* name, void* ctx), void* ctx); +/* THE SYMBOLS OF AN IMAGE'S CODE AND DATA (0.2): every function and object + * symbol its symbol table states (the section table's `.symtab', else its + * dynamic symbols), as `value' (an address of the image, unrelocated: add the + * module's base), `size' and name, in the table's order. The table lies in the + * file and is never placed in memory by loading, which is why a program that + * names an address in itself asks the party holding the file (openkal.debug, + * SPEC 12.3). Returns how many were given. */ +okld_u64 okld_symbols(const void* bytes, okld_size size, + void (*each)(okld_u64 value, okld_u64 symbol_size, const char* name, void* ctx), + void* ctx); + /* The `KAL_IFACE_*' positions a set of imported names requires: each `kal_' * name belongs to the interface its prefix names. */ okld_u64 okld_interface_of(const char* kal_name); diff --git a/ld/mcpp.toml b/ld/mcpp.toml index e2e5c93..a5cf4fe 100644 --- a/ld/mcpp.toml +++ b/ld/mcpp.toml @@ -1,7 +1,7 @@ [package] namespace = "mcpplibs" name = "openkal-ld" -version = "0.1.9" +version = "0.2.0" description = "libokld: the ok format (openkal image format) --- its definition, a reader of bytes, a checker, a signer, and a loader. Freestanding: no C library, no exceptions." license = "Apache-2.0" authors = ["mcpplibs"] diff --git a/ld/src/read.cpp b/ld/src/read.cpp index 9b2a7bc..4fd11ab 100644 --- a/ld/src/read.cpp +++ b/ld/src/read.cpp @@ -87,6 +87,9 @@ extern "C" okld_u64 okld_interface_of(const char* n) { { "kal_env_", 3 }, { "kal_time_", 4 }, { "kal_random_", 5 }, { "kal_fs_", 6 }, { "kal_process_", 7 }, { "kal_task_", 8 }, { "kal_exec_", 9 }, { "kal_terminal_", 10 }, { "kal_net_", 11 }, { "kal_datagram_", 12 }, { "kal_space_", 13 }, { "kal_timeout_", 14 }, + // openkal 0.18 and 0.20. openkal.map was missing here until 0.2, so an + // image that imported it required nothing for it in its note. + { "kal_map_", 15 }, { "kal_region_", 16 }, { "kal_prof_", 17 }, { "kal_debug_", 18 }, }; for (const auto& e : table) if (starts(n, e.prefix)) return 1ull << e.bit; return 0; // kal_version, kal_interfaces: the specification's self-description @@ -254,3 +257,54 @@ extern "C" int okld_set_requirements(void* bytes, okld_size n, okld_u64 interfac copy(d + 8, &interfaces, 8); return 0; } + +// --- the symbols of an image's code and data (0.2) ----------------------------- +extern "C" okld_u64 okld_symbols(const void* bytes, okld_size size, + void (*each)(okld_u64, okld_u64, const char*, void*), void* ctx) { + using namespace okld; + view v; + char why[8]; + if (!bytes || !each || !open(bytes, size, v, why, sizeof why)) return 0; + // The section table's symbol table, with its string table; else the + // dynamic one, whose names are the exports alone. + const elf::sym* syms = nullptr; + elf::u64 count = 0; + const char* strings = nullptr; + elf::u64 strings_size = 0; + if (v.eh->shoff && v.eh->shentsize == sizeof(elf::shdr)) { + const auto* sh = v.at(v.eh->shoff, v.eh->shnum); + for (unsigned i = 0; sh && i < v.eh->shnum && !syms; ++i) { + if (sh[i].type != elf::sht_symtab || sh[i].link >= v.eh->shnum) continue; + const auto& str = sh[sh[i].link]; + const auto n = sh[i].size / sizeof(elf::sym); + const auto* t = v.at(sh[i].offset, n); + const auto* s = v.at(str.offset, str.size); + if (t && s) { syms = t; count = n; strings = s; strings_size = str.size; } + } + } + if (!syms) { + syms = v.symbols(); + count = syms ? v.symbol_count() : 0; + } + okld_u64 given = 0; + for (elf::u64 k = 0; k < count; ++k) { + const auto& s = syms[k]; + const unsigned type = s.info & 0xf; + if ((type != 1 /* STT_OBJECT */ && type != 2 /* STT_FUNC */) || s.value == 0 + || s.shndx == elf::shn_undef || s.shndx == elf::shn_abs) continue; + const char* name = nullptr; + if (strings) { + if (s.name >= strings_size) continue; + name = strings + s.name; + bool ended = false; + for (elf::u64 i = s.name; i < strings_size; ++i) if (strings[i] == 0) { ended = true; break; } + if (!ended) continue; + } else { + name = v.string(s.name); + } + if (!name || !*name) continue; + each(s.value, s.size, name, ctx); + ++given; + } + return given; +} diff --git a/run/build.mcpp b/run/build.mcpp index bf12fb4..b59cb44 100644 --- a/run/build.mcpp +++ b/run/build.mcpp @@ -26,8 +26,12 @@ int main() { } // The arguments and the image's own name are the runner's to answer (SPEC // 12.3); a context's entry is entered in the image's convention through a - // box; every other name is the implementation's. - const std::set own{ "kal_env_arg_count", "kal_env_arg", "kal_env_image", "kal_env_loader" }; + // box; every other name is the implementation's. Since 0.2 the runner + // also answers openkal.prof and openkal.debug, which are a loader's (SPEC + // 12.3), the word of interfaces (the implementation's and those two), and + // the program's end (where a measurement records what it measured). + std::set own{ "kal_env_arg_count", "kal_env_arg", "kal_env_image", "kal_env_loader", + "kal_interfaces", "kal_exit", "kal_abort" }; const std::set callbacks{ "kal_task_start", "kal_task_start_sized" }; std::vector main_names, space_names; std::string group, line; @@ -35,13 +39,19 @@ int main() { if (line.rfind("# openkal.", 0) == 0) { group = line.substr(2, line.find_first_of(" \t", 2) - 2); continue; } if (line.rfind("kal_", 0) != 0) continue; while (!line.empty() && (line.back() == '\r' || line.back() == ' ')) line.pop_back(); + if (group == "openkal.prof" || group == "openkal.debug") own.insert(line); (group == "openkal.space" ? space_names : main_names).push_back(line); } auto emit = [&](const char* file, const std::vector& names) { std::ofstream o(std::filesystem::path(out) / file); o << "// Generated by openkal-run's build.mcpp from " << surface.generic_string() << ".\n"; - for (const auto& n : names) - o << (own.contains(n) ? "OKRUN_OWN(" : callbacks.contains(n) ? "OKRUN_CALLBACK(" : "OKRUN_BIND(") << n << ")\n"; + // The second argument is the name's position, by which a measurement + // of the calls an image makes (okprof, 0.2) keeps its counts. + for (std::size_t i = 0; i < names.size(); ++i) { + const auto& n = names[i]; + o << (own.contains(n) ? "OKRUN_OWN(" : callbacks.contains(n) ? "OKRUN_CALLBACK(" : "OKRUN_BIND(") + << n << ", " << i << ")\n"; + } }; emit("surface.inc", main_names); emit("surface_space.inc", space_names); diff --git a/run/include/okrun.h b/run/include/okrun.h index 7e27ff8..b38c894 100644 --- a/run/include/okrun.h +++ b/run/include/okrun.h @@ -96,6 +96,16 @@ void* okrun_symbol(struct okrun_library* library, const char* name); * nothing else holds. */ void okrun_close(struct okrun_library* library); +/* --- a measurement (0.2) --------------------------------------------------- */ + +/* What the next image run is measured for, as OKRUN_PROF would say it: any of + * `calls' (each name an image imports is counted with its time), `counters' + * (the counters and phases the program registers through openkal.prof) and + * `perfmap' (/tmp/perf-.map, for this system's profiler), separated by + * commas, then `:' for the report (`okprof-.json'). Empty or + * null measures nothing. Without a call, OKRUN_PROF is read. */ +void okrun_profile(const char* spec); + /* The names this runner binds (the specification's surface), `count' of them. */ const char* const* okrun_names(unsigned long* count); diff --git a/run/mcpp.toml b/run/mcpp.toml index 0d1f6dc..411745e 100644 --- a/run/mcpp.toml +++ b/run/mcpp.toml @@ -1,7 +1,7 @@ [package] namespace = "mcpplibs" name = "openkal-run" -version = "0.1.9" +version = "0.2.0" description = "libokrun: the runner's core --- an ok image's names bound to this process's one openkal implementation, in the image's calling convention, and its entry entered." license = "Apache-2.0" authors = ["mcpplibs"] @@ -11,23 +11,23 @@ repo = "https://github.com/mcpplibs/openkal-image" kind = "lib" [dependencies] -openkal = "0.19.0" +openkal = "0.20.0" openkal-ld = { path = "../ld" } [target.linux.dependencies] -openkal-linux = "0.20.0" +openkal-linux = "0.21.0" [target.macos.dependencies] -openkal-macos = "0.17.1" +openkal-macos = "0.18.0" [target.windows.dependencies] -openkal-windows = "0.15.0" +openkal-windows = "0.16.0" # The interfaces each system's implementation provides decide one unit: where # openkal.space is provided its names are bound, and elsewhere an image that # requires it is refused at load, naming it. [build] -sources = ["src/okrun.cppm", "src/run.cpp", "src/bindings.cpp"] +sources = ["src/okrun.cppm", "src/run.cpp", "src/bindings.cpp", "src/prof.cpp"] include_dirs = ["include"] flags = [ { glob = "src/*.cpp", cxxflags = ["-fno-exceptions", "-fno-rtti"] }, diff --git a/run/src/bindings.cpp b/run/src/bindings.cpp index ffe00cd..e5b3b37 100644 --- a/run/src/bindings.cpp +++ b/run/src/bindings.cpp @@ -17,6 +17,27 @@ struct present { OKRUN_IMAGE_ABI static R call(A... a) { return F(a...); } }; +// THE SAME, COUNTED (okprof `calls', 0.2): each call's duration is added to +// the name's count by its position. Bound instead of `present' only when a +// measurement asked for calls, so an image that is not measured pays nothing. +template struct returns_nothing { static constexpr bool value = false; }; +template <> struct returns_nothing { static constexpr bool value = true; }; +template struct counted; +template +struct counted { + OKRUN_IMAGE_ABI static R call(A... a) { + const kal_u64 t0 = kal_time_monotonic(); + if constexpr (returns_nothing::value) { + F(a...); + record_call(I, kal_time_monotonic() - t0); + } else { + R r = F(a...); + record_call(I, kal_time_monotonic() - t0); + return r; + } + } +}; + // A function the image passes to be entered as a new context: it is the // image's, so it is entered in the image's convention through a box. using image_entry = void (OKRUN_IMAGE_ABI*)(void*); @@ -77,9 +98,36 @@ OKRUN_IMAGE_ABI kal_intptr own_kal_env_loader(char* out, kal_uintptr cap) { return kal_env_image(out, cap); } -#define OKRUN_BIND(n) { #n, reinterpret_cast(&present<&::n>::call) }, -#define OKRUN_OWN(n) { #n, reinterpret_cast(&own_##n) }, -#define OKRUN_CALLBACK(n) { #n, reinterpret_cast(&callback_##n) }, +// THE LOADER'S OWN INTERFACES (SPEC 12.3, 0.2): openkal.prof and openkal.debug +// are answered here, and the word of interfaces states them beside the +// implementation's. The program's end is where a measurement writes what it +// measured, so `kal_exit' and `kal_abort' pass through here first. +OKRUN_IMAGE_ABI kal_u64 own_kal_interfaces(void) { + return kal_interfaces() | KAL_IFACE_PROF | KAL_IFACE_DEBUG; +} +OKRUN_IMAGE_ABI void own_kal_exit(int status) { + profile_finish(); + kal_exit(status); +} +OKRUN_IMAGE_ABI void own_kal_abort(const char* msg, kal_uintptr len) { + profile_finish(); + kal_abort(msg, len); +} +OKRUN_IMAGE_ABI int own_kal_prof_active(void) { return profile_active() ? 1 : 0; } +OKRUN_IMAGE_ABI int own_kal_prof_counter(const char* name, kal_uintptr len, const volatile kal_u64* cell) { + return profile_counter(name, len, cell); +} +OKRUN_IMAGE_ABI void own_kal_prof_mark(const char* name, kal_uintptr len, int begin) { + profile_mark(name, len, begin); +} +OKRUN_IMAGE_ABI kal_intptr own_kal_debug_symbolize(kal_uintptr address, char* out, kal_uintptr cap, + kal_uintptr* start) { + return symbolize(address, out, cap, start); +} + +#define OKRUN_BIND(n, i) { #n, reinterpret_cast(&present<&::n>::call) }, +#define OKRUN_OWN(n, i) { #n, reinterpret_cast(&own_##n) }, +#define OKRUN_CALLBACK(n, i) { #n, reinterpret_cast(&callback_##n) }, // At namespace scope, so that it is made once at start and no guard is asked // of a C++ run-time this library may be linked without. @@ -87,11 +135,18 @@ const binding g_table[] = { #include "surface.inc" }; +#undef OKRUN_BIND +#define OKRUN_BIND(n, i) { #n, reinterpret_cast(&counted<&::n, i>::call) }, +const binding g_counted[] = { +#include "surface.inc" +}; +static_assert(sizeof g_table / sizeof g_table[0] <= kProfiledNames, "the call counts hold every name"); + } // namespace const binding* bindings(unsigned long& count) { count = sizeof g_table / sizeof g_table[0]; - return g_table; + return profile_calls() ? g_counted : g_table; } } // namespace okrun diff --git a/run/src/bindings_space.cpp b/run/src/bindings_space.cpp index 355acf9..65a7252 100644 --- a/run/src/bindings_space.cpp +++ b/run/src/bindings_space.cpp @@ -13,9 +13,9 @@ template struct present; template struct present { OKRUN_IMAGE_ABI static R call(A... a) { return F(a...); } }; -#define OKRUN_BIND(n) { #n, reinterpret_cast(&present<&::n>::call) }, -#define OKRUN_OWN(n) OKRUN_BIND(n) -#define OKRUN_CALLBACK(n) OKRUN_BIND(n) +#define OKRUN_BIND(n, i) { #n, reinterpret_cast(&present<&::n>::call) }, +#define OKRUN_OWN(n, i) OKRUN_BIND(n, i) +#define OKRUN_CALLBACK(n, i) OKRUN_BIND(n, i) const binding g_space[] = { #include "surface_space.inc" }; diff --git a/run/src/okrun.cppm b/run/src/okrun.cppm index 80b26a2..8efe8b0 100644 --- a/run/src/okrun.cppm +++ b/run/src/okrun.cppm @@ -17,3 +17,4 @@ export using ::okrun_open_library; export using ::okrun_open_library_bytes; export using ::okrun_symbol; export using ::okrun_close; +export using ::okrun_profile; diff --git a/run/src/prof.cpp b/run/src/prof.cpp new file mode 100644 index 0000000..ba437f6 --- /dev/null +++ b/run/src/prof.cpp @@ -0,0 +1,526 @@ +// A measurement of an image, and the names of its addresses (openkal-image 0.2). +// +// THE LOADER IS WHERE BOTH BELONG (SPEC 12.3). It binds every name an image +// calls across openkal, so it can count those calls without the image's help; +// it read the image's file, so it can name an address from the symbol table the +// image never placed in memory; and it is where the program ends, so it is +// where what was measured is written. openkal.prof lets the program add what +// only it knows (its own counters and phases) and openkal.debug lets it ask for +// a name; both are answered here. +// +// WHAT IS MEASURED IS WHAT WAS ASKED FOR, AND NOTHING WHEN NOTHING WAS. +// OKRUN_PROF (or `okrun --prof', which sets the same) names any of +// +// calls each name an image imports is bound through an entry that +// counts its calls and their time, by name +// counters the counters and phases the program registers are kept +// perfmap /tmp/perf-.map is written, where this system's profiler +// (perf) reads the names of an anonymous region's addresses +// +// separated by commas, and `:' after them receives the report, +// `okprof-.json' (the working directory otherwise). Unset, an image is bound +// as it always was and `kal_prof_active' answers zero. +#include +#include +#include "runner.h" + +extern "C" void okrun_profile(const char* spec); + +namespace okrun { +namespace { + +kal_uintptr length(const char* s) { kal_uintptr n = 0; while (s && s[n]) ++n; return n; } +void copy_text(char* d, kal_uintptr cap, const char* s, kal_uintptr n) { + if (cap == 0) return; + kal_uintptr i = 0; + for (; i < n && i + 1 < cap && s[i]; ++i) d[i] = s[i]; + d[i] = 0; +} +bool same_text(const char* a, kal_uintptr an, const char* b) { + kal_uintptr i = 0; + for (; i < an && b[i]; ++i) if (a[i] != b[i]) return false; + return i == an && b[i] == 0; +} +bool contains(const char* s, kal_uintptr n, const char* word) { + const kal_uintptr w = length(word); + for (kal_uintptr i = 0; i + w <= n; ++i) { + kal_uintptr k = 0; + while (k < w && s[i + k] == word[k]) ++k; + if (k == w && (i + w == n || s[i + w] == ',' || s[i + w] == ':') && (i == 0 || s[i - 1] == ',')) return true; + } + return false; +} + +// --- one lock, held briefly ------------------------------------------------------ +volatile int g_lock = 0; +void lock() { while (__atomic_exchange_n(&g_lock, 1, __ATOMIC_ACQUIRE)) kal_task_yield(); } +void unlock() { __atomic_store_n(&g_lock, 0, __ATOMIC_RELEASE); } + +// --- what was asked for ------------------------------------------------------------ +enum : int { asked_active = 1, asked_calls = 2, asked_counters = 4, asked_perfmap = 8 }; +int g_mode = -1; // -1: not read yet +char g_dir[512] = {}; +kal_u64 g_started = 0; + +void read_spec(const char* spec, kal_uintptr n) { + int mode = 0; + if (n > 0) { + mode = asked_active; + if (contains(spec, n, "calls")) mode |= asked_calls; + if (contains(spec, n, "counters")) mode |= asked_counters; + if (contains(spec, n, "perfmap")) mode |= asked_perfmap; + if (mode == asked_active) mode |= asked_calls | asked_counters; // a bare `1' + for (kal_uintptr i = 0; i < n; ++i) + if (spec[i] == ':') { copy_text(g_dir, sizeof g_dir, spec + i + 1, n - i - 1); break; } + } + g_started = kal_time_monotonic(); + __atomic_store_n(&g_mode, mode, __ATOMIC_RELEASE); +} + +int mode() { + int m = __atomic_load_n(&g_mode, __ATOMIC_ACQUIRE); + if (m >= 0) return m; + static const char name[] = "OKRUN_PROF"; + char spec[600]; + const kal_intptr n = kal_env_var(name, sizeof name - 1, spec, sizeof spec - 1); + read_spec(spec, n > 0 && n < static_cast(sizeof spec) ? static_cast(n) : 0); + return __atomic_load_n(&g_mode, __ATOMIC_ACQUIRE); +} + +// --- calls --------------------------------------------------------------------------- +kal_u64 g_calls[kProfiledNames]; +kal_u64 g_call_ns[kProfiledNames]; + +// --- counters and phases -------------------------------------------------------------- +struct counter { char name[96]; const volatile kal_u64* cell; }; +constexpr unsigned kCounters = 512; +counter g_counters[kCounters]; +unsigned g_counter_count = 0; + +struct phase { char name[96]; kal_u64 count, ns; }; +constexpr unsigned kPhases = 256; +phase g_phases[kPhases]; +unsigned g_phase_count = 0; + +// The phases open on each context: the context's word (kal_task_current), and +// a stack of (phase, start). A table rather than thread-local storage, because +// this library may be linked into a program that brings no C++ run-time to +// allocate it. +struct open_phase { unsigned phase; kal_u64 start; }; +struct context_phases { kal_uintptr context; unsigned depth; open_phase stack[16]; }; +constexpr unsigned kContexts = 512; +context_phases g_open[kContexts]; + +unsigned phase_of(const char* name, kal_uintptr len) { + for (unsigned i = 0; i < g_phase_count; ++i) if (same_text(name, len, g_phases[i].name)) return i; + if (g_phase_count == kPhases) return kPhases; + copy_text(g_phases[g_phase_count].name, sizeof g_phases[0].name, name, len); + return g_phase_count++; +} + +context_phases* context_of(kal_uintptr me) { + context_phases* free_slot = nullptr; + for (auto& c : g_open) { + if (c.context == me) return &c; + if (!free_slot && c.context == 0) free_slot = &c; + } + if (free_slot) { free_slot->context = me; free_slot->depth = 0; } + return free_slot; +} + +// --- images, and the names of their addresses ------------------------------------------ +struct symbol { kal_u64 value, size; kal_u32 name; }; +struct image_record { + char name[256]; + const void* bytes; // bytes that stay where they are, or null + kal_u64 size; + char source[512]; // a file to read them again from, or empty + kal_u64 offset; + const void* at_hand; // bytes valid until the runner releases them + bool mapped; // its lines are in the perf map + symbol* symbols; // made at the first question, sorted by value + kal_u64 symbol_count; + char* names; + kal_u64 names_size; + bool tried; +}; +constexpr unsigned kImages = 64; +image_record g_images[kImages]; +unsigned g_image_count = 0; + +struct gather { symbol* symbols; kal_u64 count, cap; char* names; kal_u64 used, names_cap; }; + +void count_symbol(okld_u64, okld_u64, const char* name, void* ctx) { + auto* g = static_cast(ctx); + ++g->count; + g->names_cap += length(name) + 1; +} +void keep_symbol(okld_u64 value, okld_u64 size, const char* name, void* ctx) { + auto* g = static_cast(ctx); + if (g->count == g->cap) return; + const kal_uintptr n = length(name); + if (g->used + n + 1 > g->names_cap) return; + g->symbols[g->count++] = symbol{ value, size, static_cast(g->used) }; + for (kal_uintptr i = 0; i <= n; ++i) g->names[g->used + i] = name[i]; + g->used += n + 1; +} + +void sift(symbol* a, kal_u64 start, kal_u64 end) { + kal_u64 root = start; + while (2 * root + 1 < end) { + kal_u64 child = 2 * root + 1; + if (child + 1 < end && a[child].value < a[child + 1].value) ++child; + if (a[root].value >= a[child].value) return; + const symbol t = a[root]; a[root] = a[child]; a[child] = t; + root = child; + } +} +void sort_symbols(symbol* a, kal_u64 n) { + if (n < 2) return; + for (kal_u64 i = n / 2; i-- > 0;) sift(a, i, n); + for (kal_u64 end = n - 1; end > 0; --end) { + const symbol t = a[0]; a[0] = a[end]; a[end] = t; + sift(a, 0, end); + } +} + +void index_symbols(image_record& r, const void* bytes, kal_u64 size) { + gather g{}; + okld_symbols(bytes, size, &count_symbol, &g); + if (g.count == 0) return; + g.symbols = static_cast(kal_alloc(g.count * sizeof(symbol), 16)); + g.names = static_cast(kal_alloc(g.names_cap, 16)); + if (!g.symbols || !g.names) { + if (g.symbols) kal_free(g.symbols, g.count * sizeof(symbol), 16); + if (g.names) kal_free(g.names, g.names_cap, 16); + return; + } + g.cap = g.count; + g.count = 0; + okld_symbols(bytes, size, &keep_symbol, &g); + sort_symbols(g.symbols, g.count); + r.symbols = g.symbols; + r.symbol_count = g.count; + r.names = g.names; + r.names_size = g.names_cap; +} + +// The bytes of an image again: those that stayed, or its file's region. +bool reread(image_record& r, void*& region, kal_uintptr& region_len, const void*& bytes) { + region = nullptr; + region_len = 0; + if (r.bytes) { bytes = r.bytes; return true; } + if (!r.source[0]) return false; + kal_dir dir{}; + const char* rest = nullptr; + if (!locate_path(r.source, dir, rest)) return false; + kal_file f{}; + if (kal_fs_open(dir, rest, length(rest), KAL_OPEN_READ, &f) != kal_ok) return false; + const kal_u64 aligned = r.offset & ~(KAL_MAP_ALIGNMENT - 1); + region_len = static_cast(r.offset - aligned + r.size); + const int e = kal_map_file(f, aligned, region_len, 0, ®ion); + kal_fs_close_file(f); + if (e != kal_ok) { region = nullptr; return false; } + bytes = static_cast(region) + (r.offset - aligned); + return true; +} + +image_record* record_of(const char* name) { + for (unsigned i = 0; i < g_image_count; ++i) + if (same_text(name, length(name), g_images[i].name)) return &g_images[i]; + return nullptr; +} + +struct find_base { const char* name; void* base; }; +int base_of(void* base, const void*, okld_u32, const char* name, void* ctx) { + auto* f = static_cast(ctx); + if (!same_text(f->name, length(f->name), name)) return 0; + f->base = base; + return 1; +} + +// --- writing --------------------------------------------------------------------------- +struct out_buffer { + char* data = nullptr; kal_uintptr used = 0, cap = 0; + void put(const char* s, kal_uintptr n) { + if (used + n > cap) { + kal_uintptr grown = cap ? cap * 2 : 65536; + while (grown < used + n) grown *= 2; + char* d = static_cast(kal_alloc(grown, 16)); + if (!d) return; + for (kal_uintptr i = 0; i < used; ++i) d[i] = data[i]; + if (data) kal_free(data, cap, 16); + data = d; cap = grown; + } + for (kal_uintptr i = 0; i < n; ++i) data[used + i] = s[i]; + used += n; + } + void put(const char* s) { put(s, length(s)); } + void number(kal_u64 v, int base = 10) { + char b[24]; int n = 0; + do { const int d = static_cast(v % static_cast(base)); b[n++] = static_cast(d < 10 ? '0' + d : 'a' + d - 10); v /= static_cast(base); } while (v); + char r[24]; + for (int i = 0; i < n; ++i) r[i] = b[n - 1 - i]; + put(r, static_cast(n)); + } + void quoted(const char* s) { + put("\""); + for (; s && *s; ++s) { + if (*s == '"' || *s == '\\') { const char e[2] = { '\\', *s }; put(e, 2); } + else if (static_cast(*s) < 0x20) put("?"); + else put(s, 1); + } + put("\""); + } + void release() { if (data) kal_free(data, cap, 16); data = nullptr; used = cap = 0; } +}; + +bool write_file(const char* path, const out_buffer& b) { + kal_dir dir{}; + const char* rest = nullptr; + if (!locate_path(path, dir, rest)) return false; + kal_file f{}; + if (kal_fs_open(dir, rest, length(rest), KAL_OPEN_WRITE | KAL_OPEN_CREATE | KAL_OPEN_TRUNCATE, &f) != kal_ok) + return false; + const kal_intptr w = kal_stream_write(kal_fs_stream(f), b.data, b.used); + kal_fs_close_file(f); + return w == static_cast(b.used); +} + +kal_u64 own_id() { + kal_process self{}; + if (kal_process_self(&self) != kal_ok) return 0; + kal_program_info info{}; + info.self_size = sizeof info; + kal_process_info(self, KAL_PINFO_ID, &info); + kal_process_close(self); + return info.id; +} + +bool on_linux() { + char system[16]; + const kal_intptr n = kal_env_system(system, sizeof system); + return n == 5 && same_text(system, 5, "linux"); +} + +void write_perf_map(image_record& r, const void* bytes, kal_u64 size) { + find_base fb{ r.name, nullptr }; + okld_iterate(&base_of, &fb); + if (!fb.base) return; + index_symbols(r, bytes, size); + if (!r.symbols) return; + out_buffer b; + for (kal_u64 i = 0; i < r.symbol_count; ++i) { + b.number(reinterpret_cast(fb.base) + r.symbols[i].value, 16); + b.put(" "); + b.number(r.symbols[i].size ? r.symbols[i].size : 1, 16); + b.put(" "); + b.put(r.names + r.symbols[i].name); + b.put("\n"); + } + // Appended: each image a run loads adds its lines. + char path[64] = "/tmp/perf-"; + kal_uintptr at = 10; + kal_u64 id = own_id(); + char digits[24]; int nd = 0; + do { digits[nd++] = static_cast('0' + id % 10); id /= 10; } while (id); + while (nd) path[at++] = digits[--nd]; + const char tail[] = ".map"; + for (kal_uintptr i = 0; i < sizeof tail; ++i) path[at + i] = tail[i]; + kal_dir dir{}; + const char* rest = nullptr; + if (locate_path(path, dir, rest)) { + kal_file f{}; + if (kal_fs_open(dir, rest, length(rest), KAL_OPEN_WRITE | KAL_OPEN_CREATE | KAL_OPEN_APPEND, &f) == kal_ok) { + kal_stream_write(kal_fs_stream(f), b.data, b.used); + kal_fs_close_file(f); + } + } + b.release(); +} + +bool g_finished = false; + +} // namespace + +bool profile_active() { return (mode() & asked_active) != 0; } +bool profile_calls() { return (mode() & asked_calls) != 0; } + +void record_call(unsigned long index, kal_u64 ns) { + if (index >= kProfiledNames) return; + __atomic_fetch_add(&g_calls[index], 1, __ATOMIC_RELAXED); + __atomic_fetch_add(&g_call_ns[index], ns, __ATOMIC_RELAXED); +} + +int profile_counter(const char* name, kal_uintptr len, const volatile kal_u64* cell) { + if (!(mode() & asked_counters) || !name || !cell) return kal_ok; + lock(); + if (g_counter_count == kCounters) { unlock(); return kal_err_no_memory; } + copy_text(g_counters[g_counter_count].name, sizeof g_counters[0].name, name, len); + g_counters[g_counter_count].cell = cell; + ++g_counter_count; + unlock(); + return kal_ok; +} + +void profile_mark(const char* name, kal_uintptr len, int begin) { + if (!(mode() & asked_counters) || !name) return; + const kal_u64 now = kal_time_monotonic(); + const kal_uintptr me = kal_task_current(); + lock(); + const unsigned p = phase_of(name, len); + context_phases* c = p < kPhases ? context_of(me) : nullptr; + if (c) { + if (begin) { + if (c->depth < 16) c->stack[c->depth++] = open_phase{ p, now }; + } else { + for (unsigned d = c->depth; d-- > 0;) { + if (c->stack[d].phase != p) continue; + g_phases[p].count += 1; + g_phases[p].ns += now - c->stack[d].start; + c->depth = d; + break; + } + if (c->depth == 0) c->context = 0; + } + } + unlock(); +} + +void profile_loaded(const char* name, const void* bytes, kal_u64 size, const char* source, kal_u64 offset) { + lock(); + image_record* r = record_of(name); + if (!r && g_image_count < kImages) r = &g_images[g_image_count++]; + if (r) { + *r = image_record{}; + copy_text(r->name, sizeof r->name, name, length(name)); + r->bytes = source ? nullptr : bytes; + r->at_hand = bytes; + r->size = size; + if (source) copy_text(r->source, sizeof r->source, source, length(source)); + r->offset = offset; + } + unlock(); +} + +// The images loaded so far, while their bytes are at hand: a perf map's lines +// are written for each (their places are known once they are loaded). +void profile_settle() { + const bool map = (mode() & asked_perfmap) && on_linux(); + for (unsigned i = 0; i < g_image_count; ++i) { + image_record& r = g_images[i]; + if (!r.at_hand) continue; + if (map && !r.mapped) { + find_base fb{ r.name, nullptr }; + okld_iterate(&base_of, &fb); + if (!fb.base) continue; // not loaded yet; asked again later + write_perf_map(r, r.at_hand, r.size); + r.mapped = true; + } + r.at_hand = r.bytes; + } +} + +kal_intptr symbolize(kal_uintptr address, char* out, kal_uintptr cap, kal_uintptr* start) { + okld_module* m = okld_containing(reinterpret_cast(address)); + if (!m) return -kal_err_not_found; + const char* name = okld_name(m); + const kal_u64 base = reinterpret_cast(okld_base(m)); + lock(); + image_record* r = name ? record_of(name) : nullptr; + if (r && !r->tried) { + r->tried = true; + void* region = nullptr; kal_uintptr region_len = 0; const void* bytes = nullptr; + if (reread(*r, region, region_len, bytes)) { + if (!r->symbols) index_symbols(*r, bytes, r->size); + if (region) kal_map_release(region, region_len); + } + } + if (!r || !r->symbols) { unlock(); return -kal_err_not_found; } + const kal_u64 at = address - base; + kal_u64 lo = 0, hi = r->symbol_count; + while (lo < hi) { // the last symbol whose value is at or below `at' + const kal_u64 mid = lo + (hi - lo) / 2; + if (r->symbols[mid].value <= at) lo = mid + 1; else hi = mid; + } + if (lo == 0) { unlock(); return -kal_err_not_found; } + const symbol& s = r->symbols[lo - 1]; + if (s.size && at >= s.value + s.size) { unlock(); return -kal_err_not_found; } + const char* text = r->names + s.name; + const kal_uintptr n = length(text); + for (kal_uintptr i = 0; out && i < n && i < cap; ++i) out[i] = text[i]; + if (start) *start = static_cast(base + s.value); + unlock(); + return static_cast(n); +} + +// THE REPORT, written once, where the program ends. +void profile_finish() { + if (!(mode() & asked_active)) return; + lock(); + if (g_finished) { unlock(); return; } + g_finished = true; + unlock(); + out_buffer b; + b.put("{\n \"format\": \"okprof-1\",\n \"image\": "); + b.quoted(g_image_count ? g_images[0].name : ""); + b.put(",\n \"id\": "); + const kal_u64 id = own_id(); + b.number(id); + b.put(",\n \"wall_ns\": "); + b.number(kal_time_monotonic() - g_started); + b.put(",\n \"calls\": ["); + unsigned long count = 0; + const binding* t = bindings(count); + bool first = true; + for (unsigned long i = 0; i < count && i < kProfiledNames; ++i) { + if (!g_calls[i]) continue; + b.put(first ? "\n { \"name\": " : ",\n { \"name\": "); + first = false; + b.quoted(t[i].name); + b.put(", \"count\": "); b.number(g_calls[i]); + b.put(", \"ns\": "); b.number(g_call_ns[i]); + b.put(" }"); + } + b.put(first ? "],\n \"counters\": [" : "\n ],\n \"counters\": ["); + first = true; + for (unsigned i = 0; i < g_counter_count; ++i) { + b.put(first ? "\n { \"name\": " : ",\n { \"name\": "); + first = false; + b.quoted(g_counters[i].name); + b.put(", \"value\": "); b.number(*g_counters[i].cell); + b.put(" }"); + } + b.put(first ? "],\n \"phases\": [" : "\n ],\n \"phases\": ["); + first = true; + for (unsigned i = 0; i < g_phase_count; ++i) { + b.put(first ? "\n { \"name\": " : ",\n { \"name\": "); + first = false; + b.quoted(g_phases[i].name); + b.put(", \"count\": "); b.number(g_phases[i].count); + b.put(", \"ns\": "); b.number(g_phases[i].ns); + b.put(" }"); + } + b.put(first ? "]\n}\n" : "\n ]\n}\n"); + char path[700]; + kal_uintptr at = 0; + if (g_dir[0]) { copy_text(path, sizeof path - 64, g_dir, length(g_dir)); at = length(path); path[at++] = '/'; } + const char stem[] = "okprof-"; + for (kal_uintptr i = 0; i + 1 < sizeof stem; ++i) path[at++] = stem[i]; + char digits[24]; int nd = 0; + kal_u64 v = id; + do { digits[nd++] = static_cast('0' + v % 10); v /= 10; } while (v); + while (nd) path[at++] = digits[--nd]; + const char tail[] = ".json"; + for (kal_uintptr i = 0; i < sizeof tail; ++i) path[at + i] = tail[i]; + write_file(path, b); + b.release(); +} + +} // namespace okrun + +// `okrun --prof=' and a program that carries an image: what OKRUN_PROF +// would have said, stated before the image is entered. +extern "C" void okrun_profile(const char* spec) { + okrun::read_spec(spec ? spec : "", okrun::length(spec)); +} diff --git a/run/src/run.cpp b/run/src/run.cpp index 5b94634..851608f 100644 --- a/run/src/run.cpp +++ b/run/src/run.cpp @@ -167,8 +167,10 @@ const char* interface_name(int bit) { "openkal.abort", "openkal.stream", "openkal.memory", "openkal.env", "openkal.time", "openkal.random", "openkal.fs", "openkal.process", "openkal.task", "openkal.exec", "openkal.terminal", "openkal.net", "openkal.datagram", "openkal.space", "openkal.timeout", + "openkal.map", "openkal.region", "openkal.prof", "openkal.debug", }; - return bit < 15 ? names[bit] : "an interface this runner does not know"; + return bit < static_cast(sizeof names / sizeof names[0]) ? names[bit] + : "an interface this runner does not know"; } bool key_is_trusted(const okrun_options* options, const unsigned char key[32]) { @@ -199,7 +201,9 @@ bool admit(const void* bytes, unsigned long long size, const char* name, const o } // WHAT THE IMAGE REQUIRES, AGAINST WHAT THIS IMPLEMENTATION PROVIDES: the // loader reports at load what a linker reports at link (SPEC 12.4). - unsigned long long provided = kal_interfaces(); + // The loader's own interfaces are provided whatever the implementation + // provides (SPEC 12.3, 0.2). + unsigned long long provided = kal_interfaces() | KAL_IFACE_PROF | KAL_IFACE_DEBUG; unsigned long n = 0; if (!space_bindings(n) || n == 0) provided &= ~(1ull << 13); if (const unsigned long long missing = info.interfaces & ~provided) { @@ -242,6 +246,12 @@ struct search { // read for an image another needs until the loader gives it back. file_bytes ran{}; file_bytes needed[16] = {}; + // Where each was read from, for a measurement and openkal.debug (0.2): + // the file run (empty for bytes a program carries), and each needed file's + // path and the name it was needed by. + char ran_path[1024] = {}; + char needed_path[16][1024] = {}; + char needed_name[16][256] = {}; }; // The file a run's image or okb was read from, given back once the image is @@ -290,14 +300,162 @@ void in_parallel(void (*job)(okld_u64, void*), okld_u64 count, void* ctx, void*) if (running[i]) kal_task_join(started[i]); } +// --- an okb's admission, remembered (0.2) ------------------------------------ +// +// EVERY START OF AN OKB READ EVERY BYTE OF THE IMAGES IT RAN, to compare them +// with the digests okb.json states: 0.04 to 0.14 s for a 146 MB okb on CI's +// machines, and a language server that starts a worker per module it builds +// started 176 of them. The comparison is remembered per user, per okb.json (its +// SHA-256) and per member, with the identity, the length and the modification +// time of the file it was made on: a start of the same file finds the record and +// admits the member by its shape (stored, aligned) without reading it; a file +// replaced or changed in place has another identity or time, and is compared +// again. The record is in the user's own cache directory, which only the user +// can write --- a party that could write a record could as well write the okb +// it describes. +struct memo_place { kal_dir dir{}; bool usable = false; char key[65] = {}; char stamp[96] = {}; }; + +const char hex_digits[] = "0123456789abcdef"; +void to_hex(const unsigned char* b, int n, char* out) { + for (int i = 0; i < n; ++i) { out[2 * i] = hex_digits[b[i] >> 4]; out[2 * i + 1] = hex_digits[b[i] & 15]; } + out[2 * n] = 0; +} +char* put_number(char* o, unsigned long long v) { + char d[24]; int n = 0; + do { d[n++] = static_cast('0' + v % 10); v /= 10; } while (v); + while (n) *o++ = d[--n]; + return o; +} + +// The user's cache directory, as this system places it, then `okrun/admitted'. +bool memo_directory(kal_dir& out) { + char system[16]; + const kal_intptr sn = kal_env_system(system, sizeof system); + const bool windows = sn == 7 && same_bytes(system, "windows", 7); + const bool macos = sn == 5 && same_bytes(system, "macos", 5); + char base[1024]; kal_intptr n = -1; + const char* tail = ""; + if (windows) { n = kal_env_var("LOCALAPPDATA", 12, base, sizeof base - 64); } + else { + n = macos ? -1 : kal_env_var("XDG_CACHE_HOME", 14, base, sizeof base - 64); + if (n <= 0) { n = kal_env_var("HOME", 4, base, sizeof base - 64); tail = macos ? "/Library/Caches" : "/.cache"; } + } + if (n <= 0 || n >= static_cast(sizeof base - 64)) return false; + base[n] = 0; + char path[1200]; + if (!join(path, sizeof path, base, "")) return false; + unsigned long at = length(path); + if (at && separator(path[at - 1])) --at; + for (const char* t = tail; *t && at + 1 < sizeof path; ++t) path[at++] = *t; + path[at] = 0; + kal_dir root{}; + const char* rest = nullptr; + if (!locate(path, root, rest)) return false; + // Each component made in turn; one that exists is entered. + kal_dir d = root; + char part[1100]; unsigned long p = 0; + auto enter_part = [&]() -> bool { + if (p == 0) return true; + kal_dir next{}; + kal_fs_mkdir(d, part, p); + if (kal_fs_open_dir(d, part, p, &next) != kal_ok) return false; + if (!(d.h == root.h)) kal_fs_close_dir(d); + d = next; + p = 0; + return true; + }; + for (const char* c = rest; ; ++c) { + if (*c == 0 || separator(*c)) { if (!enter_part()) return false; if (*c == 0) break; continue; } + if (p + 1 < sizeof part) part[p++] = *c; + } + static const char* const below[] = { "okrun", "admitted" }; + for (const char* c : below) { + p = 0; + while (c[p]) { part[p] = c[p]; ++p; } + if (!enter_part()) return false; + } + out = d; + return true; +} + +void memo_open(search& s, memo_place& m) { + if (!s.ran_path[0] || !s.json) return; + kal_dir dir{}; const char* rest = nullptr; + if (!locate(s.ran_path, dir, rest)) return; + kal_node_info info{}; + info.self_size = sizeof info; + if (kal_fs_info(dir, rest, length(rest), 0, KAL_INFO_IDENTITY | KAL_INFO_SIZE | KAL_INFO_MODIFIED, &info) != kal_ok + || !(info.present & KAL_INFO_IDENTITY) || !(info.present & KAL_INFO_MODIFIED)) + return; + if (!memo_directory(m.dir)) return; + unsigned char digest[32]; + okld_sha256(s.json, s.json_size, digest); + to_hex(digest, 32, m.key); + char* o = m.stamp; + o = put_number(o, info.identity[0]); *o++ = ' '; + o = put_number(o, info.identity[1]); *o++ = ' '; + o = put_number(o, info.size); *o++ = ' '; + o = put_number(o, info.modified_ns); *o++ = '\n'; *o = 0; + m.usable = true; +} + +// The record's name: the manifest's digest and the member's. +void memo_name(const memo_place& m, const char* member, char (&out)[160]) { + unsigned char d[32]; + okld_sha256(member, length(member), d); + char h[65]; + to_hex(d, 8, h); + unsigned long at = 0; + for (int i = 0; i < 64; ++i) out[at++] = m.key[i]; + out[at++] = '-'; + for (int i = 0; i < 16; ++i) out[at++] = h[i]; + out[at] = 0; +} + +bool memo_holds(const memo_place& m, const char* member) { + if (!m.usable) return false; + char name[160]; + memo_name(m, member, name); + kal_file f{}; + if (kal_fs_open(m.dir, name, length(name), KAL_OPEN_READ, &f) != kal_ok) return false; + char have[96] = {}; + const kal_intptr n = kal_stream_read(kal_fs_stream(f), have, sizeof have - 1); + kal_fs_close_file(f); + const unsigned long want = length(m.stamp); + return n == static_cast(want) && same_bytes(have, m.stamp, want); +} + +void memo_record(const memo_place& m, const char* member) { + if (!m.usable) return; + char name[160]; + memo_name(m, member, name); + kal_file f{}; + if (kal_fs_open(m.dir, name, length(name), KAL_OPEN_WRITE | KAL_OPEN_CREATE | KAL_OPEN_TRUNCATE, &f) != kal_ok) return; + kal_stream_write(kal_fs_stream(f), m.stamp, length(m.stamp)); + kal_fs_close_file(f); +} + +memo_place g_memo; +bool g_memo_opened = false; + // A member of the archive by name, admitted (stored, aligned, its stated content). const void* archive_member(search& s, const char* member, unsigned long long& size) { okld_zip_member m{}; okld_why why{}; if (okld_zip_find(s.archive, s.archive_size, member, &m, &why) != 0) return nullptr; - if (okld_okb_admit_chunked(s.archive, s.archive_size, &m, s.json, s.json_size, &in_parallel, nullptr, &why) != 0) { - copy_text(s.refusal, sizeof s.refusal, why.text); - return nullptr; + if (!g_memo_opened) { g_memo_opened = true; memo_open(s, g_memo); } + if (memo_holds(g_memo, member)) { + // Compared before, on this file as it is: its shape alone is asked. + if (okld_okb_admit(s.archive, s.archive_size, &m, nullptr, 0, &why) != 0) { + copy_text(s.refusal, sizeof s.refusal, why.text); + return nullptr; + } + } else { + if (okld_okb_admit_chunked(s.archive, s.archive_size, &m, s.json, s.json_size, &in_parallel, nullptr, &why) != 0) { + copy_text(s.refusal, sizeof s.refusal, why.text); + return nullptr; + } + memo_record(g_memo, member); } size = m.size; return s.archive + m.data; @@ -370,7 +528,8 @@ const void* find_needed(const char* name, okld_size* size, void* ctx) { if (!bytes && s.refusal[0]) return nullptr; } file_bytes f; - if (!bytes && find_file(s, name, f)) { bytes = f.data; n = f.size; } + char found[1024] = {}; + if (!bytes && find_file(s, name, f, found, sizeof found)) { bytes = f.data; n = f.size; } if (!bytes) return nullptr; char why[256]; if (!admit(bytes, n, name, s.options, why, sizeof why)) { @@ -381,9 +540,17 @@ const void* find_needed(const char* name, okld_size* size, void* ctx) { if (f.data) { // Held until the loader gives the bytes back (`release_needed'). file_bytes* slot = nullptr; - for (auto& h : s.needed) if (!h.data) { slot = &h; break; } + unsigned k = 0; + for (auto& h : s.needed) { if (!h.data) { slot = &h; break; } ++k; } if (!slot) { release_file(f); copy_text(s.refusal, sizeof s.refusal, "more than sixteen files are needed at once"); return nullptr; } *slot = f; + copy_text(s.needed_path[k], sizeof s.needed_path[k], found); + copy_text(s.needed_name[k], sizeof s.needed_name[k], name); + } else if (s.archive && bytes >= s.archive && bytes < s.archive + s.archive_size) { + profile_loaded(name, bytes, n, s.ran_path[0] ? s.ran_path : nullptr, + static_cast(static_cast(bytes) - s.archive)); + } else { + profile_loaded(name, bytes, n, nullptr, 0); } *size = n; return bytes; @@ -393,8 +560,16 @@ const void* find_needed(const char* name, okld_size* size, void* ctx) { // read with; the archive's and the group's bytes are not this search's. void release_needed(const void* bytes, okld_size, void* ctx) { auto& s = *static_cast(ctx); - for (auto& h : s.needed) - if (h.data && h.data == bytes) { release_file(h); return; } + for (unsigned k = 0; k < 16; ++k) { + auto& h = s.needed[k]; + if (!h.data || h.data != bytes) continue; + // The image is loaded by now: a measurement reads its symbols while its + // bytes are at hand, and openkal.debug knows where to read them again. + profile_loaded(s.needed_name[k], h.data, h.size, s.needed_path[k], 0); + profile_settle(); + release_file(h); + return; + } } void prepare_host(search& s) { @@ -431,10 +606,21 @@ int enter(search& s, const void* bytes, unsigned long long size, const char* nam } void* entry = okld_entry(m); if (!entry) { text(why, cap) << name << ": the image has no entry"; return -1; } + // Where the entry image's bytes can be read again (0.2): the file run, at + // the image's offset in it, or nowhere for bytes a program carries, which + // stay where they are. + { + const auto* b = static_cast(bytes); + const bool fromRan = s.ran.data && b >= s.ran.data && b < s.ran.data + s.ran.size; + profile_loaded(name, bytes, size, fromRan && s.ran_path[0] ? s.ran_path : nullptr, + fromRan ? static_cast(b - s.ran.data) : 0); + } + profile_settle(); release_ran(s); okld_initialise(m, &call_image, nullptr); const int status = reinterpret_cast(entry)(); okld_finalise(m, &call_image, nullptr); + profile_finish(); kal_exit(status); } @@ -517,6 +703,8 @@ search g_search; image_state& state() { return g_state; } +bool locate_path(const char* path, kal_dir& dir, const char*& rest) { return locate(path, dir, rest); } + } // namespace okrun using namespace okrun; @@ -564,6 +752,7 @@ extern "C" int okrun_run_file(const char* path, int argc, const char* const* arg file_bytes f; if (!read_all(path, f, why, cap)) return -1; g_search.ran = f; + copy_text(g_search.ran_path, sizeof g_search.ran_path, path); const int status = okrun_run_bytes(f.data, f.size, path, argc, argv, options, why, cap); release_ran(g_search); return status; diff --git a/run/src/runner.h b/run/src/runner.h index 5e5741a..3b1e690 100644 --- a/run/src/runner.h +++ b/run/src/runner.h @@ -1,4 +1,6 @@ #pragma once +#include +#include namespace okrun { @@ -16,6 +18,31 @@ struct image_state { image_state& state(); const binding* bindings(unsigned long& count); + +// --- a measurement, and the names of addresses (0.2, src/prof.cpp) --------- +// +// What OKRUN_PROF (or `okrun --prof') asks for, read once: `calls' binds each +// name through a counting entry, `counters' keeps the counters and phases a +// program registers (openkal.prof), `perfmap' writes /tmp/perf-.map for a +// profiler of this system; a directory after `:' receives the report. +inline constexpr unsigned long kProfiledNames = 256; +bool profile_active(); +bool profile_calls(); +void record_call(unsigned long index, kal_u64 ns); +int profile_counter(const char* name, kal_uintptr len, const volatile kal_u64* cell); +void profile_mark(const char* name, kal_uintptr len, int begin); +void profile_finish(); +// An image's bytes, while they are at hand, for the symbols a measurement and +// openkal.debug name (`source' and `offset' say where they can be read again: +// a file the image is read from, at an offset; null for bytes that stay where +// they are --- an image a program carries). +void profile_loaded(const char* name, const void* bytes, kal_u64 size, + const char* source, kal_u64 offset); +void profile_settle(); +kal_intptr symbolize(kal_uintptr address, char* out, kal_uintptr cap, kal_uintptr* start); +// A name as openkal names things: the supplied directory that is the longest +// prefix of it, and the rest (src/run.cpp). +bool locate_path(const char* path, kal_dir& dir, const char*& rest); const binding* space_bindings(unsigned long& count); inline bool same(const char* a, const char* b) { diff --git a/tools/mcpp.toml b/tools/mcpp.toml index 305da60..a50cf62 100644 --- a/tools/mcpp.toml +++ b/tools/mcpp.toml @@ -1,7 +1,7 @@ [package] namespace = "mcpplibs" name = "openkal-tools" -version = "0.1.9" +version = "0.2.0" description = "okrun, which runs an ok image on this system, and okld, the format's tool on a build machine (check, note, sign, inspect)." license = "Apache-2.0" authors = ["mcpplibs"] @@ -19,5 +19,10 @@ main = "src/okrun.cpp" kind = "bin" main = "src/okld.cpp" +# What an image did across openkal, from the runner's reports (0.2). +[targets.okprof] +kind = "bin" +main = "src/okprof.cpp" + [build] sources = [] diff --git a/tools/src/okld.cpp b/tools/src/okld.cpp index 42f77c9..c78147a 100644 --- a/tools/src/okld.cpp +++ b/tools/src/okld.cpp @@ -55,9 +55,10 @@ std::string hex(const unsigned char* p, std::size_t n) { std::string interfaces(okld_u64 bits) { static const char* const names[] = { "abort", "stream", "memory", "env", "time", "random", "fs", "process", - "task", "exec", "terminal", "net", "datagram", "space", "timeout" }; + "task", "exec", "terminal", "net", "datagram", "space", "timeout", + "map", "region", "prof", "debug" }; std::string s; - for (int b = 0; b < 15; ++b) if (bits & (1ull << b)) { if (!s.empty()) s += ' '; s += "openkal."; s += names[b]; } + for (int b = 0; b < static_cast(sizeof names / sizeof names[0]); ++b) if (bits & (1ull << b)) { if (!s.empty()) s += ' '; s += "openkal."; s += names[b]; } return s.empty() ? "none" : s; } diff --git a/tools/src/okprof.cpp b/tools/src/okprof.cpp new file mode 100644 index 0000000..a6100ec --- /dev/null +++ b/tools/src/okprof.cpp @@ -0,0 +1,339 @@ +// okprof --- what an ok image did across openkal, measured by the runner that +// ran it (openkal-image 0.2). +// +// okprof record [-o ] [--what ] -- [arguments...] +// runs the program with OKRUN_PROF set, so that every image an okrun +// beneath it runs (okrun itself, or a program that starts one, as an +// editor starts its language server) writes /okprof-.json when it +// ends. is any of calls, counters, perfmap (default calls,counters). +// okprof report ... +// the calls each image made across openkal, by name and by interface, +// with their counts and time; the counters and phases the program stated. +// Several reports are summed. +// okprof diff +// the same names side by side, ordered by how much their time moved: the +// input of a gate that compares two runs (an image and the same program +// built for its system; two versions). +// +// A report is the runner's (run/src/prof.cpp); this program only reads and +// starts. It does not sample: a profiler of the system does, and on Linux the +// runner's `perfmap' names an image's addresses for it. +#include +#include +import std; + +namespace { + +struct row { std::string name; std::uint64_t count = 0, ns = 0, value = 0; }; +struct report { + std::map calls, counters, phases; + std::uint64_t wall_ns = 0; + int files = 0; +}; + +// --- reading a report ---------------------------------------------------------- +// The runner writes one shape; this reads that shape: three arrays of flat +// objects whose values are strings and unsigned numbers. +struct reader { + std::string_view s; std::size_t at = 0; + void space() { while (at < s.size() && std::isspace(static_cast(s[at]))) ++at; } + bool eat(char c) { space(); if (at < s.size() && s[at] == c) { ++at; return true; } return false; } + std::optional text() { + space(); + if (at >= s.size() || s[at] != '"') return std::nullopt; + std::string out; + for (++at; at < s.size() && s[at] != '"'; ++at) { + if (s[at] == '\\' && at + 1 < s.size()) ++at; + out += s[at]; + } + if (at >= s.size()) return std::nullopt; + ++at; + return out; + } + std::optional number() { + space(); + std::uint64_t v = 0; bool any = false; + while (at < s.size() && s[at] >= '0' && s[at] <= '9') { v = v * 10 + static_cast(s[at++] - '0'); any = true; } + return any ? std::optional(v) : std::nullopt; + } + void skip_value() { + space(); + if (at < s.size() && s[at] == '"') { text(); return; } + if (at < s.size() && (s[at] == '[' || s[at] == '{')) { + int depth = 0; + for (; at < s.size(); ++at) { + if (s[at] == '"') { text(); --at; continue; } + if (s[at] == '[' || s[at] == '{') ++depth; + if (s[at] == ']' || s[at] == '}') { if (--depth == 0) { ++at; return; } } + } + return; + } + while (at < s.size() && s[at] != ',' && s[at] != '}' && s[at] != ']') ++at; + } +}; + +bool read_rows(reader& r, std::map& into) { + if (!r.eat('[')) return false; + if (r.eat(']')) return true; + do { + if (!r.eat('{')) return false; + row one; + if (!r.eat('}')) { + do { + auto key = r.text(); + if (!key || !r.eat(':')) return false; + if (*key == "name") { auto v = r.text(); if (!v) return false; one.name = *v; } + else if (*key == "count") { auto v = r.number(); if (!v) return false; one.count = *v; } + else if (*key == "ns") { auto v = r.number(); if (!v) return false; one.ns = *v; } + else if (*key == "value") { auto v = r.number(); if (!v) return false; one.value = *v; } + else r.skip_value(); + } while (r.eat(',')); + if (!r.eat('}')) return false; + } + auto& have = into[one.name]; + have.name = one.name; + have.count += one.count; + have.ns += one.ns; + have.value += one.value; + } while (r.eat(',')); + return r.eat(']'); +} + +bool read_report(const std::filesystem::path& file, report& into) { + std::ifstream in(file, std::ios::binary); + if (!in) return false; + const std::string text((std::istreambuf_iterator(in)), {}); + reader r{ text }; + if (!r.eat('{')) return false; + do { + auto key = r.text(); + if (!key || !r.eat(':')) return false; + if (*key == "calls") { if (!read_rows(r, into.calls)) return false; } + else if (*key == "counters") { if (!read_rows(r, into.counters)) return false; } + else if (*key == "phases") { if (!read_rows(r, into.phases)) return false; } + else if (*key == "wall_ns") { auto v = r.number(); if (v) into.wall_ns += *v; } + else r.skip_value(); + } while (r.eat(',')); + ++into.files; + return true; +} + +bool read_reports(const std::vector& args, report& into) { + for (auto const& a : args) { + std::error_code ec; + if (std::filesystem::is_directory(a, ec)) { + for (auto const& e : std::filesystem::directory_iterator(a, ec)) { + const auto name = e.path().filename().string(); + if (name.starts_with("okprof-") && name.ends_with(".json") && !read_report(e.path(), into)) { + std::println(std::cerr, "okprof: {} is not a report", e.path().string()); + return false; + } + } + } else if (!read_report(a, into)) { + std::println(std::cerr, "okprof: {} is not a report", a); + return false; + } + } + if (into.files == 0) { std::println(std::cerr, "okprof: no report was found"); return false; } + return true; +} + +// The interface a name belongs to, as libokld reads it (`kal_alloc' is +// openkal.memory, `kal_stdout' openkal.stream). +std::string interface_of(const std::string& name) { + static const char* const names[] = { + "abort", "stream", "memory", "env", "time", "random", "fs", "process", "task", "exec", + "terminal", "net", "datagram", "space", "timeout", "map", "region", "prof", "debug" }; + const okld_u64 bits = okld_interface_of(name.c_str()); + for (unsigned i = 0; i < sizeof names / sizeof names[0]; ++i) + if (bits & (okld_u64{1} << i)) return std::string("openkal.") + names[i]; + if (name == "kal_version" || name == "kal_interfaces") return "openkal.version"; + return "?"; +} + +std::string ms(std::uint64_t ns) { return std::format("{:.3f} ms", static_cast(ns) / 1e6); } + +int report_cmd(const std::vector& args) { + report r; + if (!read_reports(args, r)) return 2; + std::println("{} report{}; the images ran {} in all", r.files, r.files == 1 ? "" : "s", ms(r.wall_ns)); + std::vector calls; + for (auto const& [k, v] : r.calls) calls.push_back(v); + std::ranges::sort(calls, [](const row& a, const row& b) { return a.ns > b.ns; }); + std::map byInterface; + std::uint64_t total = 0; + for (auto const& c : calls) { + auto& i = byInterface[interface_of(c.name)]; + i.name = interface_of(c.name); + i.count += c.count; + i.ns += c.ns; + total += c.ns; + } + std::println("\ncalls across openkal, by interface ({} in all)", ms(total)); + std::vector ifaces; + for (auto const& [k, v] : byInterface) ifaces.push_back(v); + std::ranges::sort(ifaces, [](const row& a, const row& b) { return a.ns > b.ns; }); + for (auto const& i : ifaces) + std::println(" {:<20} {:>12} calls {:>14}", i.name, i.count, ms(i.ns)); + std::println("\ncalls across openkal, by name"); + for (auto const& c : calls) + std::println(" {:<32} {:>12} calls {:>14} {:>9} ns each", c.name, c.count, ms(c.ns), + c.count ? c.ns / c.count : 0); + if (!r.counters.empty()) { + std::println("\ncounters the program stated"); + for (auto const& [k, v] : r.counters) std::println(" {:<40} {:>16}", k, v.value); + } + if (!r.phases.empty()) { + std::println("\nphases the program stated"); + for (auto const& [k, v] : r.phases) std::println(" {:<40} {:>8} times {:>14}", k, v.count, ms(v.ns)); + } + return 0; +} + +int diff_cmd(const std::vector& args) { + if (args.size() != 2) { std::println(std::cerr, "okprof diff "); return 2; } + report a, b; + if (!read_reports({args[0]}, a) || !read_reports({args[1]}, b)) return 2; + std::set names; + for (auto const& [k, v] : a.calls) names.insert(k); + for (auto const& [k, v] : b.calls) names.insert(k); + struct moved { std::string name; std::int64_t delta; row before, after; }; + std::vector rows; + for (auto const& n : names) { + const row before = a.calls.contains(n) ? a.calls.at(n) : row{n}; + const row after = b.calls.contains(n) ? b.calls.at(n) : row{n}; + rows.push_back({n, static_cast(after.ns) - static_cast(before.ns), before, after}); + } + std::ranges::sort(rows, [](const moved& x, const moved& y) { return std::llabs(x.delta) > std::llabs(y.delta); }); + std::println("{:<32} {:>12} {:>12} {:>14} {:>14} {:>9}", "name", "calls before", "calls after", + "time before", "time after", "ratio"); + for (auto const& m : rows) + std::println("{:<32} {:>12} {:>12} {:>14} {:>14} {:>9}", m.name, m.before.count, m.after.count, + ms(m.before.ns), ms(m.after.ns), + m.before.ns ? std::format("{:.2f}", static_cast(m.after.ns) / static_cast(m.before.ns)) + : std::string("new")); + std::println("\nwall time {} -> {}", ms(a.wall_ns), ms(b.wall_ns)); + return 0; +} + +// --- starting the program --------------------------------------------------------- +// A name as openkal names things: the supplied directory that is the longest +// prefix of it, and the rest. A name without a separator is looked for on PATH. +bool locate(const std::string& path, kal_dir& dir, std::string& rest) { + auto separator = [](char c) { return c == '/' || c == '\\'; }; + const bool absolute = !path.empty() && (separator(path[0]) || (path.size() > 1 && path[1] == ':')); + if (!absolute) { + char name[8]; kal_uintptr len = 0; + if (kal_fs_preopen(0, &dir, name, sizeof name, &len) != kal_ok) return false; + rest = path; + return true; + } + std::size_t best = 0; bool found = false; + for (kal_uintptr i = 0; i < kal_fs_preopen_count(); ++i) { + kal_dir d{}; char name[1024]; kal_uintptr len = 0; + if (kal_fs_preopen(i, &d, name, sizeof name, &len) != kal_ok || len == 0 || len >= sizeof name) continue; + const std::string_view n(name, len); + if (!path.starts_with(n)) continue; + if (!separator(n.back()) && path.size() > len && !separator(path[len])) continue; + if (found && len <= best) continue; + best = len; found = true; dir = d; + } + if (!found) return false; + rest = path.substr(best); + while (!rest.empty() && separator(rest.front())) rest.erase(0, 1); + return true; +} + +std::optional on_path(const std::string& name) { + if (name.find('/') != std::string::npos || name.find('\\') != std::string::npos) return name; + char list[8192]; + const kal_intptr n = kal_env_var("PATH", 4, list, sizeof list - 1); + if (n <= 0 || n >= static_cast(sizeof list)) return std::nullopt; + char system[16]; + const kal_intptr sn = kal_env_system(system, sizeof system); + const bool windows = sn == 7 && std::string_view(system, 7) == "windows"; + const char sep = windows ? ';' : ':'; + for (auto part : std::views::split(std::string_view(list, static_cast(n)), sep)) { + std::string dirName(part.begin(), part.end()); + if (dirName.empty()) continue; + for (const char* ext : {"", ".exe"}) { + if (!windows && *ext) continue; + const std::string candidate = dirName + "/" + name + ext; + kal_dir d{}; std::string rest; + if (!locate(candidate, d, rest)) continue; + kal_node_info info{}; + info.self_size = sizeof info; + if (kal_fs_info(d, rest.c_str(), rest.size(), 0, KAL_INFO_KIND, &info) == kal_ok && info.kind == kal_node_file) + return candidate; + } + } + return std::nullopt; +} + +int record_cmd(std::vector args) { + std::string dir = ".", what = "calls,counters"; + while (!args.empty() && args.front() != "--") { + if (args.front() == "-o" && args.size() > 1) { dir = args[1]; args.erase(args.begin(), args.begin() + 2); continue; } + if (args.front() == "--what" && args.size() > 1) { what = args[1]; args.erase(args.begin(), args.begin() + 2); continue; } + std::println(std::cerr, "okprof record: unknown option {}", args.front()); + return 2; + } + if (args.empty() || args.size() < 2) { std::println(std::cerr, "okprof record [-o ] [--what ] -- [arguments...]"); return 2; } + args.erase(args.begin()); // "--" + std::error_code ec; + std::filesystem::create_directories(dir, ec); + const auto program = on_path(args.front()); + if (!program) { std::println(std::cerr, "okprof: {} is not found", args.front()); return 2; } + kal_dir base{}; std::string rest; + if (!locate(*program, base, rest)) { std::println(std::cerr, "okprof: {} is beneath no directory this program was given", *program); return 2; } + + // The environment, with OKRUN_PROF stated (replacing any it had). + const std::string spec = std::format("OKRUN_PROF={}:{}", what, std::filesystem::absolute(dir, ec).generic_string()); + std::vector env; + for (kal_uintptr i = 0; i < kal_env_var_count(); ++i) { + char entry[16384]; + const kal_intptr n = kal_env_var_at(i, entry, sizeof entry); + if (n <= 0 || n >= static_cast(sizeof entry)) continue; + std::string e(entry, static_cast(n)); + if (e.starts_with("OKRUN_PROF=")) continue; + env.push_back(std::move(e)); + } + env.push_back(spec); + std::vector argv, envp; + std::vector argl, envl; + for (auto const& a : args) { argv.push_back(a.c_str()); argl.push_back(a.size()); } + for (auto const& e : env) { envp.push_back(e.c_str()); envl.push_back(e.size()); } + kal_dir work{}; + { + char name[8]; kal_uintptr len = 0; + kal_fs_preopen(0, &work, name, sizeof name, &len); + } + const kal_spawn how{ base, work, nullptr, nullptr, 0, 0 }; + const kal_spawn_streams streams{ 0, 0, 0 }; + kal_process child{}; + const int e = kal_process_spawn(&how, rest.c_str(), rest.size(), argv.data(), argl.data(), argv.size(), + envp.data(), envl.data(), envp.size(), &streams, &child); + if (e != kal_ok) { std::println(std::cerr, "okprof: {} could not be started ({})", *program, e); return 2; } + int status = 0, terminated = 0; + kal_process_wait(child, &status, &terminated); + kal_process_close(child); + std::println(std::cerr, "okprof: the reports are in {}; `okprof report {}' reads them", dir, dir); + return terminated ? 128 + status : status; +} + +} // namespace + +int main(int argc, char** argv) { + std::vector args(argv + 1, argv + argc); + if (args.empty()) { + std::println(std::cerr, "okprof record|report|diff ... (see the head of tools/src/okprof.cpp)"); + return 2; + } + const std::string command = args.front(); + args.erase(args.begin()); + if (command == "record") return record_cmd(std::move(args)); + if (command == "report") return report_cmd(args); + if (command == "diff") return diff_cmd(args); + std::println(std::cerr, "okprof: unknown command {}", command); + return 2; +} diff --git a/tools/src/okrun.cpp b/tools/src/okrun.cpp index dabb352..43e5a23 100644 --- a/tools/src/okrun.cpp +++ b/tools/src/okrun.cpp @@ -5,6 +5,9 @@ // --trust a public key (64 hexadecimal digits) whose images run; // given at least once, an image signed by another is refused // --debugger announce the image to gdb and lldb (GDB JIT interface) +// --prof=[:dir] measure the image: calls, counters, perfmap (comma +// separated); the report goes to /okprof-.json, +// which `okprof report' reads (OKRUN_PROF says the same) // // The image is told its arguments are the image's (or the okb's) name and what // follows it. The images it needs are found beside it, in `lib//' and @@ -27,6 +30,8 @@ int main(int argc, char** argv) { if (same(argv[i], "--")) { ++i; break; } if (same(argv[i], "--require-signature")) { options.require_signature = 1; continue; } if (same(argv[i], "--debugger")) { options.debugger = 1; continue; } + if (argv[i][2] == 'p' && argv[i][3] == 'r' && argv[i][4] == 'o' && argv[i][5] == 'f' + && argv[i][6] == '=') { okrun_profile(argv[i] + 7); continue; } if (same(argv[i], "--trust") && i + 1 < argc && options.trusted_count < 16 && length(argv[i + 1]) == 64) { const char* k = argv[++i]; for (int b = 0; b < 32; ++b) { From f909e0b2b3425451182e7410b2c1259e298aa9be Mon Sep 17 00:00:00 2001 From: Sunrisepeak Date: Mon, 2 Nov 2026 06:06:10 +0900 Subject: [PATCH 2/2] ci: the report's directory relative to the working one (an MSYS path is not a Windows program's) Co-authored-by: speak-agent <248744407+speak-agent@users.noreply.github.com> --- .github/workflows/ci.yml | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 74122b8..ea488e2 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -205,7 +205,9 @@ jobs: grep -q "prof: not measured" <<<"$p" || { echo "::error::an unmeasured image was told it is measured"; exit 1; } grep -q "debug: prof_named_function" <<<"$p" || { echo "::error::the loader did not name the image's function"; exit 1; } rm -rf prof && mkdir prof - "$OKRUN" "--prof=calls,counters:$(pwd)/prof" "out/$a/prof.oke" > prof.out + # A directory relative to the working one: under Git Bash `$(pwd)` is + # an MSYS spelling (/d/a/...), which a Windows program does not open. + "$OKRUN" "--prof=calls,counters:prof" "out/$a/prof.oke" > prof.out grep -q "prof: measured" prof.out || { echo "::error::a measured image was told it is not"; exit 1; } report=$(ls prof/okprof-*.json | head -1) [ -n "$report" ] || { echo "::error::a measured image wrote no report"; exit 1; }