Conversation
XProf ships a compiled profiler_plugin_c_api.so and libpywrap_events_db_c_api.so behind a py3-none wheel (the "abi: py3" in the queue note is upstream's own retag, not a pure-Python build): they are ctypes/C-API loaded, not linked against a specific CPython, so one build serves every interpreter. Confirmed this is not the jaxlib (PR #526, parked) XLA-codegen wall: xprof's native targets (xprof/pywrap, xprof/convert/events_db/python) depend only on xla/pjrt, xla/tsl/* and xla/hlo/ir - none of it reachable from xla/codegen/intrinsic/cpp, which is the genrule that has no RISC-V LLVM backend wired in and is used exclusively by xla/backends/cpu/codegen, xla/service/cpu and xla/service/gpu, none of which xprof touches. The embedded/llo_analysis path that would pull in more of XLA sits behind enable_embedded_features, which defaults off in the OSS build. The frontend (Angular/rollup, trace_viewer_v2 wasm) has no riscv64-specific code and node.js publishes no riscv64 release, so it is built once on its own job with upstream's hermetic toolchain (confirmed byte-identical across upstream's macOS/Linux/Windows wheels) and only the two native extensions are built on the riscv64 runner, following build-ray.yml's precedent for splitting an arch-independent JS asset out of a Bazel build. Bazel bootstrap follows build-ai-edge-litert.yml/build-ray.yml (rules_python 0.33.2 and rules_java 7.6.5 both need a riscv64 stand-in; xprof's own .bazelversion, 7.4.1, resolves bazel_features 1.11.0 and does not need that fix). --config=clang_local is xprof's own off-switch for XLA's hermetic C++ toolchain, already used upstream for non-x86_64/aarch64 hosts.
luhenry
added a commit
that referenced
this pull request
Sep 24, 2026
The reference_build job compiles the same Arrow/gRPC/protobuf/abseil/XLA-TSL tree the riscv64 job does, on a GitHub-hosted runner with a much smaller disk than the self-hosted riscv64 boxes; build-daft.yml/build-lancedb.yml hit the same ceiling on a comparably sized dependency tree.
Contributor
|
The riscv64 bazel bootstrap died mid-compile: Fatal glibc error: pthread_mutex_lock.c:94 (___pthread_mutex_lock): assertion failed: mutex->__data.__owner == 0 inside a Javac multiplex-worker process building bazel's own libtop_level_status_events.jar, which took the whole self-build down (ERROR: Could not build Bazel). --strategy=Javac=standalone forks a fresh javac per compile action instead of routing them through one persistent worker JVM, which is what races on this image's glibc. Applied to both the from-source bazel bootstrap and the xprof build itself, since either could hit the same worker path.
Checked the workflow against the actual xprof-v2.23.1 git tag (rather than main, which I had been reading): xprof/convert/events_db (the Arrow/Parquet events database) does not exist yet at that tag, so //xprof/convert/events_db/python:libpywrap_events_db_c_api.so is not a buildable target and the real released wheel does not carry it either (confirmed by unzipping it - only profiler_plugin_c_api.so is present). CI caught this: 'no such package xprof/convert/events_db/python'. Removes the target from the build, the copy step and the wheel sanity check; xprof/pywrap:profiler_plugin_c_api.so's own dependency chain (checked against the tag too) is unaffected.
--config=clang_local only disables the hermetic cc toolchain and toolchain resolution; it doesn't select clang. Without CC set, Bazel's local_config_cc autodetection falls back to plain gcc, which silently built most of the tree but failed on com_google_highway's riscv64 copts: they unconditionally add the Clang-only -menable-experimental-extensions flag, which gcc has never implemented.
luhenry
added a commit
that referenced
this pull request
Sep 24, 2026
…stalled xprof's build-xprof.yml (PR #2270) installed clang and passed --config=clang_local per gotcha 132 but never exported CC/CXX, so Bazel's local toolchain autodetection kept resolving to plain gcc. The build got 10+ minutes deep before failing on com_google_highway's riscv64 copts, which unconditionally add the Clang-only -menable-experimental-extensions flag.
…lvm-raw The build died after ~3.5h with clang-21 exiting 132 (SIGILL) while compiling @llvm-project//mlir:IR's lib/IR/BuiltinDialectBytecode.cpp [for tool], instantiating mlir::getChecked<AffineMapAttr, AffineMap &> (BytecodeImplementation.h:472). This is not stack or memory exhaustion: the backtrace is 87 frames deep and ends in OverloadCandidateSet::CompleteCandidates <- NoteCandidates, a stack overflow would be SIGSEGV, and it reproduces deterministically on x86_64 with the same clang (21.1.8, RESF 21.1.8-1.el10, Rocky 10) from a CMake build of the same llvm-project c9b25a6437fd - also under `ulimit -s unlimited`, with -std=c++20, and without PCH. A 30-line reduction crashes too. Cause: MLIR's is_detected probe of T::getChecked(emitError, params...) fails overload resolution against the inherited StorageUserBase::getChecked( function_ref, MLIRContext *, Args...) template for every type without a no-context getChecked. Clang 21's CWG2369 handling skips the function_ref parameter, rejects the MLIRContext * one, and leaves a conversion slot uninitialized that building the candidate notes then reads (llvm/llvm-project#188249, #198278). Upstream fixed it on the MLIR side in llvm/llvm-project#208359 (9722a2ceddb6, 2026-07-09), after the LLVM that xprof 2.23.1's XLA pins. Carry that change unmodified as patches/xprof/2.23.1/0001 and apply it to Bazel's fetched @llvm-raw between a --nobuild pass and the real build. Verified locally: with it the crashing TU and 853 MLIR IR/bytecode/dialect objects compile with clang 21.1.8 (XLA's own llvm patches applied too); without it the same TU still crashes. The fetch/patch/build sequence was checked with bazel 7.4.1 against an @llvm-raw -> llvm_configure-style symlink overlay.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
xprof2.23.1Compiles
xprof/pywrap:profiler_plugin_c_api.so, a Bazel target built against XLA's TSL profiler/platform libraries (not XLA's compiler). Upstream publishes no riscv64 wheel.Mirrors upstream's own documented build (
bazel run plugin:build_pip_package, see the README's "Build from Source" section) andplugin/build_pip_package.sh, which upstream's actual PyPI release also runs (confirmed by unzipping the released wheels: they carry the sameprofiler_plugin_c_api.so, arch-specific, behind apy3-nonetag from a custombdist_wheel.get_tag).This is not the jaxlib (PR #526, parked) XLA-codegen wall: jaxlib dies in
xla/codegen/intrinsic/cpp'sembed_bitcode, which links every LLVM backend except RISC-V. xprof's native target depends only onxla/pjrt,xla/tsl/*andxla/hlo/ir, none of which is reachable fromxla/codegen/intrinsic(used exclusively byxla/backends/cpu/codegen,xla/service/cpuandxla/service/gpu). The one path that would pull in more of XLA,xprof/embedded/llo_analysis, sits behindenable_embedded_features, which defaultsFalsein the OSS build.Differs from upstream
trace_viewer_v2wasm) is built once onubuntu-24.04with upstream's own hermetic toolchain, since it has no riscv64-specific code and node.js publishes no riscv64 release (same reasoning asbuild-ray.yml's dashboard job). Confirmed byte-identical across upstream's released macOS/Linux/Windows wheels, so building it once and reusing it does not diverge from what upstream itself ships..sotarget is then built on the riscv64 runner and swapped into that reference package beforepip wheel, instead of running the fullplugin:build_pip_packagesh_binary there - it would otherwise pull the Angular/npm build back into the riscv64 job.--config=clang_local: xprof's own off-switch for XLA's hermetic C++ toolchain (no riscv64 build), already used upstream for non-x86_64/aarch64 hosts (xla/tsl:ci_linux_aarch64).bazel-<ver>-dist.zip), as no release publishes a riscv64 binary; rules_python 0.33.2 and rules_java 7.6.5 both need the same riscv64 stand-ins asbuild-ai-edge-litert.yml/build-ray.yml.--strategy=Javac=standaloneworks around a Javac persistent-worker/glibc crash on this image (Fatal glibc error: pthread_mutex_lock.c: assertion failed: mutex->__data.__owner == 0).Matrix: none - the wheel is
py3-none(the.sois loaded via a C API, not linked against a specific CPython), so one build serves every interpreter.Testing
License: OK - Apache-2.0, nothing vendored under a different licence ships in the wheel (gRPC, protobuf, Abseil, google-cloud-cpp are all build-time-only, none of it ships pre-built into the wheel beyond the one static-linked
.so).Built on cp312 (the sole build interpreter); full CI run in progress.