From 2451fc56e468e8715816e9ee041c2285086ac2a9 Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Thu, 7 May 2026 12:23:47 +0000 Subject: [PATCH 01/20] Add spark-gluten-clickhouse entry (Spark + Gluten with the CH backend) Adds a spark-gluten-clickhouse/ entry that runs the ClickBench query suite against Apache Spark with Apache Gluten configured to use the ClickHouse backend ('ch'), in which Gluten loads libch.so (a fork of ClickHouse v23.1) into the Spark executor JVM and runs the columnar plan natively through it. Compared with spark-gluten/ (which uses the Velox backend), this exercises a meaningfully different execution path: Catalyst -> Substrait -> ClickHouse engine, rather than Catalyst -> Substrait -> Velox. No pre-built bundle is published for the CH backend (the Apache Gluten release tarball ships only the Velox bundle), so benchmark.sh builds both libch.so and the Gluten Spark plugin from source. The build is memory-hungry; a 64 GB host (c6a.8xlarge or larger) is recommended. Queries use ClickHouse-style regex backreferences (\1) since the regex evaluation runs inside libch.so, as anticipated in the spark-gluten/ README. Co-Authored-By: Claude Opus 4.7 (1M context) --- spark-gluten-clickhouse/README.md | 29 +++++ spark-gluten-clickhouse/benchmark.sh | 147 ++++++++++++++++++++++++++ spark-gluten-clickhouse/queries.sql | 43 ++++++++ spark-gluten-clickhouse/query.py | 68 ++++++++++++ spark-gluten-clickhouse/run.sh | 10 ++ spark-gluten-clickhouse/template.json | 13 +++ 6 files changed, 310 insertions(+) create mode 100644 spark-gluten-clickhouse/README.md create mode 100755 spark-gluten-clickhouse/benchmark.sh create mode 100644 spark-gluten-clickhouse/queries.sql create mode 100755 spark-gluten-clickhouse/query.py create mode 100755 spark-gluten-clickhouse/run.sh create mode 100644 spark-gluten-clickhouse/template.json diff --git a/spark-gluten-clickhouse/README.md b/spark-gluten-clickhouse/README.md new file mode 100644 index 0000000000..6dc439adac --- /dev/null +++ b/spark-gluten-clickhouse/README.md @@ -0,0 +1,29 @@ +This entry runs Apache Spark with the [Apache Gluten](https://gluten.apache.org/) plugin configured to use the **ClickHouse backend** ('ch'). Gluten loads `libch.so` (a fork of ClickHouse v23.1) into the Spark executor JVM and runs the columnar physical plan natively through it. See also [`spark-gluten/`](../spark-gluten/) (Velox backend) and the [accelerators README](../spark/README-accelerators.md). + +### Run + +`./benchmark.sh` builds everything from source (no pre-built bundle is published for the CH backend) and then runs all 43 queries. Optional first argument is the machine spec, e.g. `./benchmark.sh c6a.8xlarge`. + +## Notes + +### Build + +The CH backend is not part of Apache Gluten's release tarball — only the Velox bundle is published. As a result `benchmark.sh` builds two things from source: + +1. **`libch.so`** — built from [Kyligence/ClickHouse](https://github.com/Kyligence/ClickHouse) at the branch pinned in `gluten/cpp-ch/clickhouse.version`. The build uses Clang 18 / cmake / ninja. +2. **The Gluten Spark plugin** — built via Maven with `-P backends-clickhouse,spark-3.5`. JDK 8 is required at compile time (Gluten's POM); Spark itself runs under JDK 17. + +Building libch.so essentially compiles ClickHouse from source: it is **memory-hungry** (Gluten's docs note that 64 GB RAM is recommended). On a c6a.4xlarge (32 GB RAM) the compile may OOM; use c6a.8xlarge or larger for a clean run. + +### Configuration + +- `spark.gluten.sql.columnar.backend.lib=ch` selects the ClickHouse backend over Velox. +- `spark.gluten.sql.columnar.libpath=` points to the native library. The build location is `gluten/cpp-ch/build_ch/utils/extern-local-engine/libch.so`; `benchmark.sh` symlinks it as `libch.so` in the entry directory. +- Memory is split 50/50 between Spark heap and Gluten off-heap, identical to the Velox entry — the CH backend also runs off-heap via JNI. +- Queries use ClickHouse-style regex backreferences (`\1`) rather than Spark's `$1`, since the regex evaluation happens inside libch.so. See the discussion in [`spark-gluten/README.md`](../spark-gluten/README.md) and [Gluten issue #7545](https://github.com/apache/incubator-gluten/issues/7545). + +### Links + +- [Gluten ClickHouse-backend getting started](https://gluten.apache.org/docs/get-started/ClickHouse/). +- [Gluten release page](https://gluten.apache.org/downloads/) (Velox bundles only). +- [Kyligence/ClickHouse fork](https://github.com/Kyligence/ClickHouse) (the source of libch.so). diff --git a/spark-gluten-clickhouse/benchmark.sh b/spark-gluten-clickhouse/benchmark.sh new file mode 100755 index 0000000000..8b19fb39cc --- /dev/null +++ b/spark-gluten-clickhouse/benchmark.sh @@ -0,0 +1,147 @@ +#!/bin/bash + +# Spark + Apache Gluten with the ClickHouse backend ('ch'). Unlike the +# Velox backend, no pre-built bundle is published for the CH backend, so +# this script builds both libch.so (a ClickHouse fork) and the Gluten +# Spark plugin from source. +# +# Note: Keep in sync with spark-*/benchmark.sh (see README-accelerators.md for details) +# +# The ClickHouse compile is RAM-hungry; building on c6a.4xlarge (32 GB) +# may OOM. A larger machine (>= 64 GB RAM, c6a.8xlarge or above) is +# recommended. + +set -e + +GLUTEN_VERSION=v1.4.0 +SPARK_PROFILE=spark-3.5 + +# Install build prerequisites: +# - Java 8 to build Gluten via Maven (Gluten's pom requires JDK 8) +# - Java 17 to run Spark (auto-selected via JAVA_HOME below) +# - Clang 18, cmake, ninja, etc. to build libch.so +sudo apt-get update -y +sudo apt-get install -y python3-pip python3-venv \ + openjdk-8-jdk-headless openjdk-17-jdk-headless \ + maven git cmake ccache ninja-build nasm yasm gawk \ + lsb-release wget software-properties-common gnupg + +# Install Clang 18 (required by libch.so build). +wget -O - https://apt.llvm.org/llvm.sh | sudo bash -s -- 18 + +export CC=clang-18 +export CXX=clang++-18 + +# pyspark venv +python3 -m venv myenv +source myenv/bin/activate +pip install pyspark==3.5.2 psutil + +# Load the data +../download-hits-parquet-single + +# Clone Gluten and the Kyligence ClickHouse fork that the CH backend wraps. +GLUTEN_DIR="$PWD/gluten" +if [ ! -d "$GLUTEN_DIR" ]; then + git clone --depth 1 --branch "$GLUTEN_VERSION" \ + https://github.com/apache/gluten.git "$GLUTEN_DIR" +fi + +CH_BRANCH=$(grep '^CH_BRANCH=' "$GLUTEN_DIR/cpp-ch/clickhouse.version" | cut -d= -f2) +CH_DIR="$GLUTEN_DIR/cpp-ch/ClickHouse" +if [ ! -d "$CH_DIR" ]; then + git clone --recursive --shallow-submodules \ + --branch "$CH_BRANCH" \ + https://github.com/Kyligence/ClickHouse.git "$CH_DIR" +fi + +# Build libch.so. The wrapper at cpp-ch/build_ch invokes the inner +# ClickHouse build, whose final artifact ends up at cpp-ch/build/. +LIBCH_SO="$GLUTEN_DIR/cpp-ch/build/utils/extern-local-engine/libch.so" +if [ ! -f "$LIBCH_SO" ]; then + bash "$GLUTEN_DIR/ep/build-clickhouse/src/build_clickhouse.sh" +fi + +# Build the Gluten Spark plugin against the CH backend. JDK 8 is required +# at compile time per Gluten's pom; Spark itself runs under JDK 17 below. +# pyspark wheels ship Scala 2.12 jars, so build with scala-2.12 to match. +JAVA_HOME_8="/usr/lib/jvm/java-8-openjdk-$(dpkg --print-architecture)" +( + cd "$GLUTEN_DIR" + JAVA_HOME="$JAVA_HOME_8" PATH="$JAVA_HOME_8/bin:$PATH" \ + mvn -B clean package \ + -Pbackends-clickhouse -P"$SPARK_PROFILE" -Pscala-2.12 \ + -DskipTests -Dcheckstyle.skip +) + +# Symlink the produced uber jar (jar-with-dependencies) and libch.so into +# the entry directory; query.py expects them as ./gluten.jar and ./libch.so. +GLUTEN_JAR=$(ls "$GLUTEN_DIR"/backends-clickhouse/target/gluten-*-spark-3.5-jar-with-dependencies.jar 2>/dev/null | head -n1) +if [ -z "$GLUTEN_JAR" ]; then + echo "ERROR: could not locate built Gluten CH-backend jar" >&2 + ls "$GLUTEN_DIR/backends-clickhouse/target/" >&2 || true + exit 1 +fi +ln -sf "$GLUTEN_JAR" gluten.jar +ln -sf "$LIBCH_SO" libch.so + +# Run Spark queries under JDK 17. +export JAVA_HOME="/usr/lib/jvm/java-17-openjdk-$(dpkg --print-architecture)/" +export PATH="$JAVA_HOME/bin:$PATH" + +./run.sh 2>&1 | tee log.txt + +# Print results to stdout as required +cat log.txt | grep -P '^Time:\s+([\d\.]+)|Failure!' | sed -r -e 's/Time: //; s/^Failure!$/null/' | + awk '{ if (i % 3 == 0) { printf "[" }; printf $1; if (i % 3 != 2) { printf "," } else { print "]," }; ++i; }' + +DATA_SIZE=$(du -b hits.parquet | cut -f1) + +echo "Data size: $DATA_SIZE" +echo "Load time: 0" + +# Save results as JSON +MACHINE="${1:-c6a.8xlarge}" +SPARK_VERSION=$(pip freeze | grep '^pyspark==' | cut -d '=' -f3) +GLUTEN_TAG="${GLUTEN_VERSION#v}" + +mkdir -p results + +( +cat << EOF +{ + "system": "Spark (Gluten-on-ClickHouse)", + "date": "$(date +%Y-%m-%d)", + "machine": "${MACHINE}", + "cluster_size": 1, + "proprietary": "no", + "tuned": "no", + "comment": "Apache Gluten ${GLUTEN_TAG} with the ClickHouse backend (libch.so), Spark ${SPARK_VERSION}", + "tags": ["Java", "C++", "column-oriented", "Spark derivative", "ClickHouse", "Parquet"], + "load_time": 0, + "data_size": ${DATA_SIZE}, + "result": [ +EOF + +cat log.txt | grep -P '^Time:\s+([\d\.]+)|Failure!' | sed -r -e 's/Time: //; s/^Failure!$/null/' | + awk -v total=$(grep -cP '^Time:\s+[\d\.]+|Failure!' log.txt) ' + { + if (i % 3 == 0) printf "\t\t["; + if ($1 == "null") printf "null"; + else printf "%.3f", $1; + if (i % 3 != 2) printf ", "; + else { + if (i < total - 1) printf "],\n"; + else printf "]"; + } + i++; + }' + +cat << EOF + + ] +} +EOF +) > "results/${MACHINE}.json" + +echo "Results have been saved to results/${MACHINE}.json" diff --git a/spark-gluten-clickhouse/queries.sql b/spark-gluten-clickhouse/queries.sql new file mode 100644 index 0000000000..31f65fc898 --- /dev/null +++ b/spark-gluten-clickhouse/queries.sql @@ -0,0 +1,43 @@ +SELECT COUNT(*) FROM hits; +SELECT COUNT(*) FROM hits WHERE AdvEngineID <> 0; +SELECT SUM(AdvEngineID), COUNT(*), AVG(ResolutionWidth) FROM hits; +SELECT AVG(UserID) FROM hits; +SELECT COUNT(DISTINCT UserID) FROM hits; +SELECT COUNT(DISTINCT SearchPhrase) FROM hits; +SELECT MIN(EventDate), MAX(EventDate) FROM hits; +SELECT AdvEngineID, COUNT(*) FROM hits WHERE AdvEngineID <> 0 GROUP BY AdvEngineID ORDER BY COUNT(*) DESC; +SELECT RegionID, COUNT(DISTINCT UserID) AS u FROM hits GROUP BY RegionID ORDER BY u DESC LIMIT 10; +SELECT RegionID, SUM(AdvEngineID), COUNT(*) AS c, AVG(ResolutionWidth), COUNT(DISTINCT UserID) FROM hits GROUP BY RegionID ORDER BY c DESC LIMIT 10; +SELECT MobilePhoneModel, COUNT(DISTINCT UserID) AS u FROM hits WHERE MobilePhoneModel <> '' GROUP BY MobilePhoneModel ORDER BY u DESC LIMIT 10; +SELECT MobilePhone, MobilePhoneModel, COUNT(DISTINCT UserID) AS u FROM hits WHERE MobilePhoneModel <> '' GROUP BY MobilePhone, MobilePhoneModel ORDER BY u DESC LIMIT 10; +SELECT SearchPhrase, COUNT(*) AS c FROM hits WHERE SearchPhrase <> '' GROUP BY SearchPhrase ORDER BY c DESC LIMIT 10; +SELECT SearchPhrase, COUNT(DISTINCT UserID) AS u FROM hits WHERE SearchPhrase <> '' GROUP BY SearchPhrase ORDER BY u DESC LIMIT 10; +SELECT SearchEngineID, SearchPhrase, COUNT(*) AS c FROM hits WHERE SearchPhrase <> '' GROUP BY SearchEngineID, SearchPhrase ORDER BY c DESC LIMIT 10; +SELECT UserID, COUNT(*) FROM hits GROUP BY UserID ORDER BY COUNT(*) DESC LIMIT 10; +SELECT UserID, SearchPhrase, COUNT(*) FROM hits GROUP BY UserID, SearchPhrase ORDER BY COUNT(*) DESC LIMIT 10; +SELECT UserID, SearchPhrase, COUNT(*) FROM hits GROUP BY UserID, SearchPhrase LIMIT 10; +SELECT UserID, extract(minute FROM EventTime) AS m, SearchPhrase, COUNT(*) FROM hits GROUP BY UserID, m, SearchPhrase ORDER BY COUNT(*) DESC LIMIT 10; +SELECT UserID FROM hits WHERE UserID = 435090932899640449; +SELECT COUNT(*) FROM hits WHERE URL LIKE '%google%'; +SELECT SearchPhrase, MIN(URL), COUNT(*) AS c FROM hits WHERE URL LIKE '%google%' AND SearchPhrase <> '' GROUP BY SearchPhrase ORDER BY c DESC LIMIT 10; +SELECT SearchPhrase, MIN(URL), MIN(Title), COUNT(*) AS c, COUNT(DISTINCT UserID) FROM hits WHERE Title LIKE '%Google%' AND URL NOT LIKE '%.google.%' AND SearchPhrase <> '' GROUP BY SearchPhrase ORDER BY c DESC LIMIT 10; +SELECT * FROM hits WHERE URL LIKE '%google%' ORDER BY EventTime LIMIT 10; +SELECT SearchPhrase FROM hits WHERE SearchPhrase <> '' ORDER BY EventTime LIMIT 10; +SELECT SearchPhrase FROM hits WHERE SearchPhrase <> '' ORDER BY SearchPhrase LIMIT 10; +SELECT SearchPhrase FROM hits WHERE SearchPhrase <> '' ORDER BY EventTime, SearchPhrase LIMIT 10; +SELECT CounterID, AVG(length(URL)) AS l, COUNT(*) AS c FROM hits WHERE URL <> '' GROUP BY CounterID HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25; +SELECT REGEXP_REPLACE(Referer, '^https?://(?:www\.)?([^/]+)/.*$', '\1') AS k, AVG(length(Referer)) AS l, COUNT(*) AS c, MIN(Referer) FROM hits WHERE Referer <> '' GROUP BY k HAVING COUNT(*) > 100000 ORDER BY l DESC LIMIT 25; +SELECT SUM(ResolutionWidth), SUM(ResolutionWidth + 1), SUM(ResolutionWidth + 2), SUM(ResolutionWidth + 3), SUM(ResolutionWidth + 4), SUM(ResolutionWidth + 5), SUM(ResolutionWidth + 6), SUM(ResolutionWidth + 7), SUM(ResolutionWidth + 8), SUM(ResolutionWidth + 9), SUM(ResolutionWidth + 10), SUM(ResolutionWidth + 11), SUM(ResolutionWidth + 12), SUM(ResolutionWidth + 13), SUM(ResolutionWidth + 14), SUM(ResolutionWidth + 15), SUM(ResolutionWidth + 16), SUM(ResolutionWidth + 17), SUM(ResolutionWidth + 18), SUM(ResolutionWidth + 19), SUM(ResolutionWidth + 20), SUM(ResolutionWidth + 21), SUM(ResolutionWidth + 22), SUM(ResolutionWidth + 23), SUM(ResolutionWidth + 24), SUM(ResolutionWidth + 25), SUM(ResolutionWidth + 26), SUM(ResolutionWidth + 27), SUM(ResolutionWidth + 28), SUM(ResolutionWidth + 29), SUM(ResolutionWidth + 30), SUM(ResolutionWidth + 31), SUM(ResolutionWidth + 32), SUM(ResolutionWidth + 33), SUM(ResolutionWidth + 34), SUM(ResolutionWidth + 35), SUM(ResolutionWidth + 36), SUM(ResolutionWidth + 37), SUM(ResolutionWidth + 38), SUM(ResolutionWidth + 39), SUM(ResolutionWidth + 40), SUM(ResolutionWidth + 41), SUM(ResolutionWidth + 42), SUM(ResolutionWidth + 43), SUM(ResolutionWidth + 44), SUM(ResolutionWidth + 45), SUM(ResolutionWidth + 46), SUM(ResolutionWidth + 47), SUM(ResolutionWidth + 48), SUM(ResolutionWidth + 49), SUM(ResolutionWidth + 50), SUM(ResolutionWidth + 51), SUM(ResolutionWidth + 52), SUM(ResolutionWidth + 53), SUM(ResolutionWidth + 54), SUM(ResolutionWidth + 55), SUM(ResolutionWidth + 56), SUM(ResolutionWidth + 57), SUM(ResolutionWidth + 58), SUM(ResolutionWidth + 59), SUM(ResolutionWidth + 60), SUM(ResolutionWidth + 61), SUM(ResolutionWidth + 62), SUM(ResolutionWidth + 63), SUM(ResolutionWidth + 64), SUM(ResolutionWidth + 65), SUM(ResolutionWidth + 66), SUM(ResolutionWidth + 67), SUM(ResolutionWidth + 68), SUM(ResolutionWidth + 69), SUM(ResolutionWidth + 70), SUM(ResolutionWidth + 71), SUM(ResolutionWidth + 72), SUM(ResolutionWidth + 73), SUM(ResolutionWidth + 74), SUM(ResolutionWidth + 75), SUM(ResolutionWidth + 76), SUM(ResolutionWidth + 77), SUM(ResolutionWidth + 78), SUM(ResolutionWidth + 79), SUM(ResolutionWidth + 80), SUM(ResolutionWidth + 81), SUM(ResolutionWidth + 82), SUM(ResolutionWidth + 83), SUM(ResolutionWidth + 84), SUM(ResolutionWidth + 85), SUM(ResolutionWidth + 86), SUM(ResolutionWidth + 87), SUM(ResolutionWidth + 88), SUM(ResolutionWidth + 89) FROM hits; +SELECT SearchEngineID, ClientIP, COUNT(*) AS c, SUM(IsRefresh), AVG(ResolutionWidth) FROM hits WHERE SearchPhrase <> '' GROUP BY SearchEngineID, ClientIP ORDER BY c DESC LIMIT 10; +SELECT WatchID, ClientIP, COUNT(*) AS c, SUM(IsRefresh), AVG(ResolutionWidth) FROM hits WHERE SearchPhrase <> '' GROUP BY WatchID, ClientIP ORDER BY c DESC LIMIT 10; +SELECT WatchID, ClientIP, COUNT(*) AS c, SUM(IsRefresh), AVG(ResolutionWidth) FROM hits GROUP BY WatchID, ClientIP ORDER BY c DESC LIMIT 10; +SELECT URL, COUNT(*) AS c FROM hits GROUP BY URL ORDER BY c DESC LIMIT 10; +SELECT 1, URL, COUNT(*) AS c FROM hits GROUP BY 1, URL ORDER BY c DESC LIMIT 10; +SELECT ClientIP, ClientIP - 1, ClientIP - 2, ClientIP - 3, COUNT(*) AS c FROM hits GROUP BY ClientIP, ClientIP - 1, ClientIP - 2, ClientIP - 3 ORDER BY c DESC LIMIT 10; +SELECT URL, COUNT(*) AS PageViews FROM hits WHERE CounterID = 62 AND EventDate >= '2013-07-01' AND EventDate <= '2013-07-31' AND DontCountHits = 0 AND IsRefresh = 0 AND URL <> '' GROUP BY URL ORDER BY PageViews DESC LIMIT 10; +SELECT Title, COUNT(*) AS PageViews FROM hits WHERE CounterID = 62 AND EventDate >= '2013-07-01' AND EventDate <= '2013-07-31' AND DontCountHits = 0 AND IsRefresh = 0 AND Title <> '' GROUP BY Title ORDER BY PageViews DESC LIMIT 10; +SELECT URL, COUNT(*) AS PageViews FROM hits WHERE CounterID = 62 AND EventDate >= '2013-07-01' AND EventDate <= '2013-07-31' AND IsRefresh = 0 AND IsLink <> 0 AND IsDownload = 0 GROUP BY URL ORDER BY PageViews DESC LIMIT 10 OFFSET 1000; +SELECT TraficSourceID, SearchEngineID, AdvEngineID, CASE WHEN (SearchEngineID = 0 AND AdvEngineID = 0) THEN Referer ELSE '' END AS Src, URL AS Dst, COUNT(*) AS PageViews FROM hits WHERE CounterID = 62 AND EventDate >= '2013-07-01' AND EventDate <= '2013-07-31' AND IsRefresh = 0 GROUP BY TraficSourceID, SearchEngineID, AdvEngineID, Src, Dst ORDER BY PageViews DESC LIMIT 10 OFFSET 1000; +SELECT URLHash, EventDate, COUNT(*) AS PageViews FROM hits WHERE CounterID = 62 AND EventDate >= '2013-07-01' AND EventDate <= '2013-07-31' AND IsRefresh = 0 AND TraficSourceID IN (-1, 6) AND RefererHash = 3594120000172545465 GROUP BY URLHash, EventDate ORDER BY PageViews DESC LIMIT 10 OFFSET 100; +SELECT WindowClientWidth, WindowClientHeight, COUNT(*) AS PageViews FROM hits WHERE CounterID = 62 AND EventDate >= '2013-07-01' AND EventDate <= '2013-07-31' AND IsRefresh = 0 AND DontCountHits = 0 AND URLHash = 2868770270353813622 GROUP BY WindowClientWidth, WindowClientHeight ORDER BY PageViews DESC LIMIT 10 OFFSET 10000; +SELECT DATE_TRUNC('minute', EventTime) AS M, COUNT(*) AS PageViews FROM hits WHERE CounterID = 62 AND EventDate >= '2013-07-14' AND EventDate <= '2013-07-15' AND IsRefresh = 0 AND DontCountHits = 0 GROUP BY DATE_TRUNC('minute', EventTime) ORDER BY DATE_TRUNC('minute', EventTime) LIMIT 10 OFFSET 1000; diff --git a/spark-gluten-clickhouse/query.py b/spark-gluten-clickhouse/query.py new file mode 100755 index 0000000000..5c43a4efb8 --- /dev/null +++ b/spark-gluten-clickhouse/query.py @@ -0,0 +1,68 @@ +#!/usr/bin/env python3 + +""" +Spark + Apache Gluten using the ClickHouse backend ('ch'). The CH backend +loads libch.so (a fork of ClickHouse v23.1) into the Spark executor JVM +and runs the columnar physical plan natively. + +Note: Keep in sync with spark-*/query.py (see README-accelerators.md for details). +""" + +import os +import sys +import timeit + +import psutil +from pyspark.sql import SparkSession +import pyspark.sql.functions as F + + +query = sys.stdin.read() +print(query) + +# Calculate available memory to configure SparkSession (in MB). +# The CH backend runs off-heap (via JNI into libch.so), so split available +# memory between Spark's JVM heap and the off-heap pool the same way the +# Velox backend does. +ram = int(round(psutil.virtual_memory().available / (1024 ** 2) * 0.7)) +heap = ram // 2 +off_heap = ram - heap +print(f"SparkSession will use {heap} MB of heap and {off_heap} MB of off-heap memory (total {ram} MB)") + +builder = ( + SparkSession + .builder + .appName("ClickBench") + .config("spark.driver", "local[*]") + .config("spark.driver.memory", f"{heap}m") + .config("spark.sql.parquet.binaryAsString", True) + + # Gluten + ClickHouse backend configuration + .config("spark.jars", "gluten.jar") + .config("spark.driver.extraClassPath", "gluten.jar") + .config("spark.plugins", "org.apache.gluten.GlutenPlugin") + .config("spark.shuffle.manager", "org.apache.spark.shuffle.sort.ColumnarShuffleManager") + .config("spark.gluten.sql.columnar.backend.lib", "ch") + .config("spark.gluten.sql.columnar.libpath", os.path.abspath("libch.so")) + .config("spark.memory.offHeap.enabled", "true") + .config("spark.memory.offHeap.size", f"{off_heap}m") + .config("spark.driver.extraJavaOptions", "-Dio.netty.tryReflectionSetAccessible=true") +) + +spark = builder.getOrCreate() + +df = spark.read.parquet("hits.parquet") +df = df.withColumn("EventTime", F.col("EventTime").cast("timestamp")) +df = df.withColumn("EventDate", F.date_add(F.lit("1970-01-01"), F.col("EventDate"))) +df.createOrReplaceTempView("hits") + +for try_num in range(3): + try: + start = timeit.default_timer() + result = spark.sql(query) + result.show(100) + end = timeit.default_timer() + print("Time: ", end - start) + except Exception as e: + print(e) + print("Failure!") diff --git a/spark-gluten-clickhouse/run.sh b/spark-gluten-clickhouse/run.sh new file mode 100755 index 0000000000..8c9ca12890 --- /dev/null +++ b/spark-gluten-clickhouse/run.sh @@ -0,0 +1,10 @@ +#!/bin/bash + +# Note: Keep in sync with spark-*/run.sh (see README-accelerators.md for details) + +cat queries.sql | while read query; do + sync + echo 3 | sudo tee /proc/sys/vm/drop_caches >/dev/null + + ./query.py <<< "${query}" +done diff --git a/spark-gluten-clickhouse/template.json b/spark-gluten-clickhouse/template.json new file mode 100644 index 0000000000..6ae1ad2873 --- /dev/null +++ b/spark-gluten-clickhouse/template.json @@ -0,0 +1,13 @@ +{ + "system": "Spark (Gluten-on-ClickHouse)", + "proprietary": "no", + "hardware": "cpu", + "tuned": "no", + "tags": [ + "Java", + "C++", + "column-oriented", + "Spark derivative", + "ClickHouse" + ] +} From fe4e87fe92b86ceed85a006fbfb6bbb3362e3aca Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Fri, 8 May 2026 22:34:43 +0000 Subject: [PATCH 02/20] spark-gluten-clickhouse: bump Clang to 19 for libch.so build The pinned Kyligence/ClickHouse fork now rejects Clang < 19 in cmake/tools.cmake, so installing Clang 18 fails the configure step. Co-Authored-By: Claude Opus 4.7 (1M context) --- spark-gluten-clickhouse/benchmark.sh | 11 ++++++----- 1 file changed, 6 insertions(+), 5 deletions(-) diff --git a/spark-gluten-clickhouse/benchmark.sh b/spark-gluten-clickhouse/benchmark.sh index 8b19fb39cc..558d4d85b3 100755 --- a/spark-gluten-clickhouse/benchmark.sh +++ b/spark-gluten-clickhouse/benchmark.sh @@ -19,18 +19,19 @@ SPARK_PROFILE=spark-3.5 # Install build prerequisites: # - Java 8 to build Gluten via Maven (Gluten's pom requires JDK 8) # - Java 17 to run Spark (auto-selected via JAVA_HOME below) -# - Clang 18, cmake, ninja, etc. to build libch.so +# - Clang 19, cmake, ninja, etc. to build libch.so sudo apt-get update -y sudo apt-get install -y python3-pip python3-venv \ openjdk-8-jdk-headless openjdk-17-jdk-headless \ maven git cmake ccache ninja-build nasm yasm gawk \ lsb-release wget software-properties-common gnupg -# Install Clang 18 (required by libch.so build). -wget -O - https://apt.llvm.org/llvm.sh | sudo bash -s -- 18 +# Install Clang 19 (required by libch.so build — the pinned Kyligence/ClickHouse +# fork's cmake/tools.cmake rejects Clang < 19). +wget -O - https://apt.llvm.org/llvm.sh | sudo bash -s -- 19 -export CC=clang-18 -export CXX=clang++-18 +export CC=clang-19 +export CXX=clang++-19 # pyspark venv python3 -m venv myenv From 6443592473e32d93bf7be90c5a9932d988167f16 Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Wed, 22 Jul 2026 21:20:37 +0000 Subject: [PATCH 03/20] spark-gluten-clickhouse: convert to the shared benchmark-common harness MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The entry was written in the old self-contained benchmark.sh style, which predates lib/benchmark-common.sh. The c6a.4xlarge run (PR #861) failed immediately at `../download-hits-parquet-single: No such file or directory` — the download helpers moved into lib/ and are now invoked by the harness, not by each entry. Refactor to match the sibling spark-gluten/ (Velox) and spark-velox/ entries: - benchmark.sh: thin wrapper that sets BENCH_* env and exec's ../lib/benchmark-common.sh (RESTARTABLE=no, concurrent QPS skipped per #946). - install: the from-source build (libch.so + Gluten CH plugin), moved out of benchmark.sh. Fixes verified against Gluten v1.4.0's own build_clickhouse.sh: * clone apache/incubator-gluten (canonical repo, matches docs) at v1.4.0; * read CH_ORG as well as CH_BRANCH from cpp-ch/clickhouse.version; * the build lands in cpp-ch/build_ch/ (not build/ as the old path assumed and the prose docs still say) — glob for libch.so instead of hardcoding; * set MAVEN_OPTS per the Gluten docs so the Maven build doesn't OOM; * idempotent (skips clone/build when artifacts already present). - load/query/start/stop/check/data-size: standard harness scripts. query runs Spark under JDK 17 (Gluten is built under JDK 8). - query.py: single run per invocation, timing as the last stderr line, exit 1 on failure — the harness contract, replacing the old 3-try/"Time:" loop. - Drop run.sh (the harness drives the query loop). - .gitignore: __pycache__/ and the from-source build artifacts. Co-Authored-By: Claude Opus 4.8 (1M context) --- .gitignore | 6 ++ spark-gluten-clickhouse/README.md | 16 +-- spark-gluten-clickhouse/benchmark.sh | 155 ++------------------------- spark-gluten-clickhouse/check | 8 ++ spark-gluten-clickhouse/data-size | 4 + spark-gluten-clickhouse/install | 101 +++++++++++++++++ spark-gluten-clickhouse/load | 6 ++ spark-gluten-clickhouse/query | 13 +++ spark-gluten-clickhouse/query.py | 33 +++--- spark-gluten-clickhouse/run.sh | 10 -- spark-gluten-clickhouse/start | 2 + spark-gluten-clickhouse/stop | 2 + 12 files changed, 179 insertions(+), 177 deletions(-) create mode 100755 spark-gluten-clickhouse/check create mode 100755 spark-gluten-clickhouse/data-size create mode 100755 spark-gluten-clickhouse/install create mode 100755 spark-gluten-clickhouse/load create mode 100755 spark-gluten-clickhouse/query delete mode 100755 spark-gluten-clickhouse/run.sh create mode 100755 spark-gluten-clickhouse/start create mode 100755 spark-gluten-clickhouse/stop diff --git a/.gitignore b/.gitignore index 00320283df..323bf20ab6 100644 --- a/.gitignore +++ b/.gitignore @@ -29,3 +29,9 @@ hits.vortex # Python venvs created by install scripts myenv +__pycache__/ + +# spark-gluten-clickhouse: from-source build tree and symlinked artifacts +gluten/ +gluten.jar +libch.so diff --git a/spark-gluten-clickhouse/README.md b/spark-gluten-clickhouse/README.md index 6dc439adac..3ed249e194 100644 --- a/spark-gluten-clickhouse/README.md +++ b/spark-gluten-clickhouse/README.md @@ -1,24 +1,28 @@ -This entry runs Apache Spark with the [Apache Gluten](https://gluten.apache.org/) plugin configured to use the **ClickHouse backend** ('ch'). Gluten loads `libch.so` (a fork of ClickHouse v23.1) into the Spark executor JVM and runs the columnar physical plan natively through it. See also [`spark-gluten/`](../spark-gluten/) (Velox backend) and the [accelerators README](../spark/README-accelerators.md). +This entry runs Apache Spark with the [Apache Gluten](https://gluten.apache.org/) plugin configured to use the **ClickHouse backend** ('ch'). Gluten loads `libch.so` (a fork of ClickHouse v23.1) into the Spark executor JVM and runs the columnar physical plan natively through it. See also [`spark-gluten/`](../spark-gluten/) (Velox backend), [`spark-velox/`](../spark-velox/), and the [accelerators README](../spark/README-accelerators.md). ### Run -`./benchmark.sh` builds everything from source (no pre-built bundle is published for the CH backend) and then runs all 43 queries. Optional first argument is the machine spec, e.g. `./benchmark.sh c6a.8xlarge`. +`./benchmark.sh` sets a few env vars and delegates to the shared driver +[`../lib/benchmark-common.sh`](../lib/benchmark-common.sh), which runs the +per-system scripts (`install`, `load`, `query`, ...) and prints the results in +the format collected by play.clickhouse.com. `./install` builds everything from +source (no pre-built bundle is published for the CH backend). ## Notes ### Build -The CH backend is not part of Apache Gluten's release tarball — only the Velox bundle is published. As a result `benchmark.sh` builds two things from source: +The CH backend is not part of Apache Gluten's release tarball — only the Velox bundle is published. As a result `install` builds two things from source: -1. **`libch.so`** — built from [Kyligence/ClickHouse](https://github.com/Kyligence/ClickHouse) at the branch pinned in `gluten/cpp-ch/clickhouse.version`. The build uses Clang 18 / cmake / ninja. -2. **The Gluten Spark plugin** — built via Maven with `-P backends-clickhouse,spark-3.5`. JDK 8 is required at compile time (Gluten's POM); Spark itself runs under JDK 17. +1. **`libch.so`** — built from [Kyligence/ClickHouse](https://github.com/Kyligence/ClickHouse) at the org/branch/commit pinned in `gluten/cpp-ch/clickhouse.version`. The build uses Clang 19 / cmake / ninja (Gluten v1.4.0's CH backend requires Clang 19). +2. **The Gluten Spark plugin** — built via Maven with `-Pbackends-clickhouse -Pspark-3.5 -Pscala-2.12`. JDK 8 is required at compile time (Gluten's POM); Spark itself runs under JDK 17 (see `./query`). Building libch.so essentially compiles ClickHouse from source: it is **memory-hungry** (Gluten's docs note that 64 GB RAM is recommended). On a c6a.4xlarge (32 GB RAM) the compile may OOM; use c6a.8xlarge or larger for a clean run. ### Configuration - `spark.gluten.sql.columnar.backend.lib=ch` selects the ClickHouse backend over Velox. -- `spark.gluten.sql.columnar.libpath=` points to the native library. The build location is `gluten/cpp-ch/build_ch/utils/extern-local-engine/libch.so`; `benchmark.sh` symlinks it as `libch.so` in the entry directory. +- `spark.gluten.sql.columnar.libpath=` points to the native library. Gluten v1.4.0's build script produces it under `gluten/cpp-ch/build_ch/.../extern-local-engine/libch.so` (the prose docs still say `build/`); `install` globs for it and symlinks it as `libch.so` in the entry directory. - Memory is split 50/50 between Spark heap and Gluten off-heap, identical to the Velox entry — the CH backend also runs off-heap via JNI. - Queries use ClickHouse-style regex backreferences (`\1`) rather than Spark's `$1`, since the regex evaluation happens inside libch.so. See the discussion in [`spark-gluten/README.md`](../spark-gluten/README.md) and [Gluten issue #7545](https://github.com/apache/incubator-gluten/issues/7545). diff --git a/spark-gluten-clickhouse/benchmark.sh b/spark-gluten-clickhouse/benchmark.sh index 558d4d85b3..fb3b4d1318 100755 --- a/spark-gluten-clickhouse/benchmark.sh +++ b/spark-gluten-clickhouse/benchmark.sh @@ -1,148 +1,9 @@ #!/bin/bash - -# Spark + Apache Gluten with the ClickHouse backend ('ch'). Unlike the -# Velox backend, no pre-built bundle is published for the CH backend, so -# this script builds both libch.so (a ClickHouse fork) and the Gluten -# Spark plugin from source. -# -# Note: Keep in sync with spark-*/benchmark.sh (see README-accelerators.md for details) -# -# The ClickHouse compile is RAM-hungry; building on c6a.4xlarge (32 GB) -# may OOM. A larger machine (>= 64 GB RAM, c6a.8xlarge or above) is -# recommended. - -set -e - -GLUTEN_VERSION=v1.4.0 -SPARK_PROFILE=spark-3.5 - -# Install build prerequisites: -# - Java 8 to build Gluten via Maven (Gluten's pom requires JDK 8) -# - Java 17 to run Spark (auto-selected via JAVA_HOME below) -# - Clang 19, cmake, ninja, etc. to build libch.so -sudo apt-get update -y -sudo apt-get install -y python3-pip python3-venv \ - openjdk-8-jdk-headless openjdk-17-jdk-headless \ - maven git cmake ccache ninja-build nasm yasm gawk \ - lsb-release wget software-properties-common gnupg - -# Install Clang 19 (required by libch.so build — the pinned Kyligence/ClickHouse -# fork's cmake/tools.cmake rejects Clang < 19). -wget -O - https://apt.llvm.org/llvm.sh | sudo bash -s -- 19 - -export CC=clang-19 -export CXX=clang++-19 - -# pyspark venv -python3 -m venv myenv -source myenv/bin/activate -pip install pyspark==3.5.2 psutil - -# Load the data -../download-hits-parquet-single - -# Clone Gluten and the Kyligence ClickHouse fork that the CH backend wraps. -GLUTEN_DIR="$PWD/gluten" -if [ ! -d "$GLUTEN_DIR" ]; then - git clone --depth 1 --branch "$GLUTEN_VERSION" \ - https://github.com/apache/gluten.git "$GLUTEN_DIR" -fi - -CH_BRANCH=$(grep '^CH_BRANCH=' "$GLUTEN_DIR/cpp-ch/clickhouse.version" | cut -d= -f2) -CH_DIR="$GLUTEN_DIR/cpp-ch/ClickHouse" -if [ ! -d "$CH_DIR" ]; then - git clone --recursive --shallow-submodules \ - --branch "$CH_BRANCH" \ - https://github.com/Kyligence/ClickHouse.git "$CH_DIR" -fi - -# Build libch.so. The wrapper at cpp-ch/build_ch invokes the inner -# ClickHouse build, whose final artifact ends up at cpp-ch/build/. -LIBCH_SO="$GLUTEN_DIR/cpp-ch/build/utils/extern-local-engine/libch.so" -if [ ! -f "$LIBCH_SO" ]; then - bash "$GLUTEN_DIR/ep/build-clickhouse/src/build_clickhouse.sh" -fi - -# Build the Gluten Spark plugin against the CH backend. JDK 8 is required -# at compile time per Gluten's pom; Spark itself runs under JDK 17 below. -# pyspark wheels ship Scala 2.12 jars, so build with scala-2.12 to match. -JAVA_HOME_8="/usr/lib/jvm/java-8-openjdk-$(dpkg --print-architecture)" -( - cd "$GLUTEN_DIR" - JAVA_HOME="$JAVA_HOME_8" PATH="$JAVA_HOME_8/bin:$PATH" \ - mvn -B clean package \ - -Pbackends-clickhouse -P"$SPARK_PROFILE" -Pscala-2.12 \ - -DskipTests -Dcheckstyle.skip -) - -# Symlink the produced uber jar (jar-with-dependencies) and libch.so into -# the entry directory; query.py expects them as ./gluten.jar and ./libch.so. -GLUTEN_JAR=$(ls "$GLUTEN_DIR"/backends-clickhouse/target/gluten-*-spark-3.5-jar-with-dependencies.jar 2>/dev/null | head -n1) -if [ -z "$GLUTEN_JAR" ]; then - echo "ERROR: could not locate built Gluten CH-backend jar" >&2 - ls "$GLUTEN_DIR/backends-clickhouse/target/" >&2 || true - exit 1 -fi -ln -sf "$GLUTEN_JAR" gluten.jar -ln -sf "$LIBCH_SO" libch.so - -# Run Spark queries under JDK 17. -export JAVA_HOME="/usr/lib/jvm/java-17-openjdk-$(dpkg --print-architecture)/" -export PATH="$JAVA_HOME/bin:$PATH" - -./run.sh 2>&1 | tee log.txt - -# Print results to stdout as required -cat log.txt | grep -P '^Time:\s+([\d\.]+)|Failure!' | sed -r -e 's/Time: //; s/^Failure!$/null/' | - awk '{ if (i % 3 == 0) { printf "[" }; printf $1; if (i % 3 != 2) { printf "," } else { print "]," }; ++i; }' - -DATA_SIZE=$(du -b hits.parquet | cut -f1) - -echo "Data size: $DATA_SIZE" -echo "Load time: 0" - -# Save results as JSON -MACHINE="${1:-c6a.8xlarge}" -SPARK_VERSION=$(pip freeze | grep '^pyspark==' | cut -d '=' -f3) -GLUTEN_TAG="${GLUTEN_VERSION#v}" - -mkdir -p results - -( -cat << EOF -{ - "system": "Spark (Gluten-on-ClickHouse)", - "date": "$(date +%Y-%m-%d)", - "machine": "${MACHINE}", - "cluster_size": 1, - "proprietary": "no", - "tuned": "no", - "comment": "Apache Gluten ${GLUTEN_TAG} with the ClickHouse backend (libch.so), Spark ${SPARK_VERSION}", - "tags": ["Java", "C++", "column-oriented", "Spark derivative", "ClickHouse", "Parquet"], - "load_time": 0, - "data_size": ${DATA_SIZE}, - "result": [ -EOF - -cat log.txt | grep -P '^Time:\s+([\d\.]+)|Failure!' | sed -r -e 's/Time: //; s/^Failure!$/null/' | - awk -v total=$(grep -cP '^Time:\s+[\d\.]+|Failure!' log.txt) ' - { - if (i % 3 == 0) printf "\t\t["; - if ($1 == "null") printf "null"; - else printf "%.3f", $1; - if (i % 3 != 2) printf ", "; - else { - if (i < total - 1) printf "],\n"; - else printf "]"; - } - i++; - }' - -cat << EOF - - ] -} -EOF -) > "results/${MACHINE}.json" - -echo "Results have been saved to results/${MACHINE}.json" +export BENCH_DOWNLOAD_SCRIPT="download-hits-parquet-single" +export BENCH_RESTARTABLE=no +# Single-process engine: each query forks a fresh full-machine process with no +# shared scheduler across connections, so the concurrent-QPS test only +# oversubscribes RAM rather than measuring throughput. Skip it by default; +# override BENCH_CONCURRENT_DURATION to re-enable. See issue #946. +export BENCH_CONCURRENT_DURATION="${BENCH_CONCURRENT_DURATION:-0}" +exec ../lib/benchmark-common.sh diff --git a/spark-gluten-clickhouse/check b/spark-gluten-clickhouse/check new file mode 100755 index 0000000000..45d397cda2 --- /dev/null +++ b/spark-gluten-clickhouse/check @@ -0,0 +1,8 @@ +#!/bin/bash +set -e + +# shellcheck disable=SC1091 +source myenv/bin/activate +python3 -c 'import pyspark' >/dev/null 2>&1 +[ -f gluten.jar ] +[ -f libch.so ] diff --git a/spark-gluten-clickhouse/data-size b/spark-gluten-clickhouse/data-size new file mode 100755 index 0000000000..1a34600a86 --- /dev/null +++ b/spark-gluten-clickhouse/data-size @@ -0,0 +1,4 @@ +#!/bin/bash +set -e + +du -b hits.parquet | cut -f1 diff --git a/spark-gluten-clickhouse/install b/spark-gluten-clickhouse/install new file mode 100755 index 0000000000..447f01eb80 --- /dev/null +++ b/spark-gluten-clickhouse/install @@ -0,0 +1,101 @@ +#!/bin/bash +set -e + +# Spark + Apache Gluten with the ClickHouse backend ('ch'). Unlike the Velox +# backend, no pre-built bundle is published for the CH backend, so we build +# both libch.so (a ClickHouse fork) and the Gluten Spark plugin from source. +# +# The libch.so compile is essentially a ClickHouse build: RAM-hungry (Gluten's +# docs recommend >= 64 GB) and slow. Run on c6a.8xlarge or larger; c6a.4xlarge +# (32 GB) may OOM. +# +# Note: Keep in sync with spark-*/install (see README-accelerators.md). + +GLUTEN_VERSION=v1.4.0 +SPARK_PROFILE=spark-3.5 + +# Build prerequisites: +# - Java 8 to build Gluten via Maven (Gluten's pom requires JDK 8) +# - Java 17 to run Spark (selected in ./query via JAVA_HOME) +# - Clang 19, cmake, ninja, etc. to build libch.so (Gluten v1.4.0's CH +# backend requires Clang 19; see get-started/ClickHouse.md). +sudo apt-get update -y +sudo apt-get install -y python3-pip python3-venv \ + openjdk-8-jdk-headless openjdk-17-jdk-headless \ + maven git cmake ccache ninja-build nasm yasm gawk \ + lsb-release wget software-properties-common gnupg + +# Install Clang 19 via apt.llvm.org. +if ! command -v clang-19 >/dev/null 2>&1; then + wget -O - https://apt.llvm.org/llvm.sh | sudo bash -s -- 19 +fi + +export CC=clang-19 +export CXX=clang++-19 + +# pyspark venv. +if [ ! -d myenv ]; then + python3 -m venv myenv +fi +# shellcheck disable=SC1091 +source myenv/bin/activate +pip install -q pyspark==3.5.2 psutil + +# Clone Gluten and the Kyligence ClickHouse fork that the CH backend wraps. +# The CH fork org/branch/commit are pinned in Gluten's cpp-ch/clickhouse.version. +GLUTEN_DIR="$PWD/gluten" +if [ ! -d "$GLUTEN_DIR" ]; then + git clone --depth 1 --branch "$GLUTEN_VERSION" \ + https://github.com/apache/incubator-gluten.git "$GLUTEN_DIR" +fi + +CH_VERSION_FILE="$GLUTEN_DIR/cpp-ch/clickhouse.version" +CH_ORG=$(grep '^CH_ORG=' "$CH_VERSION_FILE" | cut -d= -f2) +CH_BRANCH=$(grep '^CH_BRANCH=' "$CH_VERSION_FILE" | cut -d= -f2) +CH_DIR="$GLUTEN_DIR/cpp-ch/ClickHouse" +if [ ! -d "$CH_DIR" ]; then + git clone --recursive --shallow-submodules \ + --branch "$CH_BRANCH" \ + "https://github.com/${CH_ORG}/ClickHouse.git" "$CH_DIR" +fi + +# Build libch.so via Gluten's own wrapper, which cmakes cpp-ch into +# cpp-ch/build_ch and builds the build_ch target. Honors the CC/CXX exported +# above. The artifact lands under cpp-ch/build_ch/.../extern-local-engine/; +# glob for it rather than hardcoding the path (Gluten's prose docs still say +# build/, but the v1.4.0 script builds into build_ch/). +LIBCH_SO=$(find "$GLUTEN_DIR/cpp-ch" -name libch.so -type f 2>/dev/null | head -n1) +if [ -z "$LIBCH_SO" ]; then + bash "$GLUTEN_DIR/ep/build-clickhouse/src/build_clickhouse.sh" + LIBCH_SO=$(find "$GLUTEN_DIR/cpp-ch" -name libch.so -type f 2>/dev/null | head -n1) +fi +if [ -z "$LIBCH_SO" ]; then + echo "ERROR: libch.so not found after build" >&2 + exit 1 +fi + +# Build the Gluten Spark plugin against the CH backend. JDK 8 is required at +# compile time per Gluten's pom; Spark itself runs under JDK 17 in ./query. +# pyspark wheels ship Scala 2.12 jars, so build with scala-2.12 to match. +JAVA_HOME_8="/usr/lib/jvm/java-8-openjdk-$(dpkg --print-architecture)" +GLUTEN_JAR=$(ls "$GLUTEN_DIR"/backends-clickhouse/target/gluten-*-spark-3.5-jar-with-dependencies.jar 2>/dev/null | head -n1) +if [ -z "$GLUTEN_JAR" ]; then + ( + cd "$GLUTEN_DIR" + export MAVEN_OPTS="-Xmx8g -XX:ReservedCodeCacheSize=2g" + JAVA_HOME="$JAVA_HOME_8" PATH="$JAVA_HOME_8/bin:$PATH" \ + mvn -B clean package \ + -Pbackends-clickhouse -P"$SPARK_PROFILE" -Pscala-2.12 \ + -DskipTests -Dcheckstyle.skip + ) + GLUTEN_JAR=$(ls "$GLUTEN_DIR"/backends-clickhouse/target/gluten-*-spark-3.5-jar-with-dependencies.jar 2>/dev/null | head -n1) +fi +if [ -z "$GLUTEN_JAR" ]; then + echo "ERROR: could not locate built Gluten CH-backend jar" >&2 + ls "$GLUTEN_DIR/backends-clickhouse/target/" >&2 || true + exit 1 +fi + +# query.py expects the jar and native library as ./gluten.jar and ./libch.so. +ln -sf "$GLUTEN_JAR" gluten.jar +ln -sf "$LIBCH_SO" libch.so diff --git a/spark-gluten-clickhouse/load b/spark-gluten-clickhouse/load new file mode 100755 index 0000000000..27b3422e88 --- /dev/null +++ b/spark-gluten-clickhouse/load @@ -0,0 +1,6 @@ +#!/bin/bash +set -e + +# Nothing to load: query.py reads hits.parquet directly. Just flush writeback +# so the parquet the harness downloaded is durably on disk before queries run. +sync diff --git a/spark-gluten-clickhouse/query b/spark-gluten-clickhouse/query new file mode 100755 index 0000000000..361e74c857 --- /dev/null +++ b/spark-gluten-clickhouse/query @@ -0,0 +1,13 @@ +#!/bin/bash +set -e + +# shellcheck disable=SC1091 +source myenv/bin/activate + +# Spark runs under JDK 17; Gluten was built under JDK 8 in ./install. +arch=$(dpkg --print-architecture) +export JAVA_HOME="/usr/lib/jvm/java-17-openjdk-${arch}" +export PATH="$JAVA_HOME/bin:$PATH" + +query=$(cat) +printf '%s' "$query" | python3 query.py diff --git a/spark-gluten-clickhouse/query.py b/spark-gluten-clickhouse/query.py index 5c43a4efb8..ca2ca2b918 100755 --- a/spark-gluten-clickhouse/query.py +++ b/spark-gluten-clickhouse/query.py @@ -2,8 +2,11 @@ """ Spark + Apache Gluten using the ClickHouse backend ('ch'). The CH backend -loads libch.so (a fork of ClickHouse v23.1) into the Spark executor JVM -and runs the columnar physical plan natively. +loads libch.so (a fork of ClickHouse v23.1) into the Spark executor JVM and +runs the columnar physical plan natively. + +Reads SQL on stdin, runs it once, prints the result on stdout and the runtime +in fractional seconds as the LAST line on stderr. Note: Keep in sync with spark-*/query.py (see README-accelerators.md for details). """ @@ -33,9 +36,9 @@ SparkSession .builder .appName("ClickBench") - .config("spark.driver", "local[*]") + .config("spark.driver", "local[*]") # To ensure using all cores .config("spark.driver.memory", f"{heap}m") - .config("spark.sql.parquet.binaryAsString", True) + .config("spark.sql.parquet.binaryAsString", True) # Correct length/text results # Gluten + ClickHouse backend configuration .config("spark.jars", "gluten.jar") @@ -56,13 +59,15 @@ df = df.withColumn("EventDate", F.date_add(F.lit("1970-01-01"), F.col("EventDate"))) df.createOrReplaceTempView("hits") -for try_num in range(3): - try: - start = timeit.default_timer() - result = spark.sql(query) - result.show(100) - end = timeit.default_timer() - print("Time: ", end - start) - except Exception as e: - print(e) - print("Failure!") +try: + start = timeit.default_timer() + result = spark.sql(query) + result.show(100) + end = timeit.default_timer() + elapsed = end - start + print(f"Time: {elapsed}") + print(f"{elapsed:.6f}", file=sys.stderr) +except Exception as e: + print(e, file=sys.stderr) + print("Failure!", file=sys.stderr) + sys.exit(1) diff --git a/spark-gluten-clickhouse/run.sh b/spark-gluten-clickhouse/run.sh deleted file mode 100755 index 8c9ca12890..0000000000 --- a/spark-gluten-clickhouse/run.sh +++ /dev/null @@ -1,10 +0,0 @@ -#!/bin/bash - -# Note: Keep in sync with spark-*/run.sh (see README-accelerators.md for details) - -cat queries.sql | while read query; do - sync - echo 3 | sudo tee /proc/sys/vm/drop_caches >/dev/null - - ./query.py <<< "${query}" -done diff --git a/spark-gluten-clickhouse/start b/spark-gluten-clickhouse/start new file mode 100755 index 0000000000..06bd986563 --- /dev/null +++ b/spark-gluten-clickhouse/start @@ -0,0 +1,2 @@ +#!/bin/bash +exit 0 diff --git a/spark-gluten-clickhouse/stop b/spark-gluten-clickhouse/stop new file mode 100755 index 0000000000..06bd986563 --- /dev/null +++ b/spark-gluten-clickhouse/stop @@ -0,0 +1,2 @@ +#!/bin/bash +exit 0 From 1405c24fef41aa6fba26bd988811b65e1e13a3a7 Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Wed, 22 Jul 2026 22:31:36 +0000 Subject: [PATCH 04/20] spark-gluten-clickhouse: use full JDK 17 so the libch.so build finds JNI/AWT MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The c6a.metal re-run got past the download fix and into the ClickHouse compile, then failed at the extern-local-engine cmake configure: -- Could NOT find JNI (missing: AWT) CMake Error: ... JAVA_AWT_INCLUDE_PATH (ADVANCED) ... set to NOTFOUND ClickHouse's extern-local-engine links against JNI including AWT, but the install used openjdk-17-jdk-headless, which ships neither jawt.h nor libjawt.so (verified against the Ubuntu noble package file lists). Switch to the full openjdk-17-jdk (ships jawt.h; depends on the non-headless JRE that ships libjawt.so) and export JAVA_HOME to it so cmake's FindJNI probes the right, AWT-complete JDK deterministically. JDK 8 stays headless — it is only the Maven build, which doesn't need AWT. Also corrects the README: the outer wrapper cmakes into cpp-ch/build_ch, but the inner ClickHouse cmake builds libch.so under cpp-ch/build/ (install globs for it under cpp-ch/ either way). Co-Authored-By: Claude Opus 4.8 (1M context) --- spark-gluten-clickhouse/README.md | 4 ++-- spark-gluten-clickhouse/install | 28 +++++++++++++++++++--------- 2 files changed, 21 insertions(+), 11 deletions(-) diff --git a/spark-gluten-clickhouse/README.md b/spark-gluten-clickhouse/README.md index 3ed249e194..e7b2427236 100644 --- a/spark-gluten-clickhouse/README.md +++ b/spark-gluten-clickhouse/README.md @@ -14,7 +14,7 @@ source (no pre-built bundle is published for the CH backend). The CH backend is not part of Apache Gluten's release tarball — only the Velox bundle is published. As a result `install` builds two things from source: -1. **`libch.so`** — built from [Kyligence/ClickHouse](https://github.com/Kyligence/ClickHouse) at the org/branch/commit pinned in `gluten/cpp-ch/clickhouse.version`. The build uses Clang 19 / cmake / ninja (Gluten v1.4.0's CH backend requires Clang 19). +1. **`libch.so`** — built from [Kyligence/ClickHouse](https://github.com/Kyligence/ClickHouse) at the org/branch/commit pinned in `gluten/cpp-ch/clickhouse.version`. The build uses Clang 19 / cmake / ninja (Gluten v1.4.0's CH backend requires Clang 19). Its `extern-local-engine` module links against JNI **including AWT**, so `install` uses the full `openjdk-17-jdk` (not `-headless`, which omits `jawt.h`/`libjawt.so` and makes cmake fail with `Could NOT find JNI (missing: AWT)`). 2. **The Gluten Spark plugin** — built via Maven with `-Pbackends-clickhouse -Pspark-3.5 -Pscala-2.12`. JDK 8 is required at compile time (Gluten's POM); Spark itself runs under JDK 17 (see `./query`). Building libch.so essentially compiles ClickHouse from source: it is **memory-hungry** (Gluten's docs note that 64 GB RAM is recommended). On a c6a.4xlarge (32 GB RAM) the compile may OOM; use c6a.8xlarge or larger for a clean run. @@ -22,7 +22,7 @@ Building libch.so essentially compiles ClickHouse from source: it is **memory-hu ### Configuration - `spark.gluten.sql.columnar.backend.lib=ch` selects the ClickHouse backend over Velox. -- `spark.gluten.sql.columnar.libpath=` points to the native library. Gluten v1.4.0's build script produces it under `gluten/cpp-ch/build_ch/.../extern-local-engine/libch.so` (the prose docs still say `build/`); `install` globs for it and symlinks it as `libch.so` in the entry directory. +- `spark.gluten.sql.columnar.libpath=` points to the native library. Gluten's wrapper cmakes into `gluten/cpp-ch/build_ch`, which drives an inner ClickHouse cmake that builds `libch.so` under `gluten/cpp-ch/build/.../extern-local-engine/`; `install` globs for it under `cpp-ch/` and symlinks it as `libch.so` in the entry directory. - Memory is split 50/50 between Spark heap and Gluten off-heap, identical to the Velox entry — the CH backend also runs off-heap via JNI. - Queries use ClickHouse-style regex backreferences (`\1`) rather than Spark's `$1`, since the regex evaluation happens inside libch.so. See the discussion in [`spark-gluten/README.md`](../spark-gluten/README.md) and [Gluten issue #7545](https://github.com/apache/incubator-gluten/issues/7545). diff --git a/spark-gluten-clickhouse/install b/spark-gluten-clickhouse/install index 447f01eb80..575de5a79d 100755 --- a/spark-gluten-clickhouse/install +++ b/spark-gluten-clickhouse/install @@ -15,13 +15,18 @@ GLUTEN_VERSION=v1.4.0 SPARK_PROFILE=spark-3.5 # Build prerequisites: -# - Java 8 to build Gluten via Maven (Gluten's pom requires JDK 8) -# - Java 17 to run Spark (selected in ./query via JAVA_HOME) +# - Java 8 (headless) to build Gluten via Maven (Gluten's pom requires JDK 8). +# - Java 17, the FULL jdk (not -headless), to run Spark AND to satisfy the +# libch.so build's JNI/AWT dependency: ClickHouse's extern-local-engine +# links against jawt.h/libjawt.so, which the -headless packages omit, so +# its cmake fails with `Could NOT find JNI (missing: AWT)` / +# `JAVA_AWT_INCLUDE_PATH-NOTFOUND`. openjdk-17-jdk ships jawt.h and pulls +# in the non-headless JRE that ships libjawt.so. # - Clang 19, cmake, ninja, etc. to build libch.so (Gluten v1.4.0's CH # backend requires Clang 19; see get-started/ClickHouse.md). sudo apt-get update -y sudo apt-get install -y python3-pip python3-venv \ - openjdk-8-jdk-headless openjdk-17-jdk-headless \ + openjdk-8-jdk-headless openjdk-17-jdk \ maven git cmake ccache ninja-build nasm yasm gawk \ lsb-release wget software-properties-common gnupg @@ -33,6 +38,8 @@ fi export CC=clang-19 export CXX=clang++-19 +ARCH=$(dpkg --print-architecture) + # pyspark venv. if [ ! -d myenv ]; then python3 -m venv myenv @@ -59,11 +66,14 @@ if [ ! -d "$CH_DIR" ]; then "https://github.com/${CH_ORG}/ClickHouse.git" "$CH_DIR" fi -# Build libch.so via Gluten's own wrapper, which cmakes cpp-ch into -# cpp-ch/build_ch and builds the build_ch target. Honors the CC/CXX exported -# above. The artifact lands under cpp-ch/build_ch/.../extern-local-engine/; -# glob for it rather than hardcoding the path (Gluten's prose docs still say -# build/, but the v1.4.0 script builds into build_ch/). +# Build libch.so via Gluten's own wrapper. It cmakes cpp-ch into +# cpp-ch/build_ch, whose build_ch target drives an inner ClickHouse cmake that +# builds `--target libch` into cpp-ch/build/, so the artifact lands under +# cpp-ch/build/.../extern-local-engine/. Glob for it rather than hardcoding — +# the two build dirs (build_ch wrapper vs build inner) are easy to confuse. +# Honors the CC/CXX exported above; JAVA_HOME points cmake's FindJNI at the +# full JDK 17 so it locates jawt.h/libjawt.so (see the JDK note above). +export JAVA_HOME="/usr/lib/jvm/java-17-openjdk-${ARCH}" LIBCH_SO=$(find "$GLUTEN_DIR/cpp-ch" -name libch.so -type f 2>/dev/null | head -n1) if [ -z "$LIBCH_SO" ]; then bash "$GLUTEN_DIR/ep/build-clickhouse/src/build_clickhouse.sh" @@ -77,7 +87,7 @@ fi # Build the Gluten Spark plugin against the CH backend. JDK 8 is required at # compile time per Gluten's pom; Spark itself runs under JDK 17 in ./query. # pyspark wheels ship Scala 2.12 jars, so build with scala-2.12 to match. -JAVA_HOME_8="/usr/lib/jvm/java-8-openjdk-$(dpkg --print-architecture)" +JAVA_HOME_8="/usr/lib/jvm/java-8-openjdk-${ARCH}" GLUTEN_JAR=$(ls "$GLUTEN_DIR"/backends-clickhouse/target/gluten-*-spark-3.5-jar-with-dependencies.jar 2>/dev/null | head -n1) if [ -z "$GLUTEN_JAR" ]; then ( From b737237287348dae6e17f31bfb3017099d337098 Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Thu, 23 Jul 2026 02:28:11 +0000 Subject: [PATCH 05/20] spark-gluten-clickhouse: add the delta Maven profile MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The c6a.metal run got all the way through the libch.so compile (all 12976 ninja steps linked — the JDK/AWT fix worked) and built every Gluten Maven module except `backends-clickhouse`, which failed its `enforce-delta-profile` enforcer rule: "-P delta" must be set when building Gluten with ClickHouse backend. Profile "delta" is not activated. Add `-Pdelta` to the `mvn` invocation. The `spark-3.5` profile already pins the matching `delta.version` (3.2.0, package `delta-spark`), so no extra version flag is needed. Co-Authored-By: Claude Opus 4.8 (1M context) --- spark-gluten-clickhouse/README.md | 2 +- spark-gluten-clickhouse/install | 7 ++++++- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/spark-gluten-clickhouse/README.md b/spark-gluten-clickhouse/README.md index e7b2427236..8c269afaf0 100644 --- a/spark-gluten-clickhouse/README.md +++ b/spark-gluten-clickhouse/README.md @@ -15,7 +15,7 @@ source (no pre-built bundle is published for the CH backend). The CH backend is not part of Apache Gluten's release tarball — only the Velox bundle is published. As a result `install` builds two things from source: 1. **`libch.so`** — built from [Kyligence/ClickHouse](https://github.com/Kyligence/ClickHouse) at the org/branch/commit pinned in `gluten/cpp-ch/clickhouse.version`. The build uses Clang 19 / cmake / ninja (Gluten v1.4.0's CH backend requires Clang 19). Its `extern-local-engine` module links against JNI **including AWT**, so `install` uses the full `openjdk-17-jdk` (not `-headless`, which omits `jawt.h`/`libjawt.so` and makes cmake fail with `Could NOT find JNI (missing: AWT)`). -2. **The Gluten Spark plugin** — built via Maven with `-Pbackends-clickhouse -Pspark-3.5 -Pscala-2.12`. JDK 8 is required at compile time (Gluten's POM); Spark itself runs under JDK 17 (see `./query`). +2. **The Gluten Spark plugin** — built via Maven with `-Pbackends-clickhouse -Pspark-3.5 -Pscala-2.12 -Pdelta`. The `delta` profile is required by the `backends-clickhouse` module's enforcer; `spark-3.5` pins the matching Delta version (3.2.0). JDK 8 is required at compile time (Gluten's POM); Spark itself runs under JDK 17 (see `./query`). Building libch.so essentially compiles ClickHouse from source: it is **memory-hungry** (Gluten's docs note that 64 GB RAM is recommended). On a c6a.4xlarge (32 GB RAM) the compile may OOM; use c6a.8xlarge or larger for a clean run. diff --git a/spark-gluten-clickhouse/install b/spark-gluten-clickhouse/install index 575de5a79d..c0525822ed 100755 --- a/spark-gluten-clickhouse/install +++ b/spark-gluten-clickhouse/install @@ -87,6 +87,11 @@ fi # Build the Gluten Spark plugin against the CH backend. JDK 8 is required at # compile time per Gluten's pom; Spark itself runs under JDK 17 in ./query. # pyspark wheels ship Scala 2.12 jars, so build with scala-2.12 to match. +# The delta profile is mandatory: backends-clickhouse's pom carries an +# `enforce-delta-profile` enforcer rule that fails the build with +# `"-P delta" must be set when building Gluten with ClickHouse backend` unless +# it is active. The spark-3.5 profile pins the matching delta.version (3.2.0), +# so -Pdelta needs no extra version flag. JAVA_HOME_8="/usr/lib/jvm/java-8-openjdk-${ARCH}" GLUTEN_JAR=$(ls "$GLUTEN_DIR"/backends-clickhouse/target/gluten-*-spark-3.5-jar-with-dependencies.jar 2>/dev/null | head -n1) if [ -z "$GLUTEN_JAR" ]; then @@ -95,7 +100,7 @@ if [ -z "$GLUTEN_JAR" ]; then export MAVEN_OPTS="-Xmx8g -XX:ReservedCodeCacheSize=2g" JAVA_HOME="$JAVA_HOME_8" PATH="$JAVA_HOME_8/bin:$PATH" \ mvn -B clean package \ - -Pbackends-clickhouse -P"$SPARK_PROFILE" -Pscala-2.12 \ + -Pbackends-clickhouse -P"$SPARK_PROFILE" -Pscala-2.12 -Pdelta \ -DskipTests -Dcheckstyle.skip ) GLUTEN_JAR=$(ls "$GLUTEN_DIR"/backends-clickhouse/target/gluten-*-spark-3.5-jar-with-dependencies.jar 2>/dev/null | head -n1) From 7ef0a1595f0262573b039f3f6a7b81306c869e89 Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Thu, 23 Jul 2026 07:32:06 +0000 Subject: [PATCH 06/20] spark-gluten-clickhouse: preload libch.so to dodge the static TLS error Run 4 (with -Pdelta) got all the way through the libch.so compile, the Maven backends-clickhouse build, and data loading (~14.8 GB), then failed at query time when the driver JVM lazily loaded the native engine: java.lang.UnsatisfiedLinkError: .../libch.so: cannot allocate memory in static TLS block at org.apache.gluten.backendsapi.clickhouse.CHListenerApi.initialize ... CHBackend.onDriverStart ... GlutenDriverPlugin.init libch.so carries initial-exec-model TLS (from its statically linked deps), and the static TLS block is sized at process startup, so a lazy `dlopen` from the already-running JVM has no room left. Gluten's own docs work around this with `spark.executorEnv.LD_PRELOAD=`, but in local[*] mode the driver JVM *is* the executor and is launched before any Spark config is read, so executorEnv never applies. Preload it via the JVM's environment instead: set `LD_PRELOAD` in query.py before `getOrCreate()`. This does not affect the already-started Python process, but pyspark copies os.environ into the JVM it spawns, so the JVM preloads libch.so at startup while the static TLS block still has room; Gluten's later `System.load()` then reuses the already-loaded library. Also set `spark.executorEnv.LD_PRELOAD` for correctness under a future cluster-mode run. Co-Authored-By: Claude Opus 4.8 (1M context) --- spark-gluten-clickhouse/README.md | 1 + spark-gluten-clickhouse/query.py | 19 +++++++++++++++++++ 2 files changed, 20 insertions(+) diff --git a/spark-gluten-clickhouse/README.md b/spark-gluten-clickhouse/README.md index 8c269afaf0..013bc7bdad 100644 --- a/spark-gluten-clickhouse/README.md +++ b/spark-gluten-clickhouse/README.md @@ -24,6 +24,7 @@ Building libch.so essentially compiles ClickHouse from source: it is **memory-hu - `spark.gluten.sql.columnar.backend.lib=ch` selects the ClickHouse backend over Velox. - `spark.gluten.sql.columnar.libpath=` points to the native library. Gluten's wrapper cmakes into `gluten/cpp-ch/build_ch`, which drives an inner ClickHouse cmake that builds `libch.so` under `gluten/cpp-ch/build/.../extern-local-engine/`; `install` globs for it under `cpp-ch/` and symlinks it as `libch.so` in the entry directory. - Memory is split 50/50 between Spark heap and Gluten off-heap, identical to the Velox entry — the CH backend also runs off-heap via JNI. +- `libch.so` is preloaded into the JVM via `LD_PRELOAD` (set in `query.py`). Because the library carries initial-exec-model TLS, a lazy `dlopen` from the running JVM otherwise fails with `cannot allocate memory in static TLS block`; preloading it at JVM startup — while the static TLS block still has room — avoids this. Gluten's docs use `spark.executorEnv.LD_PRELOAD` for this, but in `local[*]` mode the driver JVM is the executor and launches before that config is read, so the preload is done through the JVM's environment instead. - Queries use ClickHouse-style regex backreferences (`\1`) rather than Spark's `$1`, since the regex evaluation happens inside libch.so. See the discussion in [`spark-gluten/README.md`](../spark-gluten/README.md) and [Gluten issue #7545](https://github.com/apache/incubator-gluten/issues/7545). ### Links diff --git a/spark-gluten-clickhouse/query.py b/spark-gluten-clickhouse/query.py index ca2ca2b918..5db97c99df 100755 --- a/spark-gluten-clickhouse/query.py +++ b/spark-gluten-clickhouse/query.py @@ -50,8 +50,27 @@ .config("spark.memory.offHeap.enabled", "true") .config("spark.memory.offHeap.size", f"{off_heap}m") .config("spark.driver.extraJavaOptions", "-Dio.netty.tryReflectionSetAccessible=true") + + # Cluster-mode equivalent of the LD_PRELOAD below; a no-op in local[*] but + # kept so the config is correct if this is ever run on real executors. + .config("spark.executorEnv.LD_PRELOAD", os.path.abspath("libch.so")) ) +# Gluten's CH backend loads libch.so into the JVM via JNI (System.load). The +# library carries initial-exec-model TLS (from its statically linked deps), so +# a lazy dlopen from the already-running JVM fails with +# java.lang.UnsatisfiedLinkError: libch.so: cannot allocate memory in static +# TLS block +# because the static TLS block is sized at process startup. Gluten's docs work +# around this with spark.executorEnv.LD_PRELOAD=, but in local[*] mode +# the driver JVM *is* the executor and is launched (by pyspark below) before any +# Spark config is read, so executorEnv never applies. Instead, preload it via +# the driver JVM's environment: setting LD_PRELOAD here does not affect this +# already-started Python process, but pyspark's launcher copies os.environ into +# the JVM it spawns, so the JVM preloads libch.so at startup while the static +# TLS block still has room. System.load() then reuses the already-loaded lib. +os.environ["LD_PRELOAD"] = os.path.abspath("libch.so") + spark = builder.getOrCreate() df = spark.read.parquet("hits.parquet") From 64189eb245f454fa8c2958b011076bfb7a3745ed Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Thu, 23 Jul 2026 22:36:12 +0000 Subject: [PATCH 07/20] spark-gluten-clickhouse: fix static-TLS via GLIBC_TUNABLES, not LD_PRELOAD The previous `LD_PRELOAD=libch.so` workaround for `UnsatisfiedLinkError: libch.so: cannot allocate memory in static TLS block` made the build succeed and data load, but every query then hung indefinitely (the run burned the full 10h job timeout with zero query output). Preloading forces libch.so's global constructors and its statically-linked allocator onto the whole JVM from process start, which deadlocks JVM startup in `local[*]` mode. Switch to glibc's `rtld.optional_static_tls` tunable, exported via `GLIBC_TUNABLES` into the JVM's environment (inherited from `os.environ`, also set as `spark.executorEnv` for cluster mode). It enlarges the per-thread static-TLS surplus reserved at process start so Gluten's lazy `System.load(libch.so)` succeeds without preloading anything. Also add temporary diagnostic scaffolding to `query.py` so a hang fails fast and loud instead of silently: per-step STEP markers on stderr, a watchdog that forces a JVM thread dump (SIGQUIT) then kills the JVM, and a sentinel file that short-circuits the remaining tries/queries after the first hang (so a fully-wedged backend costs one timeout, not 43x3). To be removed once queries run cleanly. Co-Authored-By: Claude Opus 4.8 --- spark-gluten-clickhouse/query.py | 135 ++++++++++++++++++++++++++----- 1 file changed, 117 insertions(+), 18 deletions(-) diff --git a/spark-gluten-clickhouse/query.py b/spark-gluten-clickhouse/query.py index 5db97c99df..a61e83b924 100755 --- a/spark-gluten-clickhouse/query.py +++ b/spark-gluten-clickhouse/query.py @@ -11,8 +11,12 @@ Note: Keep in sync with spark-*/query.py (see README-accelerators.md for details). """ +import faulthandler import os +import signal import sys +import threading +import time import timeit import psutil @@ -20,6 +24,87 @@ import pyspark.sql.functions as F +# --- Diagnostic scaffolding (temporary) -------------------------------------- +# The CH backend is built + loaded remotely on c6a.metal and cannot be +# reproduced on the (aarch64) dev box, so each iteration costs a ~1h build. +# The harness sends query stdout to /dev/null and only surfaces stderr when a +# query *exits non-zero*; a hang therefore produces no output and burns the +# full 10h job timeout. These helpers make a hang fail fast and loud instead: +# * STEP markers on stderr pinpoint how far we got before wedging. +# * A watchdog forces a JVM thread dump (SIGQUIT) then kills the JVM so the +# captured-stderr pipe closes and the invocation exits non-zero. +# * A sentinel file short-circuits the remaining tries/queries once one hang +# is observed, so a fully-wedged backend costs one timeout, not 43x3. +# Remove once queries run cleanly. +# Generous per-query cap: any real ClickBench query finishes in well under this +# on c6a.metal, and the sentinel means a total hang costs only ONE timeout (not +# 43x3), so a large value is safe and won't false-trip a slow-but-valid query. +QUERY_TIMEOUT = int(os.environ.get("QUERY_TIMEOUT", "600")) +HANG_SENTINEL = "query_hang.sentinel" + + +def mark(msg): + print(f"=== STEP: {msg} ===", file=sys.stderr, flush=True) + + +def _dump_and_die(): + print( + f"\n=== WATCHDOG: query.py exceeded {QUERY_TIMEOUT}s; dumping stacks ===", + file=sys.stderr, + flush=True, + ) + jvms = [] + try: + for child in psutil.Process().children(recursive=True): + try: + if "java" in child.name().lower(): + jvms.append(child) + print( + f"=== sending SIGQUIT to JVM pid {child.pid} for thread dump ===", + file=sys.stderr, + flush=True, + ) + child.send_signal(signal.SIGQUIT) + except Exception as exc: # noqa: BLE001 + print(f"watchdog: {exc}", file=sys.stderr, flush=True) + except Exception as exc: # noqa: BLE001 + print(f"watchdog: could not enumerate children: {exc}", file=sys.stderr, flush=True) + + time.sleep(10) # let the JVM flush its thread dump to our stderr + + print("=== Python stacks ===", file=sys.stderr, flush=True) + faulthandler.dump_traceback(file=sys.stderr) + sys.stderr.flush() + + # Kill the JVM so the pipe the harness reads from (2>&1) closes and the + # command substitution capturing our stderr returns instead of blocking. + for child in jvms: + try: + child.kill() + except Exception: # noqa: BLE001 + pass + + try: + open(HANG_SENTINEL, "w").close() + except Exception: # noqa: BLE001 + pass + os._exit(1) + + +if os.path.exists(HANG_SENTINEL): + print( + "=== a prior query hung (see earlier thread dump); fast-failing ===", + file=sys.stderr, + flush=True, + ) + sys.exit(1) + +watchdog = threading.Timer(QUERY_TIMEOUT, _dump_and_die) +watchdog.daemon = True +watchdog.start() +# ----------------------------------------------------------------------------- + + query = sys.stdin.read() print(query) @@ -32,6 +117,28 @@ off_heap = ram - heap print(f"SparkSession will use {heap} MB of heap and {off_heap} MB of off-heap memory (total {ram} MB)") +# Gluten's CH backend loads libch.so into the JVM lazily via JNI (System.load, +# from CHListenerApi.initialize). libch.so carries initial-exec-model TLS (from +# its statically linked deps), and glibc sizes the static TLS block at process +# startup, leaving only a small surplus — so a lazy dlopen from the running JVM +# fails with: +# java.lang.UnsatisfiedLinkError: libch.so: cannot allocate memory in static +# TLS block +# Gluten's docs suggest LD_PRELOAD=, but preloading forces libch.so's +# global constructors and (statically linked) allocator onto the whole JVM from +# process start, which deadlocked JVM startup here (query never returned). The +# cleaner fix is glibc's `rtld.optional_static_tls` tunable: it enlarges the +# per-thread static-TLS surplus reserved at startup, so the *lazy* System.load +# succeeds without preloading anything. It must be in the environment before +# the dynamic loader runs, i.e. before the JVM starts; setting it here does not +# affect this already-running Python process, but pyspark's launcher copies +# os.environ into the JVM it spawns, so the JVM starts with the enlarged +# surplus. 16 MiB is generous (libch.so's real IE-TLS footprint is far smaller) +# but cheap next to a wasted ~1h remote build; lower if per-thread TLS memory +# becomes a concern. +_TLS_SURPLUS = "glibc.rtld.optional_static_tls=16777216" +os.environ["GLIBC_TUNABLES"] = _TLS_SURPLUS + builder = ( SparkSession .builder @@ -51,42 +158,34 @@ .config("spark.memory.offHeap.size", f"{off_heap}m") .config("spark.driver.extraJavaOptions", "-Dio.netty.tryReflectionSetAccessible=true") - # Cluster-mode equivalent of the LD_PRELOAD below; a no-op in local[*] but - # kept so the config is correct if this is ever run on real executors. - .config("spark.executorEnv.LD_PRELOAD", os.path.abspath("libch.so")) + # Cluster-mode equivalent of the GLIBC_TUNABLES above: a no-op in local[*] + # (the driver JVM is the executor and already inherits os.environ) but kept + # so real executors get the same enlarged static-TLS surplus. + .config("spark.executorEnv.GLIBC_TUNABLES", _TLS_SURPLUS) ) -# Gluten's CH backend loads libch.so into the JVM via JNI (System.load). The -# library carries initial-exec-model TLS (from its statically linked deps), so -# a lazy dlopen from the already-running JVM fails with -# java.lang.UnsatisfiedLinkError: libch.so: cannot allocate memory in static -# TLS block -# because the static TLS block is sized at process startup. Gluten's docs work -# around this with spark.executorEnv.LD_PRELOAD=, but in local[*] mode -# the driver JVM *is* the executor and is launched (by pyspark below) before any -# Spark config is read, so executorEnv never applies. Instead, preload it via -# the driver JVM's environment: setting LD_PRELOAD here does not affect this -# already-started Python process, but pyspark's launcher copies os.environ into -# the JVM it spawns, so the JVM preloads libch.so at startup while the static -# TLS block still has room. System.load() then reuses the already-loaded lib. -os.environ["LD_PRELOAD"] = os.path.abspath("libch.so") - +mark("building SparkSession (JVM launch + libch.so load)") spark = builder.getOrCreate() +mark("SparkSession ready; reading hits.parquet") df = spark.read.parquet("hits.parquet") df = df.withColumn("EventTime", F.col("EventTime").cast("timestamp")) df = df.withColumn("EventDate", F.date_add(F.lit("1970-01-01"), F.col("EventDate"))) df.createOrReplaceTempView("hits") +mark("temp view created; executing query") try: start = timeit.default_timer() result = spark.sql(query) result.show(100) end = timeit.default_timer() elapsed = end - start + mark("query complete") print(f"Time: {elapsed}") print(f"{elapsed:.6f}", file=sys.stderr) except Exception as e: print(e, file=sys.stderr) print("Failure!", file=sys.stderr) sys.exit(1) +finally: + watchdog.cancel() From cd152de3c17f386f05eafbf76f37cdd029217ad3 Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Fri, 24 Jul 2026 01:38:25 +0000 Subject: [PATCH 08/20] spark-gluten-clickhouse: use a safe static-TLS surplus (2 MiB, not 16) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The GLIBC_TUNABLES fix was right in principle but the value was reckless. `glibc.rtld.optional_static_tls=16777216` (16 MiB) makes glibc segfault every multi-threaded process at thread creation — including pyspark's own launcher JVM (`org.apache.spark.launcher.Main`). That surfaced in the last run as `spark-class: line 97: CMD: bad array subscript` followed by `[JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port number`, so all 43 queries returned `[null,null,null]`. Measured the crash threshold locally (glibc 2.43): 8 MiB segfaults, 6 MiB is fine. Drop to 2 MiB (2097152) — ~1260x the failing glibc default (1664 B), comfortably more than libch.so's real IE-TLS footprint, and well under the crash threshold on both glibc 2.39 (noble) and 2.43. Verified a 128-thread process starts cleanly with this value. The fail-fast diagnostics from the prior commit did their job: the run finished in 62 min with the exact error instead of burning the 10h job timeout. Kept in place for one more run in case 2 MiB needs tuning. Co-Authored-By: Claude Opus 4.8 --- spark-gluten-clickhouse/query.py | 16 ++++++++++++---- 1 file changed, 12 insertions(+), 4 deletions(-) diff --git a/spark-gluten-clickhouse/query.py b/spark-gluten-clickhouse/query.py index a61e83b924..6c95e6bc12 100755 --- a/spark-gluten-clickhouse/query.py +++ b/spark-gluten-clickhouse/query.py @@ -133,10 +133,18 @@ def _dump_and_die(): # the dynamic loader runs, i.e. before the JVM starts; setting it here does not # affect this already-running Python process, but pyspark's launcher copies # os.environ into the JVM it spawns, so the JVM starts with the enlarged -# surplus. 16 MiB is generous (libch.so's real IE-TLS footprint is far smaller) -# but cheap next to a wasted ~1h remote build; lower if per-thread TLS memory -# becomes a concern. -_TLS_SURPLUS = "glibc.rtld.optional_static_tls=16777216" +# surplus. +# +# Value matters: an over-large surplus makes glibc *segfault* every +# multi-threaded process at thread creation (measured threshold ~7 MiB on +# glibc 2.43; 8 MiB crashes, 6 MiB is fine). A 16 MiB attempt crashed pyspark's +# own launcher JVM (`org.apache.spark.launcher.Main`), which surfaced as +# `spark-class: line 97: CMD: bad array subscript` + +# `[JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port +# number`. 2 MiB is ~1260x the failing glibc default (1664 B) — comfortably +# more than libch.so's real IE-TLS footprint — while staying well under the +# crash threshold on both glibc 2.39 (noble) and 2.43. +_TLS_SURPLUS = "glibc.rtld.optional_static_tls=2097152" os.environ["GLIBC_TUNABLES"] = _TLS_SURPLUS builder = ( From 33091eb68a2a157906e4ef0fe81490a32c65f89c Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Fri, 24 Jul 2026 15:32:39 +0000 Subject: [PATCH 09/20] spark-gluten-clickhouse: drop static-TLS surplus to 512 KiB (2 MiB crashed the JVM) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The 2 MiB surplus reliably crashes JVM startup. The surplus is reserved as native memory in every JVM the environment touches, including pyspark's tiny `-Xmx128m` launcher JVM (`org.apache.spark.launcher.Main`); past a sharp cliff the JVM dies with "Cannot create worker GC thread. Out of system resources" (native TLS colliding with the compressed-oops heap region). Because spark-class runs the launcher inside a process substitution, bash swallows the crash and it only surfaced as `spark-class: line 97: CMD: bad array subscript` + `[JAVA_GATEWAY_EXITED]`, so all 43 queries returned `[null,null,null]` again. Measured the cliff locally (OpenJDK 17, 192 GC threads to mimic c6a.metal, 5 trials each): it's sharp and machine-independent — every value <= 1 MiB starts 5/5 for both the -Xmx128m launcher and a big-heap -Xmx64g gateway JVM, and 2 MiB fails 5/5. Use 512 KiB: 4x under the cliff, still 315x the failing glibc default (1664 B) and well clear of libch.so's small IE-model TLS need. The fail-fast diagnostics again did their job (62 min with the exact error, not a 10h hang) and stay in for one more run in case the gateway JVM, which we have not yet reached, needs attention. Co-Authored-By: Claude Opus 4.8 --- spark-gluten-clickhouse/query.py | 27 +++++++++++++++++---------- 1 file changed, 17 insertions(+), 10 deletions(-) diff --git a/spark-gluten-clickhouse/query.py b/spark-gluten-clickhouse/query.py index 6c95e6bc12..2f1961a683 100755 --- a/spark-gluten-clickhouse/query.py +++ b/spark-gluten-clickhouse/query.py @@ -135,16 +135,23 @@ def _dump_and_die(): # os.environ into the JVM it spawns, so the JVM starts with the enlarged # surplus. # -# Value matters: an over-large surplus makes glibc *segfault* every -# multi-threaded process at thread creation (measured threshold ~7 MiB on -# glibc 2.43; 8 MiB crashes, 6 MiB is fine). A 16 MiB attempt crashed pyspark's -# own launcher JVM (`org.apache.spark.launcher.Main`), which surfaced as -# `spark-class: line 97: CMD: bad array subscript` + -# `[JAVA_GATEWAY_EXITED] Java gateway process exited before sending its port -# number`. 2 MiB is ~1260x the failing glibc default (1664 B) — comfortably -# more than libch.so's real IE-TLS footprint — while staying well under the -# crash threshold on both glibc 2.39 (noble) and 2.43. -_TLS_SURPLUS = "glibc.rtld.optional_static_tls=2097152" +# Value matters, and the safe window is narrow. The surplus is reserved as +# native memory in *every* JVM the env touches — including pyspark's own tiny +# launcher JVM (`org.apache.spark.launcher.Main`, run at -Xmx128m by +# spark-class to build the real command). Too large and the JVM dies at +# startup with "Cannot create worker GC thread. Out of system resources" +# (native TLS collides with the compressed-oops heap region); because +# spark-class runs the launcher inside a process substitution, bash swallows +# the crash and it surfaces only as `spark-class: line 97: CMD: bad array +# subscript` -> `[JAVA_GATEWAY_EXITED] Java gateway process exited before +# sending its port number`, so all queries return null. Measured locally +# (OpenJDK 17, 192 GC threads to mimic c6a.metal): the cliff is sharp and +# machine-independent — every value <= 1 MiB starts 5/5 for both the -Xmx128m +# launcher and a big-heap (-Xmx64g) gateway JVM, and 2 MiB fails 5/5 (which is +# exactly what killed the prior run). 512 KiB sits 4x under that cliff yet is +# 315x the failing glibc default (1664 B) — well clear of libch.so's small +# IE-model TLS footprint. +_TLS_SURPLUS = "glibc.rtld.optional_static_tls=524288" os.environ["GLIBC_TUNABLES"] = _TLS_SURPLUS builder = ( From 4af952eb87fdeec8c02cc4c3ef2b4468f72f1e3d Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Sat, 25 Jul 2026 00:55:07 +0000 Subject: [PATCH 10/20] spark-gluten-clickhouse: diagnose the libch.so init crash (mem cap + hs_err dump) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 512 KiB cleared the static-TLS/launcher problem: the JVM now launches and Gluten's CH backend classes load (CHBackend, CHDeltaComponent discovered). But the JVM then dies during `JavaSparkContext` construction — where the CH backend initializes libch.so — with only `Py4JNetworkError: Answer from Java side is empty` (the process vanished) and, tellingly, NO SIGSEGV/hs_err banner on the captured stderr. That points to an external SIGKILL rather than a caught crash; earlyoom is enabled on the runner. The Velox sibling uses the identical 0.7*available memory split and survives, so the request itself isn't the difference — but the CH engine may eagerly commit its off-heap arena. This commit disambiguates in one run: - Cap heap to 24 GiB and off-heap to 48 GiB (far below c6a.metal's 384 GB) so an OOM kill cannot be the cause. If queries now run, it was memory. - Pin `-XX:ErrorFile=./hs_err_pid%p.log` and dump the crash log on getOrCreate() failure. If an hs_err appears, it's a native SIGSEGV and the frame names the culprit; if none appears, it confirms SIGKILL/OOM. - Print the resolved memory numbers to stderr. - Drop the hang sentinel on the first init crash too, so the remaining invocations fast-fail (one crash dump is enough; bounds the run). All temporary — reverts to the 0.7 split and sheds the scaffolding once the crash is understood. Co-Authored-By: Claude Opus 4.8 --- spark-gluten-clickhouse/query.py | 68 ++++++++++++++++++++++++++++---- 1 file changed, 61 insertions(+), 7 deletions(-) diff --git a/spark-gluten-clickhouse/query.py b/spark-gluten-clickhouse/query.py index 2f1961a683..aad794bd62 100755 --- a/spark-gluten-clickhouse/query.py +++ b/spark-gluten-clickhouse/query.py @@ -93,7 +93,8 @@ def _dump_and_die(): if os.path.exists(HANG_SENTINEL): print( - "=== a prior query hung (see earlier thread dump); fast-failing ===", + "=== a prior query hung or crashed at init (see earlier diagnostics); " + "fast-failing ===", file=sys.stderr, flush=True, ) @@ -112,10 +113,20 @@ def _dump_and_die(): # The CH backend runs off-heap (via JNI into libch.so), so split available # memory between Spark's JVM heap and the off-heap pool the same way the # Velox backend does. -ram = int(round(psutil.virtual_memory().available / (1024 ** 2) * 0.7)) -heap = ram // 2 -off_heap = ram - heap -print(f"SparkSession will use {heap} MB of heap and {off_heap} MB of off-heap memory (total {ram} MB)") +# +# DIAGNOSTIC (temporary): the Velox sibling uses 0.7*available (half heap / +# half off-heap) and survives on c6a.metal, but the CH backend's JVM dies +# during JavaSparkContext construction (where libch.so initializes) with no +# hs_err banner on stderr — consistent with an external SIGKILL (earlyoom is +# enabled on the runner) if the CH engine eagerly commits its off-heap arena. +# Cap heap+off-heap far below available so an OOM kill can't be the cause; if +# this run's queries execute, the 0.7 split needs revisiting. Restore it once +# the init crash is understood. +avail_mb = int(psutil.virtual_memory().available / (1024 ** 2)) +heap = min(avail_mb // 2, 24576) # <= 24 GiB driver heap +off_heap = min(avail_mb - heap, 49152) # <= 48 GiB off-heap for the CH engine +print(f"avail={avail_mb} MB -> heap={heap} MB, off_heap={off_heap} MB", file=sys.stderr, flush=True) +print(f"SparkSession will use {heap} MB of heap and {off_heap} MB of off-heap memory") # Gluten's CH backend loads libch.so into the JVM lazily via JNI (System.load, # from CHListenerApi.initialize). libch.so carries initial-exec-model TLS (from @@ -171,7 +182,11 @@ def _dump_and_die(): .config("spark.gluten.sql.columnar.libpath", os.path.abspath("libch.so")) .config("spark.memory.offHeap.enabled", "true") .config("spark.memory.offHeap.size", f"{off_heap}m") - .config("spark.driver.extraJavaOptions", "-Dio.netty.tryReflectionSetAccessible=true") + # -XX:ErrorFile pins any JVM crash log to cwd so the except handler below + # can surface it (temporary; part of the init-crash diagnosis). + .config("spark.driver.extraJavaOptions", + "-Dio.netty.tryReflectionSetAccessible=true " + "-XX:ErrorFile=./hs_err_pid%p.log") # Cluster-mode equivalent of the GLIBC_TUNABLES above: a no-op in local[*] # (the driver JVM is the executor and already inherits os.environ) but kept @@ -179,8 +194,47 @@ def _dump_and_die(): .config("spark.executorEnv.GLIBC_TUNABLES", _TLS_SURPLUS) ) + +def _dump_crash_artifacts(): + """Surface a JVM crash log if one exists; its absence implies SIGKILL (OOM).""" + import glob + files = sorted(glob.glob("hs_err_pid*.log") + glob.glob("/tmp/hs_err_pid*.log")) + if not files: + print( + "=== no hs_err file: JVM was SIGKILLed, not a caught crash " + "(points to earlyoom/OOM, not a native SIGSEGV) ===", + file=sys.stderr, flush=True, + ) + return + newest = files[-1] + print(f"=== JVM crash log {newest} (first 90 lines) ===", file=sys.stderr, flush=True) + try: + with open(newest) as fh: + for i, line in enumerate(fh): + if i >= 90: + break + print(line.rstrip(), file=sys.stderr) + except OSError as exc: + print(f" (could not read {newest}: {exc})", file=sys.stderr) + sys.stderr.flush() + + mark("building SparkSession (JVM launch + libch.so load)") -spark = builder.getOrCreate() +try: + spark = builder.getOrCreate() +except BaseException: + watchdog.cancel() + print("=== getOrCreate() failed; scanning for JVM crash artifacts ===", + file=sys.stderr, flush=True) + _dump_crash_artifacts() + # One init crash means every subsequent query will crash identically; drop + # the sentinel so the remaining invocations fast-fail instead of each + # rebuilding a doomed JVM (bounds the run, and one crash dump is enough). + try: + open(HANG_SENTINEL, "w").close() + except OSError: + pass + raise mark("SparkSession ready; reading hits.parquet") df = spark.read.parquet("hits.parquet") From 8f4c86efc5ed64dcc1db8bf71c767f5474fe3cb8 Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Sat, 25 Jul 2026 16:26:42 +0000 Subject: [PATCH 11/20] spark-gluten-clickhouse: build libch.so without jemalloc MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The CH backend crashes with SIGSEGV in jemalloc's rtree_metadata_read during native init (CHListenerApi.initialize -> nativeInitNative), before any query runs. Root cause: jemalloc is only safe when it interposes malloc/free process-wide, which requires preloading libch.so. We load it lazily (System.load, via the static-TLS tunable) rather than LD_PRELOAD — the launcher JVM hangs when libch.so is preloaded — so jemalloc never interposes. The first time ClickHouse frees a pointer the system/JVM allocator made (seen while handling a "logger.*" config string), jemalloc's radix tree has no metadata for it and segfaults. Build ClickHouse with the system allocator instead. Gluten hardcodes -DENABLE_JEMALLOC=ON for the inner ClickHouse cmake in cpp-ch/CMakeLists.txt with no env override, so patch it to OFF before the build and verify the patch took (so a future Gluten bump fails loudly rather than silently re-enabling jemalloc). This removes the cross-allocator boundary entirely. Single-variable run: memory cap and diagnostics from the prior commit stay in place to confirm queries now execute; both come out once they do. Co-Authored-By: Claude Opus 4.8 --- spark-gluten-clickhouse/install | 20 ++++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/spark-gluten-clickhouse/install b/spark-gluten-clickhouse/install index c0525822ed..0eb99d348c 100755 --- a/spark-gluten-clickhouse/install +++ b/spark-gluten-clickhouse/install @@ -66,6 +66,26 @@ if [ ! -d "$CH_DIR" ]; then "https://github.com/${CH_ORG}/ClickHouse.git" "$CH_DIR" fi +# Build libch.so WITHOUT jemalloc. ClickHouse enables jemalloc by default and +# Gluten hardcodes -DENABLE_JEMALLOC=ON for the inner ClickHouse cmake in +# cpp-ch/CMakeLists.txt (there is no env hook to override it). jemalloc is only +# safe when it interposes malloc/free process-wide, which requires preloading +# libch.so. We load it lazily (System.load in query.py, via the static-TLS +# tunable) rather than LD_PRELOAD, so jemalloc does NOT interpose: the first +# time ClickHouse frees a pointer the system/JVM allocator made, jemalloc's +# radix tree has no metadata for it and SIGSEGVs in rtree_metadata_read during +# the CH backend's native init (CHListenerApi.initialize -> nativeInitNative). +# Building without jemalloc makes ClickHouse use the system allocator uniformly, +# removing the cross-allocator boundary. Patch + verify so a future Gluten bump +# that changes the flag fails loudly instead of silently re-enabling jemalloc. +CH_CMAKELISTS="$GLUTEN_DIR/cpp-ch/CMakeLists.txt" +sed -i 's/-DENABLE_JEMALLOC=ON/-DENABLE_JEMALLOC=OFF/g' "$CH_CMAKELISTS" +if grep -q -- '-DENABLE_JEMALLOC=ON' "$CH_CMAKELISTS" \ + || ! grep -q -- '-DENABLE_JEMALLOC=OFF' "$CH_CMAKELISTS"; then + echo "ERROR: could not disable jemalloc in $CH_CMAKELISTS" >&2 + exit 1 +fi + # Build libch.so via Gluten's own wrapper. It cmakes cpp-ch into # cpp-ch/build_ch, whose build_ch target drives an inner ClickHouse cmake that # builds `--target libch` into cpp-ch/build/, so the artifact lands under From 0b0b12dc8b87792035fea8c2b9e2abba7fc680fa Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Sat, 25 Jul 2026 18:14:43 +0000 Subject: [PATCH 12/20] spark-gluten-clickhouse: move to Gluten main (ClickHouse 25.12) + lift thread limits MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Two changes: 1. Track Gluten main instead of the v1.4.0 tag. The CH backend's pinned ClickHouse fork is only rebased forward on main; release tags carry an old ClickHouse (v1.4.0 was ~2 years behind). Pin apache/gluten main @ 8b70147, which builds ClickHouse 25.12.10.7 (Kyligence rebase_ch/20260425-25.12.10.7). Build requirements are unchanged (Clang 19, JDK 8 for Maven, spark-3.5, -Pdelta, the jemalloc patch still matches cpp-ch/CMakeLists.txt); bump pyspark to 3.5.5 to match the spark-3.5 profile, and clone by SHA (git clone --branch takes only refs). 2. Lift thread/process limits. With jemalloc disabled the CH backend now initializes, but SparkContext init then died with `OutOfMemoryError: unable to create native thread` (not RAM — capped to 24g/48g of 384G): libch.so's core-scaled thread pools plus the JVM's ~150 GC/JIT threads on 192 cores exhaust the process limits. Fix from three sides: raise kernel ceilings in install (threads-max, pid_max, max_map_count), raise per-process ulimits in ./query (nproc/nofile), and cap the JVM's helper threads (ParallelGCThreads=8 etc., which don't affect local[*] task parallelism). Diagnostics and the memory cap stay in for now. Co-Authored-By: Claude Opus 4.8 --- spark-gluten-clickhouse/install | 34 ++++++++++++++++++++++++-------- spark-gluten-clickhouse/query | 7 +++++++ spark-gluten-clickhouse/query.py | 7 ++++++- 3 files changed, 39 insertions(+), 9 deletions(-) diff --git a/spark-gluten-clickhouse/install b/spark-gluten-clickhouse/install index 0eb99d348c..08adaf7fe9 100755 --- a/spark-gluten-clickhouse/install +++ b/spark-gluten-clickhouse/install @@ -11,7 +11,11 @@ set -e # # Note: Keep in sync with spark-*/install (see README-accelerators.md). -GLUTEN_VERSION=v1.4.0 +# Track Gluten main, not a release tag: the CH backend's pinned ClickHouse fork +# is only rebased forward on main, so release tags carry an old ClickHouse. This +# commit pulls ClickHouse 25.12.10.7 (cpp-ch/clickhouse.version => +# Kyligence rebase_ch/20260425-25.12.10.7); v1.4.0 was on a ~2-year-old fork. +GLUTEN_COMMIT=8b701473f465e4d7eb05335b0529bad9327db8b3 # apache/gluten main @ 2026-07-24 SPARK_PROFILE=spark-3.5 # Build prerequisites: @@ -22,14 +26,22 @@ SPARK_PROFILE=spark-3.5 # its cmake fails with `Could NOT find JNI (missing: AWT)` / # `JAVA_AWT_INCLUDE_PATH-NOTFOUND`. openjdk-17-jdk ships jawt.h and pulls # in the non-headless JRE that ships libjawt.so. -# - Clang 19, cmake, ninja, etc. to build libch.so (Gluten v1.4.0's CH -# backend requires Clang 19; see get-started/ClickHouse.md). +# - Clang 19, cmake, ninja, etc. to build libch.so (Gluten's CH backend +# requires Clang 19; see get-started/ClickHouse.md). sudo apt-get update -y sudo apt-get install -y python3-pip python3-venv \ openjdk-8-jdk-headless openjdk-17-jdk \ maven git cmake ccache ninja-build nasm yasm gawk \ lsb-release wget software-properties-common gnupg +# ClickHouse (libch.so) starts core-scaled thread pools at init; on a 192-core +# box the process needs far more threads and memory-map areas than the defaults +# allow, and JavaSparkContext init otherwise dies with `OutOfMemoryError: +# unable to create native thread`. Raise the kernel ceilings once here +# (best-effort; ./query also raises the matching per-process ulimits). +sudo sysctl -w kernel.threads-max=4000000 kernel.pid_max=4000000 \ + vm.max_map_count=2000000 >/dev/null 2>&1 || true + # Install Clang 19 via apt.llvm.org. if ! command -v clang-19 >/dev/null 2>&1; then wget -O - https://apt.llvm.org/llvm.sh | sudo bash -s -- 19 @@ -46,14 +58,20 @@ if [ ! -d myenv ]; then fi # shellcheck disable=SC1091 source myenv/bin/activate -pip install -q pyspark==3.5.2 psutil +# pyspark matches the spark-3.5 profile Gluten is built against below (Spark +# 3.5.5), so the plugin jar's Spark APIs line up with the runtime. +pip install -q pyspark==3.5.5 psutil # Clone Gluten and the Kyligence ClickHouse fork that the CH backend wraps. # The CH fork org/branch/commit are pinned in Gluten's cpp-ch/clickhouse.version. GLUTEN_DIR="$PWD/gluten" if [ ! -d "$GLUTEN_DIR" ]; then - git clone --depth 1 --branch "$GLUTEN_VERSION" \ - https://github.com/apache/incubator-gluten.git "$GLUTEN_DIR" + # Shallow-fetch the pinned commit. `git clone --branch` accepts only refs, + # not SHAs, so init + fetch-by-SHA (GitHub allows reachable-SHA fetches). + git init -q "$GLUTEN_DIR" + git -C "$GLUTEN_DIR" remote add origin https://github.com/apache/gluten.git + git -C "$GLUTEN_DIR" fetch -q --depth 1 origin "$GLUTEN_COMMIT" + git -C "$GLUTEN_DIR" checkout -q FETCH_HEAD fi CH_VERSION_FILE="$GLUTEN_DIR/cpp-ch/clickhouse.version" @@ -110,8 +128,8 @@ fi # The delta profile is mandatory: backends-clickhouse's pom carries an # `enforce-delta-profile` enforcer rule that fails the build with # `"-P delta" must be set when building Gluten with ClickHouse backend` unless -# it is active. The spark-3.5 profile pins the matching delta.version (3.2.0), -# so -Pdelta needs no extra version flag. +# it is active. The spark-3.5 profile pins the matching delta.version (3.3.2 on +# current main), so -Pdelta needs no extra version flag. JAVA_HOME_8="/usr/lib/jvm/java-8-openjdk-${ARCH}" GLUTEN_JAR=$(ls "$GLUTEN_DIR"/backends-clickhouse/target/gluten-*-spark-3.5-jar-with-dependencies.jar 2>/dev/null | head -n1) if [ -z "$GLUTEN_JAR" ]; then diff --git a/spark-gluten-clickhouse/query b/spark-gluten-clickhouse/query index 361e74c857..f0a44085c2 100755 --- a/spark-gluten-clickhouse/query +++ b/spark-gluten-clickhouse/query @@ -9,5 +9,12 @@ arch=$(dpkg --print-architecture) export JAVA_HOME="/usr/lib/jvm/java-17-openjdk-${arch}" export PATH="$JAVA_HOME/bin:$PATH" +# libch.so plus Spark spin up many threads on a 192-core box; the default +# per-process limits are too low and SparkContext init dies with +# `OutOfMemoryError: unable to create native thread`. Raise the soft limits to +# the hard max (./install also bumps the kernel-wide sysctls). +ulimit -u unlimited 2>/dev/null || ulimit -u "$(ulimit -Hu)" 2>/dev/null || true +ulimit -n 1048576 2>/dev/null || ulimit -n "$(ulimit -Hn)" 2>/dev/null || true + query=$(cat) printf '%s' "$query" | python3 query.py diff --git a/spark-gluten-clickhouse/query.py b/spark-gluten-clickhouse/query.py index aad794bd62..c5aa00ed4a 100755 --- a/spark-gluten-clickhouse/query.py +++ b/spark-gluten-clickhouse/query.py @@ -182,11 +182,16 @@ def _dump_and_die(): .config("spark.gluten.sql.columnar.libpath", os.path.abspath("libch.so")) .config("spark.memory.offHeap.enabled", "true") .config("spark.memory.offHeap.size", f"{off_heap}m") + # Cap the JVM's helper threads: on 192 cores it otherwise spawns ~150 + # GC+JIT threads which, on top of ClickHouse's native pools, exhaust the + # process thread limit during SparkContext init (OutOfMemoryError: unable + # to create native thread). These do NOT limit local[*] task parallelism. # -XX:ErrorFile pins any JVM crash log to cwd so the except handler below # can surface it (temporary; part of the init-crash diagnosis). .config("spark.driver.extraJavaOptions", "-Dio.netty.tryReflectionSetAccessible=true " - "-XX:ErrorFile=./hs_err_pid%p.log") + "-XX:ErrorFile=./hs_err_pid%p.log " + "-XX:ParallelGCThreads=8 -XX:ConcGCThreads=2 -XX:CICompilerCount=4") # Cluster-mode equivalent of the GLIBC_TUNABLES above: a no-op in local[*] # (the driver JVM is the executor and already inherits os.environ) but kept From ca5de095d889d344a18375302235fefa3688822f Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Sun, 26 Jul 2026 00:23:39 +0000 Subject: [PATCH 13/20] spark-gluten-clickhouse: fix libch.so build script name on Gluten main The Gluten main run failed instantly at the build step: `ep/build-clickhouse/src/build_clickhouse.sh: No such file or directory`. The wrapper was renamed build_clickhouse.sh -> build-clickhouse.sh (hyphen) after v1.4.0. Everything else on main matched: the clone (fetch-by-SHA), the CH 25.12 fork + submodules, and the jemalloc patch all applied, and the renamed wrapper still cmakes cpp-ch -> build_ch via cpp-ch/CMakeLists.txt (so the -DENABLE_JEMALLOC=OFF patch still takes effect). Locate the wrapper with a `build[-_]clickhouse.sh` glob so either name works across Gluten versions, and fail loudly if it is missing. Co-Authored-By: Claude Opus 4.8 --- spark-gluten-clickhouse/install | 10 +++++++++- 1 file changed, 9 insertions(+), 1 deletion(-) diff --git a/spark-gluten-clickhouse/install b/spark-gluten-clickhouse/install index 08adaf7fe9..c90540f5d6 100755 --- a/spark-gluten-clickhouse/install +++ b/spark-gluten-clickhouse/install @@ -111,10 +111,18 @@ fi # the two build dirs (build_ch wrapper vs build inner) are easy to confuse. # Honors the CC/CXX exported above; JAVA_HOME points cmake's FindJNI at the # full JDK 17 so it locates jawt.h/libjawt.so (see the JDK note above). +# Note: the wrapper was renamed build_clickhouse.sh -> build-clickhouse.sh +# after v1.4.0; glob so either name works across Gluten versions. export JAVA_HOME="/usr/lib/jvm/java-17-openjdk-${ARCH}" LIBCH_SO=$(find "$GLUTEN_DIR/cpp-ch" -name libch.so -type f 2>/dev/null | head -n1) if [ -z "$LIBCH_SO" ]; then - bash "$GLUTEN_DIR/ep/build-clickhouse/src/build_clickhouse.sh" + BUILD_CH_SCRIPT=$(find "$GLUTEN_DIR/ep/build-clickhouse" \ + -name 'build[-_]clickhouse.sh' -type f 2>/dev/null | head -n1) + if [ -z "$BUILD_CH_SCRIPT" ]; then + echo "ERROR: could not find the build-clickhouse wrapper script" >&2 + exit 1 + fi + bash "$BUILD_CH_SCRIPT" LIBCH_SO=$(find "$GLUTEN_DIR/cpp-ch" -name libch.so -type f 2>/dev/null | head -n1) fi if [ -z "$LIBCH_SO" ]; then From 91dc0e7da748f63e95b4f219060e35d418d0881b Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Sun, 26 Jul 2026 17:34:08 +0000 Subject: [PATCH 14/20] spark-gluten-clickhouse: Scala 2.12 shim for Delta 3.3 Option.when With libch.so now building on Gluten main (CH 25.12), the Maven build got through 9 modules and failed only on `Gluten Backends ClickHouse`: the Scala compile of src-delta33/.../delta/Snapshot.scala errors with "value when is not a member of object Option". Option.when is a Scala 2.13 stdlib addition, but we build with -Pscala-2.12 (to match the pyspark wheel), and scala-2.12 is also Gluten's own default profile -- so the Delta 3.3 sources not compiling under 2.12 is effectively an upstream regression. Rather than switch the whole build+runtime to Scala 2.13, inject a tiny compat shim: Option.when/unless as implicit extensions on the Option companion, compiled from src-delta33 (its own source path) and imported into Snapshot.scala. Under a scala-2.13 build the real stdlib methods win and the shim is unused, so it is safe across versions. Code search confirms Snapshot.scala is the only file in backends-clickhouse using Option.when, so patching it is sufficient. Co-Authored-By: Claude Opus 4.8 --- spark-gluten-clickhouse/install | 38 +++++++++++++++++++++++++++++++++ 1 file changed, 38 insertions(+) diff --git a/spark-gluten-clickhouse/install b/spark-gluten-clickhouse/install index c90540f5d6..0b45f9b0a4 100755 --- a/spark-gluten-clickhouse/install +++ b/spark-gluten-clickhouse/install @@ -104,6 +104,44 @@ if grep -q -- '-DENABLE_JEMALLOC=ON' "$CH_CMAKELISTS" \ exit 1 fi +# Scala 2.12 compat for the Delta 3.3 backend sources. On Gluten main, +# backends-clickhouse/src-delta33/.../delta/Snapshot.scala calls Option.when, +# which only exists in Scala 2.13's stdlib -- but scala-2.12 is Gluten's default +# profile and matches the pyspark wheel we run, so the CH backend module fails +# to compile ("value when is not a member of object Option"). Rather than pull +# in a whole Scala-2.13 Spark stack, drop in Option.when/unless as implicit +# extensions on the Option companion (compiled from src-delta33, so it is on the +# module's own source path) and import it into Snapshot.scala. Under a scala-2.13 +# build the real Option.when takes precedence and this shim is simply unused, so +# it is safe across versions. Only Snapshot.scala uses Option.when in +# backends-clickhouse (verified by code search), so this one file is enough. +DELTA33_SCALA="$GLUTEN_DIR/backends-clickhouse/src-delta33/main/scala" +SNAPSHOT_SCALA="$DELTA33_SCALA/org/apache/spark/sql/delta/Snapshot.scala" +if [ -f "$SNAPSHOT_SCALA" ]; then + mkdir -p "$DELTA33_SCALA/org/apache/gluten/compat" + cat > "$DELTA33_SCALA/org/apache/gluten/compat/OptionCompat.scala" <<'SCALA' +package org.apache.gluten.compat + +// Scala 2.12 lacks Option.when/Option.unless (added in 2.13). Provide them as +// implicit extensions on the Option companion so Delta 3.3 sources compile +// under scala-2.12 (Gluten's default profile). Under scala-2.13 the real +// stdlib methods take precedence and this shim is unused. +object OptionCompat { + implicit class OptionCompanionOps(companion: Option.type) { + def when[A](cond: Boolean)(a: => A): Option[A] = if (cond) Some(a) else None + def unless[A](cond: Boolean)(a: => A): Option[A] = if (cond) None else Some(a) + } +} +SCALA + if ! grep -q 'org.apache.gluten.compat.OptionCompat' "$SNAPSHOT_SCALA"; then + sed -i '/^package /a import org.apache.gluten.compat.OptionCompat._' "$SNAPSHOT_SCALA" + fi + if ! grep -q 'org.apache.gluten.compat.OptionCompat._' "$SNAPSHOT_SCALA"; then + echo "ERROR: failed to inject the Option.when shim into Snapshot.scala" >&2 + exit 1 + fi +fi + # Build libch.so via Gluten's own wrapper. It cmakes cpp-ch into # cpp-ch/build_ch, whose build_ch target drives an inner ClickHouse cmake that # builds `--target libch` into cpp-ch/build/, so the artifact lands under From 52df6b96bc06ce33a9dfbcd07cd675965bb2c550 Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Sun, 26 Jul 2026 20:51:20 +0000 Subject: [PATCH 15/20] spark-gluten-clickhouse: skip spotless (formatter) so the CH backend builds The Scala 2.12 Option.when shim worked: backends-clickhouse now compiles (scalastyle processed 231 files, 0 errors) and the Option.when error is gone. The build then failed only at spotless:check -- the code formatter -- because the injected OptionCompat.scala lacks the Apache license header (and the import added to Snapshot.scala could likewise trip import-order formatting). Skip spotless for this build the same way we already skip checkstyle (-Dspotless.check.skip=true); it is a cosmetic formatter check, irrelevant to producing a working libch.so + plugin jar. Also add the Apache license header to the shim so it stays clean/upstreamable regardless. Co-Authored-By: Claude Opus 4.8 --- spark-gluten-clickhouse/install | 18 +++++++++++++++++- 1 file changed, 17 insertions(+), 1 deletion(-) diff --git a/spark-gluten-clickhouse/install b/spark-gluten-clickhouse/install index 0b45f9b0a4..d242687bae 100755 --- a/spark-gluten-clickhouse/install +++ b/spark-gluten-clickhouse/install @@ -120,6 +120,22 @@ SNAPSHOT_SCALA="$DELTA33_SCALA/org/apache/spark/sql/delta/Snapshot.scala" if [ -f "$SNAPSHOT_SCALA" ]; then mkdir -p "$DELTA33_SCALA/org/apache/gluten/compat" cat > "$DELTA33_SCALA/org/apache/gluten/compat/OptionCompat.scala" <<'SCALA' +/* + * Licensed to the Apache Software Foundation (ASF) under one or more + * contributor license agreements. See the NOTICE file distributed with + * this work for additional information regarding copyright ownership. + * The ASF licenses this file to You under the Apache License, Version 2.0 + * (the "License"); you may not use this file except in compliance with + * the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ package org.apache.gluten.compat // Scala 2.12 lacks Option.when/Option.unless (added in 2.13). Provide them as @@ -185,7 +201,7 @@ if [ -z "$GLUTEN_JAR" ]; then JAVA_HOME="$JAVA_HOME_8" PATH="$JAVA_HOME_8/bin:$PATH" \ mvn -B clean package \ -Pbackends-clickhouse -P"$SPARK_PROFILE" -Pscala-2.12 -Pdelta \ - -DskipTests -Dcheckstyle.skip + -DskipTests -Dcheckstyle.skip -Dspotless.check.skip=true ) GLUTEN_JAR=$(ls "$GLUTEN_DIR"/backends-clickhouse/target/gluten-*-spark-3.5-jar-with-dependencies.jar 2>/dev/null | head -n1) fi From 1a34c42125a7946b792a4d740179fd7fa3c2ac78 Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Mon, 27 Jul 2026 00:08:47 +0000 Subject: [PATCH 16/20] spark-gluten-clickhouse: raise cgroup pids.max for the native-thread ceiling Full build now succeeds on Gluten main (libch.so on CH 25.12 + all Maven modules), and the run finally reaches JVM launch -- but JavaSparkContext init still dies with `OutOfMemoryError: unable to create native thread`, unchanged by the earlier ulimit/sysctl/GC-thread-cap fixes. That pattern (ulimit raised yet still capped) points to a cgroup pids.max / systemd TasksMax ceiling on the cloud-init service cgroup, which ulimit cannot override. Print the effective ceilings (rlimits, kernel threads-max/pid_max/overcommit, and cgroup pids.max/current) as a temporary diagnostic, and raise pids.max to `max` on our cgroup and every ancestor up to the root. If TasksMax was the gate this clears it; if not, the printed numbers pin down the real limit (overcommit, stack, etc.) for a precise follow-up. Co-Authored-By: Claude Opus 4.8 --- spark-gluten-clickhouse/query | 15 +++++++++++++++ 1 file changed, 15 insertions(+) diff --git a/spark-gluten-clickhouse/query b/spark-gluten-clickhouse/query index f0a44085c2..1b2bc0956e 100755 --- a/spark-gluten-clickhouse/query +++ b/spark-gluten-clickhouse/query @@ -16,5 +16,20 @@ export PATH="$JAVA_HOME/bin:$PATH" ulimit -u unlimited 2>/dev/null || ulimit -u "$(ulimit -Hu)" 2>/dev/null || true ulimit -n 1048576 2>/dev/null || ulimit -n "$(ulimit -Hn)" 2>/dev/null || true +# The ulimit/sysctl bumps did NOT clear the native-thread error, which points to +# a cgroup pids.max (systemd TasksMax) ceiling that ulimit can't override. Print +# the effective ceilings (temporary diagnostic) and raise pids.max on our cgroup +# and every ancestor up to the root (the effective limit is the min across them). +echo "=== LIMITS: nproc(-u)=$(ulimit -u) nofile(-n)=$(ulimit -n) stack(-s)=$(ulimit -s) KiB ===" +echo "=== KERNEL: threads-max=$(cat /proc/sys/kernel/threads-max 2>/dev/null) pid_max=$(cat /proc/sys/kernel/pid_max 2>/dev/null) overcommit=$(cat /proc/sys/vm/overcommit_memory 2>/dev/null) max_map_count=$(cat /proc/sys/vm/max_map_count 2>/dev/null) ===" +cg=$(awk -F: '$1=="0"{print $3}' /proc/self/cgroup 2>/dev/null) +echo "=== CGROUP=$cg pids.max=$(cat "/sys/fs/cgroup${cg}/pids.max" 2>/dev/null) pids.current=$(cat "/sys/fs/cgroup${cg}/pids.current" 2>/dev/null) ===" +d="/sys/fs/cgroup${cg}" +while [ -n "$cg" ] && [ "$d" != "/sys/fs/cgroup" ] && [ "$d" != "/" ]; do + echo max | sudo tee "$d/pids.max" >/dev/null 2>&1 || true + d=$(dirname "$d") +done +echo "=== after raise: pids.max=$(cat "/sys/fs/cgroup${cg}/pids.max" 2>/dev/null) ===" + query=$(cat) printf '%s' "$query" | python3 query.py From 310fd6f963269e3d68498a5792e1fb15dd8701cb Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Mon, 27 Jul 2026 09:23:19 +0000 Subject: [PATCH 17/20] spark-gluten-clickhouse: send limit diagnostics to stderr; also raise overcommit The previous run's limit diagnostics never appeared: they used echo (stdout), but the harness sends this script's stdout to /dev/null and only captures stderr, so the numbers were lost and I couldn't tell whether the pids.max raise even took. The native-thread error was unchanged. Wrap the diagnostics + limit raises in { ... } >&2 so they land in the captured stderr (surfaced on the non-zero exit). Also set vm.overcommit_memory=1 to rule out a thread-stack ENOMEM under the large -Xmx, and print overcommit before and after. This run will show the real ceilings (rlimits, cgroup pids.max/current, overcommit) and apply both the pids.max and overcommit fixes. Co-Authored-By: Claude Opus 4.8 --- spark-gluten-clickhouse/query | 33 ++++++++++++++++++++------------- 1 file changed, 20 insertions(+), 13 deletions(-) diff --git a/spark-gluten-clickhouse/query b/spark-gluten-clickhouse/query index 1b2bc0956e..c089edb947 100755 --- a/spark-gluten-clickhouse/query +++ b/spark-gluten-clickhouse/query @@ -17,19 +17,26 @@ ulimit -u unlimited 2>/dev/null || ulimit -u "$(ulimit -Hu)" 2>/dev/null || true ulimit -n 1048576 2>/dev/null || ulimit -n "$(ulimit -Hn)" 2>/dev/null || true # The ulimit/sysctl bumps did NOT clear the native-thread error, which points to -# a cgroup pids.max (systemd TasksMax) ceiling that ulimit can't override. Print -# the effective ceilings (temporary diagnostic) and raise pids.max on our cgroup -# and every ancestor up to the root (the effective limit is the min across them). -echo "=== LIMITS: nproc(-u)=$(ulimit -u) nofile(-n)=$(ulimit -n) stack(-s)=$(ulimit -s) KiB ===" -echo "=== KERNEL: threads-max=$(cat /proc/sys/kernel/threads-max 2>/dev/null) pid_max=$(cat /proc/sys/kernel/pid_max 2>/dev/null) overcommit=$(cat /proc/sys/vm/overcommit_memory 2>/dev/null) max_map_count=$(cat /proc/sys/vm/max_map_count 2>/dev/null) ===" -cg=$(awk -F: '$1=="0"{print $3}' /proc/self/cgroup 2>/dev/null) -echo "=== CGROUP=$cg pids.max=$(cat "/sys/fs/cgroup${cg}/pids.max" 2>/dev/null) pids.current=$(cat "/sys/fs/cgroup${cg}/pids.current" 2>/dev/null) ===" -d="/sys/fs/cgroup${cg}" -while [ -n "$cg" ] && [ "$d" != "/sys/fs/cgroup" ] && [ "$d" != "/" ]; do - echo max | sudo tee "$d/pids.max" >/dev/null 2>&1 || true - d=$(dirname "$d") -done -echo "=== after raise: pids.max=$(cat "/sys/fs/cgroup${cg}/pids.max" 2>/dev/null) ===" +# a cgroup pids.max (systemd TasksMax) ceiling that ulimit can't override, and/or +# a memory-overcommit gate on thread stacks. Diagnose + raise both. NOTE: the +# harness sends this script's STDOUT to /dev/null and only surfaces STDERR (on a +# non-zero exit), so ALL diagnostics must go to stderr -- hence the { ... } >&2. +{ + echo "=== LIMITS: nproc(-u)=$(ulimit -u) nofile(-n)=$(ulimit -n) stack(-s)=$(ulimit -s) KiB ===" + echo "=== KERNEL: threads-max=$(cat /proc/sys/kernel/threads-max 2>/dev/null) pid_max=$(cat /proc/sys/kernel/pid_max 2>/dev/null) overcommit=$(cat /proc/sys/vm/overcommit_memory 2>/dev/null) max_map_count=$(cat /proc/sys/vm/max_map_count 2>/dev/null) ===" + cg=$(awk -F: '$1=="0"{print $3}' /proc/self/cgroup 2>/dev/null) + echo "=== CGROUP=$cg pids.max=$(cat "/sys/fs/cgroup${cg}/pids.max" 2>/dev/null) pids.current=$(cat "/sys/fs/cgroup${cg}/pids.current" 2>/dev/null) ===" + # Raise pids.max on our cgroup and every ancestor up to the root (effective + # limit is the min across them). Also allow memory overcommit so thread-stack + # reservations can't trip an ENOMEM under the big -Xmx. + d="/sys/fs/cgroup${cg}" + while [ -n "$cg" ] && [ "$d" != "/sys/fs/cgroup" ] && [ "$d" != "/" ]; do + echo max | sudo tee "$d/pids.max" >/dev/null 2>&1 || true + d=$(dirname "$d") + done + echo 1 | sudo tee /proc/sys/vm/overcommit_memory >/dev/null 2>&1 || true + echo "=== after raise: pids.max=$(cat "/sys/fs/cgroup${cg}/pids.max" 2>/dev/null) overcommit=$(cat /proc/sys/vm/overcommit_memory 2>/dev/null) ===" +} >&2 query=$(cat) printf '%s' "$query" | python3 query.py From 9157bd0993386121fc3efca2c55c06199b185d0e Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Mon, 27 Jul 2026 19:21:51 +0000 Subject: [PATCH 18/20] spark-gluten-clickhouse: measure RLIMIT_AS; cap ClickHouse thread pools Run 30253757442 finally printed the ceilings: nproc=unlimited, cgroup pids.max=max (pids.current=11), threads-max=4M, overcommit raised to 1 -- yet JavaSparkContext init STILL dies with "unable to create native thread" ~3s in, right as ClickHouse init starts. So it is neither a pids/nproc/cgroup ceiling nor overcommit. Two things remain: 1. RLIMIT_AS (address space) was never measured -- a finite value makes a thread-stack mmap fail exactly this way. Print ulimit -v/-l and the current system thread count, and lift ulimit -v. 2. libch.so sizes its thread pools to the 192 cores and creates them eagerly at init -- the leading remaining suspect. Cap ClickHouse's global and IO thread pools via the runtime_config prefix (max_thread_pool_size=512, max_io_thread_pool_size=64); 512 still far exceeds per-query parallelism so the benchmark is not throttled. Co-Authored-By: Claude Opus 4.8 --- spark-gluten-clickhouse/query | 6 +++++- spark-gluten-clickhouse/query.py | 8 ++++++++ 2 files changed, 13 insertions(+), 1 deletion(-) diff --git a/spark-gluten-clickhouse/query b/spark-gluten-clickhouse/query index c089edb947..3db41e7e73 100755 --- a/spark-gluten-clickhouse/query +++ b/spark-gluten-clickhouse/query @@ -22,7 +22,7 @@ ulimit -n 1048576 2>/dev/null || ulimit -n "$(ulimit -Hn)" 2>/dev/null || true # harness sends this script's STDOUT to /dev/null and only surfaces STDERR (on a # non-zero exit), so ALL diagnostics must go to stderr -- hence the { ... } >&2. { - echo "=== LIMITS: nproc(-u)=$(ulimit -u) nofile(-n)=$(ulimit -n) stack(-s)=$(ulimit -s) KiB ===" + echo "=== LIMITS: nproc(-u)=$(ulimit -u) nofile(-n)=$(ulimit -n) stack(-s)=$(ulimit -s) KiB addrspace(-v)=$(ulimit -v) locked(-l)=$(ulimit -l) sys-threads=$(ps -eL 2>/dev/null | wc -l) ===" echo "=== KERNEL: threads-max=$(cat /proc/sys/kernel/threads-max 2>/dev/null) pid_max=$(cat /proc/sys/kernel/pid_max 2>/dev/null) overcommit=$(cat /proc/sys/vm/overcommit_memory 2>/dev/null) max_map_count=$(cat /proc/sys/vm/max_map_count 2>/dev/null) ===" cg=$(awk -F: '$1=="0"{print $3}' /proc/self/cgroup 2>/dev/null) echo "=== CGROUP=$cg pids.max=$(cat "/sys/fs/cgroup${cg}/pids.max" 2>/dev/null) pids.current=$(cat "/sys/fs/cgroup${cg}/pids.current" 2>/dev/null) ===" @@ -37,6 +37,10 @@ ulimit -n 1048576 2>/dev/null || ulimit -n "$(ulimit -Hn)" 2>/dev/null || true echo 1 | sudo tee /proc/sys/vm/overcommit_memory >/dev/null 2>&1 || true echo "=== after raise: pids.max=$(cat "/sys/fs/cgroup${cg}/pids.max" 2>/dev/null) overcommit=$(cat /proc/sys/vm/overcommit_memory 2>/dev/null) ===" } >&2 +# RLIMIT_AS (address space) is the last unmeasured ceiling; a finite value would +# make a thread-stack mmap fail as "unable to create native thread" even with +# unlimited nproc/pids. Lift it (no-op if already unlimited). +ulimit -v unlimited 2>/dev/null || true query=$(cat) printf '%s' "$query" | python3 query.py diff --git a/spark-gluten-clickhouse/query.py b/spark-gluten-clickhouse/query.py index c5aa00ed4a..4bfab05651 100755 --- a/spark-gluten-clickhouse/query.py +++ b/spark-gluten-clickhouse/query.py @@ -180,6 +180,14 @@ def _dump_and_die(): .config("spark.shuffle.manager", "org.apache.spark.shuffle.sort.ColumnarShuffleManager") .config("spark.gluten.sql.columnar.backend.lib", "ch") .config("spark.gluten.sql.columnar.libpath", os.path.abspath("libch.so")) + # Cap ClickHouse's global/IO thread pools. On a 192-core box libch.so's + # pools are sized to the core count and their eager thread creation is the + # leading suspect for "unable to create native thread" at init (all system + # count limits are already unlimited). Passed through to ClickHouse config + # via the runtime_config prefix. 512 still far exceeds per-query + # parallelism, so this does not throttle the benchmark. + .config("spark.gluten.sql.columnar.backend.ch.runtime_config.max_thread_pool_size", "512") + .config("spark.gluten.sql.columnar.backend.ch.runtime_config.max_io_thread_pool_size", "64") .config("spark.memory.offHeap.enabled", "true") .config("spark.memory.offHeap.size", f"{off_heap}m") # Cap the JVM's helper threads: on 192 cores it otherwise spawns ~150 From 99e277dde0060296e74233bdf6c131deadf01ed7 Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Mon, 27 Jul 2026 23:25:33 +0000 Subject: [PATCH 19/20] spark-gluten-clickhouse: pin JVM to 16 CPUs to test the thread-count theory Run 30297938686 proved it is not a resource-limit ceiling: the printed limits show nproc=unlimited, pids.max=max, threads-max=4M, addrspace(-v)=unlimited, overcommit=1, max_map_count=2M -- everything unlimited -- yet JavaSparkContext init still dies with "unable to create native thread" ~3s in, and capping max_thread_pool_size didn't help. The remaining explanation: libch.so sizes its thread pools to the visible core count and eagerly spawns a huge number at init on the bare 192-core box, exhausting stack memory. Test that directly: pin the JVM (hence libch.so) to 16 CPUs with taskset, so sched_getaffinity reports 16 and ClickHouse's pools shrink ~12x. Also report the live thread count at failure, and fix the misleading "SIGKILLed" message (this is a caught Java OutOfMemoryError, not a kill). If this clears it, the real fix is a ClickHouse pool-size config that keeps full query parallelism. Co-Authored-By: Claude Opus 4.8 --- spark-gluten-clickhouse/query | 8 +++++++- spark-gluten-clickhouse/query.py | 12 +++++++++--- 2 files changed, 16 insertions(+), 4 deletions(-) diff --git a/spark-gluten-clickhouse/query b/spark-gluten-clickhouse/query index 3db41e7e73..b2ca8053f9 100755 --- a/spark-gluten-clickhouse/query +++ b/spark-gluten-clickhouse/query @@ -43,4 +43,10 @@ ulimit -n 1048576 2>/dev/null || ulimit -n "$(ulimit -Hn)" 2>/dev/null || true ulimit -v unlimited 2>/dev/null || true query=$(cat) -printf '%s' "$query" | python3 query.py +# DIAGNOSTIC: pin the JVM (and thus libch.so) to 16 CPUs. ClickHouse sizes its +# thread pools to the visible core count (sched_getaffinity), so on the bare +# 192-core box it eagerly spawns a huge number of threads at init and dies with +# "unable to create native thread" even though every system limit is unlimited. +# If capping perceived cores clears it, that confirms thread-count as the cause +# and the real fix is a ClickHouse pool-size setting (keeping full parallelism). +printf '%s' "$query" | taskset -c 0-15 python3 query.py diff --git a/spark-gluten-clickhouse/query.py b/spark-gluten-clickhouse/query.py index 4bfab05651..c687c23eee 100755 --- a/spark-gluten-clickhouse/query.py +++ b/spark-gluten-clickhouse/query.py @@ -209,13 +209,19 @@ def _dump_and_die(): def _dump_crash_artifacts(): - """Surface a JVM crash log if one exists; its absence implies SIGKILL (OOM).""" + """Surface a JVM crash log if one exists, plus the live thread count.""" import glob + import subprocess + try: + n = subprocess.run(["ps", "-eL"], capture_output=True, text=True).stdout.count("\n") + print(f"=== system thread count at failure: {n} ===", file=sys.stderr, flush=True) + except Exception as exc: # noqa: BLE001 + print(f"=== could not count threads: {exc} ===", file=sys.stderr, flush=True) files = sorted(glob.glob("hs_err_pid*.log") + glob.glob("/tmp/hs_err_pid*.log")) if not files: print( - "=== no hs_err file: JVM was SIGKILLed, not a caught crash " - "(points to earlyoom/OOM, not a native SIGSEGV) ===", + "=== no hs_err file: this is a caught Java OutOfMemoryError " + "(unable to create native thread), not a JVM/native crash ===", file=sys.stderr, flush=True, ) return From aa4c27cadd51d6a992e5cf64ad718e1522a622e1 Mon Sep 17 00:00:00 2001 From: Alexey Milovidov Date: Tue, 28 Jul 2026 01:53:06 +0000 Subject: [PATCH 20/20] spark-gluten-clickhouse: shrink static-TLS surplus 512K->128K; sample peak threads The taskset-16-CPU run confirmed it is NOT a raw thread-count explosion (all system limits proven unlimited; failure at init unchanged). The one thing our setup adds that the working Velox sibling does not is the per-thread GLIBC_TUNABLES static-TLS surplus needed to lazy-load libch.so. That surplus is paid by every thread, so the real budget is surplus x thread_count: 512 KiB let the JVM's own startup threads through but init still failed once ClickHouse added threads -- the same failure class as an over-large surplus, reached via count instead. Cut the surplus 512 KiB -> 128 KiB (4x more thread headroom, still ~79x the failing glibc default so libch.so's IE-model TLS still loads). Keep taskset constant so this is a clean single-variable test. Also add a background sampler that records the JVM subtree's PEAK thread count (the except handler counted too late, after the failed JVM tore its threads down), so the next log shows how many threads actually existed at the failure. Co-Authored-By: Claude Opus 4.8 --- spark-gluten-clickhouse/query.py | 48 +++++++++++++++++++++++++++----- 1 file changed, 41 insertions(+), 7 deletions(-) diff --git a/spark-gluten-clickhouse/query.py b/spark-gluten-clickhouse/query.py index c687c23eee..b525cba9ba 100755 --- a/spark-gluten-clickhouse/query.py +++ b/spark-gluten-clickhouse/query.py @@ -103,6 +103,32 @@ def _dump_and_die(): watchdog = threading.Timer(QUERY_TIMEOUT, _dump_and_die) watchdog.daemon = True watchdog.start() + +# Peak thread-count sampler: the except handler counts threads too late (the +# failed JVM has already torn its threads down), so sample the JVM subtree's +# thread count continuously and remember the max, to see how many threads +# actually existed at the "unable to create native thread" moment. +_peak_threads = [0] + + +def _sample_peak_threads(): + while True: + try: + total = psutil.Process().num_threads() + for child in psutil.Process().children(recursive=True): + try: + total += child.num_threads() + except Exception: # noqa: BLE001 + pass + if total > _peak_threads[0]: + _peak_threads[0] = total + except Exception: # noqa: BLE001 + pass + time.sleep(0.2) + + +_sampler = threading.Thread(target=_sample_peak_threads, daemon=True) +_sampler.start() # ----------------------------------------------------------------------------- @@ -157,12 +183,17 @@ def _dump_and_die(): # subscript` -> `[JAVA_GATEWAY_EXITED] Java gateway process exited before # sending its port number`, so all queries return null. Measured locally # (OpenJDK 17, 192 GC threads to mimic c6a.metal): the cliff is sharp and -# machine-independent — every value <= 1 MiB starts 5/5 for both the -Xmx128m -# launcher and a big-heap (-Xmx64g) gateway JVM, and 2 MiB fails 5/5 (which is -# exactly what killed the prior run). 512 KiB sits 4x under that cliff yet is -# 315x the failing glibc default (1664 B) — well clear of libch.so's small -# IE-model TLS footprint. -_TLS_SURPLUS = "glibc.rtld.optional_static_tls=524288" +# machine-independent for a FEW threads. But the surplus is per-thread, so the +# real budget is surplus x thread_count: 512 KiB let the JVM's own startup +# threads through, yet init still died with "unable to create native thread" +# once ClickHouse added its threads (all system limits were proven unlimited: +# nproc, pids.max, threads-max, addrspace, overcommit all uncapped) — the same +# class of failure as the too-large surplus, just reached via thread count +# instead. Shrinking the surplus cuts each thread's native-TLS cost, so more +# threads fit before the collision. 128 KiB is 4x smaller (4x the thread +# headroom) and still ~79x the failing glibc default (1664 B), so libch.so's +# small IE-model TLS still loads. +_TLS_SURPLUS = "glibc.rtld.optional_static_tls=131072" os.environ["GLIBC_TUNABLES"] = _TLS_SURPLUS builder = ( @@ -214,7 +245,10 @@ def _dump_crash_artifacts(): import subprocess try: n = subprocess.run(["ps", "-eL"], capture_output=True, text=True).stdout.count("\n") - print(f"=== system thread count at failure: {n} ===", file=sys.stderr, flush=True) + print( + f"=== threads at failure: system={n}, JVM-subtree peak={_peak_threads[0]} ===", + file=sys.stderr, flush=True, + ) except Exception as exc: # noqa: BLE001 print(f"=== could not count threads: {exc} ===", file=sys.stderr, flush=True) files = sorted(glob.glob("hs_err_pid*.log") + glob.glob("/tmp/hs_err_pid*.log"))