Skip to content

[SPARK-59263][BUILD] Upgrade compress-lzf to 1.2.1 #58532

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
dongjoon-hyun wants to merge 4 commits into
base: master
Choose a base branch
from
Open

[SPARK-59263][BUILD] Upgrade compress-lzf to 1.2.1 #58532

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
82d3622
[SPARK-59263] Upgrade `compress-lzf` to 1.2.1
dongjoon-hyun Sep 4, 2026
275b35f
Benchmark results for *.LZFBenchmark (JDK 25, Scala 2.13, split 1 of 1)
dongjoon-hyun Sep 4, 2026
72756e4
Benchmark results for *.LZFBenchmark (JDK 21, Scala 2.13, split 1 of 1)
dongjoon-hyun Sep 4, 2026
a5db3e5
Benchmark results for *.LZFBenchmark (JDK 17, Scala 2.13, split 1 of 1)
dongjoon-hyun Sep 4, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 6 additions & 6 deletions core/benchmarks/LZFBenchmark-jdk21-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,18 @@
Benchmark LZFCompressionCodec
================================================================================================

OpenJDK 64-Bit Server VM 21.0.12+8-LTS on Linux 6.17.0-1020-azure
OpenJDK 64-Bit Server VM 21.0.12+8-LTS on Linux 6.17.0-1022-azure
AMD EPYC 7763 64-Core Processor
Compress small objects: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------------
Compression 256000000 int values in parallel 610 616 6 419.5 2.4 1.0X
Compression 256000000 int values single-threaded 577 580 3 443.5 2.3 1.1X
Compression 256000000 int values in parallel 603 611 9 424.4 2.4 1.0X
Compression 256000000 int values single-threaded 559 564 5 457.8 2.2 1.1X

OpenJDK 64-Bit Server VM 21.0.12+8-LTS on Linux 6.17.0-1020-azure
OpenJDK 64-Bit Server VM 21.0.12+8-LTS on Linux 6.17.0-1022-azure
AMD EPYC 7763 64-Core Processor
Compress large objects: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-----------------------------------------------------------------------------------------------------------------------------
Compression 1024 array values in 1 threads 48 53 3 0.0 47285.9 1.0X
Compression 1024 array values single-threaded 33 34 1 0.0 32317.8 1.5X
Compression 1024 array values in 1 threads 49 52 2 0.0 47958.5 1.0X
Compression 1024 array values single-threaded 34 34 0 0.0 32921.4 1.5X


12 changes: 6 additions & 6 deletions core/benchmarks/LZFBenchmark-jdk25-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,18 @@
Benchmark LZFCompressionCodec
================================================================================================

OpenJDK 64-Bit Server VM 25.0.4+7-LTS on Linux 6.17.0-1020-azure
OpenJDK 64-Bit Server VM 25.0.4+7-LTS on Linux 6.17.0-1022-azure
AMD EPYC 7763 64-Core Processor
Compress small objects: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------------
Compression 256000000 int values in parallel 597 609 8 429.1 2.3 1.0X
Compression 256000000 int values single-threaded 593 596 2 431.6 2.3 1.0X
Compression 256000000 int values in parallel 603 615 9 424.3 2.4 1.0X
Compression 256000000 int values single-threaded 591 598 10 433.2 2.3 1.0X

OpenJDK 64-Bit Server VM 25.0.4+7-LTS on Linux 6.17.0-1020-azure
OpenJDK 64-Bit Server VM 25.0.4+7-LTS on Linux 6.17.0-1022-azure
AMD EPYC 7763 64-Core Processor
Compress large objects: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-----------------------------------------------------------------------------------------------------------------------------
Compression 1024 array values in 1 threads 47 51 3 0.0 45666.3 1.0X
Compression 1024 array values single-threaded 32 33 1 0.0 31432.4 1.5X
Compression 1024 array values in 1 threads 45 52 3 0.0 44214.3 1.0X
Compression 1024 array values single-threaded 33 34 1 0.0 32021.3 1.4X


16 changes: 8 additions & 8 deletions core/benchmarks/LZFBenchmark-results.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,18 +2,18 @@
Benchmark LZFCompressionCodec
================================================================================================

OpenJDK 64-Bit Server VM 17.0.20+8-LTS on Linux 6.17.0-1020-azure
AMD EPYC 9V74 80-Core Processor
OpenJDK 64-Bit Server VM 17.0.20+8-LTS on Linux 6.17.0-1022-azure
AMD EPYC 7763 64-Core Processor
Compress small objects: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
--------------------------------------------------------------------------------------------------------------------------------
Compression 256000000 int values in parallel 521 542 16 491.2 2.0 1.0X
Compression 256000000 int values single-threaded 522 538 24 490.1 2.0 1.0X
Compression 256000000 int values in parallel 597 612 12 428.8 2.3 1.0X
Compression 256000000 int values single-threaded 628 635 11 407.4 2.5 0.9X

OpenJDK 64-Bit Server VM 17.0.20+8-LTS on Linux 6.17.0-1020-azure
AMD EPYC 9V74 80-Core Processor
OpenJDK 64-Bit Server VM 17.0.20+8-LTS on Linux 6.17.0-1022-azure
AMD EPYC 7763 64-Core Processor
Compress large objects: Best Time(ms) Avg Time(ms) Stdev(ms) Rate(M/s) Per Row(ns) Relative
-----------------------------------------------------------------------------------------------------------------------------
Compression 1024 array values in 1 threads 53 56 3 0.0 51488.0 1.0X
Compression 1024 array values single-threaded 34 35 0 0.0 33270.0 1.5X
Compression 1024 array values in 1 threads 48 54 5 0.0 46539.1 1.0X
Compression 1024 array values single-threaded 33 33 0 0.0 32062.5 1.5X


2 changes: 1 addition & 1 deletion dev/deps/spark-deps-hadoop-3-hive-2.3
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,7 +61,7 @@ commons-lang3/3.20.0//commons-lang3-3.20.0.jar
commons-math3/3.6.1//commons-math3-3.6.1.jar
commons-pool/1.5.4//commons-pool-1.5.4.jar
commons-text/1.15.0//commons-text-1.15.0.jar
compress-lzf/1.2.0//compress-lzf-1.2.0.jar
compress-lzf/1.2.1//compress-lzf-1.2.1.jar
curator-client/5.9.0//curator-client-5.9.0.jar
curator-framework/5.9.0//curator-framework-5.9.0.jar
curator-recipes/5.9.0//curator-recipes-5.9.0.jar
Expand Down
2 changes: 1 addition & 1 deletion pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -868,7 +868,7 @@
<dependency>
<groupId>com.ning</groupId>
<artifactId>compress-lzf</artifactId>
<version>1.2.0</version>
<version>1.2.1</version>
</dependency>
<dependency>
<groupId>org.xerial.snappy</groupId>
Expand Down