Pgbench kernel ab benchmark - #31
Open
nmanthey wants to merge 8 commits into
Open
Conversation
On a system with multiple python environments, we might want to run with a different version. Therefore, allow the script to select a python version. This change also helps when adding support to new python versions. Signed-off-by: Norbert Manthey <nmanthey@amazon.de>
summarize_unixbench_log built the metric name from fields 1..NF-4, which pulled the numeric value, unit and part of the timing info into the metric name — producing malformed rows like 'Arithmetic_Test_(double)_385400605.9_lps' with a single sample each, instead of one 'Arithmetic_Test_(double)' metric aggregated across VMs. Use NF-6 for the metric name (matching unixbench-kernel-regression) so value=$(NF-5) and unit=$(NF-4) line up, and drop the index-section parsing that emitted duplicate/derived score metrics. Verified against sample UnixBench output: clean metric names, correct value/unit. Signed-off-by: Norbert Manthey <nmanthey@amazon.de>
UnixBench (and other CPU-bound benchmarks) run for many minutes with no new console output during the benchmark phase. The 600s hang-detection default tripped mid-run and killed all VMs as a false-positive stall. Raise the PULLAB_TASK_HANG_THRESHOLD_SEC default from 600 to 1200s so these benchmarks complete, while still catching genuine hangs within a reasonable window. The value remains env-overridable for lighter workloads that want faster detection. Signed-off-by: Norbert Manthey <nmanthey@amazon.de>
The kernel A/B tests each carried their own copy of the kernel install/reboot helpers, so any change had to be made in every test. Introduce vm-tests/lib/kernel_helpers.sh as the single home for that logic (environment validation, kernel RPM download/selection, install_kernel_rpm with grubby boot-entry management, and the get_running_kernel / assert_kernel_changed helpers). Each kernel test includes it via a kernel_helpers.sh symlink and sources it, keeping only its test-specific functions: - example-kernel-reboot-test: none (pure kernel install/reboot). - simple-source-reboot: source-RPM build helpers. - unixbench-kernel-regression: UnixBench prepare/run/summarize. The symlink is stored by the payload zip as real content, so the VM sees a normal file; no pipeline change is needed. Subsequent fixes to the kernel logic now land once in the shared lib. Signed-off-by: Norbert Manthey <nmanthey@amazon.de>
A kernel built with make binrpm-pkg and LOCALVERSION=-nogup has an RPM VERSION of 6.18.41_nogup (underscore) but installs vmlinuz-6.18.41-nogup (dash). The vmlinuz lookup missed the file because it used the RPM VERSION verbatim. Compute an alternate version string with underscores replaced by dashes, try both in the grubby --info and vmlinuz globs, and derive the kernel version for dracut/initramfs from the actual vmlinuz filename. Signed-off-by: Norbert Manthey <nmanthey@amazon.de>
AL2023 enables FIPS by default. A custom kernel built with make binrpm-pkg carries unsigned modules (e.g. ghash_clmulni_intel) that fail FIPS signature verification, causing a kernel panic reboot loop. Add fips=0 to the grubby boot-entry args and run fips-mode-setup --disable after installing the kernel, before the reboot, so unsigned modules load without panic. Signed-off-by: Norbert Manthey <nmanthey@amazon.de>
On an AL2023 AMI whose default kernel is a different series than the RPM under test (e.g. a 6.18 AMI installing a 6.1 kernel), the distro kernel<N>-tools package declares 'conflicts with kernel-uname-r < <N>', so a plain dnf/yum install is refused with 'conflicting requests'. Verified on a live 6.18 AMI: plain install fails, but 'dnf install --allowerasing' removes the conflicting kernel<N>-tools package and installs the requested kernel; both vmlinuz files remain in /boot so the target kernel boots normally. Add --allowerasing as the final fallback in install_kernel_rpm, making the kernel A/B tests robust to base-AMI kernel-series drift. Signed-off-by: Norbert Manthey <nmanthey@amazon.de>
Add a kernel A/B performance regression test using PostgreSQL pgbench. It installs two kernels in turn on a single VM and runs the same read-only and read-write pgbench workload against each, emitting benchmark-base-*.csv / benchmark-tip-*.csv for the pipeline's benchmark analyzer (which compares the distributions across VMs with Welch's t-test, Mann-Whitney U and Cohen's d). The test is self-contained: dependencies come from dependencies.txt and results use the same benchmark-*.csv schema the analyzer already consumes. The test consumes the shared kernel-management helpers via a kernel_helpers.sh symlink into vm-tests/lib/; only the PostgreSQL/pgbench functions live in its common_lib.sh. Uses postgresql16 (natively available on AL2023; binaries under /usr/bin). Signed-off-by: Norbert Manthey <nmanthey@amazon.de>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adding the pgbench postgresql benchmark as a new highlevel regression benchmark. Coefficient of variance is low enough to consider this benchmark as a stable benchmark to detect regressions (CV 0.7–1.7%). Since the same postgresql version used. Running multiple test entries in one pipeline works.
Testing Done
Ran the pgbench-kernel-regression test end-to-end on AWS (local-orchestrator path), base kernel 6.1.141-165.249.amzn2023 vs tip 6.1.150-174.273.amzn2023, on c8i.4xlarge in us-west-2 with the al2023-ami-kernel-6.1-x86_64 AMI.
Single-VM run
postgresql.readwrite.latency_avg.
Multi-VM statistical comparison (5 VMs)
Combined config (all touched tests, 2 VMs each)
One config running pgbench-kernel-regression, unixbench-kernel-regression, and simple-unixbench at min_count: 2. 6/6 VMs succeeded. pgbench produced 8 base + 8 tip rows and compared correctly.