From 6e4b40b865aa5aa635e4373da52d943545a2fabd Mon Sep 17 00:00:00 2001 From: helly25 <6420169+helly25@users.noreply.github.com> Date: Sun, 25 Oct 2026 17:13:48 +0000 Subject: [PATCH 1/2] Honor native Bazel test filtering sharding and fail-fast --- CHANGELOG.md | 4 + README.md | 4 + bashtest/README.md | 42 +++++++ bashtest/bashtest.sh | 146 ++++++++++++++++++----- bashtest/tests/external_consumer_test.sh | 73 ++++++++++++ bashtest/tests/junit_test.sh | 61 ++++++++++ tools/junit_test.py | 102 +++++++++++++++- 7 files changed, 402 insertions(+), 30 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 6f95371..8d9b4d4 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -1,5 +1,9 @@ # 0.7.1 +* Honor native Bazel test filters, sharding, and runner fail-fast. Validate filters + and shard settings before hooks run, acknowledge shard support, keep reports + local to each shard, and allow empty shards without masking an empty selection. + * Preserve assertion failures across command substitutions, subshells, and waited background jobs. Add `expect_success`, `expect_exit_code`, and opt-in `--strict-commands` checking for unhandled command failures without discarding diff --git a/README.md b/README.md index a27632d..52c67f4 100644 --- a/README.md +++ b/README.md @@ -42,6 +42,10 @@ locally or with `--sandbox_debug`. * special test function `test::test_init`: If present, then this function runs first! Tests will only be executed if it succeeds. * special test function `test::test_done`: If present, then this function runs last! +Bazel's native `--test_filter` (extended regular expression), test sharding, and +`--test_runner_fail_fast` are supported. See [selection and execution controls](bashtest/README.md#filtering-sharding-and-fail-fast) +for filter precedence, shard assignment, and teardown behavior. + ### Example 1) Write a test that sources bashtest. diff --git a/bashtest/README.md b/bashtest/README.md index 8040f88..def32ff 100644 --- a/bashtest/README.md +++ b/bashtest/README.md @@ -72,6 +72,48 @@ an explicit runner error before tests execute. The runner temporarily owns the caller's settings afterward; do not replace that trap inside a strict test. Default mode preserves existing conditional runner calls and caller error options. +## Filtering, sharding, and fail-fast + +Bazel's native filter selects bare case names without the `test::` prefix: + +```sh +bazel test //path:my_test --test_filter='^(reads|writes)_' +``` + +Bazel supplies this as `TESTBRIDGE_TEST_ONLY`. Bashtest interprets it as a Bash +extended regular expression, with substring matching unless anchored. An explicit +runner filter (`--test_arg=--test_filter=...`, `-f`, or `--gtest_filter`) replaces +that environment default, including an explicitly empty filter. If both explicit +regex and glob filters are provided, their union is selected, preserving the +existing behavior. Invalid regular expressions fail before lifecycle hooks run. +A filter selecting no cases retains exit status 2. + +`--gtest_filter` is a legacy name for one **Bash glob**. It does not implement +GoogleTest's colon-separated lists or negative-filter grammar, and `GTEST_FILTER` +is not read. Use a regex such as `^(alpha|beta)$` for multiple names. + +The macro's `shard_count` attribute and Bazel's `--test_sharding_strategy` work +with `test_runner`, including a plain `sh_test` sourcing the framework. Cases are +sorted by name in the C locale, excluding lifecycle hooks, then assigned round +robin using `TEST_TOTAL_SHARDS` and zero-based `TEST_SHARD_INDEX`. Filtering is +applied within that stable assignment; each case appears in only its owner's +console/XML report. Each shard acknowledges `TEST_SHARD_STATUS_FILE`. + +A shard with no selected cases passes if at least one case matched across the +whole suite; its hooks do not run. A completely empty shard writes a valid report +with zero cases. A shard owning only filtered-out cases reports those as skipped. +Nonempty shards run their own setup and teardown. Tests that depend on another +case's shell state must stay in one shard. Shard variables must be supplied +together as canonical decimal integers of at most nine digits, with a positive +count and an index below the count. Invalid settings or a failed shard-status +write fail before hooks execute. + +`bazel test --test_runner_fail_fast` sets `TESTBRIDGE_TEST_RUNNER_FAIL_FAST=1`. +After the first failed case, bashtest marks remaining selected cases in that +shard as skipped and still runs `test_done`. Runtime skips alone do not trigger +fail-fast, unless `--no-skip` turns them into failures. Other shards and targets +are independent; Bazel's `--notest_keep_going` controls stopping between targets. + ## Executable tools Use `tools` to build helper executables for the test execution platform and export diff --git a/bashtest/bashtest.sh b/bashtest/bashtest.sh index e44766b..21669d7 100644 --- a/bashtest/bashtest.sh +++ b/bashtest/bashtest.sh @@ -37,6 +37,10 @@ _BASHTEST_USAGE=$(cat <<'EOF' * bazel test `` [ --test_arg=-f=`` ] [ -v ] [ -u="${PWD}" ] * bazel run `` -- [ -f=`` ] [ -v ][ -u="${PWD}" ] +Native Bazel --test_filter supplies the default regex via TESTBRIDGE_TEST_ONLY. +An explicit runner filter overrides that default. Sharding and +--test_runner_fail_fast are supported; teardown still runs after a case fails. + ## Flags: * -f --test-filter `` A posix regular expression for tests to run, @@ -44,7 +48,7 @@ _BASHTEST_USAGE=$(cat <<'EOF' is considered failing if no test was run. * --gtest-output `` Write JUnit XML (also accepts GTEST_OUTPUT). Bazel sets XML_OUTPUT_FILE automatically. -* --gtest-filter Like '-f' but uses glob patterns. +* --gtest-filter One Bash glob (not GoogleTest list/exclusion syntax). * --strict-commands Record unhandled command failures using ERR traps. Requires a standalone test_runner call. * --no-skip Turn every `skip_test` into a failure. For an @@ -166,7 +170,8 @@ EOF ) _BASHTEST_REPORT_OUTPUT="${GTEST_OUTPUT-${XML_OUTPUT_FILE:+xml:${XML_OUTPUT_FILE}}}" -_BASHTEST_FILTER_REGX="" +_BASHTEST_FILTER_REGX="${TESTBRIDGE_TEST_ONLY:-}" +_BASHTEST_FILTER_FROM_ENV=1 _BASHTEST_FILTER_GLOB="" _BASHTEST_KEEP_TMPDIR="failure" _BASHTEST_UPDATE_GOLDEN="" @@ -226,9 +231,17 @@ while getopts -- '-:f:hu:v' OPTION; do OPTERR="-${OPTION}" fi case "${OPTION}" in - f|test[-_]filter) _BASHTEST_FILTER_REGX="${OPTARG}" ;; + f|test[-_]filter|gtest[-_]filter) + if [[ "${_BASHTEST_FILTER_FROM_ENV}" == 1 ]]; then + _BASHTEST_FILTER_REGX="" + _BASHTEST_FILTER_FROM_ENV=0 + fi + case "${OPTION}" in + gtest[-_]filter) _BASHTEST_FILTER_GLOB="${OPTARG}" ;; + *) _BASHTEST_FILTER_REGX="${OPTARG}" ;; + esac + ;; gtest[-_]output) _BASHTEST_REPORT_OUTPUT="${OPTARG}" ;; - gtest[-_]filter) _BASHTEST_FILTER_GLOB="${OPTARG}" ;; h|help) echo "${_BASHTEST_USAGE}"; exit 2 ;; keep[-_]tmpdir) case "${OPTARG}" in @@ -546,25 +559,72 @@ _bashtest_xml_write() { } > "${temporary}" && mv -f "${temporary}" "${_BASHTEST_XML_PATH}" } +# Match bare case names (without test::). Explicit regex/glob filters retain +# their historical union semantics; no filters selects every case. +_bashtest_selected() { + local name="${1}" + [[ -n "${_BASHTEST_FILTER_GLOB}${_BASHTEST_FILTER_REGX}" ]] || return 0 + # shellcheck disable=SC2053 # This is intentionally a Bash glob. + if [[ -n "${_BASHTEST_FILTER_GLOB}" && ${name} == ${_BASHTEST_FILTER_GLOB} ]]; then + return 0 + fi + [[ -n "${_BASHTEST_FILTER_REGX}" && ${name} =~ ${_BASHTEST_FILTER_REGX} ]] +} + +_bashtest_skip_case() { + echo "[ SKIP ] ${1}: ${2}" + ((_BASHTEST_NUM_SKIP+=1)) + _bashtest_xml_case skipped "${1}" "${2}" +} + +# Validate before any hooks run. Bounds avoid shell arithmetic overflow and +# leading-zero/octal interpretation; Bazel supplies canonical decimal integers. +_bashtest_selection_init() { + local status=0 + if [[ -n "${_BASHTEST_FILTER_REGX}" ]]; then + _bashtest_ere_match "${_BASHTEST_FILTER_REGX}" "" || status="${?}" + if [[ "${status}" == 2 ]]; then + echo >&2 "ERROR: Invalid test filter extended regular expression: ${_BASHTEST_FILTER_REGX}" + return 1 + fi + fi + _BASHTEST_TOTAL_SHARDS="${TEST_TOTAL_SHARDS-1}" + _BASHTEST_SHARD_INDEX="${TEST_SHARD_INDEX-0}" + local valid=0 + if [[ "${_BASHTEST_TOTAL_SHARDS}" =~ ^[1-9][0-9]{0,8}$ && "${_BASHTEST_SHARD_INDEX}" =~ ^(0|[1-9][0-9]{0,8})$ && "${TEST_TOTAL_SHARDS+x}" == "${TEST_SHARD_INDEX+x}" ]]; then + if (( _BASHTEST_SHARD_INDEX < _BASHTEST_TOTAL_SHARDS )); then + valid=1 + fi + fi + if [[ "${valid}" == 0 ]]; then + echo >&2 'ERROR: Invalid test shard configuration: require TEST_TOTAL_SHARDS > TEST_SHARD_INDEX >= 0 (decimal integers, at most 9 digits).' + return 1 + fi + if [[ -n "${TEST_SHARD_STATUS_FILE:-}" ]]; then + local path="${TEST_SHARD_STATUS_FILE}" + case "${OSTYPE}" in + msys*|cygwin*) path="$(cygpath -u "${path}")" ;; + esac + if ! touch "${path}"; then + echo >&2 "ERROR: Cannot acknowledge test sharding: ${path}" + return 1 + fi + fi +} + ################################################################################ function _bashtest_handler() { FUNC_NAME="${1}" TEST_NAME="${2}" - if [[ -n "${_BASHTEST_FILTER_GLOB}${_BASHTEST_FILTER_REGX}" ]]; then - PASS_GLOB= - PASS_REGX= - # shellcheck disable=SC2053 # We want glob matching here. - if [[ -n "${_BASHTEST_FILTER_GLOB}" ]] && [[ ${TEST_NAME} == ${_BASHTEST_FILTER_GLOB} ]]; then - PASS_GLOB=1 - elif [[ -n "${_BASHTEST_FILTER_REGX}" ]] && [[ ${TEST_NAME} =~ ${_BASHTEST_FILTER_REGX} ]]; then - PASS_REGX=1 - fi - if [[ -z "${PASS_GLOB}${PASS_REGX}" ]]; then - echo "[ SKIP ] ${TEST_NAME}" - ((_BASHTEST_NUM_SKIP+=1)) - _bashtest_xml_case skipped "${TEST_NAME}" "Excluded by test filter or failed initialization." - return - fi + if ! _bashtest_selected "${TEST_NAME}"; then + _bashtest_skip_case "${TEST_NAME}" "Excluded by test filter." + return + elif [[ "${_BASHTEST_INIT_FAILED}" == 1 ]]; then + _bashtest_skip_case "${TEST_NAME}" "Test initialization failed." + return + elif [[ "${_BASHTEST_STOP}" == 1 ]]; then + _bashtest_skip_case "${TEST_NAME}" "Not run after an earlier failure (fail-fast)." + return fi echo "[ TEST ] ${TEST_NAME}" _bashtest_reset_error @@ -640,6 +700,12 @@ function test_runner() { return 1 fi fi + if ! _bashtest_selection_init; then + _bashtest_xml_case error test_runner "Invalid test selection or sharding configuration." + _BASHTEST_XML_RUNNING=0 + _bashtest_xml_write || return 1 + return 1 + fi if [[ -n "${_BASHTEST_FILTER_GLOB}" ]]; then echo "Test filter glob: '${_BASHTEST_FILTER_GLOB}'." fi @@ -648,27 +714,49 @@ function test_runner() { fi echo "----------" TEST_FUNCS=() - while IFS='' read -r line; do TEST_FUNCS+=("${line}"); done < <(declare -F|LC_ALL=C sed -rne 's,^declare -f test::(.*)$,\1,p') - if _bashtest_contains_element "test_init" ${TEST_FUNCS[@]+"${TEST_FUNCS[@]}"}; then + while IFS='' read -r line; do TEST_FUNCS+=("${line}"); done < <(declare -F|LC_ALL=C sed -rne 's,^declare -f test::(.*)$,\1,p'|LC_ALL=C sort) + local ordinal=0 selected=0 assigned=0 + local owned=() + for TEST in ${TEST_FUNCS[@]+"${TEST_FUNCS[@]}"}; do + [[ "${TEST}" != test_init && "${TEST}" != test_done ]] || continue + if _bashtest_selected "${TEST}"; then + ((selected+=1)) + fi + if (( ordinal % _BASHTEST_TOTAL_SHARDS == _BASHTEST_SHARD_INDEX )); then + owned+=("${TEST}") + if _bashtest_selected "${TEST}"; then + ((assigned+=1)) + fi + fi + ((ordinal+=1)) + done + # Empty shards pass only when some case matched globally. Filters selecting + # nothing and a suite containing no cases retain the existing error status. + local empty_shard=0 + if (( selected > 0 && assigned == 0 )); then + empty_shard=1 + fi + _BASHTEST_STOP=0 + if [[ "${empty_shard}" == 0 ]] && _bashtest_contains_element "test_init" ${TEST_FUNCS[@]+"${TEST_FUNCS[@]}"}; then _bashtest_reset_error _bashtest_invoke test::test_init if [[ "${_BASHTEST_RESULT}" != 0 || "${_BASHTEST_HAS_ERROR}" != 0 ]]; then _bashtest_xml_case error test_init "Test initialization failed." _BASHTEST_INIT_FAILED=1 - _BASHTEST_FILTER_GLOB=("${TEST_FUNCS[*]}") echo "Test init (test::test_init) failed! Skipping all tests." else _bashtest_xml_case ignored test_init fi echo "----------" fi - for TEST in ${TEST_FUNCS[@]+"${TEST_FUNCS[@]}"}; do - if [[ "${TEST}" != "test_init" ]] && [[ "${TEST}" != "test_done" ]]; then - _bashtest_handler "test::${TEST}" "${TEST}" - echo "----------" + for TEST in ${owned[@]+"${owned[@]}"}; do + _bashtest_handler "test::${TEST}" "${TEST}" + if [[ "${TESTBRIDGE_TEST_RUNNER_FAIL_FAST:-}" == 1 && "${_BASHTEST_NUM_FAIL}" != 0 ]]; then + _BASHTEST_STOP=1 fi + echo "----------" done - if _bashtest_contains_element "test_done" ${TEST_FUNCS[@]+"${TEST_FUNCS[@]}"}; then + if [[ "${empty_shard}" == 0 ]] && _bashtest_contains_element "test_done" ${TEST_FUNCS[@]+"${TEST_FUNCS[@]}"}; then _bashtest_reset_error _bashtest_invoke test::test_done if [[ "${_BASHTEST_RESULT}" != 0 || "${_BASHTEST_HAS_ERROR}" != 0 ]]; then @@ -679,7 +767,7 @@ function test_runner() { fi echo "----------" fi - if [[ "${_BASHTEST_NUM_PASS}" == 0 && "${_BASHTEST_NUM_FAIL}" == 0 && "${_BASHTEST_INIT_FAILED}" == 0 ]]; then + if [[ "${empty_shard}" == 0 && "${_BASHTEST_NUM_PASS}" == 0 && "${_BASHTEST_NUM_FAIL}" == 0 && "${_BASHTEST_INIT_FAILED}" == 0 ]]; then _bashtest_xml_case error test_runner "No tests were run." fi if ! _bashtest_xml_write; then @@ -704,7 +792,7 @@ function test_runner() { echo " > bazel run ${TEST_TARGET:-} -- -u=\"\${PWD}\"" fi return 1 - elif [[ "${_BASHTEST_NUM_PASS}" == 0 ]]; then + elif [[ "${empty_shard}" == 0 && "${_BASHTEST_NUM_PASS}" == 0 ]]; then echo >&2 "ERROR: No tests were run." return 2 fi diff --git a/bashtest/tests/external_consumer_test.sh b/bashtest/tests/external_consumer_test.sh index a46477e..6d32315 100755 --- a/bashtest/tests/external_consumer_test.sh +++ b/bashtest/tests/external_consumer_test.sh @@ -141,3 +141,76 @@ for target in "${junit_targets[@]}"; do grep -F 'actual & value' "${report}" >/dev/null echo "[EXPECTED FAILURE: JUnit fixture] ${target}: Bazel collected per-function failure XML" done + +# Exercise native Bazel flags through the macro and plain sh_test. These manual +# fixtures are invoked explicitly because their unfiltered beta case must fail. +cat >> BUILD.bazel <<'EOF' +bashtest( + name = "selection_test", + srcs = ["selection.sh"], + tags = ["manual"], +) +EOF +cat > selection.sh <<'EOF' +#!/usr/bin/env bash +set -euo pipefail +# shellcheck disable=SC1090,SC1091,SC2154 +source "${mboworks_bashtest}" +# shellcheck disable=SC2317 +test::alpha() { echo CASE:alpha; if [[ "${FAIL_CASE:-}" == alpha ]]; then expect_eq yes no; fi; } +# shellcheck disable=SC2317 +test::beta() { echo CASE:beta; expect_eq filtered_case must_not_run; } +# shellcheck disable=SC2317 +test::gamma() { echo CASE:gamma; } +# shellcheck disable=SC2317 +test::test_done() { echo DONE; } +test_runner +EOF +chmod +x selection.sh +selection_targets=(//:selection_test) +case "${OSTYPE}" in + msys*|cygwin*) ;; + *) + cat >> BUILD.bazel <<'EOF' +sh_test( + name = "selection_sh_test", + srcs = ["selection.sh"], + data = ["@framework//bashtest:bashtest_sh"], + env = {"mboworks_bashtest": "$(rootpath @framework//bashtest:bashtest_sh)"}, + tags = ["manual"], +) +EOF + selection_targets+=(//:selection_sh_test) + ;; +esac +bazel test --enable_runfiles=no --test_filter='^(alpha|gamma)$' "${selection_targets[@]}" +bazel test --enable_runfiles=no --test_filter='^(alpha|gamma)$' \ + --test_sharding_strategy=forced=5 "${selection_targets[@]}" +for target in "${selection_targets[@]}"; do + reports=("${junit_logs}/${target#//:}"/shard_*/test.xml) + [[ "${#reports[@]}" == 5 ]] + alpha=0 gamma=0 empty=0 + for report in "${reports[@]}"; do + if grep -F 'name="alpha"' "${report}" >/dev/null; then ((alpha+=1)); fi + if grep -F 'name="gamma"' "${report}" >/dev/null; then ((gamma+=1)); fi + if grep -F 'tests="0" failures="0" errors="0"' "${report}" >/dev/null; then ((empty+=1)); fi + done + [[ "${alpha}" == 1 && "${gamma}" == 1 && "${empty}" == 2 ]] + echo "[SHARDING VERIFIED] ${target}: five reports, each selected case once, two empty reports" +done +if bazel test --enable_runfiles=no --test_runner_fail_fast --test_env=FAIL_CASE=alpha \ + "${selection_targets[@]}" > selection_failure.log 2>&1; then + echo >&2 'ERROR: intentionally failing fail-fast fixtures passed' + exit 1 +fi +for target in "${selection_targets[@]}"; do + report="${junit_logs}/${target#//:}/test.xml" + grep -F 'tests="3" failures="1" errors="0" skipped="2"' "${report}" >/dev/null + grep -F 'fail-fast' "${report}" >/dev/null + grep -F 'DONE' "${junit_logs}/${target#//:}/test.log" >/dev/null + if grep -F 'CASE:beta' "${junit_logs}/${target#//:}/test.log"; then + echo >&2 'ERROR: fail-fast executed a later case' + exit 1 + fi + echo "[EXPECTED FAILURE: fail-fast fixture] ${target}: later cases skipped and teardown ran" +done diff --git a/bashtest/tests/junit_test.sh b/bashtest/tests/junit_test.sh index 78e1d69..98a8dd7 100755 --- a/bashtest/tests/junit_test.sh +++ b/bashtest/tests/junit_test.sh @@ -153,4 +153,65 @@ PROBE expect_output_contains 'errors="1"' "${output}" } +# shellcheck disable=SC2317 # Functions are discovered by test_runner. +test::bazel_selection() { + local root report output status index name + root="$(test_tmpdir selection)" + report="${root}/result.xml" + cat > "${root}/probe.sh" <<'PROBE' +set -euo pipefail +# shellcheck disable=SC1090,SC1091,SC2154 +source "${mboworks_bashtest}" +test::test_init() { echo INIT; } +test::alpha() { echo CASE:alpha; if [[ "${FAIL_CASE:-}" == alpha ]]; then expect_eq yes no; fi; } +test::beta() { echo CASE:beta; } +test::gamma() { echo CASE:gamma; } +test::test_done() { echo DONE; } +test_runner +PROBE + # Nested fixtures own their selection environment and never acknowledge the + # parent target's shard status file. + local command=(env TEST_TOTAL_SHARDS=1 TEST_SHARD_INDEX=0 TEST_SHARD_STATUS_FILE= + TESTBRIDGE_TEST_RUNNER_FAIL_FAST=0 TESTBRIDGE_TEST_ONLY= "GTEST_OUTPUT=xml:${report}") + output="$("${command[@]}" TESTBRIDGE_TEST_ONLY='^alpha$' "${BASH}" "${root}/probe.sh")" + expect_output_contains 'CASE:alpha' "${output}" + expect_output_not_contains 'CASE:beta' "${output}" + output="$("${command[@]}" TESTBRIDGE_TEST_ONLY='^alpha$' "${BASH}" "${root}/probe.sh" --gtest_filter='bet*')" + expect_output_contains 'CASE:beta' "${output}" + expect_output_not_contains 'CASE:alpha' "${output}" + for index in 0 1 2 3 4; do + output="$("${command[@]}" TEST_TOTAL_SHARDS=5 TEST_SHARD_INDEX="${index}" \ + TEST_SHARD_STATUS_FILE="${root}/status-${index}" "${BASH}" "${root}/probe.sh")" + expect_success test -f "${root}/status-${index}" + case "${index}" in + 0) name=alpha ;; + 1) name=beta ;; + 2) name=gamma ;; + *) name= ;; + esac + if [[ -n "${name}" ]]; then + expect_eq "CASE:${name}" "$(printf '%s\n' "${output}" | grep '^CASE:')" + else + expect_output_not_contains 'CASE:' "${output}" + expect_output_not_contains INIT "${output}" + expect_output_not_contains DONE "${output}" + expect_output_contains 'tests="0" failures="0" errors="0"' "$(cat "${report}")" + fi + done + status=0 + output="$("${command[@]}" FAIL_CASE=alpha TESTBRIDGE_TEST_RUNNER_FAIL_FAST=1 \ + "${BASH}" "${root}/probe.sh" 2>&1)" || status="${?}" + expect_eq 1 "${status}" + expect_output_contains DONE "${output}" + expect_output_not_contains 'CASE:beta' "${output}" + expect_output_not_contains 'CASE:gamma' "${output}" + expect_output_contains 'tests="3" failures="1" errors="0" skipped="2"' "$(cat "${report}")" + status=0 + output="$("${command[@]}" TEST_TOTAL_SHARDS=2 TEST_SHARD_INDEX=2 \ + "${BASH}" "${root}/probe.sh" 2>&1)" || status="${?}" + expect_eq 1 "${status}" + expect_output_contains 'Invalid test shard configuration' "${output}" + expect_output_not_contains INIT "${output}" +} + test_runner diff --git a/tools/junit_test.py b/tools/junit_test.py index 0e2595f..055f17d 100644 --- a/tools/junit_test.py +++ b/tools/junit_test.py @@ -24,7 +24,9 @@ def run_probe(self, body, *args, output=True, runner='test_runner', **environmen script = self.root / 'probe.sh' script.write_text('set -euo pipefail\nsource "$LIBRARY"\n' + body + '\n' + runner + '\n') env = {key: value for key, value in os.environ.items() - if key not in ('GTEST_OUTPUT', 'XML_OUTPUT_FILE', 'TEST_TARGET')} + if key not in ('GTEST_OUTPUT', 'XML_OUTPUT_FILE', 'TEST_TARGET', + 'TESTBRIDGE_TEST_ONLY', 'TESTBRIDGE_TEST_RUNNER_FAIL_FAST', + 'TEST_TOTAL_SHARDS', 'TEST_SHARD_INDEX', 'TEST_SHARD_STATUS_FILE')} env.update(LIBRARY=str(ROOT / 'bashtest/bashtest.sh'), TEST_TMPDIR=str(self.root)) if output: env['XML_OUTPUT_FILE'] = str(self.xml) @@ -192,6 +194,104 @@ def test_command_assertions_work_without_strict_mode(self): self.assertEqual(result.returncode, 1, result.stderr) self.assertIsNotNone(self.report()['bad'].find('failure')) + def test_native_bazel_filter_and_explicit_override(self): + body = 'test::alpha() { echo alpha_body; }\ntest::beta() { echo beta_body; }' + for args, expected in (((), 'alpha'), (('--test_filter=^beta$',), 'beta'), + (('--gtest_filter=bet*',), 'beta')): + result = self.run_probe(body, *args, TESTBRIDGE_TEST_ONLY='^alpha$') + self.assertEqual(result.returncode, 0, result.stderr) + cases = self.report() + self.assertIsNone(cases[expected].find('skipped')) + self.assertIsNotNone(cases['beta' if expected == 'alpha' else 'alpha'].find('skipped')) + result = self.run_probe(body, '--test_filter=', TESTBRIDGE_TEST_ONLY='absent') + self.assertEqual(result.returncode, 0, result.stderr) + self.assertTrue(all(case.find('skipped') is None for case in self.report().values())) + # Preserve the existing union when both explicit filter types are given. + result = self.run_probe(body, '--test_filter=^alpha$', '--gtest_filter=bet*') + self.assertEqual(result.returncode, 0, result.stderr) + self.assertTrue(all(case.find('skipped') is None for case in self.report().values())) + + def test_invalid_filter_rejected_before_hooks(self): + result = self.run_probe('test::test_init() { echo must_not_run; }\n' + 'test::body() { :; }', TESTBRIDGE_TEST_ONLY='[') + self.assertEqual(result.returncode, 1, result.stderr) + self.assertIn(b'Invalid test filter', result.stderr) + self.assertNotIn(b'must_not_run', result.stdout) + self.assertIsNotNone(self.report()['test_runner'].find('error')) + + def test_shards_partition_cases_and_allow_empty_shards(self): + body = ("test::test_init() { echo init_hook; }\n" + "test::test_done() { echo done_hook; }\n" + "test::alpha() { :; }\ntest::beta() { :; }\ntest::gamma() { :; }") + names = [] + for index in range(5): + with self.subTest(index=index): + status = self.root / f'shard-{index}' + result = self.run_probe(body, TEST_TOTAL_SHARDS='5', TEST_SHARD_INDEX=str(index), + TEST_SHARD_STATUS_FILE=str(status)) + self.assertEqual(result.returncode, 0, result.stderr) + self.assertTrue(status.exists()) + cases = self.report() + expected = ('alpha', 'beta', 'gamma')[index:index + 1] + self.assertEqual(list(cases), list(expected)) + names.extend(cases) + self.assertEqual(b'init_hook' in result.stdout, bool(expected)) + self.assertEqual(b'done_hook' in result.stdout, bool(expected)) + self.assertEqual(names, ['alpha', 'beta', 'gamma']) + # Partition the full sorted list consistently, then filter within shards. + result = self.run_probe(body, TEST_TOTAL_SHARDS='2', TEST_SHARD_INDEX='1', + TESTBRIDGE_TEST_ONLY='^alpha$') + self.assertEqual(result.returncode, 0, result.stderr) + self.assertEqual(list(self.report()), ['beta']) + self.assertIsNotNone(self.report()['beta'].find('skipped')) + result = self.run_probe(body, TEST_TOTAL_SHARDS='2', TEST_SHARD_INDEX='1', + TESTBRIDGE_TEST_ONLY='absent') + self.assertEqual(result.returncode, 2, result.stderr) + self.assertIsNotNone(self.report()['test_runner'].find('error')) + + def test_invalid_sharding_and_handshake_failures(self): + for environment in ({'TEST_TOTAL_SHARDS': '0', 'TEST_SHARD_INDEX': '0'}, + {'TEST_TOTAL_SHARDS': '2', 'TEST_SHARD_INDEX': '-1'}, + {'TEST_TOTAL_SHARDS': '2', 'TEST_SHARD_INDEX': '2'}, + {'TEST_TOTAL_SHARDS': 'two', 'TEST_SHARD_INDEX': '0'}, + {'TEST_TOTAL_SHARDS': '2'}, {'TEST_SHARD_INDEX': '0'}, + {'TEST_TOTAL_SHARDS': '9999999999999999999999', 'TEST_SHARD_INDEX': '0'}): + with self.subTest(environment=environment): + result = self.run_probe('test::body() { echo must_not_run; }', **environment) + self.assertEqual(result.returncode, 1, result.stderr) + self.assertIn(b'Invalid test shard configuration', result.stderr) + self.assertNotIn(b'must_not_run', result.stdout) + self.assertIsNotNone(self.report()['test_runner'].find('error')) + result = self.run_probe('test::body() { :; }', TEST_TOTAL_SHARDS='1', + TEST_SHARD_INDEX='0', TEST_SHARD_STATUS_FILE=str(self.root / 'missing/status')) + self.assertEqual(result.returncode, 1, result.stderr) + self.assertIn(b'Cannot acknowledge test sharding', result.stderr) + + def test_fail_fast_reports_remaining_cases_and_still_runs_teardown(self): + body = ("test::alpha() { expect_eq yes no; return 0; }\n" + "test::beta() { echo beta_body; }\ntest::gamma() { :; }\n" + "test::test_done() { echo done_hook; }") + for flag in ('1', '0', ''): + result = self.run_probe(body, TESTBRIDGE_TEST_RUNNER_FAIL_FAST=flag) + self.assertEqual(result.returncode, 1, result.stderr) + cases = self.report() + self.assertIsNotNone(cases['alpha'].find('failure')) + self.assertIn(b'done_hook', result.stdout) + if flag == '1': + self.assertNotIn(b'beta_body', result.stdout) + self.assertIn('fail-fast', cases['beta'].find('skipped').attrib['message']) + self.assertIsNotNone(cases['gamma'].find('skipped')) + else: + self.assertIn(b'beta_body', result.stdout) + self.assertIsNone(cases['beta'].find('skipped')) + result = self.run_probe(body, TESTBRIDGE_TEST_RUNNER_FAIL_FAST='1', TESTBRIDGE_TEST_ONLY='beta') + self.assertEqual(result.returncode, 0, result.stderr) + self.assertIsNone(self.report()['beta'].find('skipped')) + result = self.run_probe('test::alpha() { skip_test absent; }\ntest::beta() { :; }', + '--no-skip', TESTBRIDGE_TEST_RUNNER_FAIL_FAST='1') + self.assertEqual(result.returncode, 1, result.stderr) + self.assertIsNotNone(self.report()['beta'].find('skipped')) + def test_skip_is_failure_under_no_skip(self): result = self.run_probe('test::skip() { skip_test "not available"; }', '--no-skip') self.assertEqual(result.returncode, 1) From 0f16881fcbc76dc0779dc11287180578a8b99ca0 Mon Sep 17 00:00:00 2001 From: helly25 <6420169+helly25@users.noreply.github.com> Date: Sun, 25 Oct 2026 17:19:00 +0000 Subject: [PATCH 2/2] Isolate nested verifier selection from the outer Bazel test --- bashtest/README.md | 5 +++++ bashtest/tests/external_consumer_test.sh | 3 +++ bashtest/tests/junit_test.sh | 22 +++++++++++++--------- 3 files changed, 21 insertions(+), 9 deletions(-) diff --git a/bashtest/README.md b/bashtest/README.md index def32ff..6a089f6 100644 --- a/bashtest/README.md +++ b/bashtest/README.md @@ -191,3 +191,8 @@ no custom report exists. Reporting begins when `test_runner` is called. Nested runner/verifier scripts should select their own output file, or disable reporting with `GTEST_OUTPUT=`, so an expected failure does not become the parent target's report. +Verifier fixtures should also choose their own selection environment: clear +`TESTBRIDGE_TEST_ONLY`, set `TESTBRIDGE_TEST_RUNNER_FAIL_FAST=0`, and use +`TEST_TOTAL_SHARDS=1 TEST_SHARD_INDEX=0 TEST_SHARD_STATUS_FILE=` for an unsharded +child. This prevents the parent's filter, shard assignment, or acknowledgement +file from changing the fixture being verified. diff --git a/bashtest/tests/external_consumer_test.sh b/bashtest/tests/external_consumer_test.sh index 6d32315..02c1fda 100755 --- a/bashtest/tests/external_consumer_test.sh +++ b/bashtest/tests/external_consumer_test.sh @@ -66,6 +66,9 @@ EOF # Also run the framework's own suite as an external dependency, as BCR does. # A standalone consumer test alone cannot catch root-workspace assumptions in it. bazel test --enable_runfiles=no --test_output=errors //:consumer_test @framework//bashtest/tests/... +# A filtered/sharded meta-test must not leak its settings into verifier fixtures. +bazel test --enable_runfiles=no --test_filter='^junit_cases_and_failures$' \ + --test_sharding_strategy=forced=5 @framework//bashtest/tests:junit_test bazel run --enable_runfiles=no //:consumer_test -- --keep-tmpdir=never # Exercise both documented update spellings with a workspace path containing spaces. diff --git a/bashtest/tests/junit_test.sh b/bashtest/tests/junit_test.sh index 98a8dd7..04e5655 100755 --- a/bashtest/tests/junit_test.sh +++ b/bashtest/tests/junit_test.sh @@ -7,6 +7,13 @@ set -euo pipefail # shellcheck disable=SC1090,SC1091,SC2154 source "${mboworks_bashtest}" +# Verifier fixtures select their own cases, independently of the outer test's +# filter/shard/fail-fast settings. Additional env arguments can override these. +_junit_probe() { + env TEST_TOTAL_SHARDS=1 TEST_SHARD_INDEX=0 TEST_SHARD_STATUS_FILE= \ + TESTBRIDGE_TEST_RUNNER_FAIL_FAST=0 TESTBRIDGE_TEST_ONLY= "${@}" +} + # shellcheck disable=SC2317 # Functions are discovered by test_runner. test::junit_cases_and_failures() { local root output status=0 report @@ -24,7 +31,7 @@ test::c_skip() { skip_test 'missing "dependency"'; } test::test_done() { expect_eq changed "${state}"; } test_runner PROBE - output="$(env GTEST_OUTPUT="xml:${report}" "${BASH}" "${root}/probe.sh" --keep-tmpdir=never 2>&1)" || status="${?}" + output="$(_junit_probe GTEST_OUTPUT="xml:${report}" "${BASH}" "${root}/probe.sh" --keep-tmpdir=never 2>&1)" || status="${?}" expect_eq 1 "${status}" expect_output_contains 'Some tests failed.' "${output}" output="$(cat "${report}")" @@ -47,7 +54,7 @@ source "${mboworks_bashtest}" test::abort() { echo 'before abort' >&2; exit 7; } test_runner PROBE - output="$(env GTEST_OUTPUT= XML_OUTPUT_FILE="${report}" "${BASH}" "${root}/probe.sh" \ + output="$(_junit_probe GTEST_OUTPUT= XML_OUTPUT_FILE="${report}" "${BASH}" "${root}/probe.sh" \ --gtest_output="xml:${report}" --keep-tmpdir=never 2>&1)" || status="${?}" expect_eq 7 "${status}" expect_output_contains 'before abort' "${output}" @@ -86,7 +93,7 @@ PROBE status=0 expected=1 [[ "${probe}" != valid_negative ]] || expected=0 - output="$(env PROBE="${probe}" GTEST_OUTPUT="xml:${report}" "${BASH}" \ + output="$(_junit_probe PROBE="${probe}" GTEST_OUTPUT="xml:${report}" "${BASH}" \ "${root}/probe.sh" --keep-tmpdir=never 2>&1)" || status="${?}" expect_eq "${expected}" "${status}" case "${probe}" in @@ -132,7 +139,7 @@ PROBE status=0 expected=1 [[ "${probe}" != handled ]] || expected=0 - output="$(env PROBE="${probe}" GTEST_OUTPUT="xml:${report}" "${BASH}" \ + output="$(_junit_probe PROBE="${probe}" GTEST_OUTPUT="xml:${report}" "${BASH}" \ "${root}/probe.sh" --strict-commands --keep-tmpdir=never 2>&1)" || status="${?}" expect_eq "${expected}" "${status}" expect_output_contains '[ PASS ] b_state' "${output}" @@ -144,7 +151,7 @@ PROBE done sed 's/^test_runner$/if test_runner; then exit 9; else exit 1; fi/' "${root}/probe.sh" > "${root}/conditional.sh" status=0 - output="$(env PROBE=handled GTEST_OUTPUT="xml:${report}" "${BASH}" \ + output="$(_junit_probe PROBE=handled GTEST_OUTPUT="xml:${report}" "${BASH}" \ "${root}/conditional.sh" --strict-commands --keep-tmpdir=never 2>&1)" || status="${?}" expect_eq 1 "${status}" expect_output_contains 'requires a standalone test_runner' "${output}" @@ -169,10 +176,7 @@ test::gamma() { echo CASE:gamma; } test::test_done() { echo DONE; } test_runner PROBE - # Nested fixtures own their selection environment and never acknowledge the - # parent target's shard status file. - local command=(env TEST_TOTAL_SHARDS=1 TEST_SHARD_INDEX=0 TEST_SHARD_STATUS_FILE= - TESTBRIDGE_TEST_RUNNER_FAIL_FAST=0 TESTBRIDGE_TEST_ONLY= "GTEST_OUTPUT=xml:${report}") + local command=(_junit_probe "GTEST_OUTPUT=xml:${report}") output="$("${command[@]}" TESTBRIDGE_TEST_ONLY='^alpha$' "${BASH}" "${root}/probe.sh")" expect_output_contains 'CASE:alpha' "${output}" expect_output_not_contains 'CASE:beta' "${output}"