Support native Bazel filters sharding and fail-fast

CHANGELOG.md

@@ -1,5 +1,9 @@
 # 0.7.1
 
+* Honor native Bazel test filters, sharding, and runner fail-fast. Validate filters
+  and shard settings before hooks run, acknowledge shard support, keep reports
+  local to each shard, and allow empty shards without masking an empty selection.
+
 * Preserve assertion failures across command substitutions, subshells, and waited
   background jobs. Add `expect_success`, `expect_exit_code`, and opt-in
   `--strict-commands` checking for unhandled command failures without discarding

README.md

@@ -42,6 +42,10 @@ locally or with `--sandbox_debug`.
 * special test function `test::test_init`: If present, then this function runs first! Tests will only be executed if it succeeds.
 * special test function `test::test_done`: If present, then this function runs last!
 
+Bazel's native `--test_filter` (extended regular expression), test sharding, and
+`--test_runner_fail_fast` are supported. See [selection and execution controls](bashtest/README.md#filtering-sharding-and-fail-fast)
+for filter precedence, shard assignment, and teardown behavior.
+
 ### Example
 
 1) Write a test that sources bashtest.

bashtest/README.md

@@ -72,6 +72,48 @@ an explicit runner error before tests execute. The runner temporarily owns the
 caller's settings afterward; do not replace that trap inside a strict test.
 Default mode preserves existing conditional runner calls and caller error options.
 
+## Filtering, sharding, and fail-fast
+
+Bazel's native filter selects bare case names without the `test::` prefix:
+
+```sh
+bazel test //path:my_test --test_filter='^(reads|writes)_'
+```
+
+Bazel supplies this as `TESTBRIDGE_TEST_ONLY`. Bashtest interprets it as a Bash
+extended regular expression, with substring matching unless anchored. An explicit
+runner filter (`--test_arg=--test_filter=...`, `-f`, or `--gtest_filter`) replaces
+that environment default, including an explicitly empty filter. If both explicit
+regex and glob filters are provided, their union is selected, preserving the
+existing behavior. Invalid regular expressions fail before lifecycle hooks run.
+A filter selecting no cases retains exit status 2.
+
+`--gtest_filter` is a legacy name for one **Bash glob**. It does not implement
+GoogleTest's colon-separated lists or negative-filter grammar, and `GTEST_FILTER`
+is not read. Use a regex such as `^(alpha|beta)$` for multiple names.
+
+The macro's `shard_count` attribute and Bazel's `--test_sharding_strategy` work
+with `test_runner`, including a plain `sh_test` sourcing the framework. Cases are
+sorted by name in the C locale, excluding lifecycle hooks, then assigned round
+robin using `TEST_TOTAL_SHARDS` and zero-based `TEST_SHARD_INDEX`. Filtering is
+applied within that stable assignment; each case appears in only its owner's
+console/XML report. Each shard acknowledges `TEST_SHARD_STATUS_FILE`.
+
+A shard with no selected cases passes if at least one case matched across the
+whole suite; its hooks do not run. A completely empty shard writes a valid report
+with zero cases. A shard owning only filtered-out cases reports those as skipped.
+Nonempty shards run their own setup and teardown. Tests that depend on another
+case's shell state must stay in one shard. Shard variables must be supplied
+together as canonical decimal integers of at most nine digits, with a positive
+count and an index below the count. Invalid settings or a failed shard-status
+write fail before hooks execute.
+
+`bazel test --test_runner_fail_fast` sets `TESTBRIDGE_TEST_RUNNER_FAIL_FAST=1`.
+After the first failed case, bashtest marks remaining selected cases in that
+shard as skipped and still runs `test_done`. Runtime skips alone do not trigger
+fail-fast, unless `--no-skip` turns them into failures. Other shards and targets
+are independent; Bazel's `--notest_keep_going` controls stopping between targets.
+
 ## Executable tools
 
 Use `tools` to build helper executables for the test execution platform and export
@@ -149,3 +191,8 @@ no custom report exists. Reporting begins when `test_runner` is called.
 Nested runner/verifier scripts should select their own output file, or disable
 reporting with `GTEST_OUTPUT=`, so an expected failure does not become the parent
 target's report.
+Verifier fixtures should also choose their own selection environment: clear
+`TESTBRIDGE_TEST_ONLY`, set `TESTBRIDGE_TEST_RUNNER_FAIL_FAST=0`, and use
+`TEST_TOTAL_SHARDS=1 TEST_SHARD_INDEX=0 TEST_SHARD_STATUS_FILE=` for an unsharded
+child. This prevents the parent's filter, shard assignment, or acknowledgement
+file from changing the fixture being verified.

bashtest/bashtest.sh

@@ -37,14 +37,18 @@ _BASHTEST_USAGE=$(cat <<'EOF'
 * bazel test `<test_target>` [ --test_arg=-f=`<pattern>` ] [ -v ] [ -u="${PWD}" ]
 * bazel run `<test_target>` -- [ -f=`<pattern>` ] [ -v ][ -u="${PWD}" ]
 
+Native Bazel --test_filter supplies the default regex via TESTBRIDGE_TEST_ONLY.
+An explicit runner filter overrides that default. Sharding and
+--test_runner_fail_fast are supported; teardown still runs after a case fails.
+
 ## Flags:
 
 * -f --test-filter `<pattern>`  A posix regular expression for tests to run,
                                 while skipping other tests. The test as a whole
                                 is considered failing if no test was run.
 * --gtest-output `<xml[:path]>` Write JUnit XML (also accepts GTEST_OUTPUT).
                                 Bazel sets XML_OUTPUT_FILE automatically.
-* --gtest-filter                Like '-f' but uses glob patterns.
+* --gtest-filter                One Bash glob (not GoogleTest list/exclusion syntax).
 * --strict-commands             Record unhandled command failures using ERR traps.
                                 Requires a standalone test_runner call.
 * --no-skip                     Turn every `skip_test` into a failure. For an
@@ -166,7 +170,8 @@ EOF
 )
 
 _BASHTEST_REPORT_OUTPUT="${GTEST_OUTPUT-${XML_OUTPUT_FILE:+xml:${XML_OUTPUT_FILE}}}"
-_BASHTEST_FILTER_REGX=""
+_BASHTEST_FILTER_REGX="${TESTBRIDGE_TEST_ONLY:-}"
+_BASHTEST_FILTER_FROM_ENV=1
 _BASHTEST_FILTER_GLOB=""
 _BASHTEST_KEEP_TMPDIR="failure"
 _BASHTEST_UPDATE_GOLDEN=""
@@ -226,9 +231,17 @@ while getopts -- '-:f:hu:v' OPTION; do
         OPTERR="-${OPTION}"
     fi
     case "${OPTION}" in
-        f|test[-_]filter) _BASHTEST_FILTER_REGX="${OPTARG}" ;;
+        f|test[-_]filter|gtest[-_]filter)
+            if [[ "${_BASHTEST_FILTER_FROM_ENV}" == 1 ]]; then
+                _BASHTEST_FILTER_REGX=""
+                _BASHTEST_FILTER_FROM_ENV=0
+            fi
+            case "${OPTION}" in
+                gtest[-_]filter) _BASHTEST_FILTER_GLOB="${OPTARG}" ;;
+                *) _BASHTEST_FILTER_REGX="${OPTARG}" ;;
+            esac
+            ;;
         gtest[-_]output) _BASHTEST_REPORT_OUTPUT="${OPTARG}" ;;
-        gtest[-_]filter) _BASHTEST_FILTER_GLOB="${OPTARG}" ;;
         h|help) echo "${_BASHTEST_USAGE}"; exit 2 ;;
         keep[-_]tmpdir)
             case "${OPTARG}" in
@@ -546,25 +559,72 @@ _bashtest_xml_write() {
     } > "${temporary}" && mv -f "${temporary}" "${_BASHTEST_XML_PATH}"
 }
 
+# Match bare case names (without test::). Explicit regex/glob filters retain
+# their historical union semantics; no filters selects every case.
+_bashtest_selected() {
+    local name="${1}"
+    [[ -n "${_BASHTEST_FILTER_GLOB}${_BASHTEST_FILTER_REGX}" ]] || return 0
+    # shellcheck disable=SC2053 # This is intentionally a Bash glob.
+    if [[ -n "${_BASHTEST_FILTER_GLOB}" && ${name} == ${_BASHTEST_FILTER_GLOB} ]]; then
+        return 0
+    fi
+    [[ -n "${_BASHTEST_FILTER_REGX}" && ${name} =~ ${_BASHTEST_FILTER_REGX} ]]
+}
+
+_bashtest_skip_case() {
+    echo "[  SKIP  ] ${1}: ${2}"
+    ((_BASHTEST_NUM_SKIP+=1))
+    _bashtest_xml_case skipped "${1}" "${2}"
+}
+
+# Validate before any hooks run. Bounds avoid shell arithmetic overflow and
+# leading-zero/octal interpretation; Bazel supplies canonical decimal integers.
+_bashtest_selection_init() {
+    local status=0
+    if [[ -n "${_BASHTEST_FILTER_REGX}" ]]; then
+        _bashtest_ere_match "${_BASHTEST_FILTER_REGX}" "" || status="${?}"
+        if [[ "${status}" == 2 ]]; then
+            echo >&2 "ERROR: Invalid test filter extended regular expression: ${_BASHTEST_FILTER_REGX}"
+            return 1
+        fi
+    fi
+    _BASHTEST_TOTAL_SHARDS="${TEST_TOTAL_SHARDS-1}"
+    _BASHTEST_SHARD_INDEX="${TEST_SHARD_INDEX-0}"
+    local valid=0
+    if [[ "${_BASHTEST_TOTAL_SHARDS}" =~ ^[1-9][0-9]{0,8}$ && "${_BASHTEST_SHARD_INDEX}" =~ ^(0|[1-9][0-9]{0,8})$ && "${TEST_TOTAL_SHARDS+x}" == "${TEST_SHARD_INDEX+x}" ]]; then
+        if (( _BASHTEST_SHARD_INDEX < _BASHTEST_TOTAL_SHARDS )); then
+            valid=1
+        fi
+    fi
+    if [[ "${valid}" == 0 ]]; then
+        echo >&2 'ERROR: Invalid test shard configuration: require TEST_TOTAL_SHARDS > TEST_SHARD_INDEX >= 0 (decimal integers, at most 9 digits).'
+        return 1
+    fi
+    if [[ -n "${TEST_SHARD_STATUS_FILE:-}" ]]; then
+        local path="${TEST_SHARD_STATUS_FILE}"
+        case "${OSTYPE}" in
+            msys*|cygwin*) path="$(cygpath -u "${path}")" ;;
+        esac
+        if ! touch "${path}"; then
+            echo >&2 "ERROR: Cannot acknowledge test sharding: ${path}"
+            return 1
+        fi
+    fi
+}
+
 ################################################################################
 function _bashtest_handler() {
     FUNC_NAME="${1}"
     TEST_NAME="${2}"
-    if [[ -n "${_BASHTEST_FILTER_GLOB}${_BASHTEST_FILTER_REGX}" ]]; then
-        PASS_GLOB=
-        PASS_REGX=
-        # shellcheck disable=SC2053 # We want glob matching here.
-        if [[ -n "${_BASHTEST_FILTER_GLOB}" ]] && [[ ${TEST_NAME} == ${_BASHTEST_FILTER_GLOB} ]]; then
-            PASS_GLOB=1
-        elif [[ -n "${_BASHTEST_FILTER_REGX}" ]] && [[ ${TEST_NAME} =~ ${_BASHTEST_FILTER_REGX} ]]; then
-            PASS_REGX=1
-        fi
-        if [[ -z "${PASS_GLOB}${PASS_REGX}" ]]; then
-            echo "[  SKIP  ] ${TEST_NAME}"
-            ((_BASHTEST_NUM_SKIP+=1))
-            _bashtest_xml_case skipped "${TEST_NAME}" "Excluded by test filter or failed initialization."
-            return
-        fi
+    if ! _bashtest_selected "${TEST_NAME}"; then
+        _bashtest_skip_case "${TEST_NAME}" "Excluded by test filter."
+        return
+    elif [[ "${_BASHTEST_INIT_FAILED}" == 1 ]]; then
+        _bashtest_skip_case "${TEST_NAME}" "Test initialization failed."
+        return
+    elif [[ "${_BASHTEST_STOP}" == 1 ]]; then
+        _bashtest_skip_case "${TEST_NAME}" "Not run after an earlier failure (fail-fast)."
+        return
     fi
     echo "[  TEST  ] ${TEST_NAME}"
     _bashtest_reset_error
@@ -640,6 +700,12 @@ function test_runner() {
             return 1
         fi
     fi
+    if ! _bashtest_selection_init; then
+        _bashtest_xml_case error test_runner "Invalid test selection or sharding configuration."
+        _BASHTEST_XML_RUNNING=0
+        _bashtest_xml_write || return 1
+        return 1
+    fi
     if [[ -n "${_BASHTEST_FILTER_GLOB}" ]]; then
         echo "Test filter glob: '${_BASHTEST_FILTER_GLOB}'."
     fi
@@ -648,27 +714,49 @@ function test_runner() {
     fi
     echo "----------"
     TEST_FUNCS=()
-    while IFS='' read -r line; do TEST_FUNCS+=("${line}"); done < <(declare -F|LC_ALL=C sed -rne 's,^declare -f test::(.*)$,\1,p')
-    if _bashtest_contains_element "test_init" ${TEST_FUNCS[@]+"${TEST_FUNCS[@]}"}; then
+    while IFS='' read -r line; do TEST_FUNCS+=("${line}"); done < <(declare -F|LC_ALL=C sed -rne 's,^declare -f test::(.*)$,\1,p'|LC_ALL=C sort)
+    local ordinal=0 selected=0 assigned=0
+    local owned=()
+    for TEST in ${TEST_FUNCS[@]+"${TEST_FUNCS[@]}"}; do
+        [[ "${TEST}" != test_init && "${TEST}" != test_done ]] || continue
+        if _bashtest_selected "${TEST}"; then
+            ((selected+=1))
+        fi
+        if (( ordinal % _BASHTEST_TOTAL_SHARDS == _BASHTEST_SHARD_INDEX )); then
+            owned+=("${TEST}")
+            if _bashtest_selected "${TEST}"; then
+                ((assigned+=1))
+            fi
+        fi
+        ((ordinal+=1))
+    done
+    # Empty shards pass only when some case matched globally. Filters selecting
+    # nothing and a suite containing no cases retain the existing error status.
+    local empty_shard=0
+    if (( selected > 0 && assigned == 0 )); then
+        empty_shard=1
+    fi
+    _BASHTEST_STOP=0
+    if [[ "${empty_shard}" == 0 ]] && _bashtest_contains_element "test_init" ${TEST_FUNCS[@]+"${TEST_FUNCS[@]}"}; then
         _bashtest_reset_error
         _bashtest_invoke test::test_init
         if [[ "${_BASHTEST_RESULT}" != 0 || "${_BASHTEST_HAS_ERROR}" != 0 ]]; then
             _bashtest_xml_case error test_init "Test initialization failed."
             _BASHTEST_INIT_FAILED=1
-            _BASHTEST_FILTER_GLOB=("${TEST_FUNCS[*]}")
             echo "Test init (test::test_init) failed! Skipping all tests."
         else
             _bashtest_xml_case ignored test_init
         fi
         echo "----------"
     fi
-    for TEST in ${TEST_FUNCS[@]+"${TEST_FUNCS[@]}"}; do
-        if [[ "${TEST}" != "test_init" ]] && [[ "${TEST}" != "test_done" ]]; then
-            _bashtest_handler "test::${TEST}" "${TEST}"
-            echo "----------"
+    for TEST in ${owned[@]+"${owned[@]}"}; do
+        _bashtest_handler "test::${TEST}" "${TEST}"
+        if [[ "${TESTBRIDGE_TEST_RUNNER_FAIL_FAST:-}" == 1 && "${_BASHTEST_NUM_FAIL}" != 0 ]]; then
+            _BASHTEST_STOP=1
         fi
+        echo "----------"
     done
-    if _bashtest_contains_element "test_done" ${TEST_FUNCS[@]+"${TEST_FUNCS[@]}"}; then
+    if [[ "${empty_shard}" == 0 ]] && _bashtest_contains_element "test_done" ${TEST_FUNCS[@]+"${TEST_FUNCS[@]}"}; then
         _bashtest_reset_error
         _bashtest_invoke test::test_done
         if [[ "${_BASHTEST_RESULT}" != 0 || "${_BASHTEST_HAS_ERROR}" != 0 ]]; then
@@ -679,7 +767,7 @@ function test_runner() {
         fi
         echo "----------"
     fi
-    if [[ "${_BASHTEST_NUM_PASS}" == 0 && "${_BASHTEST_NUM_FAIL}" == 0 && "${_BASHTEST_INIT_FAILED}" == 0 ]]; then
+    if [[ "${empty_shard}" == 0 && "${_BASHTEST_NUM_PASS}" == 0 && "${_BASHTEST_NUM_FAIL}" == 0 && "${_BASHTEST_INIT_FAILED}" == 0 ]]; then
         _bashtest_xml_case error test_runner "No tests were run."
     fi
     if ! _bashtest_xml_write; then
@@ -704,7 +792,7 @@ function test_runner() {
             echo "  > bazel run ${TEST_TARGET:-<test_target>} -- -u=\"\${PWD}\""
         fi
         return 1
-    elif [[ "${_BASHTEST_NUM_PASS}" == 0 ]]; then
+    elif [[ "${empty_shard}" == 0 && "${_BASHTEST_NUM_PASS}" == 0 ]]; then
         echo >&2 "ERROR: No tests were run."
         return 2
     fi

bashtest/tests/external_consumer_test.sh

@@ -66,6 +66,9 @@ EOF
 # Also run the framework's own suite as an external dependency, as BCR does.
 # A standalone consumer test alone cannot catch root-workspace assumptions in it.
 bazel test --enable_runfiles=no --test_output=errors //:consumer_test @framework//bashtest/tests/...
+# A filtered/sharded meta-test must not leak its settings into verifier fixtures.
+bazel test --enable_runfiles=no --test_filter='^junit_cases_and_failures$' \
+    --test_sharding_strategy=forced=5 @framework//bashtest/tests:junit_test
 bazel run --enable_runfiles=no //:consumer_test -- --keep-tmpdir=never
 
 # Exercise both documented update spellings with a workspace path containing spaces.
@@ -141,3 +144,76 @@ for target in "${junit_targets[@]}"; do
     grep -F 'actual & value' "${report}" >/dev/null
     echo "[EXPECTED FAILURE: JUnit fixture] ${target}: Bazel collected per-function failure XML"
 done
+
+# Exercise native Bazel flags through the macro and plain sh_test. These manual
+# fixtures are invoked explicitly because their unfiltered beta case must fail.
+cat >> BUILD.bazel <<'EOF'
+bashtest(
+    name = "selection_test",
+    srcs = ["selection.sh"],
+    tags = ["manual"],
+)
+EOF
+cat > selection.sh <<'EOF'
+#!/usr/bin/env bash
+set -euo pipefail
+# shellcheck disable=SC1090,SC1091,SC2154
+source "${mboworks_bashtest}"
+# shellcheck disable=SC2317
+test::alpha() { echo CASE:alpha; if [[ "${FAIL_CASE:-}" == alpha ]]; then expect_eq yes no; fi; }
+# shellcheck disable=SC2317
+test::beta() { echo CASE:beta; expect_eq filtered_case must_not_run; }
+# shellcheck disable=SC2317
+test::gamma() { echo CASE:gamma; }
+# shellcheck disable=SC2317
+test::test_done() { echo DONE; }
+test_runner
+EOF
+chmod +x selection.sh
+selection_targets=(//:selection_test)
+case "${OSTYPE}" in
+    msys*|cygwin*) ;;
+    *)
+        cat >> BUILD.bazel <<'EOF'
+sh_test(
+    name = "selection_sh_test",
+    srcs = ["selection.sh"],
+    data = ["@framework//bashtest:bashtest_sh"],
+    env = {"mboworks_bashtest": "$(rootpath @framework//bashtest:bashtest_sh)"},
+    tags = ["manual"],
+)
+EOF
+        selection_targets+=(//:selection_sh_test)
+        ;;
+esac
+bazel test --enable_runfiles=no --test_filter='^(alpha|gamma)$' "${selection_targets[@]}"
+bazel test --enable_runfiles=no --test_filter='^(alpha|gamma)$' \
+    --test_sharding_strategy=forced=5 "${selection_targets[@]}"
+for target in "${selection_targets[@]}"; do
+    reports=("${junit_logs}/${target#//:}"/shard_*/test.xml)
+    [[ "${#reports[@]}" == 5 ]]
+    alpha=0 gamma=0 empty=0
+    for report in "${reports[@]}"; do
+        if grep -F 'name="alpha"' "${report}" >/dev/null; then ((alpha+=1)); fi
+        if grep -F 'name="gamma"' "${report}" >/dev/null; then ((gamma+=1)); fi
+        if grep -F 'tests="0" failures="0" errors="0"' "${report}" >/dev/null; then ((empty+=1)); fi
+    done
+    [[ "${alpha}" == 1 && "${gamma}" == 1 && "${empty}" == 2 ]]
+    echo "[SHARDING VERIFIED] ${target}: five reports, each selected case once, two empty reports"
+done
+if bazel test --enable_runfiles=no --test_runner_fail_fast --test_env=FAIL_CASE=alpha \
+    "${selection_targets[@]}" > selection_failure.log 2>&1; then
+    echo >&2 'ERROR: intentionally failing fail-fast fixtures passed'
+    exit 1
+fi
+for target in "${selection_targets[@]}"; do
+    report="${junit_logs}/${target#//:}/test.xml"
+    grep -F 'tests="3" failures="1" errors="0" skipped="2"' "${report}" >/dev/null
+    grep -F 'fail-fast' "${report}" >/dev/null
+    grep -F 'DONE' "${junit_logs}/${target#//:}/test.log" >/dev/null
+    if grep -F 'CASE:beta' "${junit_logs}/${target#//:}/test.log"; then
+        echo >&2 'ERROR: fail-fast executed a later case'
+        exit 1
+    fi
+    echo "[EXPECTED FAILURE: fail-fast fixture] ${target}: later cases skipped and teardown ran"
+done