From 3b8ce5a8692ea5e29748f8e3b45623b338083e72 Mon Sep 17 00:00:00 2001 From: Amarjeet LNU Date: Wed, 26 Aug 2026 11:58:50 -0700 Subject: [PATCH 1/2] fix(ci,train): stop integ-tests rerunning the shallow suite The shallow suite lives under tests/integ/train/shallow, and the deep integ-tests CodeBuild project invokes pytest through tox over the whole tests/integ tree. So every PR ran the same ~100 shallow tests twice on the same commit: once in fast-integ-tests, then again inside integ-tests' parallel pass. That is pure waste, and not free. Each shallow test submits a real CreateTrainingJob, so the duplicate pass doubles this suite's draw on the training-job quota the two projects share -- the exact contention the harness's concurrency cap exists to avoid -- and adds nothing, since both passes select the same tests by the same marker expression. Fixed by passing --ignore=tests/integ/train/shallow to the pytest invocation in tox.ini. Three properties make that the right lever: - The deep project goes through tox, so it picks the ignore up from the PR's own checked-out source. The project's buildspec is not defined in this repo, and the CDK package only defines the staging copy of it, so the buildspec is not something a PR here can change. - fast-integ-tests runs `python3.10 -m pytest tests/integ/train/shallow` directly rather than through tox, so it is unaffected and still runs the whole suite. - --ignore only prunes directory recursion; it does not override an explicitly named path. `tox -- tests/integ/train/shallow` still collects all 100 tests, so no local or scheduled workflow that names the directory loses coverage. Verified by collection against the deep project's own selection, `tests/integ -m "not serial and not gpu_intensive and not us_east_1"`: 348 collected / 200 selected before, 248 / 116 after. Exactly 100 collected and 84 selected drop out -- the shallow suite and nothing else -- matching the 82 shallow tests observed in the integ-tests log for run 32994923468. Naming the directory explicitly still collects 100. (One pre-existing collection error in test_list_hyperparameters_integration.py is a stale local install, present identically before and after.) Only the parallel pass was affected: the shallow tests carry no `serial` mark, so the serial pass never collected them -- 0 references in that half of the log against 211 in the other. X-AI-Prompt: also shallow tests are duplicated : in fast-integ-tests as well as integ-tests : Can we skip or remove shallow tests from integ-tests run for sagemaker-train ? X-AI-Tool: claude-code --- sagemaker-train/tests/integ/train/shallow/README.md | 6 ++++++ sagemaker-train/tox.ini | 10 +++++++++- 2 files changed, 15 insertions(+), 1 deletion(-) diff --git a/sagemaker-train/tests/integ/train/shallow/README.md b/sagemaker-train/tests/integ/train/shallow/README.md index 07c8912185..04eb19acae 100644 --- a/sagemaker-train/tests/integ/train/shallow/README.md +++ b/sagemaker-train/tests/integ/train/shallow/README.md @@ -8,6 +8,12 @@ What this suite changes about the gate is not which job runs, but what the exist one selects: the `gpu_intensive` marks added here deselect the deep tests that submit a job and wait for it, and this suite covers those code paths instead. +`integ-tests` does not rerun this suite. It invokes pytest through tox over the +whole `tests/integ` tree, which would otherwise sweep this directory in and submit +every job here a second time on the same commit. `tox.ini` passes +`--ignore=tests/integ/train/shallow` to keep that from happening; `fast-integ-tests` +calls pytest directly, so the ignore does not apply to it. + > **Where this runs.** The `fast-integ-tests` job in `pr-checks-master.yml` does not > execute this suite on the GitHub runner — it starts the CodeBuild project > `sagemaker-python-sdk-ci-sagemaker-train-fast-integ-tests` via diff --git a/sagemaker-train/tox.ini b/sagemaker-train/tox.ini index 21962188f1..1c935ed7f0 100644 --- a/sagemaker-train/tox.ini +++ b/sagemaker-train/tox.ini @@ -94,7 +94,15 @@ commands = pip install 'torchvision==0.18.1+cpu' -f 'https://download.pytorch.org/whl/torch_stable.html' pip install 'dill>=0.3.9' - pytest {posargs} + # --ignore keeps the shallow suite out of directory sweeps like + # `tox -- tests/integ`, which is how the deep integ-tests CodeBuild project + # invokes pytest. Without it, that project reruns all ~100 shallow tests + # that fast-integ-tests has already run on the same commit -- duplicate + # CreateTrainingJob calls against the shared job quota, for no extra + # coverage. fast-integ-tests calls pytest directly rather than through tox, + # so it is unaffected. --ignore only prunes recursion, so naming the + # directory explicitly (`tox -- tests/integ/train/shallow`) still runs it. + pytest --ignore=tests/integ/train/shallow {posargs} deps = -r ../requirements/extras/test_requirements.txt ../sagemaker-core From cdcad04a7e234ae3617392f31b1b3a8510fb34da Mon Sep 17 00:00:00 2001 From: Amarjeet LNU Date: Tue, 25 Aug 2026 16:15:08 -0700 Subject: [PATCH 2/2] docs(train): drop --dist loadfile from the shallow suite's README The flag was removed from createCIShallowIntegBuildSpec in SageMakerMLFPySDKInfraCDK: it pins one file's tests to one xdist worker, and each test here holds a concurrency slot until its training job reaches a terminal state (~75s), so the 17-test RLVR file alone took 19m45s against the project's 30-minute timeout. Job names are unique per invocation rather than per test function (see unique_name in harness.py), so tests are free to spread across workers, which is what the wall-clock estimate in harness.py assumes. Documentation only -- the invocation itself lives in the CDK package. --- X-AI-Prompt: The fast-integ-tests CodeBuild job is still failing on PRs now that the CDK project is deployed -- diagnose and fix it X-AI-Tool: claude-code --- sagemaker-train/tests/integ/train/shallow/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/sagemaker-train/tests/integ/train/shallow/README.md b/sagemaker-train/tests/integ/train/shallow/README.md index 04eb19acae..ceeb4f53be 100644 --- a/sagemaker-train/tests/integ/train/shallow/README.md +++ b/sagemaker-train/tests/integ/train/shallow/README.md @@ -25,7 +25,7 @@ calls pytest directly, so the ignore does not apply to it. > — which is nearly all of them — without exposing those credentials to PR code. > > **Consequence for editing this suite:** the marker selection above (`-n 8`, -> `--dist loadfile`, `-m "not gpu_intensive and not us_east_1"`) lives in +> `-m "not gpu_intensive and not us_east_1"`) lives in > `createCIShallowIntegBuildSpec` in the `SageMakerMLFPySDKInfraCDK` package, not in > this repo. Adding a file under `shallow/` is picked up automatically, but changing > *how* the suite is invoked means a change there, which deploys through a pipeline