Skip to content

Add --resume and --timeout to databricks pipelines test #7426

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
tharunbhuk-db wants to merge 1 commit into
base: tharun/pipelines-test-pr2-output
Choose a base branch
from
+385 −13
Open

Add --resume and --timeout to databricks pipelines test #7426

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
26 changes: 25 additions & 1 deletion acceptance/pipelines/test/arguments/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,10 @@ example:
The command waits for the results. It exits with code 0 only if at least one
test was collected and none failed or errored, so a run whose tests were all
skipped succeeds; otherwise it exits with code 1. --no-wait returns after the
update starts. To stop the update, run 'pipelines stop'.
update starts.

--timeout and Ctrl+C stop waiting but not the update; pass its ID to --resume
to read the results later. To stop the update, run 'pipelines stop'.

Usage:
databricks pipelines test [flags] [KEY] [TEST_SELECTION...]
Expand All @@ -35,6 +38,8 @@ Pipeline Test Flags:
--no-wait Don't wait for the tests to complete.
--pipeline-id string ID of the pipeline to test, instead of a KEY.
--pytest-arg stringArray Argument to pass to pytest (repeatable). Only -k and -m are supported.
--resume string Read the results of an existing test update instead of starting one.
--timeout duration Stop waiting for results after this duration, for example 30m. The update keeps running.

Flags:
-h, --help help for test
Expand Down Expand Up @@ -71,3 +76,22 @@ Error: --pipeline-id "1234" is not a pipeline ID
=== JSON output is not supported
>>> musterr [CLI] pipelines test --pipeline-id 1234 -o json
Error: --output json is not supported by pipelines test

=== Incompatible run-control flags
>>> musterr [CLI] pipelines test --pipeline-id 1234 tests --resume 5678
Error: --resume cannot be used with test selections

>>> musterr [CLI] pipelines test --pipeline-id 1234 --resume 5678 --pytest-arg=-m --pytest-arg=smoke
Error: --resume cannot be used with --pytest-arg

>>> musterr [CLI] pipelines test --pipeline-id 1234 --resume 5678 --no-wait
Error: --resume cannot be used with --no-wait

>>> musterr [CLI] pipelines test --pipeline-id 1234 --no-wait --timeout 10m
Error: --timeout cannot be used with --no-wait

>>> musterr [CLI] pipelines test --pipeline-id 1234 --resume 5678' OR '1'='1
Error: --resume "5678' OR '1'='1" is not an update ID; pass the update ID that pipelines test printed

>>> musterr [CLI] pipelines test --pipeline-id 1234 --resume
Error: --resume "" is not an update ID; pass the update ID that pipelines test printed
8 changes: 8 additions & 0 deletions acceptance/pipelines/test/arguments/script
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,3 +16,11 @@ trace musterr $CLI pipelines test --pipeline-id 1234

title "JSON output is not supported"
trace musterr $CLI pipelines test --pipeline-id 1234 -o json

title "Incompatible run-control flags"
trace musterr $CLI pipelines test --pipeline-id 1234 tests --resume 5678
trace musterr $CLI pipelines test --pipeline-id 1234 --resume 5678 --pytest-arg=-m --pytest-arg=smoke
trace musterr $CLI pipelines test --pipeline-id 1234 --resume 5678 --no-wait
trace musterr $CLI pipelines test --pipeline-id 1234 --no-wait --timeout 10m
trace musterr $CLI pipelines test --pipeline-id 1234 --resume "5678' OR '1'='1"
trace musterr $CLI pipelines test --pipeline-id 1234 --resume ""
1 change: 1 addition & 0 deletions acceptance/pipelines/test/run-control/canceled/out.test.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Cloud = false
9 changes: 9 additions & 0 deletions acceptance/pipelines/test/run-control/canceled/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@

=== A canceled update is an incomplete run
>>> musterr [CLI] pipelines test --pipeline-id [UUID][0] --resume [UUID][1]
Pipeline: [UUID][0]
Update: [UUID][1]

RUN INCOMPLETE: the update was canceled before all test results were received.
Done. PASSED=0 FAILED=0 SKIPPED=0 ERROR=0 TOTAL=1
Error: update [UUID][1] is CANCELED
2 changes: 2 additions & 0 deletions acceptance/pipelines/test/run-control/canceled/script
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
title "A canceled update is an incomplete run"
trace musterr $CLI pipelines test --pipeline-id $pipeline_id --resume 00000000-0000-0000-0000-0000000000a1
10 changes: 10 additions & 0 deletions acceptance/pipelines/test/run-control/canceled/test.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
[[Server]]
Pattern = "GET /api/2.0/pipelines/{pipeline_id}/updates/{update_id}"
Response.Body = '{"update": {"update_id": "00000000-0000-0000-0000-0000000000a1", "state": "CANCELED", "test_only": true}}'

[[Server]]
Pattern = "GET /api/2.0/pipelines/{pipeline_id}/events"
Response.Body = '''
{"events": [
{"id": "e1", "event_type": "test_case_progress", "timestamp": "2026-01-01T00:00:00.000Z", "origin": {"test_case_id": "/Workspace/Users/someone@example.com/orders/tests/test_orders.py::test_slow"}, "details": {"test_case_progress": {"status": "PENDING"}}}]}
'''
1 change: 1 addition & 0 deletions acceptance/pipelines/test/run-control/not-test-update/out.test.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Cloud = false
4 changes: 4 additions & 0 deletions acceptance/pipelines/test/run-control/not-test-update/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@

=== --resume rejects an update that is not a test update
>>> musterr [CLI] pipelines test --pipeline-id [UUID][0] --resume [UUID][1]
Error: update [UUID][1] is not a test update; pass the update ID that pipelines test printed
2 changes: 2 additions & 0 deletions acceptance/pipelines/test/run-control/not-test-update/script
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
title "--resume rejects an update that is not a test update"
trace musterr $CLI pipelines test --pipeline-id $pipeline_id --resume 00000000-0000-0000-0000-0000000000a1
3 changes: 3 additions & 0 deletions acceptance/pipelines/test/run-control/not-test-update/test.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
[[Server]]
Pattern = "GET /api/2.0/pipelines/{pipeline_id}/updates/{update_id}"
Response.Body = '{"update": {"update_id": "00000000-0000-0000-0000-0000000000a1", "state": "COMPLETED", "test_only": false}}'
1 change: 1 addition & 0 deletions acceptance/pipelines/test/run-control/resume/out.test.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Cloud = false
9 changes: 9 additions & 0 deletions acceptance/pipelines/test/run-control/resume/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@

=== --resume reads the results of an existing update
>>> [CLI] pipelines test --pipeline-id [UUID][0] --resume [UUID][1]
Pipeline: [UUID][0]
Update: [UUID][1]
Running tests...
1 of 1 PASSED tests/test_orders.py::test_valid_order ........................... [PASSED in 1.00s]

Done. PASSED=1 FAILED=0 SKIPPED=0 ERROR=0 TOTAL=1
2 changes: 2 additions & 0 deletions acceptance/pipelines/test/run-control/resume/script
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
title "--resume reads the results of an existing update"
trace $CLI pipelines test --pipeline-id $pipeline_id --resume 00000000-0000-0000-0000-0000000000a1
13 changes: 13 additions & 0 deletions acceptance/pipelines/test/run-control/resume/test.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,13 @@
[[Server]]
Pattern = "GET /api/2.0/pipelines/{pipeline_id}/updates/{update_id}"
Response.Body = '{"update": {"update_id": "00000000-0000-0000-0000-0000000000a1", "state": "COMPLETED", "test_only": true}}'

[[Server]]
Pattern = "GET /api/2.0/pipelines/{pipeline_id}/events"
Response.Body = '''
{"events": [
{"id": "e1", "event_type": "test_case_progress", "timestamp": "2026-01-01T00:00:00.000Z", "origin": {"test_case_id": "/Workspace/Users/someone@example.com/orders/tests/test_orders.py::test_valid_order"}, "details": {"test_case_progress": {"status": "PENDING"}}},
{"id": "e2", "event_type": "test_case_progress", "timestamp": "2026-01-01T00:00:01.000Z", "origin": {"test_case_id": "/Workspace/Users/someone@example.com/orders/tests/test_orders.py::test_valid_order"}, "details": {"test_case_progress": {"status": "RUNNING"}}},
{"id": "e3", "event_type": "test_case_progress", "timestamp": "2026-01-01T00:00:02.000Z", "origin": {"test_case_id": "/Workspace/Users/someone@example.com/orders/tests/test_orders.py::test_valid_order"}, "details": {"test_case_progress": {"status": "PASSED"}}},
{"id": "s1", "event_type": "test_run_summary", "timestamp": "2026-01-01T00:00:03.000Z", "details": {"test_run_summary": {"total": 1, "passed": 1, "failed": 0, "skipped": 0, "errored": 0}}}]}
'''
3 changes: 3 additions & 0 deletions acceptance/pipelines/test/run-control/script.prepare
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# Each case stubs Start Update, Get Update and List Pipeline Events in its test.toml.
pipeline_id=$($CLI pipelines create --json '{"name": "orders", "root_path": "/Workspace/Users/someone@example.com/orders"}' | jq -r .pipeline_id)
rm "$OUT_REQUESTS"
1 change: 1 addition & 0 deletions acceptance/pipelines/test/run-control/timeout/out.test.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Cloud = false
7 changes: 7 additions & 0 deletions acceptance/pipelines/test/run-control/timeout/output.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,7 @@

=== --timeout stops waiting but not the update, and prints the command that reads the results
>>> musterr [CLI] pipelines test --pipeline-id [UUID][0] --resume [UUID][1] --timeout 1s
Pipeline: [UUID][0]
Update: [UUID][1]
Running tests...
Error: stopped waiting for update [UUID][1] after 1s; the update keeps running, to get its results run: databricks pipelines test --pipeline-id [UUID][0] --resume [UUID][1]
2 changes: 2 additions & 0 deletions acceptance/pipelines/test/run-control/timeout/script
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,2 @@
title "--timeout stops waiting but not the update, and prints the command that reads the results"
trace musterr $CLI pipelines test --pipeline-id $pipeline_id --resume 00000000-0000-0000-0000-0000000000a1 --timeout 1s
10 changes: 10 additions & 0 deletions acceptance/pipelines/test/run-control/timeout/test.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
[[Server]]
Pattern = "GET /api/2.0/pipelines/{pipeline_id}/updates/{update_id}"
Response.Body = '{"update": {"update_id": "00000000-0000-0000-0000-0000000000a1", "state": "RUNNING", "test_only": true}}'

[[Server]]
Pattern = "GET /api/2.0/pipelines/{pipeline_id}/events"
Response.Body = '''
{"events": [
{"id": "e1", "event_type": "test_case_progress", "timestamp": "2026-01-01T00:00:00.000Z", "origin": {"test_case_id": "/Workspace/Users/someone@example.com/orders/tests/test_orders.py::test_slow"}, "details": {"test_case_progress": {"status": "PENDING"}}}]}
'''
Loading
Loading