Sleep between create_stream retry attempts when waiting for LocalStack in Kinesis ITs by damccorm · Pull Request #41204 · apache/beam

damccorm · 2026-11-11T12:20:32Z

Note: this PR was auto-generated by tooling. Before moving it out of draft, the author has done a full review of the PR, specifically evaluating it on the following criteria:

The PR is an appropriate fix for the issue (e.g. we're not changing a test to avoid a bug)
The PR is a simple fix to the issue (e.g. we're not adding infra when we could just modify a test)
The PR is not covering for other problems (e.g. if we're bumping timeouts, it is not because we're covering for a performance problem)

Full details of the methodology of generation and review can be found here - https://docs.google.com/document/d/14lj7ECVov4T4EdwmIzNlWrRGE3lLDMOwX5xUMGzaLoc/edit?usp=sharing

If you have concerns with the PR or with the underlying methodology used to generate this, please comment below - they will be addressed by the author not the tooling :)

Summary

Fixes flaky botocore.exceptions.ConnectionClosedError: Connection was closed before we received a valid response from endpoint URL: "http://localhost:.../" failures in KinesisTest::test_only (PostCommit YAML Xlang Direct, run 43012453285) and CrossLanguageKinesisIOTest::test_kinesis_write (PostCommit Python).

Root Cause

Both _create_kinesis_stream (sdks/python/apache_beam/yaml/integration_tests.py) and KinesisHelper.create_stream (sdks/python/apache_beam/io/external/xlang_kinesisio_it_test.py) retry kinesis_client.create_stream(...) 10 times while the localstack/localstack:3.8.1 container is starting up. However, time.sleep(2) was only placed inside the try block after create_stream() succeeded, with no sleep in the except Exception: block—so when create_stream() raised ConnectionClosedError before LocalStack's HTTP server finished initializing, all 10 retry attempts executed back-to-back in a tight loop and exhausted in under 1 second.

Fix

Add time.sleep(3) in the except Exception: block of _create_kinesis_stream and KinesisHelper.create_stream (while keeping retries = 10 and the post-create time.sleep(2)) so retries wait up to 30 seconds for LocalStack's HTTP endpoint to accept connections.

Verification

python3 -m py_compile sdks/python/apache_beam/yaml/integration_tests.py sdks/python/apache_beam/io/external/xlang_kinesisio_it_test.py
Triggered beam_PostCommit_Yaml_Xlang_Direct.yml on branch fix/kinesis-it-localstack-create-stream-retry (previous run 43054337164: success)

…k in Kinesis ITs

damccorm · 2026-11-11T12:20:35Z

/gemini review

github-actions

📋 Review Summary

This PR fixes a retry timing bug in Kinesis integration tests (xlang_kinesisio_it_test.py and yaml/integration_tests.py) where time.sleep(2) was placed inside the try block rather than the except block during create_stream retries. Moving the sleep into the exception handler and increasing the retry count from 10 to 15 properly spaces out retry attempts while waiting for the LocalStack Kinesis service to initialize.

🔍 General Feedback

The changes in both test files are clean, minimal, and directly resolve the premature timeout when LocalStack startup takes several seconds.
Stream readiness polling via describe_stream already waits for the ACTIVE status in subsequent loops, so removing the sleep after a successful create_stream call is safe.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

codecov · 2026-11-11T13:31:26Z

Codecov Report

❌ Patch coverage is 0% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 59.15%. Comparing base (d7d31e9) to head (604a5e8).

Files with missing lines	Patch %	Lines
sdks/python/apache_beam/yaml/integration_tests.py	0.00%	1 Missing ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##             master   #41204      +/-   ##
============================================
- Coverage     59.26%   59.15%   -0.11%     
+ Complexity     4925     4892      -33     
============================================
  Files          1462     1461       -1     
  Lines        186313   186183     -130     
  Branches       3669     3645      -24     
============================================
- Hits         110417   110143     -274     
- Misses        72540    72685     +145     
+ Partials       3356     3355       -1

Flag	Coverage Δ
python	`80.51% <0.00%> (-0.23%)`	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

damccorm · 2026-11-13T04:49:16Z

  def create_stream(self, stream_name):
    # localstack could not have initialized in the container yet so repeat
-    retries = 10
+    retries = 15


Why do we need to increase the number of retries?

damccorm · 2026-11-13T04:50:21Z

            StreamName=stream_name,
            ShardCount=1,
        )
-        time.sleep(2)


Removing this seems unnecessary. I think this is supposed to help us wait for the stream to be active.

Done in 8cc143c — restored retries = 10 and the post-create_stream time.sleep(1), and added a comment explaining the time.sleep(2) in the except block.

damccorm · 2026-11-13T04:50:47Z

        if i == retries - 1:
          logging.error('Could not create kinesis stream')
          raise e
+        time.sleep(2)


How does sleeping here help us? Add a comment explaining. Also, if we need to sleep longer to support fewer retries, that is ok.

damccorm · 2026-11-13T04:51:03Z

 _LOCALSTACK_KINESIS_REGION = 'us-east-1'


 def _create_kinesis_stream(kinesis_client, stream_name):


Same comments apply here as in the other file

Done in 8cc143c — restored retries = 10 and the post-create_stream time.sleep(1), and added the explanatory comment here as well.

damccorm · 2026-11-13T05:11:25Z

/gemini review

github-actions

📋 Review Summary

This PR resolves flaky ConnectionClosedError failures in the LocalStack-based Kinesis integration tests (CrossLanguageKinesisIOTest and YAML KinesisTest) by adding a time.sleep(3) delay inside the except block of the create_stream retry loops. Previously, retries executed back-to-back without waiting when create_stream threw an exception during container startup.

🔍 General Feedback

The root cause analysis and fix are clean, minimal, and consistently applied across both xlang_kinesisio_it_test.py and yaml/integration_tests.py.
No issues found; the changes look good to merge.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

…calstack-create-stream-retry

damccorm · 2026-11-13T09:24:24Z

/gemini review

github-actions

📋 Review Summary

This PR fixes flaky ConnectionClosedError failures in the Kinesis LocalStack integration tests (xlang_kinesisio_it_test.py and yaml/integration_tests.py) by adding a time.sleep(3) delay between create_stream retry attempts when an exception occurs. Previously, time.sleep(2) only ran after a successful create_stream call, causing all 10 retries to execute immediately in a tight loop before the LocalStack HTTP server finished initializing.

🔍 General Feedback

The fix is minimal, targeted, and directly addresses the tight-loop retry exhaustion during LocalStack container startup across both test suites.
Explanatory comments clearly document why the sleep in the except block is necessary.

💡 Tip: Comment /gemini review on this pull request to trigger a new Gemini code review.

Sleep between create_stream retry attempts when waiting for LocalStac…

c6c69ea

…k in Kinesis ITs

damccorm added the test-failure-cleanup Automated test health and flake cleanup PRs label Nov 11, 2026

github-actions Bot added python io yaml size/xs XS: <10 lines changed labels Nov 11, 2026

github-actions Bot reviewed Nov 11, 2026

View reviewed changes

damccorm added ready-for-human Draft PR with all checks passing and all review comments resolved, ready for human review and removed ready-for-human Draft PR with all checks passing and all review comments resolved, ready for human review labels Nov 12, 2026

damccorm commented Nov 13, 2026

View reviewed changes

Keep retries=10 and post-create sleep, add comment on except sleep

8cc143c

github-actions Bot reviewed Nov 13, 2026

View reviewed changes

Merge remote-tracking branch 'upstream/master' into fix/kinesis-it-lo…

604a5e8

…calstack-create-stream-retry

github-actions Bot reviewed Nov 13, 2026

View reviewed changes

damccorm added the ready-for-human Draft PR with all checks passing and all review comments resolved, ready for human review label Nov 13, 2026

		_LOCALSTACK_KINESIS_REGION = 'us-east-1'


		def _create_kinesis_stream(kinesis_client, stream_name):

Conversation

damccorm commented Nov 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Root Cause

Fix

Verification

Uh oh!

damccorm commented Nov 11, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

📋 Review Summary

🔍 General Feedback

Uh oh!

codecov Bot commented Nov 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

damccorm Nov 13, 2026

Choose a reason for hiding this comment

Uh oh!

damccorm Nov 13, 2026

Choose a reason for hiding this comment

Uh oh!

damccorm Nov 13, 2026

Choose a reason for hiding this comment

Uh oh!

damccorm Nov 13, 2026

Choose a reason for hiding this comment

Uh oh!

damccorm Nov 13, 2026

Choose a reason for hiding this comment

Uh oh!

damccorm Nov 13, 2026

Choose a reason for hiding this comment

Uh oh!

damccorm commented Nov 13, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

📋 Review Summary

🔍 General Feedback

Uh oh!

damccorm commented Nov 13, 2026

Uh oh!

github-actions Bot left a comment

Choose a reason for hiding this comment

📋 Review Summary

🔍 General Feedback

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

1 participant

damccorm commented Nov 11, 2026 •

edited

Loading

codecov Bot commented Nov 11, 2026 •

edited

Loading