Skip to content

Fix MQTT disconnect races and isolate broker tests - #3500

Merged
richturner merged 5 commits into
mainfrom
fix/mqtt-broker-disconnect-races
Oct 30, 2026
Merged

richturner merged 5 commits into
mainfrom
fix/mqtt-broker-disconnect-races

Conversation

@wborn

@wborn wborn commented Oct 27, 2026

Copy link
Copy Markdown
Member

Description

  • Preserve last-will connection lookups during disconnect cleanup.
  • Prevent stale disconnect callbacks from removing newer connections.
  • Add deterministic regression tests for disconnect races.
  • Split MqttBrokerTest into ten independent features with shared setup, cleanup and assertion helpers.
  • Wait for subscription acknowledgements and broker cleanup, and use timed queue polls to check for unwanted events.

Checklist

  • 1. Acceptance criteria of the linked issue(s) are met
  • 2. Tests are written and all tests pass
  • 3. Changes are manually tested by you and the reviewer
  • 4. Documentation is written or updated

@wborn wborn added the Test Development of missing or unstable test label Oct 27, 2026
- Preserve last-will connection lookups during disconnect cleanup.
- Prevent stale disconnect callbacks from removing newer connections.
- Add deterministic regression tests for disconnect races.
- Split MqttBrokerTest into ten independent features with shared setup, cleanup and assertion helpers.
- Wait for subscription acknowledgements and broker cleanup, and use timed queue polls to check for unwanted events.
@wborn
wborn force-pushed the fix/mqtt-broker-disconnect-races branch from 82be5c9 to 030737f Compare October 27, 2026 12:32
@wborn wborn linked an issue Oct 27, 2026 that may be closed by this pull request
@wborn

wborn commented Oct 30, 2026

Copy link
Copy Markdown
Member Author

With the changes in this PR it seems to have been stabilized as it didn't fail any time after running the integration tests 50 times, see: https://github.com/openremote/core/actions/runs/40504589812

Only the GatewayTest failed for which I've created #3507.

@wborn
wborn requested a review from a team October 30, 2026 08:58

@richturner richturner left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like the approach here but now using more tests within the spec and calling startContainer between each in setup will impact the time it takes to run the specification.

It would be nice to try and use setupSpec and ensure each test cleans up any residual state, this would likely dramatically improve the execution time.

@wborn

wborn commented Oct 30, 2026

Copy link
Copy Markdown
Member Author

I already thought it was structured like one big test to reduce test time. 😉
The separation into different tests was done to make it more readable and easier to see what logic is broken.
Using setupSpec/cleanup seems to work and reduces the total execution time by 10s (23.3% reduction).

@wborn
wborn requested a review from richturner October 30, 2026 15:54
@richturner
richturner enabled auto-merge October 30, 2026 22:08
@sonarqubecloud

Copy link
Copy Markdown

@richturner
richturner added this pull request to the merge queue Oct 30, 2026
Merged via the queue into main with commit 18f38fb Oct 30, 2026
24 checks passed
@richturner
richturner deleted the fix/mqtt-broker-disconnect-races branch October 30, 2026 23:00
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Test Development of missing or unstable test

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unstable MqttBrokerTest

2 participants