Repository navigation
test(NODE-7697): run integration tests against disaggregated storage - #5128
Open
PavelSafronov wants to merge 5 commits into
Open
PavelSafronov wants to merge 5 commits into
PavelSafronov wants to merge 5 commits into
Conversation
- Support the unified test format `dsc` runOnRequirement (schema 1.30, specs PR #2004). - Add a DSCFilter so mocha tests can declare `requires: { dsc: 'enabled' | 'disabled' }`. - Skip replSetStepDown-based tests on DSC (no electable secondary). - Sync spec tests from specs PR #2004. - Add the `ubuntu2404-test-dsc` Evergreen variant using the DET disagg scripts. TEMP: checks out the DET fork branch until drivers-evergreen-tools#918 merges.
…overage prefix
- Pin DSC_SLS_TAG to the tag the C driver uses. Unpinned, setup.sh picks the newest SLS
image, and recent images never start (cell init job StartLog "store_id mismatch").
- Upload sls-logs.txt.gz under mongo-node-driver/dsc-logs/: "download and merge coverage"
treats everything under ${revision}/${version_id}/ as a coverage report.
…ble tests - Bump drivers-evergreen-tools to edf26f7 (#918) and drop the temporary fork checkout. - Skip tests that need mongocryptd on PATH, a deployment on localhost:27017, or localhost hostnames, which DSC tasks do not have. - Use a reserved .invalid hostname in the ENOTFOUND test: single-label names resolve to EAI_AGAIN on Ubuntu 24.04 (systemd-resolved).
Contributor
There was a problem hiding this comment.
🔵 Needs a closer look
Mainline authorization for the cross-account AWS role used to access private ECR images remains unverified and requires maintainer confirmation, since successful patch/PR runs do not establish it.
No open issues 🎉
What changed in this PR
Adds Evergreen integration testing against a disaggregated storage (DSC) replica set for the MongoDB Node.js driver, without changing driver source code.
Changes:
- Adds a daily DSC variant with deployment setup, encryption-library download, teardown, and log collection.
- Adds DSC-aware test filtering, unified run requirements, unit coverage, and upstream spec fixtures.
- Skips tests incompatible with the DSC deployment, documents the environment, and updates the DNS failure test hostname.
| File | Description |
|---|---|
test/unit/tools/unifed-utils.test.ts |
Tests DSC requirements and invalid values. |
test/tools/unified-spec-runner/unified-utils.ts |
Evaluates DSC run requirements. |
test/tools/unified-spec-runner/schema.ts |
Defines supported DSC requirement values. |
test/tools/runner/hooks/configuration.ts |
Registers the DSC filter. |
test/tools/runner/filters/dsc_filter.ts |
Filters tests using DSC environment metadata. |
test/spec/unified-test-format/invalid/runOnRequirement-dsc-type.yml |
Adds upstream invalid-type fixture. |
test/spec/unified-test-format/invalid/runOnRequirement-dsc-type.json |
Adds JSON invalid-type fixture. |
test/spec/unified-test-format/invalid/runOnRequirement-dsc-enum.yml |
Adds upstream invalid-value fixture. |
test/spec/unified-test-format/invalid/runOnRequirement-dsc-enum.json |
Adds JSON invalid-value fixture. |
test/spec/transactions/unified/commit.yml |
Syncs upstream DSC write-concern exclusion. |
test/spec/transactions/unified/commit.json |
Mirrors the transaction fixture update. |
test/spec/server-discovery-and-monitoring/unified/rediscover-quickly-after-step-down.yml |
Syncs upstream DSC step-down exclusion. |
test/spec/server-discovery-and-monitoring/unified/rediscover-quickly-after-step-down.json |
Mirrors the step-down fixture update. |
test/readme.md |
Documents DSC infrastructure and test filtering. |
test/integration/sessions/sessions.prose.test.ts |
Skips mongocryptd-dependent tests on DSC. |
test/integration/server-discovery-and-monitoring/server_description.test.ts |
Excludes mongocryptd tests from DSC. |
test/integration/node-specific/mongo_client.test.ts |
Uses a reserved .invalid DNS hostname. |
test/integration/node-specific/client_close.test.ts |
Skips a mongocryptd-dependent test on DSC. |
test/integration/initial-dns-seedlist-discovery/dns_seedlist.test.ts |
Excludes tests requiring localhost deployments. |
test/integration/crud/misc_cursors.test.ts |
Skips a localhost-dependent cursor test on DSC. |
test/integration/connections-survive-step-down/connections_survive_step_down.prose.test.ts |
Excludes getMore step-down tests from DSC. |
test/integration/change-streams/change_stream.test.ts |
Skips four step-down resumability tests on DSC. |
global.d.ts |
Types DSC test metadata. |
.evergreen/run-tests.sh |
Exports DSC status to the test runner. |
.evergreen/generate_evergreen_tasks.js |
Defines the DSC task and daily variant. |
.evergreen/config.yml |
Includes generated DSC Evergreen configuration. |
.evergreen/config.in.yml |
Adds DSC lifecycle functions and log upload. |
Review effort: Balanced
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Summary of Changes
Adds an Evergreen variant that runs the integration test suite against a replica set backed by disaggregated storage (DSC), as requested in DRIVERS-3190. There are no driver (
src/) changes.DET is bumped up to latest, which includes mongodb-labs/drivers-evergreen-tools#918 , a change that adds the DSC capabilities to DET.
Claude-speak below contains details that may not be obvious to most reviewers.
Evergreen
test-dsc-replica_setand variantubuntu2404-test-dsc("Disaggregated Storage (DSC) Tests"). The variant runs at most once a day on mainline (batchtime: 1440).start dsc: assumes thedisagg-storage-tf-project-rorole and runs the DETdisagg/setup.sh.fetch dsc crypt_shared: downloadscrypt_sharedfor the CSFLE tests.stop dsc: runs inpost, and only does anything whenDSCis set. It removes the/etc/hostsentry and uploads the SLS container logs assls-logs.txt.gz.DSC_SLS_TAG, the same tag the C driver uses, because the newest SLS images don't always start.mongo-node-driver/dsc-logs/, outside the coverage prefix. "download and merge coverage" treats every file under that prefix as a coverage report.drivers-evergreen-toolssubmodule is bumped toedf26f7, which includes the disagg scripts ([DRIVERS-3190] Scripts to start and stop dsc mongodb-labs/drivers-evergreen-tools#918).Test runner
dscrunOnRequirement (require/forbid/allow) from [DRIVERS-3190] Add disaggregated storage run on requirement specifications#2004, and throws on unknown values. Unit tests are included.DSCFilterreadsDSC=on(exported byrun-tests.sh), so integration tests can userequires: { dsc: 'disabled' }.transactions/unified/commit:dsc: forbidon the unsatisfiablew: 10test, because DSC accepts it without a write concern error (SERVER-120912).rediscover-quickly-after-step-down:dsc: forbid.runOnRequirement-dsc-*invalid tests.Tests skipped on DSC. Each skip has a comment giving its reason:
replSetStepDown:connections_survive_step_down(getMore iteration) and 4 change stream "resume on ServerSelectionError" tests. The test deployment has no electable secondary.mongocryptdspawned from PATH:server_description(iscryptd),client_close("does not execute endSessions"), sessions prose tests 18 and 19. DSC tasks don't download server binaries.dns_seedlist(its SRV records point atlocalhost:27017/27018) and themisc_cursorscursor count with a secondary read preference (its lookup only maps127.0.0.1).Other
mongo_client.test.ts: the ENOTFOUND test usesiLoveJavaScript.invalidinstead of a single-label hostname. On Ubuntu 24.04 (systemd-resolved), a single-label name fails withEAI_AGAINrather thanENOTFOUND..invalidis reserved and returnsENOTFOUNDeverywhere. This was verified on RHEL 8, Windows, macOS and Ubuntu 24.04.test/readme.md: new section on running tests against DSC.Notes for Reviewers
More Claude-speak with more context.
slslocalgives the members priorities 1, 0, 0, soreplSetStepDowncan't hand off to another member. This is documented in the DET disagg README.dsc: 'disabled'markers sit on the test rather than on itsdescribe. A test's ownmetadatareplaces its suite's metadata (metadata_ui.js), so a suite-level marker is ignored when the test sets its ownrequires.cc40ba76).Double check the following
npm run check:lint)type(NODE-xxxx)[!]: descriptionfeat(NODE-1234)!: rewriting everything in coffeescript