feat: multi-address DNS resolution for contact points and connections (DRIVER-201)#890
feat: multi-address DNS resolution for contact points and connections (DRIVER-201)#890nikagra wants to merge 9 commits into
Conversation
…VER-201) newControlReconnectionQueryPlan() now creates copies of the original contact-point nodes (with their unresolved hostname endpoints) instead of synthetic nodes with resolved IPs. This ensures the control channel carries the hostname endpoint, which is preserved in metadata after topology refresh. DNS expansion for connection fallback is handled by ChannelFactory (PR scylladb#890), so the control-reconnection path does not need to inject resolved-IP nodes into the query plan. Also adds getContactPoints() stub back to LoadBalancingPolicyWrapperTest so tests that cover the control-reconnect path continue to pass.
Before-init query plan now uses getContactPoints() (original unresolved hostname nodes) instead of getResolvedContactPoints(). The DNS expansion to all IPs happens at the ChannelFactory level (PR scylladb#890), so expanding here was redundant and broke should_connect_with_mocked_hostname by replacing hostname endpoints with resolved-IP endpoints. Also remove the should_connect_when_first_dns_entry_is_non_responsive integration test from this PR; it belongs in PR scylladb#890 where ChannelFactory expansion actually enables it to pass.
There was a problem hiding this comment.
Pull request overview
Part 2/2 of DRIVER-201: extends the EndPoint API and ChannelFactory so that a hostname mapping to multiple IPs is tried address-by-address at the connection layer, instead of only the first IP. The EndPoint.resolve() method is deprecated in favor of a new resolveAll() default method; DefaultEndPoint, SniEndPoint, and ClientRoutesEndPoint override it; ChannelFactory.connect() now iterates over candidates and only fails when all are exhausted, while keeping protocol-version downgrade scoped to a single address.
Changes:
- Add
EndPoint.resolveAll()(default impl delegating to deprecatedresolve()); override inDefaultEndPoint,SniEndPoint,ClientRoutesEndPoint. - Rework
ChannelFactory.connect()intotryNextCandidate/connectToAddressso per-address failures fall back to the next IP while protocol-version downgrades stay scoped to one address. - Add unit tests for
DefaultEndPoint.resolveAll()and a newSniEndPointTest.
Reviewed changes
Copilot reviewed 7 out of 7 changed files in this pull request and generated 4 comments.
Show a summary per file
| File | Description |
|---|---|
| core/src/main/java/com/datastax/oss/driver/api/core/metadata/EndPoint.java | Deprecates resolve(); adds default resolveAll() method. |
| core/src/main/java/com/datastax/oss/driver/internal/core/metadata/DefaultEndPoint.java | Overrides resolveAll() using InetAddress.getAllByName with single-address fallback. |
| core/src/main/java/com/datastax/oss/driver/internal/core/metadata/SniEndPoint.java | Overrides resolveAll() returning one address per sorted A-record. |
| core/src/main/java/com/datastax/oss/driver/internal/core/metadata/ClientRoutesEndPoint.java | Overrides resolveAll() to wrap the single topology-monitor address. |
| core/src/main/java/com/datastax/oss/driver/internal/core/channel/ChannelFactory.java | Adds candidate-iteration and per-address protocol-negotiation methods. |
| core/src/test/java/com/datastax/oss/driver/internal/core/metadata/DefaultEndPointTest.java | New tests for resolveAll() (resolved, unresolved expansion, unresolvable fallback). |
| core/src/test/java/com/datastax/oss/driver/internal/core/metadata/SniEndPointTest.java | New test class covering SNI resolveAll() happy path, unresolvable host, and resolve() sanity check. |
Comments suppressed due to low confidence (1)
core/src/main/java/com/datastax/oss/driver/internal/core/channel/ChannelFactory.java:303
- When
connectToAddressfails withUnsupportedProtocolVersionException.forNegotiation(i.e. all protocol downgrades exhausted),tryNextCandidatewill treat this like any other per-address failure and try the next IP, even though the protocol-negotiation failure is a server-wide condition that will recur on every other IP of the same node. This also reuses the sharedattemptedVersionsCopyOnWriteArrayListacross candidates, so on each subsequent address the downgrade loop re-attempts the same protocol versions and adds duplicate entries, and the final exception ultimately reported will list each version multiple times. Consider distinguishing non-address-specific failures (UnsupportedProtocolVersionException, authentication errors, etc.) and short-circuiting the candidate loop in those cases.
perAddressFuture.whenComplete(
(channel, error) -> {
if (error == null) {
resultFuture.complete(channel);
} else if (index + 1 < candidates.length) {
LOG.debug(
"[{}] Failed to connect to {} ({}), trying next address",
logPrefix,
candidate,
error.getMessage());
tryNextCandidate(
endPoint,
shardingInfo,
shardId,
options,
nodeMetricUpdater,
currentVersion,
isNegotiating,
attemptedVersions,
resultFuture,
candidates,
index + 1);
} else {
// Note: might be completed already if the failure happened in initializer()
resultFuture.completeExceptionally(error);
}
});
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
05553f3 to
f631971
Compare
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughThis PR adds Sequence Diagram(s)sequenceDiagram
participant ChannelFactory
participant EndPoint
participant tryNextCandidate
participant connectToAddress
participant resultFuture
ChannelFactory->>EndPoint: resolveAll()
EndPoint-->>ChannelFactory: SocketAddress[] candidates
ChannelFactory->>tryNextCandidate: attempt candidate at index 0
tryNextCandidate->>connectToAddress: connect using perAddressFuture
alt connection succeeds
connectToAddress-->>tryNextCandidate: DriverChannel
tryNextCandidate->>resultFuture: complete successfully
else connection or negotiation fails
connectToAddress-->>tryNextCandidate: complete perAddressFuture exceptionally
tryNextCandidate->>tryNextCandidate: attempt next candidate
end
tryNextCandidate->>resultFuture: fail after all candidates
Suggested reviewers: 🚥 Pre-merge checks | ✅ 4 | ❌ 1❌ Failed checks (1 warning)
✅ Passed checks (4 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
f631971 to
860a34d
Compare
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@core/src/main/java/com/datastax/oss/driver/internal/core/channel/ChannelFactory.java`:
- Around line 222-242: The code calls endPoint.resolveAll() and passes the
resulting candidates array into tryNextCandidate() which immediately indexes
candidates[0]; guard against null or empty results by validating the output of
endPoint.resolveAll()—if it returns null or candidates.length == 0, complete
resultFuture exceptionally (or create a specific error) and return; otherwise
call tryNextCandidate(...) with the non-empty candidates. Update the block
around resolveAll(), candidates, and the call to tryNextCandidate() to perform
this check and fail fast via resultFuture.completeExceptionally when
appropriate.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 7ad3d5b5-6473-4c88-8777-93861f5de639
📒 Files selected for processing (12)
core/src/main/java/com/datastax/dse/driver/api/core/auth/DseGssApiAuthProviderBase.javacore/src/main/java/com/datastax/dse/driver/internal/core/insights/InsightsClient.javacore/src/main/java/com/datastax/oss/driver/api/core/metadata/EndPoint.javacore/src/main/java/com/datastax/oss/driver/internal/core/channel/ChannelFactory.javacore/src/main/java/com/datastax/oss/driver/internal/core/metadata/ClientRoutesEndPoint.javacore/src/main/java/com/datastax/oss/driver/internal/core/metadata/DefaultEndPoint.javacore/src/main/java/com/datastax/oss/driver/internal/core/metadata/DefaultTopologyMonitor.javacore/src/main/java/com/datastax/oss/driver/internal/core/metadata/SniEndPoint.javacore/src/main/java/com/datastax/oss/driver/internal/core/ssl/DefaultSslEngineFactory.javacore/src/main/java/com/datastax/oss/driver/internal/core/ssl/SniSslEngineFactory.javacore/src/test/java/com/datastax/oss/driver/internal/core/metadata/DefaultEndPointTest.javacore/src/test/java/com/datastax/oss/driver/internal/core/metadata/SniEndPointTest.java
860a34d to
a6d0e48
Compare
There was a problem hiding this comment.
🧹 Nitpick comments (1)
core/src/test/java/com/datastax/oss/driver/internal/core/channel/ChannelFactoryResolveAllGuardTest.java (1)
37-37: ⚡ Quick winConsider adding test coverage for resolveAll() throwing an exception.
The
ChannelFactory.connect()implementation includes a catch block for exceptions thrown byresolveAll()(see context snippet 1, line 232). Adding a third test case where the mockedEndPoint.resolveAll()throws an exception (e.g.,UnknownHostException) would ensure all three defensive paths are tested:
- ✓ Returns null (covered)
- ✓ Returns empty array (covered)
- ✗ Throws exception (not covered)
📋 Suggested test case
`@Test` public void should_fail_future_when_resolve_all_throws_exception() { // Given when(defaultProfile.isDefined(DefaultDriverOption.PROTOCOL_VERSION)).thenReturn(false); when(protocolVersionRegistry.highestNonBeta()).thenReturn(DefaultProtocolVersion.V4); ChannelFactory factory = newChannelFactory(); EndPoint badEndPoint = mock(EndPoint.class); RuntimeException testException = new RuntimeException("DNS lookup failed"); when(badEndPoint.resolveAll()).thenThrow(testException); // When CompletionStage<DriverChannel> channelFuture = factory.connect( badEndPoint, null, null, DriverChannelOptions.DEFAULT, NoopNodeMetricUpdater.INSTANCE); // Then – future must complete exceptionally with the thrown exception assertThatStage(channelFuture) .isFailed(e -> assertThat(e).isSameAs(testException)); }🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@core/src/test/java/com/datastax/oss/driver/internal/core/channel/ChannelFactoryResolveAllGuardTest.java` at line 37, Add a third test in ChannelFactoryResolveAllGuardTest that verifies ChannelFactory.connect() propagates exceptions thrown by EndPoint.resolveAll(): mock an EndPoint (e.g., badEndPoint) to throw a RuntimeException (or UnknownHostException) from resolveAll(), create the factory via newChannelFactory(), call factory.connect(badEndPoint, ...) with DriverChannelOptions.DEFAULT and NoopNodeMetricUpdater.INSTANCE, and assert the returned CompletionStage<DriverChannel> completes exceptionally with the same exception; this mirrors the existing tests for null/empty resolveAll() and targets the catch path in ChannelFactory.connect().
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Nitpick comments:
In
`@core/src/test/java/com/datastax/oss/driver/internal/core/channel/ChannelFactoryResolveAllGuardTest.java`:
- Line 37: Add a third test in ChannelFactoryResolveAllGuardTest that verifies
ChannelFactory.connect() propagates exceptions thrown by EndPoint.resolveAll():
mock an EndPoint (e.g., badEndPoint) to throw a RuntimeException (or
UnknownHostException) from resolveAll(), create the factory via
newChannelFactory(), call factory.connect(badEndPoint, ...) with
DriverChannelOptions.DEFAULT and NoopNodeMetricUpdater.INSTANCE, and assert the
returned CompletionStage<DriverChannel> completes exceptionally with the same
exception; this mirrors the existing tests for null/empty resolveAll() and
targets the catch path in ChannelFactory.connect().
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: b702fd48-9ba7-4994-8bb9-351438fb02a8
📒 Files selected for processing (13)
core/src/main/java/com/datastax/dse/driver/api/core/auth/DseGssApiAuthProviderBase.javacore/src/main/java/com/datastax/dse/driver/internal/core/insights/InsightsClient.javacore/src/main/java/com/datastax/oss/driver/api/core/metadata/EndPoint.javacore/src/main/java/com/datastax/oss/driver/internal/core/channel/ChannelFactory.javacore/src/main/java/com/datastax/oss/driver/internal/core/metadata/ClientRoutesEndPoint.javacore/src/main/java/com/datastax/oss/driver/internal/core/metadata/DefaultEndPoint.javacore/src/main/java/com/datastax/oss/driver/internal/core/metadata/DefaultTopologyMonitor.javacore/src/main/java/com/datastax/oss/driver/internal/core/metadata/SniEndPoint.javacore/src/main/java/com/datastax/oss/driver/internal/core/ssl/DefaultSslEngineFactory.javacore/src/main/java/com/datastax/oss/driver/internal/core/ssl/SniSslEngineFactory.javacore/src/test/java/com/datastax/oss/driver/internal/core/channel/ChannelFactoryResolveAllGuardTest.javacore/src/test/java/com/datastax/oss/driver/internal/core/metadata/DefaultEndPointTest.javacore/src/test/java/com/datastax/oss/driver/internal/core/metadata/SniEndPointTest.java
✅ Files skipped from review due to trivial changes (5)
- core/src/main/java/com/datastax/oss/driver/internal/core/ssl/SniSslEngineFactory.java
- core/src/main/java/com/datastax/dse/driver/api/core/auth/DseGssApiAuthProviderBase.java
- core/src/main/java/com/datastax/oss/driver/internal/core/ssl/DefaultSslEngineFactory.java
- core/src/main/java/com/datastax/oss/driver/internal/core/metadata/DefaultTopologyMonitor.java
- core/src/main/java/com/datastax/dse/driver/internal/core/insights/InsightsClient.java
🚧 Files skipped from review as they are similar to previous changes (7)
- core/src/test/java/com/datastax/oss/driver/internal/core/metadata/SniEndPointTest.java
- core/src/main/java/com/datastax/oss/driver/internal/core/metadata/ClientRoutesEndPoint.java
- core/src/main/java/com/datastax/oss/driver/internal/core/metadata/DefaultEndPoint.java
- core/src/main/java/com/datastax/oss/driver/internal/core/metadata/SniEndPoint.java
- core/src/test/java/com/datastax/oss/driver/internal/core/metadata/DefaultEndPointTest.java
- core/src/main/java/com/datastax/oss/driver/api/core/metadata/EndPoint.java
- core/src/main/java/com/datastax/oss/driver/internal/core/channel/ChannelFactory.java
a6d0e48 to
f9265b3
Compare
|
🤖: Valid nitpick. Added a third test |
f9265b3 to
4448119
Compare
| Arrays.sort(aRecords, IP_COMPARATOR); | ||
| SocketAddress[] result = new SocketAddress[aRecords.length]; | ||
| for (int i = 0; i < aRecords.length; i++) { | ||
| result[i] = new InetSocketAddress(aRecords[i], proxyAddress.getPort()); |
There was a problem hiding this comment.
When the SNI proxy hostname has multiple A records, ChannelFactory now calls resolveAll() for every connection instead of resolve(). This always returns the sorted records starting at index 0, so all healthy connections go to the first proxy IP, and if that IP is down every connection pays the timeout before trying the next one. Please rotate the returned candidate order with the existing OFFSET logic so resolveAll() preserves the previous round-robin behavior.
There was a problem hiding this comment.
🤖: Valid, fixed. resolveAll() now rotates the returned candidate order using the same OFFSET round-robin counter as resolve(): after sorting the A-records by IP, it picks a rotating start index (OFFSET.getAndUpdate(...) % length) and fills the result array as addr[(start + i) % length]. So healthy connections are still spread across proxy IPs (previous round-robin behavior preserved), while the full set of records is always returned so a single connection attempt can still fall back across every IP if one is down. Single-record hostnames keep start index 0. Added resolve_all_returns_complete_and_rotated_candidate_order() to SniEndPointTest asserting both invariants (complete set every call + starting candidate rotates when multiple IPs exist).
There was a problem hiding this comment.
resolveAll() returns the complete A-record set on every call and rotates the starting candidate. The rotation added in the previous push was undermined by the shared OFFSET counter (see the Copilot thread on the same file): SSL engine setup calls resolve() once per TLS connection, advancing the same counter, which pinned the start index to 0 whenever the proxy resolved to an even number of A-records. Fixed in 46d1ae6 by giving resolveAll() a dedicated RESOLVE_ALL_OFFSET, with an interleaved-resolve() unit test to guard against regressions.
4448119 to
1c8dfa2
Compare
|
Rebased this PR (Part 2/2) on top of #889 ( Also addressed the outstanding review feedback:
Previously-addressed items (Copilot / CodeRabbit) remain in place after the rebase: N×timeout Javadoc on Verified locally on JDK 11: |
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 29 out of 29 changed files in this pull request and generated 3 comments.
Comments suppressed due to low confidence (1)
core/src/main/java/com/datastax/oss/driver/internal/core/metadata/DefaultEndPoint.java:62
NETTY_ADMIN_SIZEonly configures the number of admin event-loop threads (DefaultDriverOption.java:807-811); it does not configure anAddressResolverGroup. This link gives users an incorrect way to identify or change the resolver. Refer to a customNettyOptionsbootstrap hook instead, or omit the configuration link.
* <p><b>Note on resolver:</b> DNS lookup is performed via {@link
* InetAddress#getAllByName(String)} on the calling thread, bypassing any custom Netty {@code
* AddressResolverGroup} configured via {@link
* com.datastax.oss.driver.api.core.config.DefaultDriverOption#NETTY_ADMIN_SIZE}. This is
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In
`@core/src/main/java/com/datastax/oss/driver/api/core/config/TypedDriverOption.java`:
- Line 603: Update the public Javadoc for the reconnection-plan option in
TypedDriverOption to state that it appends DNS-expanded candidates returned by
getResolvedContactPoints(), rather than raw original contact points, and that
monitors which re-resolve addresses skip this behavior; retain the documented
default of true.
In
`@core/src/main/java/com/datastax/oss/driver/internal/core/metadata/LoadBalancingPolicyWrapper.java`:
- Around line 147-153: Prevent blocking DNS resolution from query-plan creation
by moving MetadataManager.getResolvedContactPoints() off the caller thread or
introducing bounded caching before using its results. Apply the fix to the
BEFORE_INIT/DURING_INIT path in
core/src/main/java/com/datastax/oss/driver/internal/core/metadata/LoadBalancingPolicyWrapper.java:147-153
and the control-reconnect path in
core/src/main/java/com/datastax/oss/driver/internal/core/metadata/LoadBalancingPolicyWrapper.java:164-184;
update core/src/main/resources/reference.conf:2321-2334 if needed so
fallback-to-original-contact-points is not enabled without bounded, non-blocking
resolution.
In `@core/src/main/resources/reference.conf`:
- Around line 2321-2334: The default for fallback-to-original-contact-points
must not enable the blocking DNS fallback path; change this configuration
default back to false while preserving the existing setting name and
documentation.
In
`@core/src/test/java/com/datastax/oss/driver/internal/core/metadata/MetadataManagerTest.java`:
- Around line 512-529: The test should enforce expansion to the complete DNS
result set, not merely verify that one resolved node exists. Update
should_expand_unresolved_hostname_to_all_ips to obtain
InetAddress.getAllByName("localhost"), compare the returned node count and
endpoint addresses against all expected addresses on port 9042, and retain the
resolved-address assertions.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: CHILL
Plan: Pro Plus
Run ID: 648940a1-36ee-47f0-8f02-aff008723307
📒 Files selected for processing (29)
core/src/main/java/com/datastax/dse/driver/api/core/auth/DseGssApiAuthProviderBase.javacore/src/main/java/com/datastax/dse/driver/internal/core/insights/InsightsClient.javacore/src/main/java/com/datastax/oss/driver/api/core/config/DefaultDriverOption.javacore/src/main/java/com/datastax/oss/driver/api/core/config/OptionsMap.javacore/src/main/java/com/datastax/oss/driver/api/core/config/TypedDriverOption.javacore/src/main/java/com/datastax/oss/driver/api/core/metadata/EndPoint.javacore/src/main/java/com/datastax/oss/driver/api/core/session/SessionBuilder.javacore/src/main/java/com/datastax/oss/driver/internal/core/channel/ChannelFactory.javacore/src/main/java/com/datastax/oss/driver/internal/core/loadbalancing/helper/OptionalLocalDcHelper.javacore/src/main/java/com/datastax/oss/driver/internal/core/metadata/ClientRoutesEndPoint.javacore/src/main/java/com/datastax/oss/driver/internal/core/metadata/ClientRoutesTopologyMonitor.javacore/src/main/java/com/datastax/oss/driver/internal/core/metadata/CloudTopologyMonitor.javacore/src/main/java/com/datastax/oss/driver/internal/core/metadata/DefaultEndPoint.javacore/src/main/java/com/datastax/oss/driver/internal/core/metadata/DefaultTopologyMonitor.javacore/src/main/java/com/datastax/oss/driver/internal/core/metadata/LoadBalancingPolicyWrapper.javacore/src/main/java/com/datastax/oss/driver/internal/core/metadata/MetadataManager.javacore/src/main/java/com/datastax/oss/driver/internal/core/metadata/SniEndPoint.javacore/src/main/java/com/datastax/oss/driver/internal/core/metadata/TopologyMonitor.javacore/src/main/java/com/datastax/oss/driver/internal/core/ssl/DefaultSslEngineFactory.javacore/src/main/java/com/datastax/oss/driver/internal/core/ssl/SniSslEngineFactory.javacore/src/main/resources/reference.confcore/src/test/java/com/datastax/dse/driver/internal/core/insights/InsightsClientTest.javacore/src/test/java/com/datastax/oss/driver/internal/core/channel/ChannelFactoryResolveAllGuardTest.javacore/src/test/java/com/datastax/oss/driver/internal/core/metadata/DefaultEndPointTest.javacore/src/test/java/com/datastax/oss/driver/internal/core/metadata/LoadBalancingPolicyWrapperTest.javacore/src/test/java/com/datastax/oss/driver/internal/core/metadata/MetadataManagerTest.javacore/src/test/java/com/datastax/oss/driver/internal/core/metadata/SniEndPointTest.javaintegration-tests/src/test/java/com/datastax/oss/driver/core/heartbeat/HeartbeatIT.javaintegration-tests/src/test/java/com/datastax/oss/driver/core/resolver/MockResolverIT.java
🚧 Files skipped from review as they are similar to previous changes (11)
- core/src/test/java/com/datastax/oss/driver/internal/core/channel/ChannelFactoryResolveAllGuardTest.java
- core/src/main/java/com/datastax/dse/driver/api/core/auth/DseGssApiAuthProviderBase.java
- core/src/main/java/com/datastax/oss/driver/internal/core/metadata/DefaultTopologyMonitor.java
- core/src/main/java/com/datastax/oss/driver/internal/core/metadata/ClientRoutesEndPoint.java
- core/src/main/java/com/datastax/oss/driver/internal/core/ssl/SniSslEngineFactory.java
- core/src/main/java/com/datastax/oss/driver/internal/core/metadata/SniEndPoint.java
- core/src/main/java/com/datastax/oss/driver/api/core/metadata/EndPoint.java
- core/src/main/java/com/datastax/oss/driver/internal/core/channel/ChannelFactory.java
- core/src/main/java/com/datastax/dse/driver/internal/core/insights/InsightsClient.java
- core/src/main/java/com/datastax/oss/driver/internal/core/metadata/DefaultEndPoint.java
- core/src/test/java/com/datastax/oss/driver/internal/core/metadata/DefaultEndPointTest.java
… (DRIVER-201) When RESOLVE_CONTACT_POINTS=false (the default) a hostname contact point was stored as a single unresolved InetSocketAddress, so the query plan tried only the first DNS IP. Keep contact points unresolved and expand each hostname to all its DNS IPs at query-plan time via MetadataManager.getResolvedContactPoints(), so the driver falls back to the next candidate when one IP is unreachable. Resolution is bounded, concurrent and best-effort. getResolvedContactPoints() runs on the admin event loop, where nothing should block, so each blocking InetAddress.getAllByName() call is offloaded to a cached daemon-thread pool and all unresolved hostnames are resolved concurrently against a single CONTACT_POINT_RESOLUTION_TIMEOUT deadline. A cached pool (rather than one shared thread) means each hostname resolves on its own thread, so one slow or blackholed lookup cannot starve the sibling contact points, nor the next reconnect that would otherwise queue behind it. If a hostname cannot be resolved or resolution times out, the original unresolved contact point is kept as-is rather than dropped, so the query plan is never emptier than the configured contact points and the address can still be resolved later at connection time (as it was before DNS expansion existed). This is an interim mitigation, superseded by scylladb#890's non-blocking EndPoint.resolveAll(). Default advanced.control-connection.reconnection.fallback-to-original-contact-points to true (no longer Experimental): it is the DNS re-resolution path on reconnect. Metadata nodes hold an already-resolved endpoint that is never re-resolved, so falling back to the original unresolved contact points re-expands the hostname to its current DNS IPs. Document that DNS-expanded contact points are IP-backed connection candidates that may be persisted in metadata, and that each synthetic endpoint retains the original hostname (built from the resolved InetAddress) so TLS peer host / SNI / hostname verification keep using the configured hostname. Gate the control-connection reconnection contact-point fallback behind a new TopologyMonitor.reresolvesNodeAddresses() (default false; true for the proxy-based ClientRoutesTopologyMonitor and CloudTopologyMonitor). Those monitors reach nodes through endpoints that already re-resolve on every connection attempt and maintain an authoritative node set, so appending raw contact points to their reconnection plan is unnecessary and could resurrect removed nodes (PrivateLink/Cloud regression safety). The reconnection plan also appends the contact points only once the load balancing policy is RUNNING, so the pre-init plan (already built from the resolved contact points) is not duplicated or re-resolved. Remove OptionalLocalDcHelper.checkLocalDatacenterCompatibility(): it warned when a contact point reported a different datacenter than the configured local DC. Since commit 12e6acb switched initial metadata refresh to hostId-only matching, contact-point nodes are never reused and their datacenter stays null; comparing a configured local DC against that null made the check fire as a false positive for every contact point whenever local-datacenter was set on the default profile, rather than surface a real mismatch. The node-based "configured local DC matches no node" warning (against discovered nodes whose datacenters are populated) is retained, so the only user-visible effect is that the spurious warning is no longer emitted. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
1c8dfa2 to
b4e4d14
Compare
|
Pushed
90 core unit tests pass, including new coverage that resolution runs on the dedicated resolver thread and that Note: this does not change the SNI |
…DRIVER-201) Built-in QueryPlan implementations reject add()/addAll() (poll() is their only mutator), so appending the contact-point fallback via regularQueryPlan.addAll(...) threw UnsupportedOperationException on every post-init control reconnect once the fallback defaulted on. Build the fallback as a SimpleQueryPlan and concatenate it via CompositeQueryPlan instead of mutating the policy's plan. Also keep the fallback available when the live-node plan is empty, even for re-resolving topology monitors, so control-connection reconnection can still recover when there is nothing else to try. Stub the wrapper tests with real QueryPlan implementations (SimpleQueryPlan / QueryPlan.EMPTY); the previous mutable LinkedList stubs masked the crash. Document the contact-point DNS-expansion behavior change in the upgrade guide. Addresses dkropachev's review on scylladb#889; folded into scylladb#890. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…VER-201) OptionalLocalDcHelper.checkLocalDatacenterCompatibility() warned when a contact point reported a datacenter different from the configured local DC. This has been dead code on scylla-4.x since 12e6acb: refresh matches nodes by hostId only, so contact-point nodes never get a datacenter assigned and the warning could never fire. Remove it. The separate "configured local DC matches no node" warning is retained. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…S (DRIVER-201) Contact points backed by a hostname are now always kept unresolved, so they can be expanded to all their DNS-mapped IPs later, at connection time (see the follow-up EndPoint.resolveAll() commits). SessionBuilder no longer reads RESOLVE_CONTACT_POINTS when merging contact points; the option is deprecated and has no effect. An already-resolved InetSocketAddress passed programmatically is still used as provided, with no further expansion. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ER-201) Add EndPoint.resolveAll(Executor), a default method returning a CompletionStage of all socket addresses an endpoint maps to, so a hostname with multiple A-records can be expanded to every IP. The default offloads resolve() to the executor and wraps it in a single-element array, keeping existing third-party implementations working (source- and binary-compatible; resolve() is not deprecated yet). Overrides: - DefaultEndPoint: InetAddress.getAllByName() expansion for unresolved addresses, falling back to the single stored address on DNS failure. - SniEndPoint: returns the complete A-record set, rotated with the same round-robin OFFSET as resolve() so healthy connections spread across proxy IPs. - ClientRoutesEndPoint: single-address-by-design, delegates to resolve(). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…precate resolve() (DRIVER-201) ChannelFactory.connect() now resolves via EndPoint.resolveAll() on a dedicated daemon resolver executor, so blocking DNS never runs on the admin event loop. The returned candidates are tried serially (tryNextCandidate): each address is attempted via connectToAddress(), and on a per-address failure the next candidate is tried; only when all are exhausted does the overall future fail. Protocol-version negotiation (downgrade retries) stays scoped to a single address. A null/empty resolveAll() result fails the future rather than NPE/AIOOBE. DefaultSession shuts the resolver executor down on close. resolve() is now @deprecated in favour of resolveAll(). The five internal callers that legitimately need a single canonical address (DefaultTopologyMonitor, InsightsClient, DseGssApiAuthProviderBase, DefaultSslEngineFactory, SniSslEngineFactory) are annotated @SuppressWarnings("deprecation") so the -Werror build stays green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… enable it by default (DRIVER-201) fallback-to-original-contact-points now defaults to true. On a control-connection reconnect the original (unresolved) contact points are appended after the live-node plan; EndPoint.resolveAll() re-expands each hostname to its current DNS IPs, which is the driver's DNS re-resolution path (metadata nodes hold an already-resolved endpoint that is never re-resolved). The append is gated: - only in the RUNNING state (before that newQueryPlan already builds the plan from the contact points, so appending would duplicate them); - skipped when the topology monitor re-resolves node addresses itself (TopologyMonitor.reresolvesNodeAddresses(), overridden true by CloudTopologyMonitor and route-aware in ClientRoutesTopologyMonitor), unless the regular plan is empty, to avoid resurrecting nodes those proxy-based monitors authoritatively removed. The plan is now composed (CompositeQueryPlan + SimpleQueryPlan) rather than mutated, since a RUNNING-state built-in query plan rejects add()/addAll(). HeartbeatIT pins the option to false so the extra OPTIONS message does not skew heartbeat counts. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ER-201) Add a 4.19.2.1 upgrade-guide entry covering the multi-address contact-point expansion, the deprecation of advanced.resolve-contact-points (now a no-op), and the fallback-to-original-contact-points default flip (false -> true) with how to restore the previous behavior. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
97cba7e to
bee0837
Compare
resolveAll() shared the round-robin OFFSET counter with the deprecated resolve(). SniSslEngineFactory.newSslEngine() calls resolve() on every TLS connection, so a normal SNI-over-TLS connect advanced the counter twice and resolveAll()'s start index moved in steps of two -- collapsing rotation to a single proxy IP whenever the hostname resolved to an even number of A-records (index 0 in the common two-record case). Give resolveAll() its own RESOLVE_ALL_OFFSET so SSL engine setup no longer perturbs its rotation, and add a test that interleaves resolve() calls to lock this in. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…sts (DRIVER-201) When every candidate address from resolveAll() failed, ChannelFactory propagated only the last candidate's error and discarded the earlier ones (they were logged at DEBUG only). Attach the earlier failures as suppressed exceptions on the propagated error so the full picture is available for diagnosis. Add ChannelFactoryMultiAddressTest, which the existing single-address tests did not cover: first-candidate-fails/second-succeeds fallback, and the all-candidates-exhausted path (asserting the suppressed cause). Strengthen DefaultEndPointTest to assert resolveAll() expands to the complete DNS result set (compared against an independent InetAddress.getAllByName lookup), not just a non-empty subset. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… option (DRIVER-201) The reference.conf entry for advanced.resolve-contact-points still described the old resolve-once/resolve-every-connection semantics as active above a "DEPRECATED: no effect" footer, contradicting itself. Lead with the deprecation and drop the stale active-voice prose. Clarify the CONTROL_CONNECTION_RECONNECT_CONTACT_POINTS Javadoc in TypedDriverOption: it appends the original (unresolved) contact points, expands them to all DNS IPs at connection time via EndPoint.resolveAll(), and is skipped for topology monitors that re-resolve node addresses themselves. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Problem
DRIVER-201: when a contact point or a cluster node is given as a hostname that maps to multiple IPs (e.g. a DNS round-robin / dynamic-DNS entry), the driver only ever tried the first address — at initial contact, at connection time, and on control-connection reconnect. If that first IP was unreachable the driver raised
AllNodesFailedExceptioneven though the hostname also resolved to healthy IPs.This PR fixes DRIVER-201 end-to-end: every such hostname is now resolved to all its addresses and each is tried in turn, at both the contact-point and the general connection layer.
Changes
EndPointinterfaceresolve()is now@Deprecated.resolveAll()default method returnsSocketAddress[]. The default implementation wrapsresolve()in a single-element array, so existing third-party implementations keep working (no new abstract method → source/binary compatible).DefaultEndPointresolveAll(): for unresolved addresses callsInetAddress.getAllByName()and returns oneInetSocketAddressper IP (built from the resolvedInetAddressso the original hostname is retained for TLS peer host / SNI / hostname verification). Falls back to a single-element array (the unresolved address) if DNS fails, so the connect attempt surfaces a descriptive error rather than an empty array.SniEndPointresolveAll(): re-resolves the proxy hostname on each call, sorts all A-records by IP, and returns all records so a single connection attempt can fall back across every proxy IP. The candidate order is rotated each call using the same round-robinOFFSETcounter asresolve(), so healthy connections stay spread across proxy IPs instead of always starting at index 0. (dkropachev'sCHANGES_REQUESTEDfix.)ClientRoutesEndPointresolveAll(): wraps the single topology-monitor-resolved address in a one-element array (single-address by design).ChannelFactoryconnect()now callsendPoint.resolveAll()instead ofendPoint.resolve(), and guards against anull/empty array from a customEndPointby failingresultFuture(instead of NPE/AIOOBE).tryNextCandidate()iterates the returned array; on per-address failure it logs and tries the next; only fails the overallresultFutureonce all candidates are exhausted.connectToAddress()scopes protocol-version negotiation (downgrade retries) to a single address.connect()now serially attempts every candidate, so the worst-case time to declare a node unreachable isN × connect-timeout. This is an intentional trade-off (failing on the first unreachable IP would prevent fallback) and is documented onEndPoint.resolveAll().Remove the interim query-plan resolution
Now that
ChannelFactory.connect() → resolveAll()handles multi-address fallback at connection time — for both control-connection init and pool connections — the earlier interim query-plan-time DNS expansion (the contact-point hostname expansion from the initial approach) is redundant and is removed:MetadataManager: dropsgetResolvedContactPoints()and its dedicated resolver executor, 3s timeout, and helpers. That method resolved contact-point hostnames on the admin event loop (offloaded to a bounded executor becauseInetAddress.getAllByName()blocks and the admin loop must never block). The event loop no longer blocks on DNS at all — the query plan now holds one unresolved node per contact point, andresolveAll()expands each to all its IPs at connect time.LoadBalancingPolicyWrapper/InsightsClient: revert togetContactPoints(). TheRUNNING-state reconnection fallback and theTopologyMonitor.reresolvesNodeAddresses()gate are preserved.reference.conf,SessionBuilder,DefaultEndPoint) updated to say resolution happens at connection time viaEndPoint.resolveAll().Control-connection reconnection query plan (folded from #889 review)
LoadBalancingPolicyWrapper.newControlReconnectionQueryPlan()now composes the contact-point fallback viaCompositeQueryPlan(regularPlan, new SimpleQueryPlan(contactNodes))instead of mutating the policy's plan. Built-inQueryPlans rejectadd()/addAll()(poll()is their only mutator), so the previousaddAll(...)threwUnsupportedOperationExceptionon every post-init control reconnect once the fallback defaulted on. The fallback is also kept when the live-node plan is empty, even for re-resolving topology monitors, so reconnection can still recover when there is nothing else to try. The wrapper tests now stub the policy plan with a realSimpleQueryPlan/QueryPlan.EMPTY(the earlier mutableLinkedListstub masked the crash), plus a new empty-plan + re-resolving-monitor case. (dkropachev's #889CHANGES_REQUESTEDfix.)Internal callers
Callers that legitimately need a single canonical address (
InsightsClient,DseGssApiAuthProviderBase,DefaultTopologyMonitor, the SNI / Default SSL engine factories) keep calling the deprecatedresolve()under a scoped@SuppressWarnings("deprecation").Tests
DefaultEndPointTest: already-resolved passthrough, unresolved hostname expansion, unresolvable hostname fallback.SniEndPointTest:resolveAll()happy path, unresolvable host exception,resolve()sanity check, and a rotation/completeness case asserting the full candidate set is returned every call and the starting candidate rotates when multiple IPs exist.ChannelFactoryResolveAllGuardTest: null array, empty array, andresolveAll()throwing all fail the connect future.LoadBalancingPolicyWrapperTest: realQueryPlanstubs; append-ordering, empty-plan, and re-resolving-monitor cases for the control-reconnection plan.ChannelFactorytests pass unchanged (LocalEndPointuses the default single-elementresolveAll()via the interface default).MetadataManagerTestcontact-point resolution/timeout unit tests (that behavior now lives inDefaultEndPointTest.resolveAlland theChannelFactorytests); adapted theLoadBalancingPolicyWrapper/InsightsClienttests togetContactPoints().