feat(health): add ClusterEndpointSource for BCM/file fleet inventory#3359
Conversation
Summary by CodeRabbit
WalkthroughAdds a configurable ChangesCluster endpoint discovery
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant HealthService
participant ClusterEndpointSource
participant ClusterManager
participant BmcClient
HealthService->>ClusterEndpointSource: Construct enabled source
ClusterEndpointSource->>ClusterManager: Request credentials and devices
ClusterManager-->>ClusterEndpointSource: Return inventory data
ClusterEndpointSource->>BmcClient: Build BMC clients
ClusterEndpointSource-->>HealthService: Provide endpoint source
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
There was a problem hiding this comment.
Actionable comments posted: 8
🧹 Nitpick comments (1)
crates/health/src/endpoint/cluster.rs (1)
1-414: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy liftNo test coverage for the new parsing/extraction logic.
This entire new file —
extract_manager_devices,fetch_manager_credentials,read_from_file, andbuild_endpoints— ships without tests, despite being exactly the kind of parser/normalization logic the repo's testing conventions target.sources.rsandmodel.rs(in the same crate) both carry table-driven#[test]coverage for comparable logic.Consider adding
value_scenarios!/check_valuestables forextract_manager_devices(varying response shapes: top-level array,result/data/items/deviceswrappers, missing hostname/IP, invalid IP) and forread_from_file(valid inventory, malformed JSON, missing file). As per path instructions, "prefer table-driven scenarios ... especially for parsers, validators, and conversions."🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/health/src/endpoint/cluster.rs` around lines 1 - 414, Add table-driven tests following the crate’s existing value_scenarios!/check_values conventions for extract_manager_devices, covering top-level and result/data/items/devices wrappers, missing fields, and invalid IPs; add read_from_file cases for valid inventory, malformed JSON, and missing files. Use temporary inventory paths where needed and assert parsed nodes or expected errors, without changing production behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/health/src/config.rs`:
- Around line 102-134: ClusterEndpointSourceConfig’s derived Debug
implementation exposes default_password in logs. Replace the derived Debug with
a manual implementation that redacts or omits default_password, following
StaticBmcEndpoint’s custom Debug pattern, while preserving the existing
serialization behavior unless separately required.
- Around line 104-134: Add ClusterEndpointSourceConfig::validate() to require
either a non-empty inventory_path or a configured cluster_manager_url, returning
a clear configuration error when both are absent; then invoke this validation
from Config::validate() whenever the cluster endpoint source is enabled.
In `@crates/health/src/endpoint/cluster.rs`:
- Around line 236-242: Preserve the absence of a rack identifier in the
cluster-manager parsing and endpoint construction. Update the logic producing
`rack` near the category/partition lookup to return an optional value rather
than defaulting to an empty string, and update `build_endpoints` to pass `None`
when no non-empty rack exists instead of unconditionally constructing
`Some(RackId::new(&node.rack))`; retain `Some(RackId)` only for a valid rack so
`BmcEndpoint::hash_key` falls back to the endpoint MAC.
- Around line 128-189: Remove the plaintext response logging from
fetch_manager_credentials by deleting the tracing::debug! call that emits raw.
If response diagnostics are required, create a redacted representation that
removes or masks bmcSettings.password and any equivalent credential fields
before logging it, while preserving structured logfmt fields.
- Around line 274-295: Refresh the cluster inventory during every discovery pass
instead of caching it only in ClusterEndpointSource::from_config. Update
ClusterEndpointSource and fetch_bmc_hosts to re-read via fetch_from_manager or
read_from_file, rebuild endpoints with the configured port, clients, proxy, and
cache size, and return the refreshed set so additions, removals, and IP changes
are observed without restarting.
- Around line 191-270: Make Cluster Manager parsing fail closed: update
fetch_manager_credentials to reject missing or unrecognized credential fields
instead of silently falling back, and update extract_manager_devices to return
or propagate a countable error when a non-empty getDevices response has no
recognized shape or produces zero valid nodes. Preserve explicit handling for
valid empty arrays, and ensure callers surface the parsing failure rather than
treating it as a successful empty inventory.
- Around line 88-93: The build_http function unconditionally disables TLS
certificate validation for the cluster-manager credential client. Remove
danger_accept_invalid_certs(true) and require normal trusted certificate-chain
validation, or gate the bypass behind an explicit opt-in configuration flag;
also avoid logging raw partition responses in the credential-fetching path.
In `@crates/health/src/lib.rs`:
- Around line 153-162: Make cluster endpoint initialization best-effort in
ClusterEndpointSource::from_config: handle device-inventory and URL/RPC failures
locally instead of propagating them through build_endpoint_wiring, log the
failure, and skip the cluster source (or apply the existing retry/backoff
mechanism) so other sources can start normally.
---
Nitpick comments:
In `@crates/health/src/endpoint/cluster.rs`:
- Around line 1-414: Add table-driven tests following the crate’s existing
value_scenarios!/check_values conventions for extract_manager_devices, covering
top-level and result/data/items/devices wrappers, missing fields, and invalid
IPs; add read_from_file cases for valid inventory, malformed JSON, and missing
files. Use temporary inventory paths where needed and assert parsed nodes or
expected errors, without changing production behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 8600f962-7b95-4e00-bec6-6a463bc9bd9e
📒 Files selected for processing (4)
crates/health/src/config.rscrates/health/src/endpoint/cluster.rscrates/health/src/endpoint/mod.rscrates/health/src/lib.rs
| if let Configurable::Enabled(ref source_cfg) = config.endpoint_sources.cluster { | ||
| let cluster_source = ClusterEndpointSource::from_config( | ||
| source_cfg, | ||
| &reqwest, | ||
| config.bmc_proxy_url.as_ref(), | ||
| config.cache_size, | ||
| ) | ||
| .await?; | ||
| sources.push(Arc::new(cluster_source)); | ||
| } |
There was a problem hiding this comment.
🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Map the target file and nearby functions.
ast-grep outline crates/health/src/lib.rs --view expanded || true
# Read the relevant section with line numbers.
sed -n '120,230p' crates/health/src/lib.rs | cat -n
# Find ClusterEndpointSource and run_service references in the repo.
rg -n "ClusterEndpointSource::from_config|build_endpoint_wiring|run_service|static_bmc_endpoints|carbide_api|ApiClientWrapper::new|ApiEndpointSource::new" crates/health -SRepository: NVIDIA/infra-controller
Length of output: 14211
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Inspect the health crate structure and the relevant function bodies.
ast-grep outline crates/health/src/lib.rs --view expanded
printf '\n--- relevant slice ---\n'
sed -n '140,340p' crates/health/src/lib.rs | cat -n
printf '\n--- call sites ---\n'
rg -n "ClusterEndpointSource::from_config|build_endpoint_wiring|run_service|static_bmc_endpoints|ApiClientWrapper::new|ApiEndpointSource::new" crates/ -SRepository: NVIDIA/infra-controller
Length of output: 15947
🏁 Script executed:
#!/bin/bash
set -euo pipefail
# Locate the cluster endpoint source implementation and inspect its constructor.
fd -a "cluster" crates/health/src || true
rg -n "struct ClusterEndpointSource|impl ClusterEndpointSource|from_config\(" crates/health/src -S
# Read the implementation file once identified.
file="$(rg -l "struct ClusterEndpointSource|impl ClusterEndpointSource" crates/health/src -S | head -n 1)"
echo "--- file: $file ---"
sed -n '1,260p' "$file" | cat -nRepository: NVIDIA/infra-controller
Length of output: 12309
🏁 Script executed:
#!/bin/bash
set -euo pipefail
sed -n '260,420p' crates/health/src/endpoint/cluster.rs | cat -n
printf '\n--- from_config signature and body ---\n'
rg -n "pub async fn from_config|fetch_manager_credentials|extract_manager_devices|manager_rpc" crates/health/src/endpoint/cluster.rs -n -A120 -B20Repository: NVIDIA/infra-controller
Length of output: 18990
Cluster inventory fetch still blocks startup. fetch_manager_credentials already falls back on RPC failures, but fetch_from_manager(...).await? still propagates device-inventory and URL/RPC errors, so a transient Cluster Manager outage aborts build_endpoint_wiring and prevents otherwise healthy sources from starting. Consider making this source best-effort with logging plus skip/retry-backoff.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@crates/health/src/lib.rs` around lines 153 - 162, Make cluster endpoint
initialization best-effort in ClusterEndpointSource::from_config: handle
device-inventory and URL/RPC failures locally instead of propagating them
through build_endpoint_wiring, log the failure, and skip the cluster source (or
apply the existing retry/backoff mechanism) so other sources can start normally.
🔍 Container Scan Summary
Per-CVE detail lives in the per-service |
|
There are couple of coderabbit comments which are relevant. (most marked as Major). Two which required to be fixed are password debug and tls danger_accept_invalid_certs (this can be moved to config) |
|
@coderabbitai review |
✅ Action performedReview finished.
|
There was a problem hiding this comment.
Actionable comments posted: 2
♻️ Duplicate comments (1)
crates/health/src/endpoint/cluster.rs (1)
127-188: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick winCredential path still silently falls back on missing/failed fields.
On RPC failure (Line 145-152) or missing
bmcSettings.username/.password(Line 168-174, 176-185), the function silently substitutescfg.default_username/cfg.default_passwordand returns success. A previous review flagged this exact "credential path should avoid silent fallback on missing fields" concern; the device-inventory path has since been made fail-closed (Line 200-213, 271-277), but the credential path was left as a soft fallback. If the RPC contract drifts (very plausible, per the many "field path unconfirmed" comments in this file), operators could silently end up authenticating with stale/wrong default credentials against every BMC in the cluster without any hard failure signal, only a debug/warn log line.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/health/src/endpoint/cluster.rs` around lines 127 - 188, Update fetch_manager_credentials to fail closed instead of substituting cfg.default_username or cfg.default_password when manager_rpc fails or bmcSettings fields are missing or invalid. Propagate an explicit failure using the same error-handling pattern as the device-inventory path, and update callers as needed so credential retrieval cannot return success with partial or fallback credentials.
🧹 Nitpick comments (3)
crates/health/src/endpoint/cluster.rs (3)
190-426: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick winNo test coverage for the new parsing/construction logic.
extract_manager_devices,read_from_file, andbuild_endpointscontain substantial branching (multiple fallback field paths, fail-closed checks, IPv4-only filtering, rack normalization) with no accompanying tests in this file. Given several correctness bugs in this exact logic were already caught and fixed across prior review rounds (empty-rack handling, fail-closed behavior), table-driven tests would materially reduce regression risk here. As per coding guidelines, prefer "table-driven tests using groupedcarbide-test-supportscenarios (scenarios!/value_scenarios!) or explicit cases (check_cases/check_values) for parsers, validators, conversions, and similar logic."🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/health/src/endpoint/cluster.rs` around lines 190 - 426, Add table-driven tests for extract_manager_devices, read_from_file, and build_endpoints using the repository’s carbide-test-support scenario helpers or explicit case helpers. Cover top-level and wrapped response shapes, fallback hostname/BMC fields, missing or invalid data, empty rack normalization, file parsing and credential propagation, IPv4-only filtering, and client-construction failures while preserving fail-closed behavior.Source: Coding guidelines
349-370: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick winBlocking file I/O inside an async function.
read_from_fileusesstd::fs::read_to_string, and is called directly (not viaspawn_blocking) from the asyncload_endpoints(Line 306-311), which now re-reads on every discovery poll (a behavior fixed from a prior review round). This blocks the current Tokio worker thread for the duration of the read. Local inventory files are likely small, so impact is probably minor in practice, but it's an easy fix to avoid stalling other tasks on the same runtime thread during discovery polls.♻️ Suggested fix
-fn read_from_file(cfg: &ClusterEndpointSourceConfig) -> Result<Vec<ClusterNode>, HealthError> { - let contents = std::fs::read_to_string(&cfg.inventory_path).map_err(|e| { +async fn read_from_file(cfg: &ClusterEndpointSourceConfig) -> Result<Vec<ClusterNode>, HealthError> { + let contents = tokio::fs::read_to_string(&cfg.inventory_path).await.map_err(|e| { HealthError::GenericError(format!( "Failed to read cluster inventory {}: {e}", cfg.inventory_path.display() )) })?;🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/health/src/endpoint/cluster.rs` around lines 349 - 370, Update the async load_endpoints flow so the synchronous read_from_file operation runs through Tokio’s spawn_blocking rather than directly on the runtime worker thread. Preserve the existing per-poll re-read behavior and propagate read, join, and parsing errors through the current Result path.
65-84: 🎯 Functional Correctness | 🔵 Trivial | 🏗️ Heavy liftManager RPC contract is admittedly unconfirmed guesswork.
The comment block itself flags that call names (
getDevices,getPartition) and every field path used downstream (/bmcSettings,/interfaces/ipmi0/ip,bmcAddress,category/partition, etc.) are speculative pending live/apiverification. Shipping this behind a disabled-by-default flag mitigates blast radius, but enabling it against a real Cluster Manager could still silently misparse credentials/devices in ways the current fail-closed checks don't fully cover (e.g. wrong field picked up rather than missing). Recommend gating this path on integration testing against a real head node before enabling by default, and tracking the "confirm real API" TODOs as a follow-up issue rather than inline comments.Would you like me to open a tracking issue for confirming the live Cluster Manager JSON-RPC contract?
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/health/src/endpoint/cluster.rs` around lines 65 - 84, Keep the Cluster Manager integration disabled by default and do not enable it until its JSON-RPC contract has been verified against a live head node through integration testing. Replace the speculative inline API notes around MANAGER_DEVICE_CALL and MANAGER_PARTITION_CALL with a tracked follow-up reference for confirming call names and field extractors, and preserve fail-closed behavior until validation is complete.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@crates/health/src/endpoint/cluster.rs`:
- Around line 190-280: Remove raw external JSON from the debug logs in
extract_manager_devices and fetch_from_manager. Replace item = ?item and
response = ?raw with bounded structured diagnostics such as item count and
presence of expected hostname, BMC, and wrapper fields, while preserving the
existing missing-field and response-shape context.
- Around line 36-46: Remove the `Debug` derive from `FileInventory` and
`FileCredentials`, or implement a custom redacted `Debug` implementation that
never exposes `FileCredentials.password`. Preserve their `Deserialize` behavior
and ensure any debug formatting of these types cannot reveal BMC passwords.
---
Duplicate comments:
In `@crates/health/src/endpoint/cluster.rs`:
- Around line 127-188: Update fetch_manager_credentials to fail closed instead
of substituting cfg.default_username or cfg.default_password when manager_rpc
fails or bmcSettings fields are missing or invalid. Propagate an explicit
failure using the same error-handling pattern as the device-inventory path, and
update callers as needed so credential retrieval cannot return success with
partial or fallback credentials.
---
Nitpick comments:
In `@crates/health/src/endpoint/cluster.rs`:
- Around line 190-426: Add table-driven tests for extract_manager_devices,
read_from_file, and build_endpoints using the repository’s carbide-test-support
scenario helpers or explicit case helpers. Cover top-level and wrapped response
shapes, fallback hostname/BMC fields, missing or invalid data, empty rack
normalization, file parsing and credential propagation, IPv4-only filtering, and
client-construction failures while preserving fail-closed behavior.
- Around line 349-370: Update the async load_endpoints flow so the synchronous
read_from_file operation runs through Tokio’s spawn_blocking rather than
directly on the runtime worker thread. Preserve the existing per-poll re-read
behavior and propagate read, join, and parsing errors through the current Result
path.
- Around line 65-84: Keep the Cluster Manager integration disabled by default
and do not enable it until its JSON-RPC contract has been verified against a
live head node through integration testing. Replace the speculative inline API
notes around MANAGER_DEVICE_CALL and MANAGER_PARTITION_CALL with a tracked
follow-up reference for confirming call names and field extractors, and preserve
fail-closed behavior until validation is complete.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 32dceeb1-4f18-4b58-a2f4-d7faff198064
📒 Files selected for processing (4)
crates/health/src/config.rscrates/health/src/endpoint/cluster.rscrates/health/src/endpoint/mod.rscrates/health/src/lib.rs
🚧 Files skipped from review as they are similar to previous changes (2)
- crates/health/src/config.rs
- crates/health/src/lib.rs
Adds a new EndpointSource that ingests cluster rack inventory from either
a local JSON file or Cluster manager's JSON RPC API.
File path: reads a JSON inventory file containing
hostname, BMC IP, rack, and default credentials for each node.
Cluster Manager's JSON RPC path (placeholder, pending head-node IP confirmation):
- POST /json/ cmpart.getPartition("<partition>") → cluster BMC
username + password
- POST /json/ cmdevice.getDevices → per-node hostname + BMC IP
Both paths produce the same Vec<ClusterNode> fed into build_endpoints().
MAC addresses are derived deterministically from BMC IPv4 octets
(02:00:<o1>:<o2>:<o3>:<o4>) as internal cache keys only.
config: cluster source is Disabled by default; enable with:
[endpoint_sources.cluster]
inventory_path = "/path/to/inventory.json"
# or
cluster_manager_url = "https://<head-node>:8081"
Signed-off-by: Dasa Chandramouli <dchandramoul@nvidia.com>
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
crates/health/src/lib.rs (1)
124-127: 🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy liftTLS certificate validation still disabled unconditionally — now also covers cluster BMC connections.
accept_invalid_certs(true)is hardcoded for the sharedreqwestclient, which the new cluster wiring passes straight intoClusterEndpointSource::from_config(Line 158). Per the cluster.rs snippets, this same client buildsBmcClientfor every cluster-discovered BMC, so cluster nodes now inherit the same disabled-TLS-verification posture as the other sources — expanding the blast radius of an already-identified issue rather than fixing or scoping it. This was flagged in the PR's own review comments summary ("Avoidtls danger_accept_invalid_certsor move this setting to configuration") and remains unresolved in this diff.Consider gating this behind an explicit config flag (default
false) rather than baking it in unconditionally.Also applies to: 148-164
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/health/src/lib.rs` around lines 124 - 127, Update build_endpoint_wiring and the shared ReqwestClientParams construction so accept_invalid_certs is controlled by an explicit configuration flag that defaults to false, rather than always enabled. Ensure the configured value is used for both regular endpoint sources and ClusterEndpointSource::from_config connections, preserving certificate validation unless the flag is intentionally enabled.
♻️ Duplicate comments (1)
crates/health/src/lib.rs (1)
155-163: 🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy liftCluster fetch failures still abort the whole composite discovery cycle (unresolved from prior review).
ClusterEndpointSource::load_endpointsstill doesfetch_from_manager(...).await?, andCompositeEndpointSource::fetch_bmc_hostspropagates viasrc.fetch_bmc_hosts().await?;for each source. A transient Cluster Manager outage therefore fails endpoint discovery for all sources on that cycle, not just cluster nodes — the same concern already raised on a prior commit of this function.Consider making the cluster source best-effort (log + skip/backoff on failure) so other healthy sources keep discovering endpoints.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@crates/health/src/lib.rs` around lines 155 - 163, Make cluster endpoint discovery best-effort by handling errors from ClusterEndpointSource::load_endpoints instead of propagating them through CompositeEndpointSource::fetch_bmc_hosts. Log the fetch failure and skip or apply the existing backoff behavior, allowing discovery to continue with other configured sources while preserving successful cluster results.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@crates/health/src/lib.rs`:
- Around line 124-127: Update build_endpoint_wiring and the shared
ReqwestClientParams construction so accept_invalid_certs is controlled by an
explicit configuration flag that defaults to false, rather than always enabled.
Ensure the configured value is used for both regular endpoint sources and
ClusterEndpointSource::from_config connections, preserving certificate
validation unless the flag is intentionally enabled.
---
Duplicate comments:
In `@crates/health/src/lib.rs`:
- Around line 155-163: Make cluster endpoint discovery best-effort by handling
errors from ClusterEndpointSource::load_endpoints instead of propagating them
through CompositeEndpointSource::fetch_bmc_hosts. Log the fetch failure and skip
or apply the existing backoff behavior, allowing discovery to continue with
other configured sources while preserving successful cluster results.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Enterprise
Run ID: 17a2ce48-5c30-4c5d-83b2-7972664a1f2f
📒 Files selected for processing (4)
crates/health/src/config.rscrates/health/src/endpoint/cluster.rscrates/health/src/endpoint/mod.rscrates/health/src/lib.rs
🚧 Files skipped from review as they are similar to previous changes (2)
- crates/health/src/config.rs
- crates/health/src/endpoint/cluster.rs
Adds a new EndpointSource that ingests cluster rack inventory from either a local JSON file or Cluster manager's JSON RPC API.
File path: reads a JSON inventory file containing
hostname, BMC IP, rack, and default credentials for each node.
Cluster Manager's JSON RPC path (placeholder, pending head-node IP confirmation):
Both paths produce the same Vec fed into build_endpoints(). MAC addresses are derived deterministically from BMC IPv4 octets (02:00::::) as internal cache keys only.
config: cluster source is Disabled by default; enable with:
[endpoint_sources.cluster]
inventory_path = "/path/to/inventory.json"
or
cluster_manager_url = "https://:8081"
Related issues
Fixes #3358
Type of Change
Breaking Changes
Testing
Manual testing
Manual testing performed against a single BMC endpoint using the file-based
inventory path (
inventory_path). Verified that:ClusterEndpointSourcecorrectly loads rack/node hierarchy from JSON inventorydefault_credentials[endpoint_sources.cluster]is absent from configAdditional Notes