Skip to content

feat(health): add ClusterEndpointSource for BCM/file fleet inventory#3359

Merged
ajf merged 1 commit into
NVIDIA:mainfrom
cdraman:HAAS-140
Jul 14, 2026
Merged

feat(health): add ClusterEndpointSource for BCM/file fleet inventory#3359
ajf merged 1 commit into
NVIDIA:mainfrom
cdraman:HAAS-140

Conversation

@cdraman

@cdraman cdraman commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Adds a new EndpointSource that ingests cluster rack inventory from either a local JSON file or Cluster manager's JSON RPC API.

File path: reads a JSON inventory file containing
hostname, BMC IP, rack, and default credentials for each node.

Cluster Manager's JSON RPC path (placeholder, pending head-node IP confirmation):

  • POST /json/ cmpart.getPartition("") → cluster BMC username + password
  • POST /json/ cmdevice.getDevices → per-node hostname + BMC IP

Both paths produce the same Vec fed into build_endpoints(). MAC addresses are derived deterministically from BMC IPv4 octets (02:00::::) as internal cache keys only.

config: cluster source is Disabled by default; enable with:
[endpoint_sources.cluster]
inventory_path = "/path/to/inventory.json"
or
cluster_manager_url = "https://:8081"

Related issues

Fixes #3358

Type of Change

  • Add - New feature or capability
  • Change - Changes in existing functionality
  • Fix - Bug fixes
  • Remove - Removed features or deprecated functionality
  • Internal - Internal changes (refactoring, tests, docs, etc.)

Breaking Changes

  • This PR contains breaking changes

Testing

  • Unit tests added/updated
  • Integration tests added/updated
  • Manual testing performed
  • No testing required (docs, internal refactor, etc.)

Manual testing

Manual testing performed against a single BMC endpoint using the file-based
inventory path (inventory_path). Verified that:

  • ClusterEndpointSource correctly loads rack/node hierarchy from JSON inventory
  • BMC credentials are applied per-node from default_credentials
  • Endpoints are discovered and passed into the collection pipeline
  • Redfish log events are successfully collected and forwarded over OTLP
  • Invalid or missing BMC IPs are skipped with a warning log
  • Source is correctly disabled when [endpoint_sources.cluster] is absent from config

Additional Notes

@cdraman cdraman requested a review from a team as a code owner July 10, 2026 06:45
@coderabbitai

coderabbitai Bot commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Review Change Stack

Summary by CodeRabbit

  • New Features
    • Added a new cluster-based BMC endpoint discovery source.
    • Cluster inventory can be loaded from a local file or fetched via a cluster-manager JSON-RPC service.
    • Supports configurable partitioning, default credentials, optional password, optional port, and rack mapping.
    • Cluster discovery is disabled by default and can be enabled via configuration.
    • Configuration validation now enforces that exactly one inventory source is provided when the cluster option is enabled.

Walkthrough

Adds a configurable ClusterEndpointSource that discovers BMC targets from local inventory files or cluster-manager JSON-RPC, constructs endpoint clients, and integrates the source into health-service wiring.

Changes

Cluster endpoint discovery

Layer / File(s) Summary
Cluster configuration surface
crates/health/src/config.rs, crates/health/src/endpoint/mod.rs
Adds disabled-by-default cluster configuration with inventory paths, cluster-manager access, credentials, partitions, ports, validation, and public source exposure.
Inventory discovery and normalization
crates/health/src/endpoint/cluster.rs
Loads local inventory or cluster-manager data, resolves credentials, extracts valid device fields, and normalizes both inputs into cluster nodes.
Endpoint construction and service wiring
crates/health/src/endpoint/cluster.rs, crates/health/src/lib.rs
Builds BMC endpoints from normalized nodes and adds the enabled cluster source to health-service wiring.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant HealthService
  participant ClusterEndpointSource
  participant ClusterManager
  participant BmcClient
  HealthService->>ClusterEndpointSource: Construct enabled source
  ClusterEndpointSource->>ClusterManager: Request credentials and devices
  ClusterManager-->>ClusterEndpointSource: Return inventory data
  ClusterEndpointSource->>BmcClient: Build BMC clients
  ClusterEndpointSource-->>HealthService: Provide endpoint source
Loading
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly names the new ClusterEndpointSource and the fleet inventory use case.
Description check ✅ Passed The description matches the implemented cluster/file inventory feature and its endpoint wiring.
Linked Issues check ✅ Passed The changes implement the requested file-based and Cluster Manager-based inventory ingestion paths for #3358.
Out of Scope Changes check ✅ Passed The changes stay focused on cluster inventory discovery, config, and endpoint wiring with no obvious unrelated additions.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Comment @coderabbitai help to get the list of available commands.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 8

🧹 Nitpick comments (1)
crates/health/src/endpoint/cluster.rs (1)

1-414: 📐 Maintainability & Code Quality | 🔵 Trivial | 🏗️ Heavy lift

No test coverage for the new parsing/extraction logic.

This entire new file — extract_manager_devices, fetch_manager_credentials, read_from_file, and build_endpoints — ships without tests, despite being exactly the kind of parser/normalization logic the repo's testing conventions target. sources.rs and model.rs (in the same crate) both carry table-driven #[test] coverage for comparable logic.

Consider adding value_scenarios!/check_values tables for extract_manager_devices (varying response shapes: top-level array, result/data/items/devices wrappers, missing hostname/IP, invalid IP) and for read_from_file (valid inventory, malformed JSON, missing file). As per path instructions, "prefer table-driven scenarios ... especially for parsers, validators, and conversions."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/health/src/endpoint/cluster.rs` around lines 1 - 414, Add table-driven
tests following the crate’s existing value_scenarios!/check_values conventions
for extract_manager_devices, covering top-level and result/data/items/devices
wrappers, missing fields, and invalid IPs; add read_from_file cases for valid
inventory, malformed JSON, and missing files. Use temporary inventory paths
where needed and assert parsed nodes or expected errors, without changing
production behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/health/src/config.rs`:
- Around line 102-134: ClusterEndpointSourceConfig’s derived Debug
implementation exposes default_password in logs. Replace the derived Debug with
a manual implementation that redacts or omits default_password, following
StaticBmcEndpoint’s custom Debug pattern, while preserving the existing
serialization behavior unless separately required.
- Around line 104-134: Add ClusterEndpointSourceConfig::validate() to require
either a non-empty inventory_path or a configured cluster_manager_url, returning
a clear configuration error when both are absent; then invoke this validation
from Config::validate() whenever the cluster endpoint source is enabled.

In `@crates/health/src/endpoint/cluster.rs`:
- Around line 236-242: Preserve the absence of a rack identifier in the
cluster-manager parsing and endpoint construction. Update the logic producing
`rack` near the category/partition lookup to return an optional value rather
than defaulting to an empty string, and update `build_endpoints` to pass `None`
when no non-empty rack exists instead of unconditionally constructing
`Some(RackId::new(&node.rack))`; retain `Some(RackId)` only for a valid rack so
`BmcEndpoint::hash_key` falls back to the endpoint MAC.
- Around line 128-189: Remove the plaintext response logging from
fetch_manager_credentials by deleting the tracing::debug! call that emits raw.
If response diagnostics are required, create a redacted representation that
removes or masks bmcSettings.password and any equivalent credential fields
before logging it, while preserving structured logfmt fields.
- Around line 274-295: Refresh the cluster inventory during every discovery pass
instead of caching it only in ClusterEndpointSource::from_config. Update
ClusterEndpointSource and fetch_bmc_hosts to re-read via fetch_from_manager or
read_from_file, rebuild endpoints with the configured port, clients, proxy, and
cache size, and return the refreshed set so additions, removals, and IP changes
are observed without restarting.
- Around line 191-270: Make Cluster Manager parsing fail closed: update
fetch_manager_credentials to reject missing or unrecognized credential fields
instead of silently falling back, and update extract_manager_devices to return
or propagate a countable error when a non-empty getDevices response has no
recognized shape or produces zero valid nodes. Preserve explicit handling for
valid empty arrays, and ensure callers surface the parsing failure rather than
treating it as a successful empty inventory.
- Around line 88-93: The build_http function unconditionally disables TLS
certificate validation for the cluster-manager credential client. Remove
danger_accept_invalid_certs(true) and require normal trusted certificate-chain
validation, or gate the bypass behind an explicit opt-in configuration flag;
also avoid logging raw partition responses in the credential-fetching path.

In `@crates/health/src/lib.rs`:
- Around line 153-162: Make cluster endpoint initialization best-effort in
ClusterEndpointSource::from_config: handle device-inventory and URL/RPC failures
locally instead of propagating them through build_endpoint_wiring, log the
failure, and skip the cluster source (or apply the existing retry/backoff
mechanism) so other sources can start normally.

---

Nitpick comments:
In `@crates/health/src/endpoint/cluster.rs`:
- Around line 1-414: Add table-driven tests following the crate’s existing
value_scenarios!/check_values conventions for extract_manager_devices, covering
top-level and result/data/items/devices wrappers, missing fields, and invalid
IPs; add read_from_file cases for valid inventory, malformed JSON, and missing
files. Use temporary inventory paths where needed and assert parsed nodes or
expected errors, without changing production behavior.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 8600f962-7b95-4e00-bec6-6a463bc9bd9e

📥 Commits

Reviewing files that changed from the base of the PR and between 2fa5b3b and bc2e441.

📒 Files selected for processing (4)
  • crates/health/src/config.rs
  • crates/health/src/endpoint/cluster.rs
  • crates/health/src/endpoint/mod.rs
  • crates/health/src/lib.rs

Comment thread crates/health/src/config.rs Outdated
Comment thread crates/health/src/config.rs
Comment thread crates/health/src/endpoint/cluster.rs
Comment thread crates/health/src/endpoint/cluster.rs
Comment thread crates/health/src/endpoint/cluster.rs Outdated
Comment thread crates/health/src/endpoint/cluster.rs
Comment thread crates/health/src/lib.rs
Comment on lines +153 to +162
if let Configurable::Enabled(ref source_cfg) = config.endpoint_sources.cluster {
let cluster_source = ClusterEndpointSource::from_config(
source_cfg,
&reqwest,
config.bmc_proxy_url.as_ref(),
config.cache_size,
)
.await?;
sources.push(Arc::new(cluster_source));
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Map the target file and nearby functions.
ast-grep outline crates/health/src/lib.rs --view expanded || true

# Read the relevant section with line numbers.
sed -n '120,230p' crates/health/src/lib.rs | cat -n

# Find ClusterEndpointSource and run_service references in the repo.
rg -n "ClusterEndpointSource::from_config|build_endpoint_wiring|run_service|static_bmc_endpoints|carbide_api|ApiClientWrapper::new|ApiEndpointSource::new" crates/health -S

Repository: NVIDIA/infra-controller

Length of output: 14211


🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Inspect the health crate structure and the relevant function bodies.
ast-grep outline crates/health/src/lib.rs --view expanded

printf '\n--- relevant slice ---\n'
sed -n '140,340p' crates/health/src/lib.rs | cat -n

printf '\n--- call sites ---\n'
rg -n "ClusterEndpointSource::from_config|build_endpoint_wiring|run_service|static_bmc_endpoints|ApiClientWrapper::new|ApiEndpointSource::new" crates/ -S

Repository: NVIDIA/infra-controller

Length of output: 15947


🏁 Script executed:

#!/bin/bash
set -euo pipefail

# Locate the cluster endpoint source implementation and inspect its constructor.
fd -a "cluster" crates/health/src || true
rg -n "struct ClusterEndpointSource|impl ClusterEndpointSource|from_config\(" crates/health/src -S

# Read the implementation file once identified.
file="$(rg -l "struct ClusterEndpointSource|impl ClusterEndpointSource" crates/health/src -S | head -n 1)"
echo "--- file: $file ---"
sed -n '1,260p' "$file" | cat -n

Repository: NVIDIA/infra-controller

Length of output: 12309


🏁 Script executed:

#!/bin/bash
set -euo pipefail

sed -n '260,420p' crates/health/src/endpoint/cluster.rs | cat -n

printf '\n--- from_config signature and body ---\n'
rg -n "pub async fn from_config|fetch_manager_credentials|extract_manager_devices|manager_rpc" crates/health/src/endpoint/cluster.rs -n -A120 -B20

Repository: NVIDIA/infra-controller

Length of output: 18990


Cluster inventory fetch still blocks startup. fetch_manager_credentials already falls back on RPC failures, but fetch_from_manager(...).await? still propagates device-inventory and URL/RPC errors, so a transient Cluster Manager outage aborts build_endpoint_wiring and prevents otherwise healthy sources from starting. Consider making this source best-effort with logging plus skip/retry-backoff.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/health/src/lib.rs` around lines 153 - 162, Make cluster endpoint
initialization best-effort in ClusterEndpointSource::from_config: handle
device-inventory and URL/RPC failures locally instead of propagating them
through build_endpoint_wiring, log the failure, and skip the cluster source (or
apply the existing retry/backoff mechanism) so other sources can start normally.

@github-actions

github-actions Bot commented Jul 10, 2026

Copy link
Copy Markdown

🔍 Container Scan Summary

Service Total Critical High Medium Low Other
boot-artifacts-aarch64 3 0 0 3 0 0
boot-artifacts-x86_64 3 0 0 3 0 0
forge-admin-cli-x86_64 255 13 30 79 7 126
machine-validation-runner 800 40 234 292 36 198
machine_validation 800 40 234 292 36 198
machine_validation-aarch64 800 40 234 292 36 198
nvmetal-carbide 800 40 234 292 36 198
TOTAL 3461 173 966 1253 151 918

Per-CVE detail lives in the per-service grype-* artifacts (JSON + SARIF). Severity counts only — no CVE IDs published here.

@yoks

yoks commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

There are couple of coderabbit comments which are relevant. (most marked as Major).

Two which required to be fixed are password debug and tls danger_accept_invalid_certs (this can be moved to config)

@cdraman

cdraman commented Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

@coderabbitai review

@coderabbitai

coderabbitai Bot commented Jul 13, 2026

Copy link
Copy Markdown
Contributor
✅ Action performed

Review finished.

Note: CodeRabbit is an incremental review system and does not re-review already reviewed commits. This command is applicable only when automatic reviews are paused.

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

♻️ Duplicate comments (1)
crates/health/src/endpoint/cluster.rs (1)

127-188: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick win

Credential path still silently falls back on missing/failed fields.

On RPC failure (Line 145-152) or missing bmcSettings.username/.password (Line 168-174, 176-185), the function silently substitutes cfg.default_username/cfg.default_password and returns success. A previous review flagged this exact "credential path should avoid silent fallback on missing fields" concern; the device-inventory path has since been made fail-closed (Line 200-213, 271-277), but the credential path was left as a soft fallback. If the RPC contract drifts (very plausible, per the many "field path unconfirmed" comments in this file), operators could silently end up authenticating with stale/wrong default credentials against every BMC in the cluster without any hard failure signal, only a debug/warn log line.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/health/src/endpoint/cluster.rs` around lines 127 - 188, Update
fetch_manager_credentials to fail closed instead of substituting
cfg.default_username or cfg.default_password when manager_rpc fails or
bmcSettings fields are missing or invalid. Propagate an explicit failure using
the same error-handling pattern as the device-inventory path, and update callers
as needed so credential retrieval cannot return success with partial or fallback
credentials.
🧹 Nitpick comments (3)
crates/health/src/endpoint/cluster.rs (3)

190-426: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

No test coverage for the new parsing/construction logic.

extract_manager_devices, read_from_file, and build_endpoints contain substantial branching (multiple fallback field paths, fail-closed checks, IPv4-only filtering, rack normalization) with no accompanying tests in this file. Given several correctness bugs in this exact logic were already caught and fixed across prior review rounds (empty-rack handling, fail-closed behavior), table-driven tests would materially reduce regression risk here. As per coding guidelines, prefer "table-driven tests using grouped carbide-test-support scenarios (scenarios!/value_scenarios!) or explicit cases (check_cases/check_values) for parsers, validators, conversions, and similar logic."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/health/src/endpoint/cluster.rs` around lines 190 - 426, Add
table-driven tests for extract_manager_devices, read_from_file, and
build_endpoints using the repository’s carbide-test-support scenario helpers or
explicit case helpers. Cover top-level and wrapped response shapes, fallback
hostname/BMC fields, missing or invalid data, empty rack normalization, file
parsing and credential propagation, IPv4-only filtering, and client-construction
failures while preserving fail-closed behavior.

Source: Coding guidelines


349-370: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Blocking file I/O inside an async function.

read_from_file uses std::fs::read_to_string, and is called directly (not via spawn_blocking) from the async load_endpoints (Line 306-311), which now re-reads on every discovery poll (a behavior fixed from a prior review round). This blocks the current Tokio worker thread for the duration of the read. Local inventory files are likely small, so impact is probably minor in practice, but it's an easy fix to avoid stalling other tasks on the same runtime thread during discovery polls.

♻️ Suggested fix
-fn read_from_file(cfg: &ClusterEndpointSourceConfig) -> Result<Vec<ClusterNode>, HealthError> {
-    let contents = std::fs::read_to_string(&cfg.inventory_path).map_err(|e| {
+async fn read_from_file(cfg: &ClusterEndpointSourceConfig) -> Result<Vec<ClusterNode>, HealthError> {
+    let contents = tokio::fs::read_to_string(&cfg.inventory_path).await.map_err(|e| {
         HealthError::GenericError(format!(
             "Failed to read cluster inventory {}: {e}",
             cfg.inventory_path.display()
         ))
     })?;
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/health/src/endpoint/cluster.rs` around lines 349 - 370, Update the
async load_endpoints flow so the synchronous read_from_file operation runs
through Tokio’s spawn_blocking rather than directly on the runtime worker
thread. Preserve the existing per-poll re-read behavior and propagate read,
join, and parsing errors through the current Result path.

65-84: 🎯 Functional Correctness | 🔵 Trivial | 🏗️ Heavy lift

Manager RPC contract is admittedly unconfirmed guesswork.

The comment block itself flags that call names (getDevices, getPartition) and every field path used downstream (/bmcSettings, /interfaces/ipmi0/ip, bmcAddress, category/partition, etc.) are speculative pending live /api verification. Shipping this behind a disabled-by-default flag mitigates blast radius, but enabling it against a real Cluster Manager could still silently misparse credentials/devices in ways the current fail-closed checks don't fully cover (e.g. wrong field picked up rather than missing). Recommend gating this path on integration testing against a real head node before enabling by default, and tracking the "confirm real API" TODOs as a follow-up issue rather than inline comments.

Would you like me to open a tracking issue for confirming the live Cluster Manager JSON-RPC contract?

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/health/src/endpoint/cluster.rs` around lines 65 - 84, Keep the Cluster
Manager integration disabled by default and do not enable it until its JSON-RPC
contract has been verified against a live head node through integration testing.
Replace the speculative inline API notes around MANAGER_DEVICE_CALL and
MANAGER_PARTITION_CALL with a tracked follow-up reference for confirming call
names and field extractors, and preserve fail-closed behavior until validation
is complete.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@crates/health/src/endpoint/cluster.rs`:
- Around line 190-280: Remove raw external JSON from the debug logs in
extract_manager_devices and fetch_from_manager. Replace item = ?item and
response = ?raw with bounded structured diagnostics such as item count and
presence of expected hostname, BMC, and wrapper fields, while preserving the
existing missing-field and response-shape context.
- Around line 36-46: Remove the `Debug` derive from `FileInventory` and
`FileCredentials`, or implement a custom redacted `Debug` implementation that
never exposes `FileCredentials.password`. Preserve their `Deserialize` behavior
and ensure any debug formatting of these types cannot reveal BMC passwords.

---

Duplicate comments:
In `@crates/health/src/endpoint/cluster.rs`:
- Around line 127-188: Update fetch_manager_credentials to fail closed instead
of substituting cfg.default_username or cfg.default_password when manager_rpc
fails or bmcSettings fields are missing or invalid. Propagate an explicit
failure using the same error-handling pattern as the device-inventory path, and
update callers as needed so credential retrieval cannot return success with
partial or fallback credentials.

---

Nitpick comments:
In `@crates/health/src/endpoint/cluster.rs`:
- Around line 190-426: Add table-driven tests for extract_manager_devices,
read_from_file, and build_endpoints using the repository’s carbide-test-support
scenario helpers or explicit case helpers. Cover top-level and wrapped response
shapes, fallback hostname/BMC fields, missing or invalid data, empty rack
normalization, file parsing and credential propagation, IPv4-only filtering, and
client-construction failures while preserving fail-closed behavior.
- Around line 349-370: Update the async load_endpoints flow so the synchronous
read_from_file operation runs through Tokio’s spawn_blocking rather than
directly on the runtime worker thread. Preserve the existing per-poll re-read
behavior and propagate read, join, and parsing errors through the current Result
path.
- Around line 65-84: Keep the Cluster Manager integration disabled by default
and do not enable it until its JSON-RPC contract has been verified against a
live head node through integration testing. Replace the speculative inline API
notes around MANAGER_DEVICE_CALL and MANAGER_PARTITION_CALL with a tracked
follow-up reference for confirming call names and field extractors, and preserve
fail-closed behavior until validation is complete.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 32dceeb1-4f18-4b58-a2f4-d7faff198064

📥 Commits

Reviewing files that changed from the base of the PR and between bc2e441 and 6948e53.

📒 Files selected for processing (4)
  • crates/health/src/config.rs
  • crates/health/src/endpoint/cluster.rs
  • crates/health/src/endpoint/mod.rs
  • crates/health/src/lib.rs
🚧 Files skipped from review as they are similar to previous changes (2)
  • crates/health/src/config.rs
  • crates/health/src/lib.rs

Comment thread crates/health/src/endpoint/cluster.rs Outdated
Comment thread crates/health/src/endpoint/cluster.rs
Adds a new EndpointSource that ingests cluster rack inventory from either
a local JSON file or Cluster manager's JSON RPC API.

File path: reads a JSON inventory file containing
hostname, BMC IP, rack, and default credentials for each node.

Cluster Manager's JSON RPC path (placeholder, pending head-node IP confirmation):
- POST /json/ cmpart.getPartition("<partition>") → cluster BMC
  username + password
- POST /json/ cmdevice.getDevices → per-node hostname + BMC IP

Both paths produce the same Vec<ClusterNode> fed into build_endpoints().
MAC addresses are derived deterministically from BMC IPv4 octets
(02:00:<o1>:<o2>:<o3>:<o4>) as internal cache keys only.

config: cluster source is Disabled by default; enable with:
  [endpoint_sources.cluster]
  inventory_path = "/path/to/inventory.json"
  # or
  cluster_manager_url = "https://<head-node>:8081"

Signed-off-by: Dasa Chandramouli <dchandramoul@nvidia.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Caution

Some comments are outside the diff and can’t be posted inline due to platform limitations.

⚠️ Outside diff range comments (1)
crates/health/src/lib.rs (1)

124-127: 🔒 Security & Privacy | 🟠 Major | 🏗️ Heavy lift

TLS certificate validation still disabled unconditionally — now also covers cluster BMC connections.

accept_invalid_certs(true) is hardcoded for the shared reqwest client, which the new cluster wiring passes straight into ClusterEndpointSource::from_config (Line 158). Per the cluster.rs snippets, this same client builds BmcClient for every cluster-discovered BMC, so cluster nodes now inherit the same disabled-TLS-verification posture as the other sources — expanding the blast radius of an already-identified issue rather than fixing or scoping it. This was flagged in the PR's own review comments summary ("Avoid tls danger_accept_invalid_certs or move this setting to configuration") and remains unresolved in this diff.

Consider gating this behind an explicit config flag (default false) rather than baking it in unconditionally.

Also applies to: 148-164

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/health/src/lib.rs` around lines 124 - 127, Update
build_endpoint_wiring and the shared ReqwestClientParams construction so
accept_invalid_certs is controlled by an explicit configuration flag that
defaults to false, rather than always enabled. Ensure the configured value is
used for both regular endpoint sources and ClusterEndpointSource::from_config
connections, preserving certificate validation unless the flag is intentionally
enabled.
♻️ Duplicate comments (1)
crates/health/src/lib.rs (1)

155-163: 🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy lift

Cluster fetch failures still abort the whole composite discovery cycle (unresolved from prior review).

ClusterEndpointSource::load_endpoints still does fetch_from_manager(...).await?, and CompositeEndpointSource::fetch_bmc_hosts propagates via src.fetch_bmc_hosts().await?; for each source. A transient Cluster Manager outage therefore fails endpoint discovery for all sources on that cycle, not just cluster nodes — the same concern already raised on a prior commit of this function.

Consider making the cluster source best-effort (log + skip/backoff on failure) so other healthy sources keep discovering endpoints.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@crates/health/src/lib.rs` around lines 155 - 163, Make cluster endpoint
discovery best-effort by handling errors from
ClusterEndpointSource::load_endpoints instead of propagating them through
CompositeEndpointSource::fetch_bmc_hosts. Log the fetch failure and skip or
apply the existing backoff behavior, allowing discovery to continue with other
configured sources while preserving successful cluster results.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Outside diff comments:
In `@crates/health/src/lib.rs`:
- Around line 124-127: Update build_endpoint_wiring and the shared
ReqwestClientParams construction so accept_invalid_certs is controlled by an
explicit configuration flag that defaults to false, rather than always enabled.
Ensure the configured value is used for both regular endpoint sources and
ClusterEndpointSource::from_config connections, preserving certificate
validation unless the flag is intentionally enabled.

---

Duplicate comments:
In `@crates/health/src/lib.rs`:
- Around line 155-163: Make cluster endpoint discovery best-effort by handling
errors from ClusterEndpointSource::load_endpoints instead of propagating them
through CompositeEndpointSource::fetch_bmc_hosts. Log the fetch failure and skip
or apply the existing backoff behavior, allowing discovery to continue with
other configured sources while preserving successful cluster results.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Enterprise

Run ID: 17a2ce48-5c30-4c5d-83b2-7972664a1f2f

📥 Commits

Reviewing files that changed from the base of the PR and between 6948e53 and f7f3b50.

📒 Files selected for processing (4)
  • crates/health/src/config.rs
  • crates/health/src/endpoint/cluster.rs
  • crates/health/src/endpoint/mod.rs
  • crates/health/src/lib.rs
🚧 Files skipped from review as they are similar to previous changes (2)
  • crates/health/src/config.rs
  • crates/health/src/endpoint/cluster.rs

@ajf ajf merged commit 10dfa89 into NVIDIA:main Jul 14, 2026
59 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

feat(health): add ClusterEndpointSource for fleet inventory ingestion

3 participants