Skip to content

feat(collector): support milvus collector (#3877) - #4303

Open
paultanay wants to merge 10 commits into
apache:masterfrom
paultanay:feature/milvus-collector
Open

feat(collector): support milvus collector (#3877)#4303
paultanay wants to merge 10 commits into
apache:masterfrom
paultanay:feature/milvus-collector

Conversation

@paultanay

@paultanay paultanay commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

What's changed?

Reworked the Milvus monitor to use HertzBeat's native protocol: http + parseType: prometheus - no custom module, no new Java classes.

  • hertzbeat-manager/src/main/resources/define/app-milvus.yml - monitoring template scraping Milvus's Prometheus endpoint (port 9091, /metrics), covering 12 metric groups: proxy request count, slow query count, search/insert vector counts, search and mutation latency histograms (histogram bucket series), QueryNode collection/entity counts, and RootCoord collection/partition/DML/DDL counts
  • home/docs/help/milvus.md - English help documentation, metric tables aligned exactly to the template

Fixes #3877

@github-actions github-actions Bot added doc Improvements or additions to documentation monitoring-template collector backend labels Aug 8, 2026
@paultanay
paultanay marked this pull request as ready for review August 8, 2026 16:18
* Scrapes the Prometheus-format /metrics endpoint exposed by Milvus on port 9091.
*/
@Slf4j
public class MilvusCollectImpl extends AbstractCollect {

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you for your work! Before going further, could we reconsider the approach?

Milvus exposes a standard Prometheus /metrics endpoint, and HertzBeat already handles that with protocol: http + parseType: prometheus. Going that route would let app-milvus.yml work without a new hertzbeat-collector-milvus module, a new MilvusProtocol, or changes to Metrics.java / DispatchConstants.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the clear guidance @Duansg! reworked the implementation to use protocol: http with parseType: prometheus. Please take another look!

@Duansg
Duansg marked this pull request as draft August 9, 2026 15:42
@paultanay
paultanay force-pushed the feature/milvus-collector branch from f64a96f to 443ebdb Compare August 11, 2026 18:14
@paultanay
paultanay requested a review from Duansg August 11, 2026 18:16
Aias00
Aias00 previously approved these changes Aug 17, 2026

@Aias00 Aias00 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review: feat(collector): support milvus collector (#3877)

Verdict: ✅ APPROVED — useful new monitor template; please address two doc/metric notes before or after merge.

What this PR does

Adds a Milvus vector-DB monitor template (app-milvus.yml) scraping the Prometheus /metrics endpoint on port 9091, plus the matching help doc. Covers proxy request/slow-query/vector counts, search & mutation latency, QueryNode loaded collections/entities, and RootCoord collection/partition/DML/DDL counts across 12 priority groups.

Assessment

  • Structure: Valid template — ASF license header, trilingual name/help, sensible params (host/port/timeout), and consistent protocol: http + parseType: prometheus per group. Port default 9091 and path /metrics match Milvus defaults.
  • Risk: Low — additive, no code path touched.

Notes (non-blocking)

  1. Doc/metric mismatch: home/docs/help/milvus.md lists metrics that the template does not define (e.g. milvus_proxy_sq_latency_sum, milvus_proxy_receive_bytes_count, milvus_proxy_send_bytes_count, milvus_querynode_search_req_latency_sum, milvus_querynode_search_group_nq_sum, milvus_querynode_evicted_memory_size). The "Collected Metrics" tables read like Milvus's full metric surface rather than what this template actually scrapes. Please align the doc tables with the 12 metric names present in app-milvus.yml so users aren't misled.
  2. Histogram metric names: milvus_proxy_sq_latency and milvus_proxy_mutation_latency are modeled with an le label (classic histogram-bucket shape). Prometheus histograms expose ..._bucket / ..._sum / ..._count series; a bare milvus_proxy_sq_latency series usually does not exist, so these two groups may silently collect nothing. Please verify Milvus emits a bare series with the le label, or retarget the metric name to milvus_proxy_sq_latency_bucket (and add _sum/_count as needed).

Nice addition overall.

@paultanay
paultanay marked this pull request as ready for review August 17, 2026 21:17
@paultanay
paultanay requested a review from Aias00 August 17, 2026 21:17
@paultanay

Copy link
Copy Markdown
Contributor Author

addressed remaining suggestions
@Aias00 @Duansg could you please take a look! Thanks...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

backend doc Improvements or additions to documentation monitoring-template

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Feature] <collector> support milvus collector

3 participants