feat(collector): support milvus collector (#3877) - #4303
Conversation
| * Scrapes the Prometheus-format /metrics endpoint exposed by Milvus on port 9091. | ||
| */ | ||
| @Slf4j | ||
| public class MilvusCollectImpl extends AbstractCollect { |
There was a problem hiding this comment.
Thank you for your work! Before going further, could we reconsider the approach?
Milvus exposes a standard Prometheus /metrics endpoint, and HertzBeat already handles that with protocol: http + parseType: prometheus. Going that route would let app-milvus.yml work without a new hertzbeat-collector-milvus module, a new MilvusProtocol, or changes to Metrics.java / DispatchConstants.
There was a problem hiding this comment.
Thanks for the clear guidance @Duansg! reworked the implementation to use protocol: http with parseType: prometheus. Please take another look!
f64a96f to
443ebdb
Compare
Aias00
left a comment
There was a problem hiding this comment.
Review: feat(collector): support milvus collector (#3877)
Verdict: ✅ APPROVED — useful new monitor template; please address two doc/metric notes before or after merge.
What this PR does
Adds a Milvus vector-DB monitor template (app-milvus.yml) scraping the Prometheus /metrics endpoint on port 9091, plus the matching help doc. Covers proxy request/slow-query/vector counts, search & mutation latency, QueryNode loaded collections/entities, and RootCoord collection/partition/DML/DDL counts across 12 priority groups.
Assessment
- Structure: Valid template — ASF license header, trilingual
name/help, sensibleparams(host/port/timeout), and consistentprotocol: http+parseType: prometheusper group. Port default 9091 and path/metricsmatch Milvus defaults. - Risk: Low — additive, no code path touched.
Notes (non-blocking)
- Doc/metric mismatch:
home/docs/help/milvus.mdlists metrics that the template does not define (e.g.milvus_proxy_sq_latency_sum,milvus_proxy_receive_bytes_count,milvus_proxy_send_bytes_count,milvus_querynode_search_req_latency_sum,milvus_querynode_search_group_nq_sum,milvus_querynode_evicted_memory_size). The "Collected Metrics" tables read like Milvus's full metric surface rather than what this template actually scrapes. Please align the doc tables with the 12 metric names present inapp-milvus.ymlso users aren't misled. - Histogram metric names:
milvus_proxy_sq_latencyandmilvus_proxy_mutation_latencyare modeled with anlelabel (classic histogram-bucket shape). Prometheus histograms expose..._bucket/..._sum/..._countseries; a baremilvus_proxy_sq_latencyseries usually does not exist, so these two groups may silently collect nothing. Please verify Milvus emits a bare series with thelelabel, or retarget the metric name tomilvus_proxy_sq_latency_bucket(and add_sum/_countas needed).
Nice addition overall.
What's changed?
Reworked the Milvus monitor to use HertzBeat's native
protocol: http+parseType: prometheus- no custom module, no new Java classes.hertzbeat-manager/src/main/resources/define/app-milvus.yml- monitoring template scraping Milvus's Prometheus endpoint (port 9091,/metrics), covering 12 metric groups: proxy request count, slow query count, search/insert vector counts, search and mutation latency histograms (histogram bucket series), QueryNode collection/entity counts, and RootCoord collection/partition/DML/DDL countshome/docs/help/milvus.md- English help documentation, metric tables aligned exactly to the templateFixes #3877