Sidecar metrics collection over plugin gRPC has no deadline — one stuck operation hangs the whole instance /metrics

[tenhishadow]

Environment

CloudNativePG 1.30.0, plugin-barman-cloud v0.14.0, PostgreSQL 17/18 clusters, VersityGW (posix) S3 endpoint.

What happens

The instance sidecar collects plugin metrics over the plugin gRPC with no deadline on the collect path. Consequence: if any plugin operation gets stuck (in my case a retention/catalog-maintenance run that never completes against a posix-backend S3 gateway — filed separately), the next scrape blocks on the collector and the entire /metrics endpoint of that instance hangs indefinitely. Prometheus marks the target down and every metric of that instance disappears, not just the plugin's own collectors.

Observed in production: the primary instance's exporter dark for 6.7 hours while the database itself was healthy — WAL-archiver and backup-staleness alerting was blind precisely on the instance where it matters.

Expected

• A per-collect deadline (a few seconds) on the plugin metrics gRPC call.
• On timeout/error: return the rest of the metrics plus an error counter (e.g. ..._collector_errors_total), instead of hanging the whole endpoint. A misbehaving plugin should degrade its own metrics, never the instance's.

Reproduction sketch

1. ObjectStore against VersityGW (posix backend) with any retentionPolicy set.
2. Wait for a retention run to start (~30 min cadence) — it never completes against this backend.
3. Scrape the instance sidecar's metrics port: the request hangs until the client gives up; up goes to 0 for the instance.

Companion issue describes the retention hang itself; this one is about the missing deadline that turns any such hang into a full metrics outage.

[tenhishadow]

Companion issue describing the stuck retention run that triggers this: #1044

[dosubot]

Hi @tenhishadow! I'm Dosu and I'm helping the CloudNativePG team.

I looked into both issues and can confirm the code path you've identified. Here's a summary of what's happening:

The metrics collect path has no deadline. In CloudNativePG's pluginCollector.Collect, the plugin loading step has a 5-second timeout [1], but the actual CollectMetrics gRPC call passes the context straight through with no explicit deadline [2]. The only timeout in play is whatever Prometheus has configured for its scrape (default 10s), and even that only cancels the HTTP handler — it doesn't interrupt the in-flight gRPC call.

On the plugin side, the Collect RPC fetches the ObjectStore CR via a Kubernetes API call using the same context [3]. This shouldn't be the bottleneck in your case though — the hang you're seeing in #1044 is in CatalogMaintenanceRunnable.cycle(), which calls barmanCommand.DeleteBackupsByPolicy with no per-cycle timeout [4]. While the retention cycle is stuck, the metrics gRPC server itself still serves — but the connection between the two (both running as runnables in the same controller-runtime manager [5]) means a stuck retention process could indirectly starve resources or block shared state.

The fix would likely need to touch two layers:

1. CloudNativePG side — add a per-collect deadline (e.g. a few seconds) to the CollectMetrics gRPC context in pluginCollector.Collect [2], and surface a collector-error counter on timeout instead of blocking.
2. Plugin side — add a timeout to the cycle() call in CatalogMaintenanceRunnable so that a stuck barman operation doesn't run forever (Retention run never completes against a POSIX-backend S3 gateway (VersityGW) and stalls the sidecar metrics endpoint #1044's domain) [4].

Both issues are valid and complementary — #1044 is the root cause (retention hang), and this issue (#1045) is about blast-radius containment (a misbehaving plugin shouldn't take down the entire /metrics endpoint).

To reply, just mention @dosu.

---

Share context across your team and agents. Try Dosu.

[tenhishadow]

Fair point from the analysis above: the missing-deadline half of this lives in CloudNativePG core (pluginCollector.Collect in cloudnative-pg/cloudnative-pg), not in this plugin — only the cycle() timeout half is in this repo. If the maintainers prefer, I'm happy to re-file the core part as a separate issue against cloudnative-pg/cloudnative-pg and keep this one scoped to the plugin side; just say the word.