P3 - container-runtime/copilot-runtime-bazel-cache-memory-usage-vs-limits-dotcom on catalog_service:copilot-runtime-bazel-cache,kube_cluster:dotcom-3-gc-cus-02,kube_container_name:copilot-runtime-bazel-cache,kube_deployment:copilot-r... - 2026-10-18T14:11Z

Alert Details

Source: [P3] [Triggered] container-runtime/copilot-runtime-bazel-cache-memory-usage-vs-limits-dotcom on catalog_service:copilot-runtime-bazel-cache,kube_cluster:dotcom-3-gc-cus-02,kube_container_name:copilot-runtime-bazel-cache,kube_deployment:copilot-runtime-bazel-cache,kube_namespace:copilot-runtime-bazel-cache-production,site:gc-cus-02
Status: CRITICAL
Triggered at: 2026-10-18T14:11Z

This issue was created because the "create_github_issue_on_alert:true" tag was applied to the monitor. This issue was created by ISP.

%%%

Container copilot-runtime-bazel-cache in deployment copilot-runtime-bazel-cache (namespace copilot-runtime-bazel-cache-production, service copilot-runtime-bazel-cache) in cluster dotcom-3-gc-cus-02 — its busiest pod is using 95.906% of this container's memory limit and is at risk of being OOM-killed and restarted.

What this means: the busiest pod's memory usage for this container, relative to the container's own limit. A leak or spike in a single replica will trip this, and crossing the limit triggers an OOM kill.

What to do:

Open the Kubernetes dashboard for this container and check per-pod memory — identify the pod climbing toward its limit and whether it is a steady leak or a spike.
Investigate top memory consumers / a possible leak; raise this container's memory limit if the workload legitimately needs more.
If this is expected, right-size the memory request/limit; otherwise escalate to the service owner.

Dashboard: Kubernetes Workload Observability

Playbook: https://github.com/github/ops/blob/master/docs/playbooks/kubernetes/workload-alerts/memory-usage-vs-limits.md

NOTE: Fleet-wide platform-default check (priority 3). When this fires, a GitHub issue is opened against your service's Service Catalog owning team.

HOW TO OPT OUT

Pick ONE of the following exclusions:
- !catalog_service:<your-service-slug> — excludes all your workloads
- !kube_deployment:<your-deployment> — excludes one deployment
- !kube_namespace:<your-namespace> — excludes an entire namespace
- !kube_container_name:<container> — excludes one container, for example an injected sidecar
Open a PR in github/datadog-monitoring adding your chosen exclusion inside every {...} filter block in this monitor's query. The CPU monitors (usage-vs-limits and throttling) have four filter blocks and the memory monitor has three — the exclusion must appear in every one of them or the filter is inconsistent. The error monitors have one. Repeat on all six monitor families for a full opt-out.

Cluster-level opt-outs are not accepted — this is a per-service safety net.

Questions? #compute-platform-support

@webhook-isp-alert

avg(last_15m):max:kubernetes.memory.usage_pct{!catalog_service:github,!catalog_service:token-scanning-service,!catalog_service:pyroscope,!catalog_service:aqueduct,!kube_cluster_profile:test,!kube_cluster_stamp:test*,kube_cluster_profile:dotcom} by {kube_deployment,kube_namespace,kube_cluster,site,kube_container_name} * (avg:kubernetes_state.container.memory_limit{!catalog_service:github,!catalog_service:token-scanning-service,!catalog_service:pyroscope,!catalog_service:aqueduct,!kube_cluster_profile:test,!kube_cluster_stamp:test*,kube_cluster_profile:dotcom,catalog_service:*} by {kube_deployment,kube_namespace,catalog_service,kube_cluster,site,kube_container_name} / avg:kubernetes_state.container.memory_limit{!catalog_service:github,!catalog_service:token-scanning-service,!catalog_service:pyroscope,!catalog_service:aqueduct,!kube_cluster_profile:test,!kube_cluster_stamp:test*,kube_cluster_profile:dotcom,catalog_service:*} by {kube_deployment,kube_namespace,catalog_service,kube_cluster,site,kube_container_name}) * 100 > 95

The monitor was last triggered at Sun Oct 18 2026 14:11:10 UTC.

[Monitor Status] · [Edit Monitor] · [Related Logs]
%%%

On-Call

(none)

Alert Update: Triggered

Status: CRITICAL
Time: 2026-10-22T07:04Z

%%%

Container copilot-runtime-bazel-cache in deployment copilot-runtime-bazel-cache (namespace copilot-runtime-bazel-cache-production, service copilot-runtime-bazel-cache) in cluster dotcom-3-gc-cus-02 — its busiest pod is using 99.974% of this container's memory limit and is at risk of being OOM-killed and restarted.

What this means: the busiest pod's memory usage for this container, relative to the container's own limit. A leak or spike in a single replica will trip this, and crossing the limit triggers an OOM kill.

What to do:

Open the Kubernetes dashboard for this container and check per-pod memory — identify the pod climbing toward its limit and whether it is a steady leak or a spike.
Investigate top memory consumers / a possible leak; raise this container's memory limit if the workload legitimately needs more.
If this is expected, right-size the memory request/limit; otherwise escalate to the service owner.

Dashboard: Kubernetes Workload Observability

Playbook: https://github.com/github/ops/blob/master/docs/playbooks/kubernetes/workload-alerts/memory-usage-vs-limits.md

NOTE: Fleet-wide platform-default check (priority 3). When this fires, a GitHub issue is opened against your service's Service Catalog owning team. For github/github deployments, the issue goes to the deployment's owner in fleetwide-aliases.yaml, or to monolith platform if the deployment isn't listed.

HOW TO OPT OUT

Pick ONE of the following exclusions:
- !catalog_service:<your-service-slug> — excludes all your workloads (not for github/github: every monolith deployment shares catalog_service:github, so monolith teams must exclude their kube_deployment instead)
- !kube_deployment:<your-deployment> — excludes one deployment
- !kube_namespace:<your-namespace> — excludes an entire namespace
- !kube_container_name:<container> — excludes one container, for example an injected sidecar
Open a PR in github/datadog-monitoring adding your chosen exclusion inside every {...} filter block in this monitor's query. The CPU monitors (usage-vs-limits and throttling) have four filter blocks and the memory monitor has three — the exclusion must appear in every one of them or the filter is inconsistent. The error monitors have one. Repeat on all six monitor families for a full opt-out.

Cluster-level opt-outs are not accepted — this is a per-service safety net.

Questions? #compute-platform-support

@webhook-isp-alert

This notification has been sent because a downtime ended.

The monitor was last triggered at Sun Oct 18 2026 14:11:10 UTC.

[Monitor Status] · [Edit Monitor] · [Related Logs]
%%%

Alert Update: Recovered

Status: RECOVERED
Time: 2026-11-02T15:44Z

%%%

Recovered: memory usage for container copilot-runtime-bazel-cache in copilot-runtime-bazel-cache (namespace copilot-runtime-bazel-cache-production) in cluster dotcom-3-gc-cus-02 is back below the threshold.

What this means: the busiest pod's memory usage for this container, relative to the container's own limit. A leak or spike in a single replica will trip this, and crossing the limit triggers an OOM kill.

What to do:

Open the Kubernetes dashboard for this container and check per-pod memory — identify the pod climbing toward its limit and whether it is a steady leak or a spike.
Investigate top memory consumers / a possible leak; raise this container's memory limit if the workload legitimately needs more.
If this is expected, right-size the memory request/limit; otherwise escalate to the service owner.

Dashboard: Kubernetes Workload Observability

Playbook: https://github.com/github/ops/blob/master/docs/playbooks/kubernetes/workload-alerts/memory-usage-vs-limits.md

NOTE: Fleet-wide platform-default check (priority 3). When this fires, a GitHub issue is opened against your service's Service Catalog owning team. For github/github deployments, the issue goes to the deployment's owner in fleetwide-aliases.yaml, or to monolith platform if the deployment isn't listed.

HOW TO OPT OUT

Pick ONE of the following exclusions:
- !catalog_service:<your-service-slug> — excludes all your workloads (not for github/github: every monolith deployment shares catalog_service:github, so monolith teams must exclude their kube_deployment instead)
- !kube_deployment:<your-deployment> — excludes one deployment
- !kube_namespace:<your-namespace> — excludes an entire namespace
- !kube_container_name:<container> — excludes one container, for example an injected sidecar
Open a PR in github/datadog-monitoring adding your chosen exclusion inside every {...} filter block in this monitor's query. The CPU monitors (usage-vs-limits and throttling) have four filter blocks and the memory monitor has three — the exclusion must appear in every one of them or the filter is inconsistent. The error monitors have one. Repeat on all six monitor families for a full opt-out.

Cluster-level opt-outs are not accepted — this is a per-service safety net.

Questions? #compute-platform-support

@webhook-isp-alert

avg(last_15m):max:kubernetes.memory.usage_pct{!kube_namespace:*staging*,!kube_cluster_profile:test,!kube_cluster_stamp:test*,kube_cluster_profile:dotcom} by {kube_deployment,kube_namespace,kube_cluster,site,kube_container_name} * (avg:kubernetes_state.container.memory_limit{!kube_namespace:*staging*,!kube_cluster_profile:test,!kube_cluster_stamp:test*,kube_cluster_profile:dotcom,catalog_service:*} by {kube_deployment,kube_namespace,catalog_service,kube_cluster,site,kube_container_name} / avg:kubernetes_state.container.memory_limit{!kube_namespace:*staging*,!kube_cluster_profile:test,!kube_cluster_stamp:test*,kube_cluster_profile:dotcom,catalog_service:*} by {kube_deployment,kube_namespace,catalog_service,kube_cluster,site,kube_container_name}) * 100 > 95

The monitor was last triggered at Sun Oct 18 2026 14:11:10 UTC.

[Monitor Status] · [Edit Monitor] · [Related Logs]
%%%

Alert Update: Triggered

Status: CRITICAL
Time: 2026-11-02T15:59Z

%%%

Container copilot-runtime-bazel-cache in deployment copilot-runtime-bazel-cache (namespace copilot-runtime-bazel-cache-production, service copilot-runtime-bazel-cache) in cluster dotcom-3-gc-cus-02 — its busiest pod is using 96.375% of this container's memory limit and is at risk of being OOM-killed and restarted.

What this means: the busiest pod's memory usage for this container, relative to the container's own limit. A leak or spike in a single replica will trip this, and crossing the limit triggers an OOM kill.

What to do:

Open the Kubernetes dashboard for this container and check per-pod memory — identify the pod climbing toward its limit and whether it is a steady leak or a spike.
Investigate top memory consumers / a possible leak; raise this container's memory limit if the workload legitimately needs more.
If this is expected, right-size the memory request/limit; otherwise escalate to the service owner.

Dashboard: Kubernetes Workload Observability

Playbook: https://github.com/github/ops/blob/master/docs/playbooks/kubernetes/workload-alerts/memory-usage-vs-limits.md

NOTE: Fleet-wide platform-default check (priority 3). When this fires, a GitHub issue is opened against your service's Service Catalog owning team. For github/github deployments, the issue goes to the deployment's owner in fleetwide-aliases.yaml, or to monolith platform if the deployment isn't listed.

HOW TO OPT OUT

Pick ONE of the following exclusions:
- !catalog_service:<your-service-slug> — excludes all your workloads (not for github/github: every monolith deployment shares catalog_service:github, so monolith teams must exclude their kube_deployment instead)
- !kube_deployment:<your-deployment> — excludes one deployment
- !kube_namespace:<your-namespace> — excludes an entire namespace
- !kube_container_name:<container> — excludes one container, for example an injected sidecar
Open a PR in github/datadog-monitoring adding your chosen exclusion inside every {...} filter block in this monitor's query. The CPU monitors (usage-vs-limits and throttling) have four filter blocks and the memory monitor has three — the exclusion must appear in every one of them or the filter is inconsistent. The error monitors have one. Repeat on all six monitor families for a full opt-out.

Cluster-level opt-outs are not accepted — this is a per-service safety net.

Questions? #compute-platform-support

@webhook-isp-alert

avg(last_15m):max:kubernetes.memory.usage_pct{!kube_namespace:*staging*,!kube_cluster_profile:test,!kube_cluster_stamp:test*,kube_cluster_profile:dotcom} by {kube_deployment,kube_namespace,kube_cluster,site,kube_container_name} * (avg:kubernetes_state.container.memory_limit{!kube_namespace:*staging*,!kube_cluster_profile:test,!kube_cluster_stamp:test*,kube_cluster_profile:dotcom,catalog_service:*} by {kube_deployment,kube_namespace,catalog_service,kube_cluster,site,kube_container_name} / avg:kubernetes_state.container.memory_limit{!kube_namespace:*staging*,!kube_cluster_profile:test,!kube_cluster_stamp:test*,kube_cluster_profile:dotcom,catalog_service:*} by {kube_deployment,kube_namespace,catalog_service,kube_cluster,site,kube_container_name}) * 100 > 95

The monitor was last triggered at Mon Nov 02 2026 15:59:10 UTC.

[Monitor Status] · [Edit Monitor] · [Related Logs]
%%%

Alert Update: Recovered

Status: RECOVERED
Time: 2026-11-09T13:02Z

%%%

Recovered: memory usage for container copilot-runtime-bazel-cache in copilot-runtime-bazel-cache (namespace copilot-runtime-bazel-cache-production) in cluster dotcom-3-gc-cus-02 is back below the threshold.

What this means: the busiest pod's memory usage for this container, relative to the container's own limit. A leak or spike in a single replica will trip this, and crossing the limit triggers an OOM kill.

What to do:

Open the Kubernetes dashboard for this container and check per-pod memory — identify the pod climbing toward its limit and whether it is a steady leak or a spike.
Investigate top memory consumers / a possible leak; raise this container's memory limit if the workload legitimately needs more.
If this is expected, right-size the memory request/limit; otherwise escalate to the service owner.

Dashboard: Kubernetes Workload Observability

Playbook: https://github.com/github/ops/blob/master/docs/playbooks/kubernetes/workload-alerts/memory-usage-vs-limits.md

NOTE: Fleet-wide platform-default check (priority 3). When this fires, a GitHub issue is opened against your service's Service Catalog owning team. For github/github deployments, the issue goes to the deployment's owner in fleetwide-aliases.yaml, or to monolith platform if the deployment isn't listed.

HOW TO OPT OUT

Pick ONE of the following exclusions:
- !catalog_service:<your-service-slug> — excludes all your workloads (not for github/github: every monolith deployment shares catalog_service:github, so monolith teams must exclude their kube_deployment instead)
- !kube_deployment:<your-deployment> — excludes one deployment
- !kube_namespace:<your-namespace> — excludes an entire namespace
- !kube_container_name:<container> — excludes one container, for example an injected sidecar
Open a PR in github/datadog-monitoring adding your chosen exclusion inside every {...} filter block in this monitor's query. The CPU monitors (usage-vs-limits and throttling) have four filter blocks and the memory monitor has three — the exclusion must appear in every one of them or the filter is inconsistent. The error monitors have one. Repeat on all six monitor families for a full opt-out.

Cluster-level opt-outs are not accepted — this is a per-service safety net.

Questions? #compute-platform-support

@webhook-isp-alert

avg(last_15m):max:kubernetes.memory.usage_pct{!kube_namespace:*staging*,!kube_cluster_profile:test,!kube_cluster_stamp:test*,kube_cluster_profile:dotcom} by {kube_deployment,kube_namespace,kube_cluster,site,kube_container_name} * (avg:kubernetes_state.container.memory_limit{!kube_namespace:*staging*,!kube_cluster_profile:test,!kube_cluster_stamp:test*,kube_cluster_profile:dotcom,catalog_service:*} by {kube_deployment,kube_namespace,catalog_service,kube_cluster,site,kube_container_name} / avg:kubernetes_state.container.memory_limit{!kube_namespace:*staging*,!kube_cluster_profile:test,!kube_cluster_stamp:test*,kube_cluster_profile:dotcom,catalog_service:*} by {kube_deployment,kube_namespace,catalog_service,kube_cluster,site,kube_container_name}) * 100 > 95

The monitor was last triggered at Mon Nov 02 2026 15:59:10 UTC.

[Monitor Status] · [Edit Monitor] · [Related Logs]
%%%

Alert Update: Triggered

Status: CRITICAL
Time: 2026-11-09T13:38Z

%%%

Container copilot-runtime-bazel-cache in deployment copilot-runtime-bazel-cache (namespace copilot-runtime-bazel-cache-production, service copilot-runtime-bazel-cache) in cluster dotcom-3-gc-cus-02 — its busiest pod is using 96.082% of this container's memory limit and is at risk of being OOM-killed and restarted.

What this means: the busiest pod's memory usage for this container, relative to the container's own limit. A leak or spike in a single replica will trip this, and crossing the limit triggers an OOM kill.

What to do:

Open the Kubernetes dashboard for this container and check per-pod memory — identify the pod climbing toward its limit and whether it is a steady leak or a spike.
Investigate top memory consumers / a possible leak; raise this container's memory limit if the workload legitimately needs more.
If this is expected, right-size the memory request/limit; otherwise escalate to the service owner.

Dashboard: Kubernetes Workload Observability

Playbook: https://github.com/github/ops/blob/master/docs/playbooks/kubernetes/workload-alerts/memory-usage-vs-limits.md

NOTE: Fleet-wide platform-default check (priority 3). When this fires, a GitHub issue is opened against your service's Service Catalog owning team. For github/github deployments, the issue goes to the deployment's owner in fleetwide-aliases.yaml, or to monolith platform if the deployment isn't listed.

HOW TO OPT OUT

Pick ONE of the following exclusions:
- !catalog_service:<your-service-slug> — excludes all your workloads (not for github/github: every monolith deployment shares catalog_service:github, so monolith teams must exclude their kube_deployment instead)
- !kube_deployment:<your-deployment> — excludes one deployment
- !kube_namespace:<your-namespace> — excludes an entire namespace
- !kube_container_name:<container> — excludes one container, for example an injected sidecar
Open a PR in github/datadog-monitoring adding your chosen exclusion inside every {...} filter block in this monitor's query. The CPU monitors (usage-vs-limits and throttling) have four filter blocks and the memory monitor has three — the exclusion must appear in every one of them or the filter is inconsistent. The error monitors have one. Repeat on all six monitor families for a full opt-out.

Cluster-level opt-outs are not accepted — this is a per-service safety net.

Questions? #compute-platform-support

@webhook-isp-alert

avg(last_15m):max:kubernetes.memory.usage_pct{!kube_namespace:*staging*,!kube_cluster_profile:test,!kube_cluster_stamp:test*,kube_cluster_profile:dotcom} by {kube_deployment,kube_namespace,kube_cluster,site,kube_container_name} * (avg:kubernetes_state.container.memory_limit{!kube_namespace:*staging*,!kube_cluster_profile:test,!kube_cluster_stamp:test*,kube_cluster_profile:dotcom,catalog_service:*} by {kube_deployment,kube_namespace,catalog_service,kube_cluster,site,kube_container_name} / avg:kubernetes_state.container.memory_limit{!kube_namespace:*staging*,!kube_cluster_profile:test,!kube_cluster_stamp:test*,kube_cluster_profile:dotcom,catalog_service:*} by {kube_deployment,kube_namespace,catalog_service,kube_cluster,site,kube_container_name}) * 100 > 95

The monitor was last triggered at Mon Nov 09 2026 13:38:10 UTC.

[Monitor Status] · [Edit Monitor] · [Related Logs]
%%%

Alert Update: Recovered

Status: RECOVERED
Time: 2026-11-12T11:06Z

%%%

Recovered: memory usage for container copilot-runtime-bazel-cache in copilot-runtime-bazel-cache (namespace copilot-runtime-bazel-cache-production) in cluster dotcom-3-gc-cus-02 is back below the threshold.

What this means: the busiest pod's memory usage for this container, relative to the container's own limit. A leak or spike in a single replica will trip this, and crossing the limit triggers an OOM kill.

What to do:

Open the Kubernetes dashboard for this container and check per-pod memory — identify the pod climbing toward its limit and whether it is a steady leak or a spike.
Investigate top memory consumers / a possible leak; raise this container's memory limit if the workload legitimately needs more.
If this is expected, right-size the memory request/limit; otherwise escalate to the service owner.

Dashboard: Kubernetes Workload Observability

Playbook: https://github.com/github/ops/blob/master/docs/playbooks/kubernetes/workload-alerts/memory-usage-vs-limits.md

NOTE: Fleet-wide platform-default check (priority 3). When this fires, a GitHub issue is opened against your service's Service Catalog owning team. For github/github deployments, the issue goes to the deployment's owner in fleetwide-aliases.yaml, or to monolith platform if the deployment isn't listed.

HOW TO OPT OUT

Pick ONE of the following exclusions:
- !catalog_service:<your-service-slug> — excludes all your workloads (not for github/github: every monolith deployment shares catalog_service:github, so monolith teams must exclude their kube_deployment instead)
- !kube_deployment:<your-deployment> — excludes one deployment
- !kube_namespace:<your-namespace> — excludes an entire namespace
- !kube_container_name:<container> — excludes one container, for example an injected sidecar
Open a PR in github/datadog-monitoring adding your chosen exclusion inside every {...} filter block in this monitor's query. The CPU monitors (usage-vs-limits and throttling) have four filter blocks and the memory monitor has three — the exclusion must appear in every one of them or the filter is inconsistent. The error monitors have one. Repeat on all six monitor families for a full opt-out.

Cluster-level opt-outs are not accepted — this is a per-service safety net.

Questions? #compute-platform-support

@webhook-isp-alert

avg(last_15m):max:kubernetes.memory.usage_pct{!kube_namespace:*staging*,!kube_cluster_profile:test,!kube_cluster_stamp:test*,kube_cluster_profile:dotcom} by {kube_deployment,kube_namespace,kube_cluster,site,kube_container_name} * (avg:kubernetes_state.container.memory_limit{!kube_namespace:*staging*,!kube_cluster_profile:test,!kube_cluster_stamp:test*,kube_cluster_profile:dotcom,catalog_service:*} by {kube_deployment,kube_namespace,catalog_service,kube_cluster,site,kube_container_name} / avg:kubernetes_state.container.memory_limit{!kube_namespace:*staging*,!kube_cluster_profile:test,!kube_cluster_stamp:test*,kube_cluster_profile:dotcom,catalog_service:*} by {kube_deployment,kube_namespace,catalog_service,kube_cluster,site,kube_container_name}) * 100 > 95

The monitor was last triggered at Mon Nov 09 2026 13:38:10 UTC.

[Monitor Status] · [Edit Monitor] · [Related Logs]
%%%

Navigation Menu

P3 - container-runtime/copilot-runtime-bazel-cache-memory-usage-vs-limits-dotcom on catalog_service:copilot-runtime-bazel-cache,kube_cluster:dotcom-3-gc-cus-02,kube_container_name:copilot-runtime-bazel-cache,kube_deployment:copilot-r... - 2026-10-18T14:11Z #5185

Alert Details

Tags

On-Call

Alert Update: Triggered

Alert Update: Recovered

Alert Update: Triggered

Alert Update: Recovered

Alert Update: Triggered

Alert Update: Recovered

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

P3 - container-runtime/copilot-runtime-bazel-cache-memory-usage-vs-limits-dotcom on catalog_service:copilot-runtime-bazel-cache,kube_cluster:dotcom-3-gc-cus-02,kube_container_name:copilot-runtime-bazel-cache,kube_deployment:copilot-r... - 2026-10-18T14:11Z #5185

Description

Alert Details

Tags

On-Call

Activity

github-internal-statuspage-new commented on Oct 22, 2026

Alert Update: Triggered

github-internal-statuspage-new commented on Nov 2, 2026

Alert Update: Recovered

github-internal-statuspage-new commented on Nov 2, 2026

Alert Update: Triggered

github-internal-statuspage-new commented on Nov 9, 2026

Alert Update: Recovered

github-internal-statuspage-new commented on Nov 9, 2026

Alert Update: Triggered

github-internal-statuspage-new commented on Nov 12, 2026

Alert Update: Recovered

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions