From a79460118d285b06016361183425685161898f44 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=EC=B5=9C=EC=98=81=EA=B8=B0?= <dch0202@rsquare.co.kr>
Date: Tue, 14 Jul 2026 18:25:58 +0900
Subject: [PATCH 1/2] knowledge: ingest 2 verified insights (kubelet resource
 metrics on embedded k8s; host-to-pod memory limit sizing)

---
 .dev-loop/INGEST_REPORT.md                    | 90 ++++++++++++++++++
 log.md                                        |  2 +
 .../containers/resource-limits-and-probes.md  |  8 +-
 wiki/infrastructure/index.md                  |  1 +
 ...ontainer-metrics-when-cadvisor-is-empty.md | 93 +++++++++++++++++++
 5 files changed, 192 insertions(+), 2 deletions(-)
 create mode 100644 .dev-loop/INGEST_REPORT.md
 create mode 100644 wiki/infrastructure/observability/container-metrics-when-cadvisor-is-empty.md

diff --git a/.dev-loop/INGEST_REPORT.md b/.dev-loop/INGEST_REPORT.md
new file mode 100644
index 0000000..e9bcc5a
--- /dev/null
+++ b/.dev-loop/INGEST_REPORT.md
@@ -0,0 +1,90 @@
+# Knowledge flush — 2 insight(s)
+
+Cross-Check: every mechanism claim was independently re-verified against primary
+sources (chart values.yaml, kubernetes source, kubernetes-mixin source, psycopg2
+docs) fetched live during this flush — not taken from the harvesting session.
+
+Queue note: a third queued row (QueryPie streaming / fetchmany + write_only,
+hash `637877dfc33bc0f0`) was already flushed as **PR #1** (open) but had not
+been removed from its session queue file; it was retired without re-ingesting.
+
+## Verified best-practice
+
+### Insight 1 — container_* metrics empty on OrbStack k8s despite kubelet targets up
+
+Claim chain and verification:
+
+1. **kube-prometheus-stack ships the kubelet resource-metrics scrape disabled,
+   with a stale default path.**
+   Source: `charts/kube-prometheus-stack/values.yaml` (prometheus-community/helm-charts,
+   fetched live) — `resource: false`, `resourcePath: "/metrics/resource/v1alpha1"`,
+   with the in-file comment "From kubernetes 1.18, /metrics/resource/v1alpha1
+   renamed to /metrics/resource". → **verified**
+2. **`/metrics/resource` exposes `container_cpu_usage_seconds_total` and
+   `container_memory_working_set_bytes` with labels `container, pod, namespace`
+   only — no `image` label.**
+   Source: `pkg/kubelet/metrics/collectors/resource_metrics.go` (kubernetes/kubernetes,
+   fetched live; both metrics STABLE). → **verified**
+3. **The bundled Compute Resources dashboards cannot show these series.**
+   Source: `dashboards/resources/queries/pod.libsonnet` (kubernetes-monitoring/
+   kubernetes-mixin, fetched live) — memory queries select `cadvisorSelector`
+   (`metrics_path="/metrics/cadvisor"`) AND `image!=""`; the resource-endpoint
+   series fail both. → **verified**
+4. **OrbStack's kubelet cAdvisor endpoint emits only `machine_*`, zero
+   `container_*`, while the scrape reports success.**
+   First-hand probe (2026-07-14): `kubectl get --raw .../proxy/metrics/cadvisor`
+   → 0 `container_` lines, `machine_scrape_error 0`; `/metrics/resource` →
+   real working-set values. No official OrbStack doc states this; orbstack#1561
+   corroborates the metrics-integration gap. → **field-tested** (marked as such
+   in the page body; the page's other rows are verified).
+
+### Insight 2 — "ran fine on the host" is not evidence a pod memory limit fits
+
+1. **Exceeding a container memory limit ⇒ OOMKill; limits must come from
+   measurement.** Already sourced on the existing page
+   (kubernetes.io manage-resources-containers). → **verified** (pre-existing)
+2. **A default (client-side) psycopg2 cursor loads the entire result set
+   client-side, so memory scales with data size; server-side cursors transfer
+   controlled amounts.** Source: https://www.psycopg.org/docs/usage.html#server-side-cursors,
+   fetched live — "the Psycopg cursor usually fetches all the records returned
+   by the backend … a proportionally large amount of memory will be allocated
+   by the client" / "transfer to the client only a controlled amount of data".
+   → **verified**
+3. **Concrete failure/measurement**: 800k-row extract OOMKilled in a 1Gi pod
+   after 20 min; measured 1.5Gi+ peak once raised to 3Gi (mac-server k8s
+   migration, 2026-07-13). → **field-tested** context, used as the motivating
+   example only.
+
+## Existing-layer check
+
+- Pages read: root `INDEX.md`, `infrastructure/index.md`,
+  `infrastructure/containers/resource-limits-and-probes.md`,
+  `infrastructure/observability/logs-metrics-signals.md`; PR #1's diff
+  (`databases/query-optimization/streaming-large-result-sets.md`) reviewed for
+  overlap with insight 2.
+- **Insight 1**: no existing page covers scraping/metric-source selection for
+  container metrics. `logs-metrics-signals` is about instrumenting your own
+  service (different trigger) — not a duplicate. → new page, `related:` linked
+  both ways with `resource-limits-and-probes` (its "set limits from
+  measurement" rule depends on these very metrics) and one-way to
+  `logs-metrics-signals`.
+- **Insight 2**: same trigger family as `resource-limits-and-probes`
+  ("pods OOMKilled") and same directive spirit ("measure, don't copy") → **merged**
+  into that page (1 edge-case row + 1 instead-of row + psycopg2 source), no new
+  page. The streaming *fix* itself lives in PR #1's databases page; a
+  `related:` link to it was **deliberately not added** because that page is not
+  on `main` yet (PR #1 unmerged) — flagged here instead so the owner can add
+  the link when/if PR #1 merges.
+- Conflicts: none found.
+
+## Routing decision
+
+- **Insight 1** → `infrastructure/observability/container-metrics-when-cadvisor-is-empty.md`
+  (**new page**). Category `observability` fits (metrics collection/source
+  selection); no new category needed. Registered in `infrastructure/index.md`
+  with a load-when line keyed on "embedded/local k8s + container_* empty +
+  targets up"; `log.md` ingest entry appended.
+- **Insight 2** → `infrastructure/containers/resource-limits-and-probes.md`
+  (**merge/revise**, no new page). Category `containers` already owns the
+  OOMKill/limit-sizing trigger. `last_verified` bumped to 2026-07-14; `log.md`
+  revise entry appended.
diff --git a/log.md b/log.md
index bcc206d..9d607da 100644
--- a/log.md
+++ b/log.md
@@ -32,3 +32,5 @@ Append-only. Format: `## [YYYY-MM-DD] <ingest|revise|lint|gap|contradiction|drif
 ## [2026-07-11] ingest | security +1 (new category api-exposure): exposing-an-origin-http-api — code-vs-edge control split for putting an origin HTTP API behind a public tunnel/proxy. Captures the non-obvious 5xx-bypasses-request-middleware gotcha (set security headers in the exception handler too — verify with a forced 500), header-only fail-closed auth, docs/OpenAPI off at the origin by default, sanitized 5xx bodies, and the code(validate/headers/errors)-vs-edge(TLS/HSTS/rate-limit/WAF) boundary. Derived from hardening korea-data-suite for external exposure + independent pentest; sources OWASP Secure Headers, MDN, Starlette middleware, FastAPI docs.
 ## [2026-07-12] ingest | databases +1 (new category sqlite): concurrent-access-for-a-read-api — WAL + busy_timeout for a read-heavy API with a background writer; set journal_mode once (per-connection pragma cost ~0.4ms measured vs 0.04ms busy_timeout), single writer, run the batch writer as a separate process (interpreter-lock isolation), scale reads with worker processes not threads. Derived from load-testing korea-data-suite (measured connect/query/pragma costs, GIL-bound concurrency, separate-sync-process deployment). Sources: SQLite WAL/pragma/FAQ docs.
 ## [2026-07-12] revise | security/secrets-in-code +1 edge case: third-party HTTP client (httpx/requests) logs the full request URL — including a query-param API key — at INFO, so root/DEBUG logging leaks it; keep the client logger above INFO. Found when a standalone sync process set logging.basicConfig(INFO) and httpx wrote the data.go.kr serviceKey to the log file. last_verified bumped to 2026-07-12.
+## [2026-07-14] ingest | infrastructure +1 (observability): container-metrics-when-cadvisor-is-empty — on embedded/local k8s (OrbStack) the kubelet cAdvisor endpoint can emit only machine_* series while scrapes stay "up"; probe endpoint bodies via kubectl get --raw, enable kps kubelet.serviceMonitor.resource with resourcePath /metrics/resource (chart default v1alpha1 path 404s post-1.18), and chart the series on a custom dashboard because mixin queries hard-filter metrics_path="/metrics/cadvisor" + image!="". Mechanism verified against kps values.yaml, kubernetes resource_metrics.go (labels container/pod/namespace only), kubernetes-mixin pod.libsonnet; OrbStack empty-cAdvisor claim field-tested (first-hand probe 2026-07-14, corroborated by orbstack#1561).
+## [2026-07-14] revise | infrastructure/containers/resource-limits-and-probes +1 edge case & instead-of: a workload migrated from a host/VM into a memory-limited pod — "ran fine on the host" carries no limit-fit information (host enforced no limit); measure the real workload peak in-container before sizing, and bound data-proportional consumers (default client-side DB cursors buffer the whole result set — psycopg2 usage docs added as source). Derived from mac-server k8s migration (800k-row extract OOMKilled at 1Gi, measured 1.5Gi+ peak at 3Gi, 2026-07-13). last_verified bumped to 2026-07-14.
diff --git a/wiki/infrastructure/containers/resource-limits-and-probes.md b/wiki/infrastructure/containers/resource-limits-and-probes.md
index 2afe8ef..76a6b58 100644
--- a/wiki/infrastructure/containers/resource-limits-and-probes.md
+++ b/wiki/infrastructure/containers/resource-limits-and-probes.md
@@ -8,8 +8,9 @@ sources:
   - https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/
   - https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/
   - https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/
-last_verified: 2026-07-10
-related: [infrastructure-deploy-rollout-and-rollback, backend-java-runtime-threads-and-memory]
+  - https://www.psycopg.org/docs/usage.html#server-side-cursors
+last_verified: 2026-07-14
+related: [infrastructure-deploy-rollout-and-rollback, backend-java-runtime-threads-and-memory, infrastructure-observability-container-metrics-when-cadvisor-is-empty]
 ---
 
 # Resource Limits and Health Probes in Deployment Manifests
@@ -68,6 +69,7 @@ storm; traffic reaches pods that are not ready to serve.
 | Runtime OOMKilled while its configured heap is below the limit | Non-heap memory (metaspace, thread stacks, direct buffers) pushed the total over: [backend-java-runtime-threads-and-memory] |
 | Pod evicted rather than OOMKilled | Node memory pressure evicting by QoS order — set memory request = limit to move the workload to Guaranteed |
 | App has no HTTP server to probe | Use an exec or TCP probe against the process itself; the process-only rule for liveness is unchanged |
+| Process moved from a host/VM into a pod is OOMKilled at a limit that "should be plenty" | "Ran fine on the host" is not evidence — the host enforced no limit. Run the real workload once and measure its peak (`kubectl top`, `container_memory_working_set_bytes`) before sizing the limit. If memory scales with input size (e.g. a default client-side DB cursor buffers the entire result set on execute), bound the working set (server-side/streaming cursor, chunked `fetchmany`) or size the limit to the measured peak of the largest real input |
 
 ## Instead of
 
@@ -77,9 +79,11 @@ storm; traffic reaches pods that are not ready to serve.
 | Copy resource limits from a blog post or another service | Measure this service's usage; set requests from p99 + headroom | Limits sized for someone else's workload produce OOMKills or throttling for yours |
 | Leave requests unset "for flexibility" | Set requests on every container | Unset requests = BestEffort: first evicted, and the scheduler overcommits the node |
 | Stretch liveness timeouts so the app survives boot | Add a startup probe sized for worst-case boot; keep liveness at steady-state timing | Boot-sized liveness leaves real steady-state hangs undetected for the whole boot window |
+| Size a migrated workload's memory limit from "it ran fine on the host" | Measure the workload's actual peak in-container on real data, then set limit = peak + headroom | The host never enforced a limit, so its success carries no information about the peak; data-proportional consumers (whole-result-set DB cursors, in-memory file builds) only reveal their peak under the real input |
 
 ## Sources
 
 - https://kubernetes.io/docs/tasks/configure-pod-container/configure-liveness-readiness-startup-probes/ — probe purposes, failureThreshold/periodSeconds, startup probes protecting slow boots, readiness removes from endpoints without restart
 - https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ — requests drive scheduling, limits enforced at runtime; memory over limit = OOM kill, CPU over limit = throttling
 - https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/ — Guaranteed/Burstable/BestEffort assignment and eviction order
+- https://www.psycopg.org/docs/usage.html#server-side-cursors — a default (client-side) cursor loads the whole result set client-side; server-side cursors transfer data in controlled amounts
diff --git a/wiki/infrastructure/index.md b/wiki/infrastructure/index.md
index 4940602..4eb98f2 100644
--- a/wiki/infrastructure/index.md
+++ b/wiki/infrastructure/index.md
@@ -46,3 +46,4 @@ Match your situation to a "load when" line; load only matching pages.
 |------|-----------|
 | [logs-metrics-signals](observability/logs-metrics-signals.md) | Instrumenting a new or existing service (logs, metrics, correlation ids); an incident revealed you couldn't see what happened; choosing between a log line and a metric; a metric label would carry unbounded values (user ids/UUIDs) |
 | [alerting](observability/alerting.md) | Creating or reviewing alerts; the team ignores a noisy pager; deciding whether a condition pages, tickets, or stays on a dashboard |
+| [container-metrics-when-cadvisor-is-empty](observability/container-metrics-when-cadvisor-is-empty.md) | Prometheus/kube-prometheus-stack on an embedded/local Kubernetes (OrbStack or similar) shows no `container_*` series while kubelet targets are up; deciding between cAdvisor and the kubelet `/metrics/resource` endpoint |
diff --git a/wiki/infrastructure/observability/container-metrics-when-cadvisor-is-empty.md b/wiki/infrastructure/observability/container-metrics-when-cadvisor-is-empty.md
new file mode 100644
index 0000000..7deb2b1
--- /dev/null
+++ b/wiki/infrastructure/observability/container-metrics-when-cadvisor-is-empty.md
@@ -0,0 +1,93 @@
+---
+id: infrastructure-observability-container-metrics-when-cadvisor-is-empty
+domain: infrastructure
+category: observability
+applies_to: [kubernetes, prometheus, local-dev]
+confidence: verified
+sources:
+  - https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml
+  - https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/metrics/collectors/resource_metrics.go
+  - https://github.com/kubernetes-monitoring/kubernetes-mixin/blob/master/dashboards/resources/queries/pod.libsonnet
+  - https://github.com/orbstack/orbstack/issues/1561
+last_verified: 2026-07-14
+related: [infrastructure-containers-resource-limits-and-probes, infrastructure-observability-logs-metrics-signals]
+---
+
+# Collecting Container Metrics When the Kubelet cAdvisor Endpoint Is Empty
+
+## When this applies
+
+Running kube-prometheus-stack (or any Prometheus scraping the kubelet) on an
+embedded/local Kubernetes distribution (OrbStack; potentially other
+lightweight runtimes) and per-container `container_*` series are absent —
+while every kubelet scrape target reports **up** and dashboards render only
+node-level panels.
+
+## Do this
+
+1. Confirm the gap at the source, not via target health — a cAdvisor endpoint
+   that returns only `machine_*` series is still a *successful* scrape:
+
+   ```sh
+   kubectl get --raw /api/v1/nodes/<node>/proxy/metrics/cadvisor | grep -c '^container_'
+   kubectl get --raw /api/v1/nodes/<node>/proxy/metrics/resource | grep -c '^container_'
+   ```
+
+   If cAdvisor shows 0 but `/metrics/resource` shows real values, switch the
+   scrape source instead of debugging the scrape.
+
+2. Enable the kubelet resource-metrics scrape in kube-prometheus-stack values,
+   and override the path — the chart's default still points at the pre-1.18
+   versioned path, which 404s on current kubelets:
+
+   ```yaml
+   kubelet:
+     serviceMonitor:
+       resource: true
+       resourcePath: "/metrics/resource"   # chart default is /metrics/resource/v1alpha1
+   ```
+
+3. Chart these series on a custom dashboard, not the bundled
+   "Compute Resources" dashboards. `/metrics/resource` exposes
+   `container_cpu_usage_seconds_total` and `container_memory_working_set_bytes`
+   with only `container, pod, namespace` labels (no `image`), while the bundled
+   kubernetes-mixin queries select `metrics_path="/metrics/cadvisor"` and filter
+   `image!=""` — both conditions exclude the resource-endpoint series.
+
+4. Treat `/metrics/resource` as the core-usage subset it is: CPU seconds and
+   working-set bytes per container/pod/node (it is what metrics-server reads).
+   Filesystem, network, and throttling detail exists only in cAdvisor; if you
+   need those on such a cluster, they are simply unavailable from the kubelet.
+
+## Edge cases
+
+| Case | Then |
+|------|------|
+| Resource-metrics target shows **down** with 404 after enabling `resource: true` | The path override was missed — set `resourcePath: "/metrics/resource"` (the unversioned GA path) |
+| Bundled dashboards stay empty after the scrape works | Expected — mixin queries require `metrics_path="/metrics/cadvisor"` and `image!=""`; build panels that query the series without those selectors |
+| Verifying whether the same gap exists on another local distro | Run the two `kubectl get --raw` probes from step 1 first; only the endpoint contents distinguish "scrape broken" from "endpoint empty" |
+| Production/managed cluster (EKS/GKE/AKS) missing `container_*` | Different failure — cAdvisor works there; check kubelet scrape auth, relabeling drops, and job selectors before touching `resourcePath` |
+
+## Instead of
+
+| If you are about to | Do this instead | Why |
+|---------------------|-----------------|-----|
+| Debug the scrape because targets are up but `container_*` is empty | Probe the endpoint bodies with `kubectl get --raw` (step 1) | A cAdvisor endpoint emitting only `machine_*` scrapes "successfully"; target health cannot detect an empty metric family |
+| Enable `resource: true` alone and trust the chart default path | Also set `resourcePath: "/metrics/resource"` | The chart default `/metrics/resource/v1alpha1` predates the k8s 1.18 rename and 404s on current kubelets |
+| Wait for the bundled Compute Resources dashboards to light up | Build a custom dashboard on the resource-endpoint series | Mixin queries hard-filter on `metrics_path="/metrics/cadvisor"` and `image!=""`, which the resource endpoint's label set can never satisfy |
+
+## Notes on evidence
+
+The mechanism rows above are verified against the primary sources below. The
+OrbStack-specific claim — its kubelet cAdvisor endpoint emits `machine_*` but
+zero `container_*` series — is **field-tested** (first-hand probe on OrbStack
+1.x, 2026-07-14: cAdvisor `container_` count 0 with `machine_scrape_error 0`,
+`/metrics/resource` returning real working-set values), corroborated by the
+OrbStack tracker discussion of missing metrics integrations.
+
+## Sources
+
+- https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml — `kubelet.serviceMonitor.resource` disabled by default; `resourcePath` default `/metrics/resource/v1alpha1` with the "renamed in 1.18" comment
+- https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/metrics/collectors/resource_metrics.go — `/metrics/resource` exposes `container_cpu_usage_seconds_total` / `container_memory_working_set_bytes` with labels `container, pod, namespace` only (STABLE)
+- https://github.com/kubernetes-monitoring/kubernetes-mixin/blob/master/dashboards/resources/queries/pod.libsonnet — bundled dashboard queries select `cadvisorSelector` + `image!=""`
+- https://github.com/orbstack/orbstack/issues/1561 — OrbStack metrics-server/cAdvisor integration gaps (corroboration)

From a3ada1b0e1e1dac28995974037b9bc0668c7c0a1 Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?=EC=B5=9C=EC=98=81=EA=B8=B0?= <dch0202@rsquare.co.kr>
Date: Tue, 14 Jul 2026 18:28:56 +0900
Subject: [PATCH 2/2] knowledge: add Decision Log to ingest report

---
 .dev-loop/INGEST_REPORT.md | 10 ++++++++++
 1 file changed, 10 insertions(+)

diff --git a/.dev-loop/INGEST_REPORT.md b/.dev-loop/INGEST_REPORT.md
index e9bcc5a..d56e156 100644
--- a/.dev-loop/INGEST_REPORT.md
+++ b/.dev-loop/INGEST_REPORT.md
@@ -1,5 +1,15 @@
 # Knowledge flush — 2 insight(s)
 
+## Decision Log
+
+- **의도**: dev-loop 큐의 pending 인사이트 2건(OrbStack kubelet 메트릭 소스 전환, host→pod 메모리 리밋 산정)을 1차 소스로 재검증해 위키에 반영. 이슈처럼 큰 변경이 아니라 지식 ingest — 페이지 1 신규 + 1 병합.
+- **배제한 대안**:
+  - insight 1을 `observability/logs-metrics-signals.md`에 병합 → 배제: 그 페이지는 "내 서비스 계측"이 트리거고, 이 건은 "클러스터 메트릭 수집 소스 선택"이라 트리거가 다름(one case per page 규칙).
+  - insight 2를 신규 페이지로 생성 → 배제: `resource-limits-and-probes.md`가 이미 "OOMKilled + 측정 기반 리밋" 트리거를 소유 — 엣지케이스/Instead-of 행 병합이 중복 없는 형태.
+  - PR #1의 `streaming-large-result-sets` 페이지로의 `related:` 링크 → 보류: PR #1 미머지 상태라 main에 없는 페이지로의 링크는 깨질 수 있음(본문 Existing-layer check에 플래그).
+- **리뷰어가 볼 곳**: 신규 페이지의 4개 소스가 각 주장과 대응하는지(values.yaml 기본값, resource_metrics.go 라벨셋, mixin `image!=""`+cadvisorSelector, orbstack#1561), OrbStack 주장이 field-tested로 정직하게 표기됐는지, `resource-limits-and-probes.md` 추가 2행이 기존 규칙 4("measure, don't copy")와 중복이 아닌 보강인지.
+- [추정] 없음 — 모든 메커니즘 주장은 이번 flush에서 라이브로 가져온 1차 소스로 검증됨(본문 Verified best-practice 참조).
+
 Cross-Check: every mechanism claim was independently re-verified against primary
 sources (chart values.yaml, kubernetes source, kubernetes-mixin source, psycopg2
 docs) fetched live during this flush — not taken from the harvesting session.