From 632873e0ddddf97b90ee66b41e5554b21eab375f Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=EC=B5=9C=EC=98=81=EA=B8=B0?= Date: Wed, 15 Jul 2026 13:07:01 +0900 Subject: [PATCH 1/4] knowledge: ingest 3 verified insight(s) --- .dev-loop/INGEST_REPORT.md | 45 +++++++++++ log.md | 3 + .../containers/entrypoint-log-capture.md | 73 +++++++++++++++++ .../infrastructure/containers/image-builds.md | 2 +- .../containers/resource-limits-and-probes.md | 2 +- wiki/infrastructure/index.md | 2 + .../kubelet-container-metrics.md | 79 +++++++++++++++++++ .../observability/logs-metrics-signals.md | 2 +- .../shells/portable-shell-scripts.md | 2 +- wiki/platforms/tools/bsd-vs-gnu-cli.md | 7 +- 10 files changed, 212 insertions(+), 5 deletions(-) create mode 100644 .dev-loop/INGEST_REPORT.md create mode 100644 wiki/infrastructure/containers/entrypoint-log-capture.md create mode 100644 wiki/infrastructure/observability/kubelet-container-metrics.md diff --git a/.dev-loop/INGEST_REPORT.md b/.dev-loop/INGEST_REPORT.md new file mode 100644 index 0000000..12c8140 --- /dev/null +++ b/.dev-loop/INGEST_REPORT.md @@ -0,0 +1,45 @@ +# Knowledge flush — 3 insight(s) + +## Verified best-practice + +**1. Empty `container_*` metrics though kubelet targets are up → scrape `/metrics/resource` (kube-prometheus-stack).** +Claim: on OrbStack-style embedded k8s the kubelet cAdvisor endpoint emits no `container_*` series while scrapes stay healthy; the fix is `kubelet.serviceMonitor.resource: true` plus an explicit `resourcePath: "/metrics/resource"`, and the mixin Compute Resources dashboards will still drop these series because of their `image!=""` filter. +Verified against: +- https://kubernetes.io/docs/reference/instrumentation/node-metrics/ — `/metrics/resource` is the current kubelet resource-metrics endpoint (metrics-server ≥0.6 uses it). +- https://raw.githubusercontent.com/prometheus-community/helm-charts/main/charts/kube-prometheus-stack/values.yaml — confirmed `resource: false` default and `resourcePath: "/metrics/resource/v1alpha1"` default with the chart's own comment "From kubernetes 1.18, /metrics/resource/v1alpha1 renamed to /metrics/resource" — exactly the 404 trap the insight describes. +- kubernetes-mixin source read via `gh api` (`dashboards/resources/queries/pod.libsonnet`) — pod CPU/memory queries all carry `image!=""` (and `container!=""`); `/metrics/resource` series have no `image` label. +The OrbStack-specific behavior (cAdvisor emitting only `machine_*`) has no external doc; it is marked as a dated field observation in the page. Overall **confidence: verified** (mechanism fully doc-backed + session reproduction with PromQL round-trip). + +**2. PID-1 entrypoint bash with `exec > >(tee -a f)` loses log output → EXIT-trap fd close + `wait "$TEE_PID"` + `trap 'exit 143' TERM`.** +Verified against: +- https://mywiki.wooledge.org/ProcessSubstitution — process substitution "will continue to run when your script exits (unless you manage your child processes)"; since bash 4.4 it can be managed with `wait "$!"`. In a container the PID-1 exit tears the orphan down before flush instead of letting it finish. +- https://tiswww.case.edu/php/chet/bash/bashref.html — 128+n exit status for signal-terminated commands (143 = SIGTERM). +- Session reproduction (OrbStack container): without trap 0/10 runs captured output; with trap 10/10. +**confidence: verified**. (gnu.org bash manual rate-limited me with HTTP 429 during the flush, so the mirror + Wooledge are cited instead — no unverified URL was included.) + +**3. BSD/GNU `date` ms-timestamp feature detection → test the actual `%3N` output shape, not `%N` presence.** +Verified against: +- https://www.gnu.org/software/coreutils/manual/html_node/Time-conversion-specifiers.html — "%N nanoseconds … This is a GNU extension." +- https://www.gnu.org/software/coreutils/manual/html_node/Padding-and-other-flags.html — field width between `%` and the specifier is a GNU extension. +- **Independent reproduction on a second machine during this flush** (macOS 26.5.1, beyond the original 14.8.3 measurement): `date +%N` → `070788000`, `date +%3N` → literal `3N`. The trap is real: `%N` working makes naive detection pass, then `%3N` corrupts timestamps. +**confidence: verified**. + +## Existing-layer check + +Read: root `INDEX.md`, `wiki/infrastructure/index.md`, `wiki/platforms/index.md`, and every overlapping page: `observability/logs-metrics-signals.md`, `observability/alerting.md` (index line), `containers/resource-limits-and-probes.md`, `containers/image-builds.md`, `shells/portable-shell-scripts.md`, `tools/bsd-vs-gnu-cli.md`. + +- Insight 1: no overlap — `logs-metrics-signals` covers instrumentation principles (signal choice, cardinality), not scrape-source configuration. No conflict. **Created new page**; related-linked both ways to `logs-metrics-signals` and `resource-limits-and-probes`. +- Insight 2: no overlap — `portable-shell-scripts` covers shell portability, `image-builds` covers Dockerfiles; neither covers PID-1 runtime log/signal behavior. No conflict. **Created new page**; related-linked both ways to `image-builds` and `portable-shell-scripts` (cross-domain), one-way to `logs-metrics-signals`. +- Insight 3: direct overlap with `platforms/tools/bsd-vs-gnu-cli.md` (same trigger, same directive family — its "Relative date" row already covers `date -d` vs `-v`). **Merged**: one command-table row (sub-second timestamp), one edge-case row (`%N` printing digits on macOS must not imply GNU), two sources, `last_verified` bumped to 2026-07-15. No new page. + +Conflicts flagged: none. + +## Routing decision + +| Insight | Target | New category? | +|---------|--------|---------------| +| 1 — kubelet container metrics | `infrastructure/observability/kubelet-container-metrics.md` (new page) | No — observability already owns metrics collection; index "load when" added | +| 2 — PID-1 tee log loss | `infrastructure/containers/entrypoint-log-capture.md` (new page) | No — containers owns container-runtime behavior; platforms/shells was rejected because the trigger is PID-1 container semantics, not shell portability | +| 3 — date `%3N` detection | `platforms/tools/bsd-vs-gnu-cli.md` (merge) | No — exact existing page for BSD-vs-GNU flag differences | + +Nothing left `unverified`. Queue rows retired to `~/.dev-loop/queue/.processed.jsonl` after PR creation. diff --git a/log.md b/log.md index bcc206d..41a31ac 100644 --- a/log.md +++ b/log.md @@ -32,3 +32,6 @@ Append-only. Format: `## [YYYY-MM-DD] >(tee …) loses log output because the shell never waits for process substitutions and the container dies with PID 1; fix is EXIT-trap fd close + wait "$TEE_PID" plus trap 'exit 143' TERM. Sources: Wooledge ProcessSubstitution (wait "$!" since bash 4.4), bash ref manual 128+n; field reproduction 0/10 → 10/10 +## [2026-07-15] revise | platforms-tools-bsd-vs-gnu-cli +1 row +1 edge case: sub-second timestamps — recent macOS date prints digits for %N but emits %3N literally as "3N", so feature-detect on the actual %3N output shape, never on %N presence. GNU manual confirms %N and field width are GNU extensions; reproduced on macOS 14.8.3 and 26.5.1. last_verified bumped to 2026-07-15 diff --git a/wiki/infrastructure/containers/entrypoint-log-capture.md b/wiki/infrastructure/containers/entrypoint-log-capture.md new file mode 100644 index 0000000..992ce0e --- /dev/null +++ b/wiki/infrastructure/containers/entrypoint-log-capture.md @@ -0,0 +1,73 @@ +--- +id: infrastructure-containers-entrypoint-log-capture +domain: infrastructure +category: containers +applies_to: [docker, kubernetes, bash] +confidence: verified +sources: + - https://mywiki.wooledge.org/ProcessSubstitution + - https://tiswww.case.edu/php/chet/bash/bashref.html +last_verified: 2026-07-15 +related: [infrastructure-containers-image-builds, infrastructure-observability-logs-metrics-signals, platforms-shells-portable-shell-scripts] +--- + +# Log Loss from Process-Substituted tee in Container Entrypoint Scripts + +## When this applies + +A container's entrypoint bash script (running as PID 1) duplicates its output to +a file with `exec > >(tee -a "$LOG") 2>&1`, and log lines are missing: short jobs +capture nothing, long jobs lose the tail (the completion-summary lines). + +## Do this + +1. Capture the tee PID and wait for it in an EXIT trap. Bash runs process + substitutions asynchronously and does not wait for them; `wait "$!"` on a + process substitution is supported since bash 4.4. Outside a container the + orphaned tee usually finishes anyway — as PID 1, the container is torn down + the instant the script exits, killing tee before it flushes: + + ```bash + exec > >(tee -a "$LOG") 2>&1 + TEE_PID=$! + flush_logs() { + exec 1>&- 2>&- # close our ends so tee sees EOF + wait "$TEE_PID" + } + trap flush_logs EXIT + ``` + +2. Also trap TERM so a `docker stop` / pod deletion goes through the same path: + + ```bash + trap 'exit 143' TERM # 128+15, the conventional SIGTERM exit status + ``` + + Converting the signal into an `exit` guarantees the EXIT trap (and the tee + flush) runs, and reports the conventional 128+n code. + +3. Verify by re-running the container several times and checking the log file + for the final line each run — this failure is timing-dependent, so a single + successful run proves nothing. + +## Edge cases + +| Case | Then | +|------|------| +| Base image bash is older than 4.4 (`wait "$!"` on a process substitution fails) | Use a named pipe: `mkfifo`, start `tee < pipe &` as a normal background job, `exec > pipe`, and `wait` its real PID | +| Script also traps other signals (INT, HUP) | Route each through `exit 128+n` the same way, so every termination path reaches the EXIT trap | +| Logs must survive even a SIGKILL (OOMKill, forced deletion) | No trap runs on KILL — write to a mounted volume via unbuffered append (or ship lines as they are produced) instead of relying on exit-time flushing | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Trust `exec > >(tee -a f)` because it works in a terminal | Add the EXIT-trap wait before it runs as PID 1 | The shell never waits for process substitutions; a terminal leaves the orphan running to finish, a container kills it at PID 1 exit | +| Debug the missing lines as a tee buffering option problem | Reproduce with repeated short runs and add the trap | The writer is killed before flushing — buffering flags don't fix a killed process | +| Exit 0 from the TERM trap for a "clean" shutdown | `exit 143` (128+15) | Orchestrators and CI distinguish signalled termination from success by the 128+n convention | + +## Sources + +- https://mywiki.wooledge.org/ProcessSubstitution — process substitution "will continue to run when your script exits (unless you manage your child processes)"; since bash 4.4 it can be managed with `wait "$!"` +- https://tiswww.case.edu/php/chet/bash/bashref.html — exit status is 128+n for a command terminated by signal n +- Field reproduction (OrbStack container, 2026-07-15): without the trap the log captured the script output in 0 of 10 runs; with the EXIT-trap wait, 10 of 10 diff --git a/wiki/infrastructure/containers/image-builds.md b/wiki/infrastructure/containers/image-builds.md index 3c500b6..96ba9ba 100644 --- a/wiki/infrastructure/containers/image-builds.md +++ b/wiki/infrastructure/containers/image-builds.md @@ -8,7 +8,7 @@ sources: - https://docs.docker.com/build/building/best-practices/ - https://docs.docker.com/build/building/secrets/ last_verified: 2026-07-10 -related: [infrastructure-ci-cd-pipeline-structure, infrastructure-ci-cd-secrets-handling] +related: [infrastructure-ci-cd-pipeline-structure, infrastructure-ci-cd-secrets-handling, infrastructure-containers-entrypoint-log-capture] --- # Structuring Container Image Builds diff --git a/wiki/infrastructure/containers/resource-limits-and-probes.md b/wiki/infrastructure/containers/resource-limits-and-probes.md index 2afe8ef..106251e 100644 --- a/wiki/infrastructure/containers/resource-limits-and-probes.md +++ b/wiki/infrastructure/containers/resource-limits-and-probes.md @@ -9,7 +9,7 @@ sources: - https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ - https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/ last_verified: 2026-07-10 -related: [infrastructure-deploy-rollout-and-rollback, backend-java-runtime-threads-and-memory] +related: [infrastructure-deploy-rollout-and-rollback, backend-java-runtime-threads-and-memory, infrastructure-observability-kubelet-container-metrics] --- # Resource Limits and Health Probes in Deployment Manifests diff --git a/wiki/infrastructure/index.md b/wiki/infrastructure/index.md index 4940602..1cdc792 100644 --- a/wiki/infrastructure/index.md +++ b/wiki/infrastructure/index.md @@ -27,6 +27,7 @@ Match your situation to a "load when" line; load only matching pages. |------|-----------| | [image-builds](containers/image-builds.md) | Writing or reviewing a Dockerfile; images rebuild everything on small changes, build slowly, or are too large; choosing an image tagging scheme | | [resource-limits-and-probes](containers/resource-limits-and-probes.md) | Writing or reviewing Kubernetes-style deployment manifests; pods OOMKilled, evicted, or CPU-throttled; a dependency outage triggered a restart storm; traffic hitting pods that are not ready | +| [entrypoint-log-capture](containers/entrypoint-log-capture.md) | A container entrypoint bash script tees its output to a log file and lines are missing (short jobs capture nothing, long jobs lose the tail); wiring stdout duplication or signal traps into a PID-1 shell script | ## data @@ -46,3 +47,4 @@ Match your situation to a "load when" line; load only matching pages. |------|-----------| | [logs-metrics-signals](observability/logs-metrics-signals.md) | Instrumenting a new or existing service (logs, metrics, correlation ids); an incident revealed you couldn't see what happened; choosing between a log line and a metric; a metric label would carry unbounded values (user ids/UUIDs) | | [alerting](observability/alerting.md) | Creating or reviewing alerts; the team ignores a noisy pager; deciding whether a condition pages, tickets, or stays on a dashboard | +| [kubelet-container-metrics](observability/kubelet-container-metrics.md) | Prometheus kubelet targets are `up` but `container_*` CPU/memory queries return nothing (common on OrbStack/k3s/kind); configuring kube-prometheus-stack kubelet scraping; built-in Compute Resources dashboards blank while PromQL returns data | diff --git a/wiki/infrastructure/observability/kubelet-container-metrics.md b/wiki/infrastructure/observability/kubelet-container-metrics.md new file mode 100644 index 0000000..5166c91 --- /dev/null +++ b/wiki/infrastructure/observability/kubelet-container-metrics.md @@ -0,0 +1,79 @@ +--- +id: infrastructure-observability-kubelet-container-metrics +domain: infrastructure +category: observability +applies_to: [kubernetes, prometheus] +confidence: verified +sources: + - https://kubernetes.io/docs/reference/instrumentation/node-metrics/ + - https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml + - https://github.com/kubernetes-monitoring/kubernetes-mixin/blob/master/dashboards/resources/queries/pod.libsonnet +last_verified: 2026-07-15 +related: [infrastructure-observability-logs-metrics-signals, infrastructure-containers-resource-limits-and-probes] +--- + +# Empty container_* Metrics Though Kubelet Targets Are Up (kube-prometheus-stack) + +## When this applies + +Prometheus (kube-prometheus-stack) shows every kubelet target as `up`, but +`container_cpu_*` / `container_memory_*` queries return nothing — typically on a +lightweight or embedded Kubernetes (OrbStack, k3s, kind, colima) whose kubelet +cAdvisor endpoint does not produce per-container series. + +## Do this + +1. Confirm what cAdvisor actually emits — target health cannot detect this, + because the scrape itself succeeds while returning only `machine_*` series: + + ```sh + kubectl get --raw /api/v1/nodes//proxy/metrics/cadvisor | grep -c '^container_' + ``` + + 0 matches with a healthy target means the source is empty, not the scrape. + +2. Scrape the kubelet's resource-metrics endpoint instead. In + kube-prometheus-stack values, two lines: + + ```yaml + kubelet: + serviceMonitor: + resource: true + resourcePath: "/metrics/resource" + ``` + + `resourcePath` must be set explicitly: the chart default is the pre-1.18 path + `/metrics/resource/v1alpha1` (the chart's own comment notes the rename), which + 404s on current kubelets and leaves the new target down. + +3. Verify end to end: the `kubelet` target for the resource path is `up`, and + `container_memory_working_set_bytes` returns live values in PromQL. + +4. View these series on a custom dashboard (or panels without an `image` filter). + `/metrics/resource` series carry a different label set than cAdvisor — notably + no `image` label — and the bundled kubernetes-mixin "Compute Resources" + dashboards filter on `image!=""`, which silently drops every resource-endpoint + series. + +## Edge cases + +| Case | Then | +|------|------| +| PromQL returns data but the built-in dashboards stay blank | That is the `image!=""` (and `container!=""`) mixin filter, not a scrape problem — chart the same query without the image filter | +| cAdvisor emits `container_*` fine on the same cluster | Keep cAdvisor as the source; `/metrics/resource` has fewer series (CPU/memory only — no network, filesystem, or throttling metrics) | +| Recording rules / alerts reference cAdvisor label sets | Rules built on `image`/`id` labels will not match resource-endpoint series — rewrite them against the reduced label set before trusting the alerts | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Judge metric availability from target health (`up == 1`) | Curl the endpoint and count the series you need | A scrape can succeed while the exporter emits none of the series you care about | +| Debug the ServiceMonitor because dashboards are empty | Run the dashboard's PromQL by hand first | If data exists, the gap is the dashboard's label filter, not collection | +| Enable `resource: true` alone | Also set `resourcePath: "/metrics/resource"` | The chart default path is the removed v1alpha1 endpoint — the target 404s | + +## Sources + +- https://kubernetes.io/docs/reference/instrumentation/node-metrics/ — `/metrics/resource` is the current kubelet resource-metrics endpoint +- https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml — `kubelet.serviceMonitor.resource` (default `false`) and `resourcePath` (default `/metrics/resource/v1alpha1`, comment: renamed in Kubernetes 1.18) +- https://github.com/kubernetes-monitoring/kubernetes-mixin/blob/master/dashboards/resources/queries/pod.libsonnet — pod resource queries filter `image!=""` and `container!=""` +- Field observation (OrbStack, 2026-07-14): kubelet cAdvisor endpoint served only `machine_*` series (`container_*` count 0, `machine_scrape_error` 0) while targets were `up`; after the two-line change, `container_memory_working_set_bytes` returned live per-pod values diff --git a/wiki/infrastructure/observability/logs-metrics-signals.md b/wiki/infrastructure/observability/logs-metrics-signals.md index 87e9108..282f5a6 100644 --- a/wiki/infrastructure/observability/logs-metrics-signals.md +++ b/wiki/infrastructure/observability/logs-metrics-signals.md @@ -9,7 +9,7 @@ sources: - https://prometheus.io/docs/practices/naming/ - https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html last_verified: 2026-07-10 -related: [infrastructure-observability-alerting] +related: [infrastructure-observability-alerting, infrastructure-observability-kubelet-container-metrics] --- # Instrumenting a Service with Logs and Metrics diff --git a/wiki/platforms/shells/portable-shell-scripts.md b/wiki/platforms/shells/portable-shell-scripts.md index dcab402..718cdc3 100644 --- a/wiki/platforms/shells/portable-shell-scripts.md +++ b/wiki/platforms/shells/portable-shell-scripts.md @@ -11,7 +11,7 @@ sources: - https://google.github.io/styleguide/shellguide.html - https://www.shellcheck.net/ last_verified: 2026-07-10 -related: [platforms-tools-bsd-vs-gnu-cli, platforms-toolchains-version-management] +related: [platforms-tools-bsd-vs-gnu-cli, platforms-toolchains-version-management, infrastructure-containers-entrypoint-log-capture] --- # Shell Scripts That Must Run on More Than One Machine or Shell diff --git a/wiki/platforms/tools/bsd-vs-gnu-cli.md b/wiki/platforms/tools/bsd-vs-gnu-cli.md index 21181ef..8110697 100644 --- a/wiki/platforms/tools/bsd-vs-gnu-cli.md +++ b/wiki/platforms/tools/bsd-vs-gnu-cli.md @@ -9,7 +9,7 @@ sources: - https://man.freebsd.org/cgi/man.cgi?date(1) - https://man.freebsd.org/cgi/man.cgi?sed(1) - https://man.freebsd.org/cgi/man.cgi?seq(1) -last_verified: 2026-07-10 +last_verified: 2026-07-15 related: [platforms-shells-portable-shell-scripts] --- @@ -32,6 +32,7 @@ flags do not. Apply the portable fix per command: | `timeout` | In coreutils | Absent on stock macOS — exits 127 | Install coreutils and call `gtimeout`, or background the command and `kill` it from a sleep watchdog | | `seq -s,` | Separator between numbers only | Separator also emitted after the last number, before the terminator | Join after generating (`seq 1 3 \| paste -sd, -`) and count items by lines, never by separators | | PCRE grep | `grep -P` | Not supported | `grep -E` with ERE, or `perl -ne 'print if /…/'` | +| Sub-second timestamp | `date +%3N` → milliseconds (`%N` and the field width are both GNU extensions) | Recent macOS prints digits for `%N` but emits `%3N` literally as `3N` (timestamps become `…08.3NZ`) | Feature-detect on the actual `%3N` output, not on `%N`: `case "$(date +%3N)" in [0-9][0-9][0-9]) …GNU path… ;; *) …fallback… ;; esac`; or `gdate +%3N`, or settle for epoch seconds | | Resolve path | `readlink -f` | Absent before macOS 13 | `cd "$(dirname "$f")" && pwd -P` for directories, `python3 -c 'import os,sys;print(os.path.realpath(sys.argv[1]))'`, or coreutils `greadlink -f` | | File metadata | `stat -c '%s'` | `stat -f '%z'` | Detect once: `stat -c %s "$f" 2>/dev/null \|\| stat -f %z "$f"` | @@ -49,6 +50,7 @@ General strategy by situation: | Case | Then | |------|------| | `command -v timeout` succeeds on macOS | Someone installed coreutils unprefixed — confirm `timeout --version` reports GNU coreutils before relying on GNU exit-code semantics (124 on timeout) | +| `date +%N` prints digits on macOS | Do not infer GNU userland from one specifier working — width-modified forms (`%3N`) still fail on BSD `date`; feature-detect the exact format string you will use, by its output shape | | Any flags passed to `echo` (`-e`, `-n`) | `echo` flag handling differs across shells and userlands — use `printf` for anything beyond a bare literal string | | Script needs bash 4+ features on macOS | Stock `/bin/bash` on macOS is 3.2 — use `#!/usr/bin/env bash` so a brew-installed bash is picked up, and state the required bash version in the script header | @@ -63,6 +65,9 @@ General strategy by situation: ## Sources - https://www.gnu.org/software/coreutils/manual/html_node/index.html — GNU date/timeout/seq/stat behavior +- https://www.gnu.org/software/coreutils/manual/html_node/Time-conversion-specifiers.html — `%N` (nanoseconds) is a GNU extension +- https://www.gnu.org/software/coreutils/manual/html_node/Padding-and-other-flags.html — field width between `%` and the specifier is a GNU extension +- Field reproduction (macOS 14.8.3 and 26.5.1, 2026-07-15): `date +%N` → digits, `date +%3N` → literal `3N` on both - https://man.freebsd.org/cgi/man.cgi?date(1) — BSD `date -v` adjustment flag - https://man.freebsd.org/cgi/man.cgi?sed(1) — BSD `sed -i` backup-extension argument - https://man.freebsd.org/cgi/man.cgi?seq(1) — BSD `seq -s` separator emitted after the last number (see man-page example) From cc6bb71d1d56a6644f6a94774c17802f23d583c3 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=EC=B5=9C=EC=98=81=EA=B8=B0?= Date: Wed, 15 Jul 2026 13:18:19 +0900 Subject: [PATCH 2/4] knowledge: apply adversarial cross-check corrections (dashboard metrics_path pin, TERM-trap foreground deferral) --- .dev-loop/INGEST_REPORT.md | 5 ++++- log.md | 4 ++-- .../containers/entrypoint-log-capture.md | 22 ++++++++++++++----- .../kubelet-container-metrics.md | 21 ++++++++++-------- 4 files changed, 34 insertions(+), 18 deletions(-) diff --git a/.dev-loop/INGEST_REPORT.md b/.dev-loop/INGEST_REPORT.md index 12c8140..772fc1a 100644 --- a/.dev-loop/INGEST_REPORT.md +++ b/.dev-loop/INGEST_REPORT.md @@ -1,5 +1,7 @@ # Knowledge flush — 3 insight(s) +Cross-Check: 독립 서브에이전트 적대검증 1회 — date 페이지 CONFIRMED(GNU/macOS 양쪽 실측), kubelet 페이지는 대시보드 제외 원인이 `metrics_path="/metrics/cadvisor"` 핀 + `image!=""` 복합임을 적발(수정 반영, swap/start_time 노출도 보정), entrypoint 페이지는 TERM trap 근거가 실증상 반대(foreground 자식 뒤 trap은 지연→SIGKILL 137; 무trap SIGTERM에도 EXIT trap 실행)임을 적발 → background 자식+`wait`+시그널 포워딩 패턴으로 재작성. 재현 아티팩트: bash 4.4/5.2 PID-1 docker 실험, 업스트림 소스 대조. + ## Verified best-practice **1. Empty `container_*` metrics though kubelet targets are up → scrape `/metrics/resource` (kube-prometheus-stack).** @@ -10,7 +12,8 @@ Verified against: - kubernetes-mixin source read via `gh api` (`dashboards/resources/queries/pod.libsonnet`) — pod CPU/memory queries all carry `image!=""` (and `container!=""`); `/metrics/resource` series have no `image` label. The OrbStack-specific behavior (cAdvisor emitting only `machine_*`) has no external doc; it is marked as a dated field observation in the page. Overall **confidence: verified** (mechanism fully doc-backed + session reproduction with PromQL round-trip). -**2. PID-1 entrypoint bash with `exec > >(tee -a f)` loses log output → EXIT-trap fd close + `wait "$TEE_PID"` + `trap 'exit 143' TERM`.** +**2. PID-1 entrypoint bash with `exec > >(tee -a f)` loses log output → EXIT-trap fd close + `wait "$TEE_PID"`; long foreground jobs need background-child + `wait` + TERM forwarding.** +Note: the queued candidate's original `trap 'exit 143' TERM` directive was **corrected during cross-check** — adversarial reproduction (bash 4.4/5.2 as PID 1) showed bash runs the EXIT trap on an untrapped SIGTERM anyway, while a TERM trap behind a foreground child is deferred until SIGKILL (exit 137, nothing flushed). The page teaches the working pattern instead. Verified against: - https://mywiki.wooledge.org/ProcessSubstitution — process substitution "will continue to run when your script exits (unless you manage your child processes)"; since bash 4.4 it can be managed with `wait "$!"`. In a container the PID-1 exit tears the orphan down before flush instead of letting it finish. - https://tiswww.case.edu/php/chet/bash/bashref.html — 128+n exit status for signal-terminated commands (143 = SIGTERM). diff --git a/log.md b/log.md index 41a31ac..4fbe71a 100644 --- a/log.md +++ b/log.md @@ -32,6 +32,6 @@ Append-only. Format: `## [YYYY-MM-DD] >(tee …) loses log output because the shell never waits for process substitutions and the container dies with PID 1; fix is EXIT-trap fd close + wait "$TEE_PID" plus trap 'exit 143' TERM. Sources: Wooledge ProcessSubstitution (wait "$!" since bash 4.4), bash ref manual 128+n; field reproduction 0/10 → 10/10 +## [2026-07-15] ingest | infrastructure-observability-kubelet-container-metrics — empty container_* with healthy kubelet targets: verify what cAdvisor emits, enable kps kubelet.serviceMonitor.resource + explicit resourcePath: /metrics/resource (chart default is the removed v1alpha1 path), and expect the bundled dashboards to drop these series (they pin metrics_path="/metrics/cadvisor" and filter image!="" — resource-endpoint series fail both). Mechanism verified against kps values.yaml, kubernetes.io node-metrics doc, kubernetes-mixin query source; OrbStack behavior field-observed +## [2026-07-15] ingest | infrastructure-containers-entrypoint-log-capture — PID-1 entrypoint bash with exec > >(tee …) loses log output because the shell never waits for process substitutions and the container dies with PID 1; fix is EXIT-trap fd close + wait "$TEE_PID"; for long foreground jobs, background-child + wait + TERM forwarding (a bare TERM trap behind a foreground child is deferred until SIGKILL — adversarial cross-check corrected the queued directive). Sources: Wooledge ProcessSubstitution (wait "$!" since bash 4.4), bash ref manual 128+n; field reproduction 0/10 → 10/10 ## [2026-07-15] revise | platforms-tools-bsd-vs-gnu-cli +1 row +1 edge case: sub-second timestamps — recent macOS date prints digits for %N but emits %3N literally as "3N", so feature-detect on the actual %3N output shape, never on %N presence. GNU manual confirms %N and field width are GNU extensions; reproduced on macOS 14.8.3 and 26.5.1. last_verified bumped to 2026-07-15 diff --git a/wiki/infrastructure/containers/entrypoint-log-capture.md b/wiki/infrastructure/containers/entrypoint-log-capture.md index 992ce0e..a327056 100644 --- a/wiki/infrastructure/containers/entrypoint-log-capture.md +++ b/wiki/infrastructure/containers/entrypoint-log-capture.md @@ -37,14 +37,22 @@ capture nothing, long jobs lose the tail (the completion-summary lines). trap flush_logs EXIT ``` -2. Also trap TERM so a `docker stop` / pod deletion goes through the same path: +2. For a script whose main work is a long foreground job, run that job as a + background child and `wait` it before adding any TERM trap. Bash defers + user traps until the foreground command completes — a TERM trap behind a + foreground child never fires on `docker stop`, the grace period expires, + and SIGKILL ends the container at exit 137 with nothing flushed. `wait` + is interruptible by traps, so this shape reacts immediately: ```bash - trap 'exit 143' TERM # 128+15, the conventional SIGTERM exit status + trap 'kill -TERM "$MAIN_PID" 2>/dev/null; wait "$MAIN_PID"; exit 143' TERM + main_job & MAIN_PID=$! + wait "$MAIN_PID" ``` - Converting the signal into an `exit` guarantees the EXIT trap (and the tee - flush) runs, and reports the conventional 128+n code. + `exit 143` (128+15) reports the conventional SIGTERM status. For simple + sequential scripts, no TERM trap is needed for the flush itself — bash runs + the EXIT trap on an untrapped SIGTERM in non-interactive shells. 3. Verify by re-running the container several times and checking the log file for the final line each run — this failure is timing-dependent, so a single @@ -55,7 +63,7 @@ capture nothing, long jobs lose the tail (the completion-summary lines). | Case | Then | |------|------| | Base image bash is older than 4.4 (`wait "$!"` on a process substitution fails) | Use a named pipe: `mkfifo`, start `tee < pipe &` as a normal background job, `exec > pipe`, and `wait` its real PID | -| Script also traps other signals (INT, HUP) | Route each through `exit 128+n` the same way, so every termination path reaches the EXIT trap | +| Script also traps other signals (INT, HUP) | End each handler with `exit 128+n`, so every trapped path reports its conventional signal code and reaches the EXIT trap | | Logs must survive even a SIGKILL (OOMKill, forced deletion) | No trap runs on KILL — write to a mounted volume via unbuffered append (or ship lines as they are produced) instead of relying on exit-time flushing | ## Instead of @@ -65,9 +73,11 @@ capture nothing, long jobs lose the tail (the completion-summary lines). | Trust `exec > >(tee -a f)` because it works in a terminal | Add the EXIT-trap wait before it runs as PID 1 | The shell never waits for process substitutions; a terminal leaves the orphan running to finish, a container kills it at PID 1 exit | | Debug the missing lines as a tee buffering option problem | Reproduce with repeated short runs and add the trap | The writer is killed before flushing — buffering flags don't fix a killed process | | Exit 0 from the TERM trap for a "clean" shutdown | `exit 143` (128+15) | Orchestrators and CI distinguish signalled termination from success by the 128+n convention | +| Add `trap 'exit 143' TERM` while the main job runs in the foreground | Background the job, `wait` it, and forward TERM from the trap (step 2) | Bash defers traps until the foreground command finishes — the trap never fires, the grace period ends in SIGKILL (exit 137), and nothing flushes | ## Sources - https://mywiki.wooledge.org/ProcessSubstitution — process substitution "will continue to run when your script exits (unless you manage your child processes)"; since bash 4.4 it can be managed with `wait "$!"` -- https://tiswww.case.edu/php/chet/bash/bashref.html — exit status is 128+n for a command terminated by signal n +- https://tiswww.case.edu/php/chet/bash/bashref.html — exit status is 128+n for a command terminated by signal n; trap execution is deferred until the running foreground command completes - Field reproduction (OrbStack container, 2026-07-15): without the trap the log captured the script output in 0 of 10 runs; with the EXIT-trap wait, 10 of 10 +- Independent adversarial re-reproduction (docker bash 4.4 and 5.2 as PID 1, 2026-07-15): tail captured 1/10 without the trap vs 10/10 with it; untrapped `docker stop` ran the EXIT trap (exit 143); `trap 'exit 143' TERM` behind a foreground child was deferred until SIGKILL (exit 137, no flush) diff --git a/wiki/infrastructure/observability/kubelet-container-metrics.md b/wiki/infrastructure/observability/kubelet-container-metrics.md index 5166c91..1433a11 100644 --- a/wiki/infrastructure/observability/kubelet-container-metrics.md +++ b/wiki/infrastructure/observability/kubelet-container-metrics.md @@ -49,18 +49,20 @@ cAdvisor endpoint does not produce per-container series. 3. Verify end to end: the `kubelet` target for the resource path is `up`, and `container_memory_working_set_bytes` returns live values in PromQL. -4. View these series on a custom dashboard (or panels without an `image` filter). - `/metrics/resource` series carry a different label set than cAdvisor — notably - no `image` label — and the bundled kubernetes-mixin "Compute Resources" - dashboards filter on `image!=""`, which silently drops every resource-endpoint - series. +4. View these series on custom panels. `/metrics/resource` series carry a + different label set than cAdvisor — no `image` label, and (with the chart's + default relabelings) `metrics_path="/metrics/resource"` — while the bundled + kubernetes-mixin "Compute Resources" dashboards pin + `metrics_path="/metrics/cadvisor"` and filter `image!=""`. Resource-endpoint + series fail both matchers, so those dashboards silently drop them; write + panels with no `metrics_path`/`image` matchers. ## Edge cases | Case | Then | |------|------| -| PromQL returns data but the built-in dashboards stay blank | That is the `image!=""` (and `container!=""`) mixin filter, not a scrape problem — chart the same query without the image filter | -| cAdvisor emits `container_*` fine on the same cluster | Keep cAdvisor as the source; `/metrics/resource` has fewer series (CPU/memory only — no network, filesystem, or throttling metrics) | +| PromQL returns data but the built-in dashboards stay blank | That is the dashboards' `metrics_path="/metrics/cadvisor"` pin plus the `image!=""`/`container!=""` filters, not a scrape problem — chart the query with neither matcher | +| cAdvisor emits `container_*` fine on the same cluster | Keep cAdvisor as the source; `/metrics/resource` has fewer series (CPU/memory/swap usage and start time only — no network, filesystem, or throttling metrics) | | Recording rules / alerts reference cAdvisor label sets | Rules built on `image`/`id` labels will not match resource-endpoint series — rewrite them against the reduced label set before trusting the alerts | ## Instead of @@ -68,12 +70,13 @@ cAdvisor endpoint does not produce per-container series. | If you are about to | Do this instead | Why | |---------------------|-----------------|-----| | Judge metric availability from target health (`up == 1`) | Curl the endpoint and count the series you need | A scrape can succeed while the exporter emits none of the series you care about | -| Debug the ServiceMonitor because dashboards are empty | Run the dashboard's PromQL by hand first | If data exists, the gap is the dashboard's label filter, not collection | +| Debug the ServiceMonitor because dashboards are empty | Run the dashboard's PromQL by hand first | If data exists, the gap is the dashboard's label matchers (`metrics_path`, `image`), not collection | | Enable `resource: true` alone | Also set `resourcePath: "/metrics/resource"` | The chart default path is the removed v1alpha1 endpoint — the target 404s | ## Sources - https://kubernetes.io/docs/reference/instrumentation/node-metrics/ — `/metrics/resource` is the current kubelet resource-metrics endpoint - https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml — `kubelet.serviceMonitor.resource` (default `false`) and `resourcePath` (default `/metrics/resource/v1alpha1`, comment: renamed in Kubernetes 1.18) -- https://github.com/kubernetes-monitoring/kubernetes-mixin/blob/master/dashboards/resources/queries/pod.libsonnet — pod resource queries filter `image!=""` and `container!=""` +- https://github.com/kubernetes-monitoring/kubernetes-mixin/blob/master/dashboards/resources/queries/pod.libsonnet — pod resource queries filter `image!=""` and `container!=""`; the chart's rendered dashboards additionally pin `metrics_path="/metrics/cadvisor"`, and its default `resourceRelabelings` label resource-endpoint series `metrics_path="/metrics/resource"` +- https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/metrics/collectors/resource_metrics.go — resource-endpoint container series are declared with labels `container, pod, namespace` only (no `image`); exposes CPU, memory, swap, and start time - Field observation (OrbStack, 2026-07-14): kubelet cAdvisor endpoint served only `machine_*` series (`container_*` count 0, `machine_scrape_error` 0) while targets were `up`; after the two-line change, `container_memory_working_set_bytes` returned live per-pod values From 5186e8b62bc0a302a55c57fcc8d792a9281dacd8 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=EC=B5=9C=EC=98=81=EA=B8=B0?= Date: Wed, 15 Jul 2026 13:19:26 +0900 Subject: [PATCH 3/4] knowledge: add Decision Log to ingest report --- .dev-loop/INGEST_REPORT.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/.dev-loop/INGEST_REPORT.md b/.dev-loop/INGEST_REPORT.md index 772fc1a..c8c4eb7 100644 --- a/.dev-loop/INGEST_REPORT.md +++ b/.dev-loop/INGEST_REPORT.md @@ -46,3 +46,13 @@ Conflicts flagged: none. | 3 — date `%3N` detection | `platforms/tools/bsd-vs-gnu-cli.md` (merge) | No — exact existing page for BSD-vs-GNU flag differences | Nothing left `unverified`. Queue rows retired to `~/.dev-loop/queue/.processed.jsonl` after PR creation. + +## Decision Log + +- **의도**: dev-loop 세션에서 수확된 ★ Insight 후보 3건을 검증·중복점검·라우팅을 거쳐 위키에 반영 (knowledge-flush 파이프라인, 사용자가 flush 실행 및 PR 1건 생성을 명시 지시). +- **배제한 대안**: + - 인사이트 1·2를 기존 페이지(logs-metrics-signals / portable-shell-scripts)에 병합 — 트리거가 다름(스크레이프 소스 설정 vs 계측 원칙, PID-1 컨테이너 시맨틱 vs 셸 이식성)이라 "one case per page" 원칙에 따라 신규 페이지로 분리. + - 신규 카테고리 생성 — observability/containers/tools 기존 카테고리가 정확히 커버해 불필요. + - 큐 원문 directive 그대로 반영 — 인사이트 2의 `trap 'exit 143' TERM` 단독 처방은 적대검증 실험에서 반박되어(foreground 자식 뒤 trap 지연→SIGKILL) 교정 후 반영. +- **리뷰어가 볼 곳**: `entrypoint-log-capture.md`의 step 2(background-child+wait 패턴 — 교차검증으로 재작성된 부분), `kubelet-container-metrics.md`의 대시보드 제외 원인 서술(metrics_path 핀 + image 필터 복합), `bsd-vs-gnu-cli.md` 추가 행의 case 패턴 분기. +- [추정] RTB 팀 표준 리뷰어 3인은 이 개인 레포의 collaborator가 아닐 수 있어 리뷰어 지정이 GitHub에서 거부될 수 있음 — 그 경우 레포 소유자 리뷰(스킬 기본 모델)로 진행. From 366b0a9b54935ec253441aa81feab3815aebbe31 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=EC=B5=9C=EC=98=81=EA=B8=B0?= Date: Wed, 15 Jul 2026 13:21:59 +0900 Subject: [PATCH 4/4] knowledge: drop kubelet metrics page (duplicate of open PR #2), report 2 insights --- .dev-loop/INGEST_REPORT.md | 66 +++++++-------- log.md | 1 - .../containers/resource-limits-and-probes.md | 2 +- wiki/infrastructure/index.md | 1 - .../kubelet-container-metrics.md | 82 ------------------- .../observability/logs-metrics-signals.md | 2 +- 6 files changed, 32 insertions(+), 122 deletions(-) delete mode 100644 wiki/infrastructure/observability/kubelet-container-metrics.md diff --git a/.dev-loop/INGEST_REPORT.md b/.dev-loop/INGEST_REPORT.md index c8c4eb7..921ff10 100644 --- a/.dev-loop/INGEST_REPORT.md +++ b/.dev-loop/INGEST_REPORT.md @@ -1,58 +1,52 @@ -# Knowledge flush — 3 insight(s) +# Knowledge flush — 2 insight(s) (+1 retired as already flushed) -Cross-Check: 독립 서브에이전트 적대검증 1회 — date 페이지 CONFIRMED(GNU/macOS 양쪽 실측), kubelet 페이지는 대시보드 제외 원인이 `metrics_path="/metrics/cadvisor"` 핀 + `image!=""` 복합임을 적발(수정 반영, swap/start_time 노출도 보정), entrypoint 페이지는 TERM trap 근거가 실증상 반대(foreground 자식 뒤 trap은 지연→SIGKILL 137; 무trap SIGTERM에도 EXIT trap 실행)임을 적발 → background 자식+`wait`+시그널 포워딩 패턴으로 재작성. 재현 아티팩트: bash 4.4/5.2 PID-1 docker 실험, 업스트림 소스 대조. +Cross-Check: 독립 서브에이전트 적대검증 1회 — date 페이지 CONFIRMED(GNU coreutils 컨테이너·macOS 26.5.1 양쪽 실측, case 패턴 분기 검증), entrypoint 페이지는 큐 원문의 TERM trap 근거가 실증상 반대(무trap SIGTERM에도 bash EXIT trap 실행; foreground 자식 뒤 TERM trap은 지연→SIGKILL 137, flush 0건)임을 적발 → background 자식+`wait`+시그널 포워딩 패턴으로 재작성 후 반영. 재현 아티팩트: bash 4.4/5.2 PID-1 docker 실험 각 10회. -## Verified best-practice +## Decision Log -**1. Empty `container_*` metrics though kubelet targets are up → scrape `/metrics/resource` (kube-prometheus-stack).** -Claim: on OrbStack-style embedded k8s the kubelet cAdvisor endpoint emits no `container_*` series while scrapes stay healthy; the fix is `kubelet.serviceMonitor.resource: true` plus an explicit `resourcePath: "/metrics/resource"`, and the mixin Compute Resources dashboards will still drop these series because of their `image!=""` filter. -Verified against: -- https://kubernetes.io/docs/reference/instrumentation/node-metrics/ — `/metrics/resource` is the current kubelet resource-metrics endpoint (metrics-server ≥0.6 uses it). -- https://raw.githubusercontent.com/prometheus-community/helm-charts/main/charts/kube-prometheus-stack/values.yaml — confirmed `resource: false` default and `resourcePath: "/metrics/resource/v1alpha1"` default with the chart's own comment "From kubernetes 1.18, /metrics/resource/v1alpha1 renamed to /metrics/resource" — exactly the 404 trap the insight describes. -- kubernetes-mixin source read via `gh api` (`dashboards/resources/queries/pod.libsonnet`) — pod CPU/memory queries all carry `image!=""` (and `container!=""`); `/metrics/resource` series have no `image` label. -The OrbStack-specific behavior (cAdvisor emitting only `machine_*`) has no external doc; it is marked as a dated field observation in the page. Overall **confidence: verified** (mechanism fully doc-backed + session reproduction with PromQL round-trip). +- **의도**: dev-loop 큐의 pending 인사이트를 검증·중복점검·라우팅해 위키에 반영 (knowledge-flush 파이프라인, 사용자가 flush 실행과 PR 1건 생성을 명시 지시). +- **배제한 대안**: + - 인사이트(kubelet /metrics/resource)의 신규 페이지 생성 — **배제: 열려 있는 PR #2가 동일 인사이트를 `container-metrics-when-cadvisor-is-empty.md`로 이미 반영**(metrics_path 핀 + image!="" 복합 필터까지 커버 확인). 큐 row가 retire되지 않아 재등장한 것 — 이번 flush에서 재반영 없이 retire만 함. 작성했던 중복 페이지는 커밋 이력에서 제거. + - 인사이트(PID-1 tee)를 `platforms/shells/portable-shell-scripts.md`에 병합 — 배제: 트리거가 셸 이식성이 아니라 PID-1 컨테이너 시맨틱이라 `infrastructure/containers` 신규 페이지가 맞음 (one case per page). + - 큐 원문 directive(`trap 'exit 143' TERM` 단독) 그대로 반영 — 배제: 적대검증 실험에서 반박되어 교정 후 반영. +- **리뷰어가 볼 곳**: `entrypoint-log-capture.md` step 2(background-child+wait 패턴 — 교차검증으로 재작성된 부분)와 Sources의 두 독립 재현 기록, `bsd-vs-gnu-cli.md` 추가 행의 case 패턴 분기. `wiki/infrastructure/index.md`는 PR #2도 수정하므로 머지 순서에 따라 사소한 충돌 가능. +- [추정] RTB 팀 표준 리뷰어 3인은 이 개인 레포 collaborator가 아니어서 리뷰어 지정이 거부됨(gh 확인: `d43103-rsquare not found`) — 레포 소유자 리뷰(스킬 기본 모델)로 진행. + +## Verified best-practice -**2. PID-1 entrypoint bash with `exec > >(tee -a f)` loses log output → EXIT-trap fd close + `wait "$TEE_PID"`; long foreground jobs need background-child + `wait` + TERM forwarding.** -Note: the queued candidate's original `trap 'exit 143' TERM` directive was **corrected during cross-check** — adversarial reproduction (bash 4.4/5.2 as PID 1) showed bash runs the EXIT trap on an untrapped SIGTERM anyway, while a TERM trap behind a foreground child is deferred until SIGKILL (exit 137, nothing flushed). The page teaches the working pattern instead. +**1. PID-1 entrypoint bash with `exec > >(tee -a f)` loses log output → EXIT-trap fd close + `wait "$TEE_PID"`; long foreground jobs need background-child + `wait` + TERM forwarding.** Verified against: -- https://mywiki.wooledge.org/ProcessSubstitution — process substitution "will continue to run when your script exits (unless you manage your child processes)"; since bash 4.4 it can be managed with `wait "$!"`. In a container the PID-1 exit tears the orphan down before flush instead of letting it finish. -- https://tiswww.case.edu/php/chet/bash/bashref.html — 128+n exit status for signal-terminated commands (143 = SIGTERM). +- https://mywiki.wooledge.org/ProcessSubstitution — process substitution "will continue to run when your script exits (unless you manage your child processes)"; managed with `wait "$!"` since bash 4.4 (also confirmed in bash CHANGES, bash-4.4 section). +- https://tiswww.case.edu/php/chet/bash/bashref.html — 128+n exit status for signal-terminated commands (143 = SIGTERM); trap execution deferred until the running foreground command completes. - Session reproduction (OrbStack container): without trap 0/10 runs captured output; with trap 10/10. -**confidence: verified**. (gnu.org bash manual rate-limited me with HTTP 429 during the flush, so the mirror + Wooledge are cited instead — no unverified URL was included.) +- Independent adversarial re-reproduction during this flush (docker, bash 4.4 and 5.2 as PID 1): 1/10 without trap vs 10/10 with; untrapped `docker stop` ran the EXIT trap (exit 143); `trap 'exit 143' TERM` behind a foreground child was deferred until SIGKILL (exit 137, no flush). +The queued candidate's original `trap 'exit 143' TERM` directive was **corrected during cross-check** — the page teaches the working background-child pattern instead. **confidence: verified.** (gnu.org bash manual rate-limited with HTTP 429 during this flush; the case.edu mirror + Wooledge are cited instead — no unverified URL included.) -**3. BSD/GNU `date` ms-timestamp feature detection → test the actual `%3N` output shape, not `%N` presence.** +**2. BSD/GNU `date` ms-timestamp feature detection → test the actual `%3N` output shape, not `%N` presence.** Verified against: - https://www.gnu.org/software/coreutils/manual/html_node/Time-conversion-specifiers.html — "%N nanoseconds … This is a GNU extension." - https://www.gnu.org/software/coreutils/manual/html_node/Padding-and-other-flags.html — field width between `%` and the specifier is a GNU extension. -- **Independent reproduction on a second machine during this flush** (macOS 26.5.1, beyond the original 14.8.3 measurement): `date +%N` → `070788000`, `date +%3N` → literal `3N`. The trap is real: `%N` working makes naive detection pass, then `%3N` corrupts timestamps. -**confidence: verified**. +- Independent reproduction on a second machine during this flush (macOS 26.5.1, beyond the original 14.8.3 measurement): `date +%N` → digits, `date +%3N` → literal `3N`. Adversarial re-check additionally ran GNU coreutils (debian container): `date +%3N` → exactly 3 digits, and confirmed the `case … [0-9][0-9][0-9])` pattern takes the GNU path on GNU and falls through on macOS. +**confidence: verified.** + +**Retired without re-ingesting**: the OrbStack kubelet `/metrics/resource` insight (hash `ab44b921a1f3f759`) — already flushed as open **PR #2** (`container-metrics-when-cadvisor-is-empty.md`) but never removed from its session queue file; verified PR #2's page already covers the chart defaults, the label-set difference, and the `metrics_path="/metrics/cadvisor"` + `image!=""` dashboard exclusion, so nothing new to add. ## Existing-layer check -Read: root `INDEX.md`, `wiki/infrastructure/index.md`, `wiki/platforms/index.md`, and every overlapping page: `observability/logs-metrics-signals.md`, `observability/alerting.md` (index line), `containers/resource-limits-and-probes.md`, `containers/image-builds.md`, `shells/portable-shell-scripts.md`, `tools/bsd-vs-gnu-cli.md`. +Read: root `INDEX.md`, `wiki/infrastructure/index.md`, `wiki/platforms/index.md`, and every overlapping page: `observability/logs-metrics-signals.md`, `containers/resource-limits-and-probes.md`, `containers/image-builds.md`, `shells/portable-shell-scripts.md`, `tools/bsd-vs-gnu-cli.md`, plus open PR #2's `container-metrics-when-cadvisor-is-empty.md` (via the PR branch). -- Insight 1: no overlap — `logs-metrics-signals` covers instrumentation principles (signal choice, cardinality), not scrape-source configuration. No conflict. **Created new page**; related-linked both ways to `logs-metrics-signals` and `resource-limits-and-probes`. -- Insight 2: no overlap — `portable-shell-scripts` covers shell portability, `image-builds` covers Dockerfiles; neither covers PID-1 runtime log/signal behavior. No conflict. **Created new page**; related-linked both ways to `image-builds` and `portable-shell-scripts` (cross-domain), one-way to `logs-metrics-signals`. -- Insight 3: direct overlap with `platforms/tools/bsd-vs-gnu-cli.md` (same trigger, same directive family — its "Relative date" row already covers `date -d` vs `-v`). **Merged**: one command-table row (sub-second timestamp), one edge-case row (`%N` printing digits on macOS must not imply GNU), two sources, `last_verified` bumped to 2026-07-15. No new page. +- Insight 1 (PID-1 tee): no overlap — `portable-shell-scripts` covers shell portability, `image-builds` covers Dockerfiles; neither covers PID-1 runtime log/signal behavior. No conflict. **Created new page**; related-linked both ways to `image-builds` and `portable-shell-scripts` (cross-domain), one-way to `logs-metrics-signals`. +- Insight 2 (date `%3N`): direct overlap with `platforms/tools/bsd-vs-gnu-cli.md` (same trigger family — its "Relative date" row already covers `date -d` vs `-v`). **Merged**: one command-table row (sub-second timestamp), one edge-case row (`%N` printing digits on macOS must not imply GNU), two sources, `last_verified` bumped to 2026-07-15. No new page. +- Kubelet insight: duplicate of open PR #2 → dropped (see Decision Log). -Conflicts flagged: none. +Conflicts flagged: none in content. Noted: `wiki/infrastructure/index.md` is also touched by open PR #2 (different rows) — trivial merge-order conflict possible. ## Routing decision | Insight | Target | New category? | |---------|--------|---------------| -| 1 — kubelet container metrics | `infrastructure/observability/kubelet-container-metrics.md` (new page) | No — observability already owns metrics collection; index "load when" added | -| 2 — PID-1 tee log loss | `infrastructure/containers/entrypoint-log-capture.md` (new page) | No — containers owns container-runtime behavior; platforms/shells was rejected because the trigger is PID-1 container semantics, not shell portability | -| 3 — date `%3N` detection | `platforms/tools/bsd-vs-gnu-cli.md` (merge) | No — exact existing page for BSD-vs-GNU flag differences | +| PID-1 tee log loss | `infrastructure/containers/entrypoint-log-capture.md` (new page) | No — containers owns container-runtime behavior; platforms/shells rejected because the trigger is PID-1 container semantics, not shell portability | +| date `%3N` detection | `platforms/tools/bsd-vs-gnu-cli.md` (merge) | No — exact existing page for BSD-vs-GNU flag differences | +| kubelet `/metrics/resource` | — (retired; already in open PR #2) | — | Nothing left `unverified`. Queue rows retired to `~/.dev-loop/queue/.processed.jsonl` after PR creation. - -## Decision Log - -- **의도**: dev-loop 세션에서 수확된 ★ Insight 후보 3건을 검증·중복점검·라우팅을 거쳐 위키에 반영 (knowledge-flush 파이프라인, 사용자가 flush 실행 및 PR 1건 생성을 명시 지시). -- **배제한 대안**: - - 인사이트 1·2를 기존 페이지(logs-metrics-signals / portable-shell-scripts)에 병합 — 트리거가 다름(스크레이프 소스 설정 vs 계측 원칙, PID-1 컨테이너 시맨틱 vs 셸 이식성)이라 "one case per page" 원칙에 따라 신규 페이지로 분리. - - 신규 카테고리 생성 — observability/containers/tools 기존 카테고리가 정확히 커버해 불필요. - - 큐 원문 directive 그대로 반영 — 인사이트 2의 `trap 'exit 143' TERM` 단독 처방은 적대검증 실험에서 반박되어(foreground 자식 뒤 trap 지연→SIGKILL) 교정 후 반영. -- **리뷰어가 볼 곳**: `entrypoint-log-capture.md`의 step 2(background-child+wait 패턴 — 교차검증으로 재작성된 부분), `kubelet-container-metrics.md`의 대시보드 제외 원인 서술(metrics_path 핀 + image 필터 복합), `bsd-vs-gnu-cli.md` 추가 행의 case 패턴 분기. -- [추정] RTB 팀 표준 리뷰어 3인은 이 개인 레포의 collaborator가 아닐 수 있어 리뷰어 지정이 GitHub에서 거부될 수 있음 — 그 경우 레포 소유자 리뷰(스킬 기본 모델)로 진행. diff --git a/log.md b/log.md index 4fbe71a..695aa64 100644 --- a/log.md +++ b/log.md @@ -32,6 +32,5 @@ Append-only. Format: `## [YYYY-MM-DD] >(tee …) loses log output because the shell never waits for process substitutions and the container dies with PID 1; fix is EXIT-trap fd close + wait "$TEE_PID"; for long foreground jobs, background-child + wait + TERM forwarding (a bare TERM trap behind a foreground child is deferred until SIGKILL — adversarial cross-check corrected the queued directive). Sources: Wooledge ProcessSubstitution (wait "$!" since bash 4.4), bash ref manual 128+n; field reproduction 0/10 → 10/10 ## [2026-07-15] revise | platforms-tools-bsd-vs-gnu-cli +1 row +1 edge case: sub-second timestamps — recent macOS date prints digits for %N but emits %3N literally as "3N", so feature-detect on the actual %3N output shape, never on %N presence. GNU manual confirms %N and field width are GNU extensions; reproduced on macOS 14.8.3 and 26.5.1. last_verified bumped to 2026-07-15 diff --git a/wiki/infrastructure/containers/resource-limits-and-probes.md b/wiki/infrastructure/containers/resource-limits-and-probes.md index 106251e..2afe8ef 100644 --- a/wiki/infrastructure/containers/resource-limits-and-probes.md +++ b/wiki/infrastructure/containers/resource-limits-and-probes.md @@ -9,7 +9,7 @@ sources: - https://kubernetes.io/docs/concepts/configuration/manage-resources-containers/ - https://kubernetes.io/docs/concepts/workloads/pods/pod-qos/ last_verified: 2026-07-10 -related: [infrastructure-deploy-rollout-and-rollback, backend-java-runtime-threads-and-memory, infrastructure-observability-kubelet-container-metrics] +related: [infrastructure-deploy-rollout-and-rollback, backend-java-runtime-threads-and-memory] --- # Resource Limits and Health Probes in Deployment Manifests diff --git a/wiki/infrastructure/index.md b/wiki/infrastructure/index.md index 1cdc792..13e9abb 100644 --- a/wiki/infrastructure/index.md +++ b/wiki/infrastructure/index.md @@ -47,4 +47,3 @@ Match your situation to a "load when" line; load only matching pages. |------|-----------| | [logs-metrics-signals](observability/logs-metrics-signals.md) | Instrumenting a new or existing service (logs, metrics, correlation ids); an incident revealed you couldn't see what happened; choosing between a log line and a metric; a metric label would carry unbounded values (user ids/UUIDs) | | [alerting](observability/alerting.md) | Creating or reviewing alerts; the team ignores a noisy pager; deciding whether a condition pages, tickets, or stays on a dashboard | -| [kubelet-container-metrics](observability/kubelet-container-metrics.md) | Prometheus kubelet targets are `up` but `container_*` CPU/memory queries return nothing (common on OrbStack/k3s/kind); configuring kube-prometheus-stack kubelet scraping; built-in Compute Resources dashboards blank while PromQL returns data | diff --git a/wiki/infrastructure/observability/kubelet-container-metrics.md b/wiki/infrastructure/observability/kubelet-container-metrics.md deleted file mode 100644 index 1433a11..0000000 --- a/wiki/infrastructure/observability/kubelet-container-metrics.md +++ /dev/null @@ -1,82 +0,0 @@ ---- -id: infrastructure-observability-kubelet-container-metrics -domain: infrastructure -category: observability -applies_to: [kubernetes, prometheus] -confidence: verified -sources: - - https://kubernetes.io/docs/reference/instrumentation/node-metrics/ - - https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml - - https://github.com/kubernetes-monitoring/kubernetes-mixin/blob/master/dashboards/resources/queries/pod.libsonnet -last_verified: 2026-07-15 -related: [infrastructure-observability-logs-metrics-signals, infrastructure-containers-resource-limits-and-probes] ---- - -# Empty container_* Metrics Though Kubelet Targets Are Up (kube-prometheus-stack) - -## When this applies - -Prometheus (kube-prometheus-stack) shows every kubelet target as `up`, but -`container_cpu_*` / `container_memory_*` queries return nothing — typically on a -lightweight or embedded Kubernetes (OrbStack, k3s, kind, colima) whose kubelet -cAdvisor endpoint does not produce per-container series. - -## Do this - -1. Confirm what cAdvisor actually emits — target health cannot detect this, - because the scrape itself succeeds while returning only `machine_*` series: - - ```sh - kubectl get --raw /api/v1/nodes//proxy/metrics/cadvisor | grep -c '^container_' - ``` - - 0 matches with a healthy target means the source is empty, not the scrape. - -2. Scrape the kubelet's resource-metrics endpoint instead. In - kube-prometheus-stack values, two lines: - - ```yaml - kubelet: - serviceMonitor: - resource: true - resourcePath: "/metrics/resource" - ``` - - `resourcePath` must be set explicitly: the chart default is the pre-1.18 path - `/metrics/resource/v1alpha1` (the chart's own comment notes the rename), which - 404s on current kubelets and leaves the new target down. - -3. Verify end to end: the `kubelet` target for the resource path is `up`, and - `container_memory_working_set_bytes` returns live values in PromQL. - -4. View these series on custom panels. `/metrics/resource` series carry a - different label set than cAdvisor — no `image` label, and (with the chart's - default relabelings) `metrics_path="/metrics/resource"` — while the bundled - kubernetes-mixin "Compute Resources" dashboards pin - `metrics_path="/metrics/cadvisor"` and filter `image!=""`. Resource-endpoint - series fail both matchers, so those dashboards silently drop them; write - panels with no `metrics_path`/`image` matchers. - -## Edge cases - -| Case | Then | -|------|------| -| PromQL returns data but the built-in dashboards stay blank | That is the dashboards' `metrics_path="/metrics/cadvisor"` pin plus the `image!=""`/`container!=""` filters, not a scrape problem — chart the query with neither matcher | -| cAdvisor emits `container_*` fine on the same cluster | Keep cAdvisor as the source; `/metrics/resource` has fewer series (CPU/memory/swap usage and start time only — no network, filesystem, or throttling metrics) | -| Recording rules / alerts reference cAdvisor label sets | Rules built on `image`/`id` labels will not match resource-endpoint series — rewrite them against the reduced label set before trusting the alerts | - -## Instead of - -| If you are about to | Do this instead | Why | -|---------------------|-----------------|-----| -| Judge metric availability from target health (`up == 1`) | Curl the endpoint and count the series you need | A scrape can succeed while the exporter emits none of the series you care about | -| Debug the ServiceMonitor because dashboards are empty | Run the dashboard's PromQL by hand first | If data exists, the gap is the dashboard's label matchers (`metrics_path`, `image`), not collection | -| Enable `resource: true` alone | Also set `resourcePath: "/metrics/resource"` | The chart default path is the removed v1alpha1 endpoint — the target 404s | - -## Sources - -- https://kubernetes.io/docs/reference/instrumentation/node-metrics/ — `/metrics/resource` is the current kubelet resource-metrics endpoint -- https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml — `kubelet.serviceMonitor.resource` (default `false`) and `resourcePath` (default `/metrics/resource/v1alpha1`, comment: renamed in Kubernetes 1.18) -- https://github.com/kubernetes-monitoring/kubernetes-mixin/blob/master/dashboards/resources/queries/pod.libsonnet — pod resource queries filter `image!=""` and `container!=""`; the chart's rendered dashboards additionally pin `metrics_path="/metrics/cadvisor"`, and its default `resourceRelabelings` label resource-endpoint series `metrics_path="/metrics/resource"` -- https://github.com/kubernetes/kubernetes/blob/master/pkg/kubelet/metrics/collectors/resource_metrics.go — resource-endpoint container series are declared with labels `container, pod, namespace` only (no `image`); exposes CPU, memory, swap, and start time -- Field observation (OrbStack, 2026-07-14): kubelet cAdvisor endpoint served only `machine_*` series (`container_*` count 0, `machine_scrape_error` 0) while targets were `up`; after the two-line change, `container_memory_working_set_bytes` returned live per-pod values diff --git a/wiki/infrastructure/observability/logs-metrics-signals.md b/wiki/infrastructure/observability/logs-metrics-signals.md index 282f5a6..87e9108 100644 --- a/wiki/infrastructure/observability/logs-metrics-signals.md +++ b/wiki/infrastructure/observability/logs-metrics-signals.md @@ -9,7 +9,7 @@ sources: - https://prometheus.io/docs/practices/naming/ - https://cheatsheetseries.owasp.org/cheatsheets/Logging_Cheat_Sheet.html last_verified: 2026-07-10 -related: [infrastructure-observability-alerting, infrastructure-observability-kubelet-container-metrics] +related: [infrastructure-observability-alerting] --- # Instrumenting a Service with Logs and Metrics