choiyounggi · dch0202-rsquare · Jul 15, 2026 · Jul 15, 2026
@@ -0,0 +1,84 @@
+# Knowledge flush — 3 insight(s)
+
+Drained 3 queued `★ Insight` candidates. Each was independently researched and
+verified against primary sources before ingest. Two new pages created, one
+existing page extended. No auto-merge — this PR is for review.
+
+Cross-Check: each of the 3 candidates was verified by an independent research
+subagent against primary sources (chart values.yaml / k8s docs / kubernetes-mixin;
+Greg's Wiki + pipe(7) + Docker/bash manuals; GNU + FreeBSD date + POSIX strftime).
+All three mechanisms VERIFIED; corrections folded in (mixin also pins
+`job=cadvisor` not just `image!=""`; process-sub `$!` is bash 4.4+ not 5.1) and
+field-observed specifics (OrbStack root cause, literal `3N`) labeled as such.
+
+## Verified best-practice
+
+### 1. Empty cAdvisor `container_*` metrics on a lightweight kubelet → scrape `/metrics/resource`
+- **Claim:** With kube-prometheus-stack, pod CPU/memory series are empty though
+  kubelet targets are UP; fix by enabling `kubelet.serviceMonitor.resource: true`
+  and overriding `resourcePath: /metrics/resource`; mixin dashboards hide the
+  resource series via `image!=""`.
+- **Sources checked (real):**
+  - kps chart values.yaml — confirms both keys, `resource` default **false**, and `resourcePath` default is the stale **`/metrics/resource/v1alpha1`**: https://github.com/prometheus-community/helm-charts/blob/main/charts/kube-prometheus-stack/values.yaml
+  - k8s metrics docs — `/metrics/resource` is the current STABLE endpoint; series carry `namespace`/`pod`/`container` only (no `image`): https://kubernetes.io/docs/reference/instrumentation/metrics/ , https://kubernetes.io/docs/concepts/cluster-administration/system-metrics/
+  - k8s PR#86282 — the `v1alpha1` path was removed (404 on kubelet ≥1.20): https://github.com/kubernetes/kubernetes/pull/86282
+  - kubernetes-mixin — Compute Resources dashboards select cAdvisor series with `image!=""` **and** `job="cadvisor"`/`metrics_path="/metrics/cadvisor"`: https://github.com/kubernetes-monitoring/kubernetes-mixin
+- **Verification result:** mechanism + fix **VERIFIED** by primary sources. **Correction folded in:** `image!=""` is not the *only* reason the resource series are excluded — the mixin also pins `job=cadvisor`/`metrics_path=/metrics/cadvisor`, so a custom panel is required regardless (the page states this, not "just drop the image filter"). The OrbStack-specific root cause (cAdvisor emitting only `machine_*`) is **field-observed**, not documented upstream — labeled as such in the page and log.
+- **Confidence: verified** (fix is kubelet-endpoint-level and independent of why cAdvisor is empty).
+
+### 2. Container PID 1 entrypoint loses piped logs on exit → wait for tee in an EXIT trap
+- **Claim:** `exec > >(tee -a file)` in a PID 1 bash entrypoint loses log output on
+  container exit; capture `TEE_PID=$!` and `wait` it in an EXIT trap after closing
+  fds; add `trap 'exit 143' TERM`.
+- **Sources checked (real):**
+  - Greg's Wiki — process substitution is an **unwaited** background subshell; bash sets `$!` to its PID so you can `wait`: https://mywiki.wooledge.org/ProcessSubstitution
+  - Docker stop — SIGTERM then SIGKILL after grace period: https://docs.docker.com/reference/cli/docker/container/stop/
+  - pipe(7) — a read sees EOF only when **all** write-end fds are closed: https://www.man7.org/linux/man-pages/man7/pipe.7.html
+  - bash manual — signal-killed command exits 128+N (SIGTERM 15 → 143): https://www.gnu.org/software/bash/manual/html_node/Exit-Status.html
+  - PID 1 signal write-up — PID 1 ignores SIGTERM without a handler; orphans reaped only if PID 1 waits: https://petermalmgren.com/signal-handling-docker/
+- **Verification result:** all four mechanisms **VERIFIED**. **Correction folded in:** capturing the process-substitution PID via `$!` is **bash 4.4+** (2016), NOT 5.1 as the raw candidate implied — the page states 4.4+ and gives a named-FIFO fallback for older bash. EOF caveat (all write ends must close) captured as an edge case.
+- **Confidence: verified** (documented mechanism) — the original 10/10 capture measurement is noted as field evidence.
+
+### 3. Feature-detect `date +%3N` by 3-digit output, not by `%N` support
+- **Claim:** Detecting GNU vs BSD `date` by "does `%N` work" is wrong because modern
+  macOS supports `%N`; check that `date +%3N` outputs three digits.
+- **Sources checked (real):**
+  - GNU coreutils — `%N` nanoseconds + numeric field-width form: https://www.gnu.org/software/coreutils/manual/html_node/Options-for-date.html
+  - FreeBSD/macOS `date` — `%N` added in FreeBSD 14.1, GNU-compatible on modern macOS; no `%3N` width form documented: https://man.freebsd.org/cgi/man.cgi?query=date
+  - POSIX strftime — an unrecognized conversion specification is **undefined** (BSD passes it through literally): https://pubs.opengroup.org/onlinepubs/9699919799/functions/strftime.html
+- **Verification result:** core advice **VERIFIED and strengthened** — modern macOS returns real nanoseconds for `%N`, making it an even worse discriminator than the candidate claimed. Two specifics are **field-observed**: the exact `%3N` → literal `3N` output (consistent with POSIX "undefined", not documented as `3N`), and GNU `%3N` = "first 3 digits" (width mechanism documented; the truncation semantics universally relied on but not spelled out). The page frames `3N` as field-observed.
+- **Confidence: verified** (recommendation robust; two specifics noted as field-observed).
+
+## Existing-layer check
+
+- Read all `infrastructure/observability/` pages (`logs-metrics-signals`,
+  `alerting`) and all `infrastructure/containers/` pages (`image-builds`,
+  `resource-limits-and-probes`), plus `platforms/tools/bsd-vs-gnu-cli`,
+  `platforms/shells/portable-shell-scripts`, `platforms/environment/timezone-and-locale`.
+- **#1 (kubelet metrics):** `logs-metrics-signals` covers *what/how to instrument*
+  (signal choice, cardinality, golden signals) — a different trigger from "a
+  specific series is empty despite a healthy scrape." Merging would blur that
+  page's focus, so **created a new page** and cross-linked both ways
+  (`related:`). No conflict.
+- **#2 (PID 1 tee):** the decisive trigger condition is *container PID 1*, not OS
+  portability, so it does not belong in `platforms/shells/portable-shell-scripts`
+  (which already covers process substitution/`set -e`/EXIT traps for the
+  portability case). **Created a new page** under `infrastructure/containers`
+  (owns the PID 1 lifecycle artifact) and added a `related:` link both ways to
+  the shell page. No conflict.
+- **#3 (`date %3N`):** directly overlaps the existing `date` guidance in
+  `bsd-vs-gnu-cli` → **merged** (one Do-table row + one Instead-of row + two
+  sources + `last_verified` bump). No new page. No conflict with the existing
+  "Relative date" row (distinct sub-case).
+
+## Routing decision
+
+| Insight | Target | Action |
+|---------|--------|--------|
+| #1 kubelet resource metrics | `infrastructure/observability/kubelet-resource-metrics.md` (new page) | Created; registered in `infrastructure/index.md` observability table; `related:` ↔ `logs-metrics-signals` |
+| #2 PID 1 tee log flush | `infrastructure/containers/pid1-entrypoint-log-flush.md` (new page) | Created; registered in `infrastructure/index.md` containers table; `related:` ↔ `portable-shell-scripts`, → `resource-limits-and-probes`, `logs-metrics-signals` |
+| #3 `date +%3N` detection | `platforms/tools/bsd-vs-gnu-cli.md` (existing) | Merged one Do-row + one Instead-of row; added GNU date + POSIX strftime sources; `last_verified` → 2026-07-15 |
+
+No new category was needed — both new pages fit existing categories
+(`infrastructure/observability`, `infrastructure/containers`). `log.md` has three
+appended entries (2× ingest, 1× revise).
@@ -32,3 +32,6 @@ Append-only. Format: `## [YYYY-MM-DD] <ingest|revise|lint|gap|contradiction|drif
 ## [2026-07-11] ingest | security +1 (new category api-exposure): exposing-an-origin-http-api — code-vs-edge control split for putting an origin HTTP API behind a public tunnel/proxy. Captures the non-obvious 5xx-bypasses-request-middleware gotcha (set security headers in the exception handler too — verify with a forced 500), header-only fail-closed auth, docs/OpenAPI off at the origin by default, sanitized 5xx bodies, and the code(validate/headers/errors)-vs-edge(TLS/HSTS/rate-limit/WAF) boundary. Derived from hardening korea-data-suite for external exposure + independent pentest; sources OWASP Secure Headers, MDN, Starlette middleware, FastAPI docs.
 ## [2026-07-12] ingest | databases +1 (new category sqlite): concurrent-access-for-a-read-api — WAL + busy_timeout for a read-heavy API with a background writer; set journal_mode once (per-connection pragma cost ~0.4ms measured vs 0.04ms busy_timeout), single writer, run the batch writer as a separate process (interpreter-lock isolation), scale reads with worker processes not threads. Derived from load-testing korea-data-suite (measured connect/query/pragma costs, GIL-bound concurrency, separate-sync-process deployment). Sources: SQLite WAL/pragma/FAQ docs.
 ## [2026-07-12] revise | security/secrets-in-code +1 edge case: third-party HTTP client (httpx/requests) logs the full request URL — including a query-param API key — at INFO, so root/DEBUG logging leaks it; keep the client logger above INFO. Found when a standalone sync process set logging.basicConfig(INFO) and httpx wrote the data.go.kr serviceKey to the log file. last_verified bumped to 2026-07-12.
+## [2026-07-15] ingest | infrastructure/observability +1: kubelet-resource-metrics — cAdvisor `container_*` empty while kubelet targets UP on a lightweight kubelet (OrbStack, field-observed); fix is kubelet-independent: set `kubelet.serviceMonitor.resource: true` + override `resourcePath: /metrics/resource` (chart default `/metrics/resource/v1alpha1` 404s on k8s ≥1.20), and the resource series carry no `image` label so mixin Compute Resources dashboards (`image!=""`, `job=cadvisor`) exclude them → custom panel. Sources verified: kps values.yaml, k8s metrics docs, k8s PR#86282, kubernetes-mixin. OrbStack root-cause tagged field-observed.
+## [2026-07-15] ingest | infrastructure/containers +1: pid1-entrypoint-log-flush — a container PID 1 bash `exec > >(tee)` loses logs on exit (bash never reaps the process-sub subshell; PID 1 exit kills tee mid-flush); capture `TEE_PID=$!` (bash 4.4+, not 5.1) + EXIT trap `exec 1>&- 2>&-; wait` + `trap 'exit 143' TERM`. Sources verified: Greg's Wiki ProcessSubstitution, Docker stop, pipe(7), bash Exit-Status, PID1 signal writeup. Field-tested 10/10 capture.
+## [2026-07-15] revise | platforms/tools/bsd-vs-gnu-cli +1 Do-row + Instead-of row: millisecond timestamp — feature-detect by `date +%3N` 3-digit output, not by whether `%N` works (modern macOS supports `%N` nanoseconds so it no longer discriminates; the `%3N` width form is unrecognized and surfaces literally as `3N`). Sources +GNU date options, +POSIX strftime (unknown conversion undefined). last_verified 2026-07-15.
@@ -0,0 +1,73 @@
+---
+id: infrastructure-containers-pid1-entrypoint-log-flush
+domain: infrastructure
+category: containers
+applies_to: [containers, docker, kubernetes, bash]
+confidence: verified
+sources:
+  - https://mywiki.wooledge.org/ProcessSubstitution
+  - https://docs.docker.com/reference/cli/docker/container/stop/
+  - https://www.man7.org/linux/man-pages/man7/pipe.7.html
+  - https://www.gnu.org/software/bash/manual/html_node/Exit-Status.html
+  - https://petermalmgren.com/signal-handling-docker/
+last_verified: 2026-07-15
+related: [infrastructure-containers-resource-limits-and-probes, platforms-shells-portable-shell-scripts, infrastructure-observability-logs-metrics-signals]
+---
+
+# Container PID 1 Entrypoint: Flushing Piped Logs on Exit
+
+## When this applies
+
+A bash script runs as PID 1 (the container entrypoint) and duplicates its stdout
+to a file with process substitution — `exec > >(tee -a "$LOG")`. Log lines are
+lost when the container exits: short jobs lose everything, long jobs lose the tail
+(the final summary lines).
+
+## Do this
+
+Process substitution `>(tee …)` runs in an **unwaited background subshell** — bash
+does not reap it on exit. A container terminates the instant PID 1 exits, so the
+`tee` is killed before it flushes its buffer. Make the entrypoint wait for the tee
+to drain before it exits.
+
+1. Capture the tee's PID on the line immediately after starting it (bash **4.4+**,
+   where `$!` is set to the process-substitution PID):
+   ```bash
+   exec > >(tee -a "$LOG") 2>&1
+   TEE_PID=$!
+   ```
+   Capture it right away — any other backgrounded job overwrites `$!`.
+2. In an EXIT trap, close the write ends first so `tee` sees EOF, then wait for it:
+   ```bash
+   trap 'exec 1>&- 2>&-; wait "$TEE_PID"' EXIT
+   ```
+   Order matters — closing fds 1/2 delivers the pipe EOF that makes `tee` finish its
+   last read and flush; the `wait` then blocks until it has exited.
+3. Make container stop run the EXIT trap. On `docker stop`/pod termination, PID 1
+   receives SIGTERM then (after the grace period) SIGKILL, and PID 1 **ignores**
+   SIGTERM unless a handler is registered. Add `trap 'exit 143' TERM` (128+15) so a
+   stop signal exits cleanly and fires the EXIT trap's flush.
+
+## Edge cases
+
+| Case | Then |
+|------|------|
+| bash < 4.4 | `$!` does not expose the process-substitution PID — start the writer explicitly through a named FIFO (`mkfifo`; `tee -a "$LOG" < "$FIFO" & TEE_PID=$!`) and redirect to the FIFO |
+| Another child still holds a copy of fd 1/2 | Closing the fds only in the shell does not deliver EOF — `tee` blocks until every write end is closed; ensure children exit or close their fds first |
+| Only the tail is missing (long jobs) vs everything (short jobs) | Same root cause — exit-before-flush; the fix (wait in EXIT trap) covers both |
+
+## Instead of
+
+| If you are about to | Do this instead | Why |
+|---------------------|-----------------|-----|
+| `exec > >(tee …)` with no trap | Capture `TEE_PID=$!` and `wait` it in an EXIT trap after closing the fds | Bash never reaps the process-substitution subshell; PID 1 exit kills it mid-flush |
+| Assume logs were captured because the completion line printed to the console | Wait for the tee in the EXIT trap | Console output and the tee's file buffer are independent; PID 1 exit discards the unflushed buffer |
+| Rely on `docker stop` alone to end the script | Add `trap 'exit 143' TERM` | PID 1 ignores SIGTERM without a handler, so the EXIT trap (and its flush) never runs before SIGKILL |
+
+## Sources
+
+- https://mywiki.wooledge.org/ProcessSubstitution — process substitution runs in an unwaited subshell; bash 4.4+ sets `$!` to its PID so you can `wait`
+- https://docs.docker.com/reference/cli/docker/container/stop/ — stop sends SIGTERM, then SIGKILL after the grace period
+- https://www.man7.org/linux/man-pages/man7/pipe.7.html — a read sees EOF only once all write-end fds are closed
+- https://www.gnu.org/software/bash/manual/html_node/Exit-Status.html — a command killed by signal N exits 128+N (SIGTERM 15 → 143)
+- https://petermalmgren.com/signal-handling-docker/ — PID 1 ignores SIGTERM unless a handler is registered; orphans reaped only if PID 1 waits
@@ -27,6 +27,7 @@ Match your situation to a "load when" line; load only matching pages.
 |------|-----------|
 | [image-builds](containers/image-builds.md) | Writing or reviewing a Dockerfile; images rebuild everything on small changes, build slowly, or are too large; choosing an image tagging scheme |
 | [resource-limits-and-probes](containers/resource-limits-and-probes.md) | Writing or reviewing Kubernetes-style deployment manifests; pods OOMKilled, evicted, or CPU-throttled; a dependency outage triggered a restart storm; traffic hitting pods that are not ready |
+| [pid1-entrypoint-log-flush](containers/pid1-entrypoint-log-flush.md) | A container entrypoint bash script pipes stdout via `exec > >(tee …)` and loses log lines on exit (short jobs lose all, long jobs lose the tail); making PID 1 flush piped logs and run its EXIT trap on container stop (SIGTERM) |
 
 ## data
 
@@ -46,3 +47,4 @@ Match your situation to a "load when" line; load only matching pages.
 |------|-----------|
 | [logs-metrics-signals](observability/logs-metrics-signals.md) | Instrumenting a new or existing service (logs, metrics, correlation ids); an incident revealed you couldn't see what happened; choosing between a log line and a metric; a metric label would carry unbounded values (user ids/UUIDs) |
 | [alerting](observability/alerting.md) | Creating or reviewing alerts; the team ignores a noisy pager; deciding whether a condition pages, tickets, or stays on a dashboard |
+| [kubelet-resource-metrics](observability/kubelet-resource-metrics.md) | Pod CPU/memory (`container_*`) series are empty though kubelet scrape targets are UP; wiring pod-resource dashboards on a lightweight kubelet (OrbStack/kind/k3s); a kube-prometheus-stack resource-metrics target 404s |