diff --git a/.dev-loop/INGEST_REPORT.md b/.dev-loop/INGEST_REPORT.md new file mode 100644 index 0000000..921ff10 --- /dev/null +++ b/.dev-loop/INGEST_REPORT.md @@ -0,0 +1,52 @@ +# Knowledge flush — 2 insight(s) (+1 retired as already flushed) + +Cross-Check: 독립 서브에이전트 적대검증 1회 — date 페이지 CONFIRMED(GNU coreutils 컨테이너·macOS 26.5.1 양쪽 실측, case 패턴 분기 검증), entrypoint 페이지는 큐 원문의 TERM trap 근거가 실증상 반대(무trap SIGTERM에도 bash EXIT trap 실행; foreground 자식 뒤 TERM trap은 지연→SIGKILL 137, flush 0건)임을 적발 → background 자식+`wait`+시그널 포워딩 패턴으로 재작성 후 반영. 재현 아티팩트: bash 4.4/5.2 PID-1 docker 실험 각 10회. + +## Decision Log + +- **의도**: dev-loop 큐의 pending 인사이트를 검증·중복점검·라우팅해 위키에 반영 (knowledge-flush 파이프라인, 사용자가 flush 실행과 PR 1건 생성을 명시 지시). +- **배제한 대안**: + - 인사이트(kubelet /metrics/resource)의 신규 페이지 생성 — **배제: 열려 있는 PR #2가 동일 인사이트를 `container-metrics-when-cadvisor-is-empty.md`로 이미 반영**(metrics_path 핀 + image!="" 복합 필터까지 커버 확인). 큐 row가 retire되지 않아 재등장한 것 — 이번 flush에서 재반영 없이 retire만 함. 작성했던 중복 페이지는 커밋 이력에서 제거. + - 인사이트(PID-1 tee)를 `platforms/shells/portable-shell-scripts.md`에 병합 — 배제: 트리거가 셸 이식성이 아니라 PID-1 컨테이너 시맨틱이라 `infrastructure/containers` 신규 페이지가 맞음 (one case per page). + - 큐 원문 directive(`trap 'exit 143' TERM` 단독) 그대로 반영 — 배제: 적대검증 실험에서 반박되어 교정 후 반영. +- **리뷰어가 볼 곳**: `entrypoint-log-capture.md` step 2(background-child+wait 패턴 — 교차검증으로 재작성된 부분)와 Sources의 두 독립 재현 기록, `bsd-vs-gnu-cli.md` 추가 행의 case 패턴 분기. `wiki/infrastructure/index.md`는 PR #2도 수정하므로 머지 순서에 따라 사소한 충돌 가능. +- [추정] RTB 팀 표준 리뷰어 3인은 이 개인 레포 collaborator가 아니어서 리뷰어 지정이 거부됨(gh 확인: `d43103-rsquare not found`) — 레포 소유자 리뷰(스킬 기본 모델)로 진행. + +## Verified best-practice + +**1. PID-1 entrypoint bash with `exec > >(tee -a f)` loses log output → EXIT-trap fd close + `wait "$TEE_PID"`; long foreground jobs need background-child + `wait` + TERM forwarding.** +Verified against: +- https://mywiki.wooledge.org/ProcessSubstitution — process substitution "will continue to run when your script exits (unless you manage your child processes)"; managed with `wait "$!"` since bash 4.4 (also confirmed in bash CHANGES, bash-4.4 section). +- https://tiswww.case.edu/php/chet/bash/bashref.html — 128+n exit status for signal-terminated commands (143 = SIGTERM); trap execution deferred until the running foreground command completes. +- Session reproduction (OrbStack container): without trap 0/10 runs captured output; with trap 10/10. +- Independent adversarial re-reproduction during this flush (docker, bash 4.4 and 5.2 as PID 1): 1/10 without trap vs 10/10 with; untrapped `docker stop` ran the EXIT trap (exit 143); `trap 'exit 143' TERM` behind a foreground child was deferred until SIGKILL (exit 137, no flush). +The queued candidate's original `trap 'exit 143' TERM` directive was **corrected during cross-check** — the page teaches the working background-child pattern instead. **confidence: verified.** (gnu.org bash manual rate-limited with HTTP 429 during this flush; the case.edu mirror + Wooledge are cited instead — no unverified URL included.) + +**2. BSD/GNU `date` ms-timestamp feature detection → test the actual `%3N` output shape, not `%N` presence.** +Verified against: +- https://www.gnu.org/software/coreutils/manual/html_node/Time-conversion-specifiers.html — "%N nanoseconds … This is a GNU extension." +- https://www.gnu.org/software/coreutils/manual/html_node/Padding-and-other-flags.html — field width between `%` and the specifier is a GNU extension. +- Independent reproduction on a second machine during this flush (macOS 26.5.1, beyond the original 14.8.3 measurement): `date +%N` → digits, `date +%3N` → literal `3N`. Adversarial re-check additionally ran GNU coreutils (debian container): `date +%3N` → exactly 3 digits, and confirmed the `case … [0-9][0-9][0-9])` pattern takes the GNU path on GNU and falls through on macOS. +**confidence: verified.** + +**Retired without re-ingesting**: the OrbStack kubelet `/metrics/resource` insight (hash `ab44b921a1f3f759`) — already flushed as open **PR #2** (`container-metrics-when-cadvisor-is-empty.md`) but never removed from its session queue file; verified PR #2's page already covers the chart defaults, the label-set difference, and the `metrics_path="/metrics/cadvisor"` + `image!=""` dashboard exclusion, so nothing new to add. + +## Existing-layer check + +Read: root `INDEX.md`, `wiki/infrastructure/index.md`, `wiki/platforms/index.md`, and every overlapping page: `observability/logs-metrics-signals.md`, `containers/resource-limits-and-probes.md`, `containers/image-builds.md`, `shells/portable-shell-scripts.md`, `tools/bsd-vs-gnu-cli.md`, plus open PR #2's `container-metrics-when-cadvisor-is-empty.md` (via the PR branch). + +- Insight 1 (PID-1 tee): no overlap — `portable-shell-scripts` covers shell portability, `image-builds` covers Dockerfiles; neither covers PID-1 runtime log/signal behavior. No conflict. **Created new page**; related-linked both ways to `image-builds` and `portable-shell-scripts` (cross-domain), one-way to `logs-metrics-signals`. +- Insight 2 (date `%3N`): direct overlap with `platforms/tools/bsd-vs-gnu-cli.md` (same trigger family — its "Relative date" row already covers `date -d` vs `-v`). **Merged**: one command-table row (sub-second timestamp), one edge-case row (`%N` printing digits on macOS must not imply GNU), two sources, `last_verified` bumped to 2026-07-15. No new page. +- Kubelet insight: duplicate of open PR #2 → dropped (see Decision Log). + +Conflicts flagged: none in content. Noted: `wiki/infrastructure/index.md` is also touched by open PR #2 (different rows) — trivial merge-order conflict possible. + +## Routing decision + +| Insight | Target | New category? | +|---------|--------|---------------| +| PID-1 tee log loss | `infrastructure/containers/entrypoint-log-capture.md` (new page) | No — containers owns container-runtime behavior; platforms/shells rejected because the trigger is PID-1 container semantics, not shell portability | +| date `%3N` detection | `platforms/tools/bsd-vs-gnu-cli.md` (merge) | No — exact existing page for BSD-vs-GNU flag differences | +| kubelet `/metrics/resource` | — (retired; already in open PR #2) | — | + +Nothing left `unverified`. Queue rows retired to `~/.dev-loop/queue/.processed.jsonl` after PR creation. diff --git a/log.md b/log.md index bcc206d..695aa64 100644 --- a/log.md +++ b/log.md @@ -32,3 +32,5 @@ Append-only. Format: `## [YYYY-MM-DD] >(tee …) loses log output because the shell never waits for process substitutions and the container dies with PID 1; fix is EXIT-trap fd close + wait "$TEE_PID"; for long foreground jobs, background-child + wait + TERM forwarding (a bare TERM trap behind a foreground child is deferred until SIGKILL — adversarial cross-check corrected the queued directive). Sources: Wooledge ProcessSubstitution (wait "$!" since bash 4.4), bash ref manual 128+n; field reproduction 0/10 → 10/10 +## [2026-07-15] revise | platforms-tools-bsd-vs-gnu-cli +1 row +1 edge case: sub-second timestamps — recent macOS date prints digits for %N but emits %3N literally as "3N", so feature-detect on the actual %3N output shape, never on %N presence. GNU manual confirms %N and field width are GNU extensions; reproduced on macOS 14.8.3 and 26.5.1. last_verified bumped to 2026-07-15 diff --git a/wiki/infrastructure/containers/entrypoint-log-capture.md b/wiki/infrastructure/containers/entrypoint-log-capture.md new file mode 100644 index 0000000..a327056 --- /dev/null +++ b/wiki/infrastructure/containers/entrypoint-log-capture.md @@ -0,0 +1,83 @@ +--- +id: infrastructure-containers-entrypoint-log-capture +domain: infrastructure +category: containers +applies_to: [docker, kubernetes, bash] +confidence: verified +sources: + - https://mywiki.wooledge.org/ProcessSubstitution + - https://tiswww.case.edu/php/chet/bash/bashref.html +last_verified: 2026-07-15 +related: [infrastructure-containers-image-builds, infrastructure-observability-logs-metrics-signals, platforms-shells-portable-shell-scripts] +--- + +# Log Loss from Process-Substituted tee in Container Entrypoint Scripts + +## When this applies + +A container's entrypoint bash script (running as PID 1) duplicates its output to +a file with `exec > >(tee -a "$LOG") 2>&1`, and log lines are missing: short jobs +capture nothing, long jobs lose the tail (the completion-summary lines). + +## Do this + +1. Capture the tee PID and wait for it in an EXIT trap. Bash runs process + substitutions asynchronously and does not wait for them; `wait "$!"` on a + process substitution is supported since bash 4.4. Outside a container the + orphaned tee usually finishes anyway — as PID 1, the container is torn down + the instant the script exits, killing tee before it flushes: + + ```bash + exec > >(tee -a "$LOG") 2>&1 + TEE_PID=$! + flush_logs() { + exec 1>&- 2>&- # close our ends so tee sees EOF + wait "$TEE_PID" + } + trap flush_logs EXIT + ``` + +2. For a script whose main work is a long foreground job, run that job as a + background child and `wait` it before adding any TERM trap. Bash defers + user traps until the foreground command completes — a TERM trap behind a + foreground child never fires on `docker stop`, the grace period expires, + and SIGKILL ends the container at exit 137 with nothing flushed. `wait` + is interruptible by traps, so this shape reacts immediately: + + ```bash + trap 'kill -TERM "$MAIN_PID" 2>/dev/null; wait "$MAIN_PID"; exit 143' TERM + main_job & MAIN_PID=$! + wait "$MAIN_PID" + ``` + + `exit 143` (128+15) reports the conventional SIGTERM status. For simple + sequential scripts, no TERM trap is needed for the flush itself — bash runs + the EXIT trap on an untrapped SIGTERM in non-interactive shells. + +3. Verify by re-running the container several times and checking the log file + for the final line each run — this failure is timing-dependent, so a single + successful run proves nothing. + +## Edge cases + +| Case | Then | +|------|------| +| Base image bash is older than 4.4 (`wait "$!"` on a process substitution fails) | Use a named pipe: `mkfifo`, start `tee < pipe &` as a normal background job, `exec > pipe`, and `wait` its real PID | +| Script also traps other signals (INT, HUP) | End each handler with `exit 128+n`, so every trapped path reports its conventional signal code and reaches the EXIT trap | +| Logs must survive even a SIGKILL (OOMKill, forced deletion) | No trap runs on KILL — write to a mounted volume via unbuffered append (or ship lines as they are produced) instead of relying on exit-time flushing | + +## Instead of + +| If you are about to | Do this instead | Why | +|---------------------|-----------------|-----| +| Trust `exec > >(tee -a f)` because it works in a terminal | Add the EXIT-trap wait before it runs as PID 1 | The shell never waits for process substitutions; a terminal leaves the orphan running to finish, a container kills it at PID 1 exit | +| Debug the missing lines as a tee buffering option problem | Reproduce with repeated short runs and add the trap | The writer is killed before flushing — buffering flags don't fix a killed process | +| Exit 0 from the TERM trap for a "clean" shutdown | `exit 143` (128+15) | Orchestrators and CI distinguish signalled termination from success by the 128+n convention | +| Add `trap 'exit 143' TERM` while the main job runs in the foreground | Background the job, `wait` it, and forward TERM from the trap (step 2) | Bash defers traps until the foreground command finishes — the trap never fires, the grace period ends in SIGKILL (exit 137), and nothing flushes | + +## Sources + +- https://mywiki.wooledge.org/ProcessSubstitution — process substitution "will continue to run when your script exits (unless you manage your child processes)"; since bash 4.4 it can be managed with `wait "$!"` +- https://tiswww.case.edu/php/chet/bash/bashref.html — exit status is 128+n for a command terminated by signal n; trap execution is deferred until the running foreground command completes +- Field reproduction (OrbStack container, 2026-07-15): without the trap the log captured the script output in 0 of 10 runs; with the EXIT-trap wait, 10 of 10 +- Independent adversarial re-reproduction (docker bash 4.4 and 5.2 as PID 1, 2026-07-15): tail captured 1/10 without the trap vs 10/10 with it; untrapped `docker stop` ran the EXIT trap (exit 143); `trap 'exit 143' TERM` behind a foreground child was deferred until SIGKILL (exit 137, no flush) diff --git a/wiki/infrastructure/containers/image-builds.md b/wiki/infrastructure/containers/image-builds.md index 3c500b6..96ba9ba 100644 --- a/wiki/infrastructure/containers/image-builds.md +++ b/wiki/infrastructure/containers/image-builds.md @@ -8,7 +8,7 @@ sources: - https://docs.docker.com/build/building/best-practices/ - https://docs.docker.com/build/building/secrets/ last_verified: 2026-07-10 -related: [infrastructure-ci-cd-pipeline-structure, infrastructure-ci-cd-secrets-handling] +related: [infrastructure-ci-cd-pipeline-structure, infrastructure-ci-cd-secrets-handling, infrastructure-containers-entrypoint-log-capture] --- # Structuring Container Image Builds diff --git a/wiki/infrastructure/index.md b/wiki/infrastructure/index.md index 4940602..13e9abb 100644 --- a/wiki/infrastructure/index.md +++ b/wiki/infrastructure/index.md @@ -27,6 +27,7 @@ Match your situation to a "load when" line; load only matching pages. |------|-----------| | [image-builds](containers/image-builds.md) | Writing or reviewing a Dockerfile; images rebuild everything on small changes, build slowly, or are too large; choosing an image tagging scheme | | [resource-limits-and-probes](containers/resource-limits-and-probes.md) | Writing or reviewing Kubernetes-style deployment manifests; pods OOMKilled, evicted, or CPU-throttled; a dependency outage triggered a restart storm; traffic hitting pods that are not ready | +| [entrypoint-log-capture](containers/entrypoint-log-capture.md) | A container entrypoint bash script tees its output to a log file and lines are missing (short jobs capture nothing, long jobs lose the tail); wiring stdout duplication or signal traps into a PID-1 shell script | ## data diff --git a/wiki/platforms/shells/portable-shell-scripts.md b/wiki/platforms/shells/portable-shell-scripts.md index dcab402..718cdc3 100644 --- a/wiki/platforms/shells/portable-shell-scripts.md +++ b/wiki/platforms/shells/portable-shell-scripts.md @@ -11,7 +11,7 @@ sources: - https://google.github.io/styleguide/shellguide.html - https://www.shellcheck.net/ last_verified: 2026-07-10 -related: [platforms-tools-bsd-vs-gnu-cli, platforms-toolchains-version-management] +related: [platforms-tools-bsd-vs-gnu-cli, platforms-toolchains-version-management, infrastructure-containers-entrypoint-log-capture] --- # Shell Scripts That Must Run on More Than One Machine or Shell diff --git a/wiki/platforms/tools/bsd-vs-gnu-cli.md b/wiki/platforms/tools/bsd-vs-gnu-cli.md index 21181ef..8110697 100644 --- a/wiki/platforms/tools/bsd-vs-gnu-cli.md +++ b/wiki/platforms/tools/bsd-vs-gnu-cli.md @@ -9,7 +9,7 @@ sources: - https://man.freebsd.org/cgi/man.cgi?date(1) - https://man.freebsd.org/cgi/man.cgi?sed(1) - https://man.freebsd.org/cgi/man.cgi?seq(1) -last_verified: 2026-07-10 +last_verified: 2026-07-15 related: [platforms-shells-portable-shell-scripts] --- @@ -32,6 +32,7 @@ flags do not. Apply the portable fix per command: | `timeout` | In coreutils | Absent on stock macOS — exits 127 | Install coreutils and call `gtimeout`, or background the command and `kill` it from a sleep watchdog | | `seq -s,` | Separator between numbers only | Separator also emitted after the last number, before the terminator | Join after generating (`seq 1 3 \| paste -sd, -`) and count items by lines, never by separators | | PCRE grep | `grep -P` | Not supported | `grep -E` with ERE, or `perl -ne 'print if /…/'` | +| Sub-second timestamp | `date +%3N` → milliseconds (`%N` and the field width are both GNU extensions) | Recent macOS prints digits for `%N` but emits `%3N` literally as `3N` (timestamps become `…08.3NZ`) | Feature-detect on the actual `%3N` output, not on `%N`: `case "$(date +%3N)" in [0-9][0-9][0-9]) …GNU path… ;; *) …fallback… ;; esac`; or `gdate +%3N`, or settle for epoch seconds | | Resolve path | `readlink -f` | Absent before macOS 13 | `cd "$(dirname "$f")" && pwd -P` for directories, `python3 -c 'import os,sys;print(os.path.realpath(sys.argv[1]))'`, or coreutils `greadlink -f` | | File metadata | `stat -c '%s'` | `stat -f '%z'` | Detect once: `stat -c %s "$f" 2>/dev/null \|\| stat -f %z "$f"` | @@ -49,6 +50,7 @@ General strategy by situation: | Case | Then | |------|------| | `command -v timeout` succeeds on macOS | Someone installed coreutils unprefixed — confirm `timeout --version` reports GNU coreutils before relying on GNU exit-code semantics (124 on timeout) | +| `date +%N` prints digits on macOS | Do not infer GNU userland from one specifier working — width-modified forms (`%3N`) still fail on BSD `date`; feature-detect the exact format string you will use, by its output shape | | Any flags passed to `echo` (`-e`, `-n`) | `echo` flag handling differs across shells and userlands — use `printf` for anything beyond a bare literal string | | Script needs bash 4+ features on macOS | Stock `/bin/bash` on macOS is 3.2 — use `#!/usr/bin/env bash` so a brew-installed bash is picked up, and state the required bash version in the script header | @@ -63,6 +65,9 @@ General strategy by situation: ## Sources - https://www.gnu.org/software/coreutils/manual/html_node/index.html — GNU date/timeout/seq/stat behavior +- https://www.gnu.org/software/coreutils/manual/html_node/Time-conversion-specifiers.html — `%N` (nanoseconds) is a GNU extension +- https://www.gnu.org/software/coreutils/manual/html_node/Padding-and-other-flags.html — field width between `%` and the specifier is a GNU extension +- Field reproduction (macOS 14.8.3 and 26.5.1, 2026-07-15): `date +%N` → digits, `date +%3N` → literal `3N` on both - https://man.freebsd.org/cgi/man.cgi?date(1) — BSD `date -v` adjustment flag - https://man.freebsd.org/cgi/man.cgi?sed(1) — BSD `sed -i` backup-extension argument - https://man.freebsd.org/cgi/man.cgi?seq(1) — BSD `seq -s` separator emitted after the last number (see man-page example)