Skip to content

fix(self-update): run external restart inside audit/notify boundary + runtime recovery tick (#971) - #976

Merged
jinon86 merged 2 commits into
mainfrom
fix/self-update-restart-signal
Aug 6, 2026
Merged

fix(self-update): run external restart inside audit/notify boundary + runtime recovery tick (#971)#976
jinon86 merged 2 commits into
mainfrom
fix/self-update-restart-signal

Conversation

@jinon86

@jinon86 jinon86 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

문제 (#971, daegyo 4h15m 침묵 장애)

수동 체이닝 크론 라인(run; rc=11 → start.sh --restart; exit 0)이 재시작 실패를 감지하고도 신호를 버림:

  • start.shnot-available-within-timeout을 정확히 보고했지만 trailing exit 0이 삼킴 → notify 없음
  • 05:45 재시도 슬롯은 코드 변경이 없어 재시작을 시도하지 않음(업데이트만 재시도)
  • self-update의 구조적 결과(degraded-no-services)는 체이닝된 재시작을 볼 수 없음

수정 — 외부 재시작을 스크립트 안으로

운영자 소유 훅 2개(self-update.services와 동일 신뢰 모델, 에이전트 쓰기 금지):

  • ~/.claude/self-update.restart-cmd — 외부 재시작 명령 (예: Termux start.sh --restart -d)
  • ~/.claude/self-update.health-cmd — 런타임 헬스 프로브 (exit 0 = healthy)
  1. degraded 경로(코드 변경·재시작 0건): restart-cmd가 있으면 스크립트 안에서 실행 → 성공 시 ok/정리, 실패 시 audit + 즉시 notify + exit 7 + 복구 스냅샷 보존 (기존처럼 degraded/11은 restart-cmd 미설정 시에만)
  2. up-to-date 틱: 두 훅이 모두 있으면 런타임 헬스를 검증하고, 다운이면 복구 재시작 1회 시도(runtime-recovered / runtime-down audit + notify) — 두 번째 데일리 슬롯이 updated-but-down 노드를 복구 가능
  3. 크론 라인은 체이닝 없는 순수 ccc-self-update.sh run으로 단순화 가능 (docs에 마이그레이션 예시)

검증

  • 테스트 16건 추가(총 73 pass): 변경+restart-cmd 성공 시 ok·실제 실행·audit / 실패 시 rc 7·restart-failures audit·즉시 notify·스냅샷 보존 / up-to-date 다운→복구 / 건강 시 미실행 / 복구 실패 시 rc 7·notify
  • bash -n clean, 기존 57건 무회귀

후속 (별도 오너 승인)

Termux 2대(daegyo·gongyung) + gongmyoung의 크론 라인을 새 훅 파일 + 순수 run으로 교체하는 배포 작업 (머지 후 self-update로 코드 전파 뒤 진행).

Closes #971

… runtime recovery tick (#971)

The hand-chained cron line (ccc-self-update run; rc=11 -> start.sh
--restart; exit 0) detected its own restart failure, reported it to a
log file, and then discarded the signal: daegyo stayed down 4h15m
until the daily fleet watch noticed. The 05:45 retry slot could never
help because it only retries the update, never the restart.

- new operator-owned hooks: self-update.restart-cmd (external restart
  command) and self-update.health-cmd (runtime probe), same trust
  model as self-update.services
- degraded path (code changed, nothing restarted): run the restart
  command inside the script — outcome audited/notified, failure exits
  7 with the recovery snapshot retained instead of a silent exit 0
- up-to-date tick: with both hooks configured, verify runtime health
  and attempt one recovery restart when the runtime is down, so the
  second daily slot can recover an updated-but-down node
- docs: supported replacement for the chained cron line

Closes #971
@jinon86
jinon86 requested a review from seoseo-ai as a code owner August 6, 2026 01:49

@seoseo-ai seoseo-ai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved after explicit operator authorization using the local seoseo-ai credential.

@jinon86
jinon86 merged commit 091fecc into main Aug 6, 2026
8 checks passed
@jinon86
jinon86 deleted the fix/self-update-restart-signal branch August 6, 2026 02:31
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

self-update: chained bridge restart detects its own failure, then exits 0 and notifies nobody (daegyo 4h15m silent outage)

2 participants