Skip to content

fix(bridge-watchdog): serialize restarts on an exclusive start lock (#970) - #982

Merged
jinon86 merged 2 commits into
mainfrom
fix/watchdog-install-lock
Aug 6, 2026
Merged

fix(bridge-watchdog): serialize restarts on an exclusive start lock (#970)#982
jinon86 merged 2 commits into
mainfrom
fix/watchdog-install-lock

Conversation

@jinon86

@jinon86 jinon86 commented Aug 6, 2026

Copy link
Copy Markdown
Contributor

문제 (#970)

5분 워치독의 유일한 race 가드는 bot.pid 신선도 디바운스 — restart vs restart만 커버. 의존성 빌드 중에는 bot.pid가 없어서 매 틱이 새 start.sh를 띄우고 각자 dependency_bootstrap을 같은 venv에 돌림 (daegyo 복구 중 cargo Text file busy, cryptography 빌드가 폰에서 5분+ 소요).

수정

down-detect → start 임계 구간 전체를 exclusive flock으로 직렬화:

  • 다른 start가 진행 중이면(락 보유) 이번 틱은 깔끔히 skip — 적재하지 않음
  • 락은 스크립트 종료 시 자동 해제, 실패한 start는 다음 틱이 재시도
  • 락 경로는 BRIDGE_WATCHDOG_LOCK으로 오버라이드 가능(테스트 심)

검증

  • 테스트 5건 추가(총 23 pass): 락 보유 시 skip·start.sh 미실행·로그 기록 / 해제 후 정상 재시작
  • bash -n clean, 기존 디바운스·stale 처리 무회귀

Closes #970

…970)

The bot.pid freshness debounce covers restart racing restart, but
during a dependency build there is no bot.pid at all — every 5-minute
tick launched another start.sh whose dependency_bootstrap raced the
first (cargo 'Text file busy' during the daegyo recovery, 2026-08-06).

Take an exclusive flock around the whole down-detect -> start critical
section: a tick that finds a start already in flight skips cleanly
instead of piling on. Lock path overridable for tests.

Closes #970
@jinon86
jinon86 requested a review from seoseo-ai as a code owner August 6, 2026 06:04

@seoseo-ai seoseo-ai left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved after explicit operator authorization using the local seoseo-ai credential.

@jinon86
jinon86 merged commit a393ab1 into main Aug 6, 2026
8 checks passed
@jinon86
jinon86 deleted the fix/watchdog-install-lock branch August 6, 2026 06:25
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

bridge-watchdog: no install lock — a dependency build longer than 5min gets concurrent start.sh runs (cargo 'Text file busy')

2 participants