Repository navigation
Stop the runner from silently dropping some of its logs - #53
Conversation
The runner never installed a controller-runtime logger, so every log.FromContext call in runner-side code was discarded - including the respond_to_user path's report that a fallback notice failed to publish. Install it globally and carry it on the root context, since a dependency can claim the global first.
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
samkim
left a comment
There was a problem hiding this comment.
This PR routes controller-runtime's logger to the runner's slog handler (a ctrl.SetLogger in main() plus a logr-carrying rootCtx in run()), so log.FromContext lines such as the respond_to_user fallback-notice failure now reach the pod's stderr. I checked the wiring: the call order is right, rootCtx descends into every FromContext user I could find, key/value lists are even, and there is no duplicate output. The change achieves its goal.
Verdict: ready after the inline fixes. Nothing blocks the behaviour; the issues are test hygiene and one inaccurate comment.
Issues, highest first:
- (minor) The new test leaves the stdlib
logoutput redirected to a dead buffer after it finishes, so later slog/log output in theinternal/cmd/runnertest process is lost. See inline on logging_test.go:18. - (minor) The test only covers the
loggingContext()helper. Revertingcontext.WithCancel(loggingContext())at main.go:402 or deletingctrl.SetLoggerat main.go:265 keeps the suite green. See inline on logging_test.go:20. - (nit) The comment at main.go:259-264 cites a transitive
init()race that does not occur in this binary: the spicedb logginginit()only touches zerolog, and no other dependency callsSetLogger. See inline on main.go:263.
Questions for the author:
- The runner has no observed dependency that claims controller-runtime's logger first. Do you want to keep both the global
SetLoggerand the per-context logger as defence in depth, or keep just one? If both stay, the comment should say so honestly. loggingContext()reads the process-globalslog.Default(). Do you want it to take the handler frommain()instead, which would remove the global mutation in the test? Todayrun()is only reached viamain(), so this is a testability choice, not a bug.- Was the fallback-notice failure line seen in a real session pod's
kubectl logs, or only in the unit test?
This review was drafted by an automated pipeline and checked before submission.
| var buf bytes.Buffer | ||
| prev := slog.Default() | ||
| slog.SetDefault(slog.New(slog.NewTextHandler(&buf, nil))) | ||
| t.Cleanup(func() { slog.SetDefault(prev) }) |
There was a problem hiding this comment.
[minor] Test cleanup leaves the stdlib log package redirected to a dead buffer
Problem: slog.SetDefault(<non-default handler>) (line 17) also calls log.SetOutput(&handlerWriter{...}) and log.SetFlags(0). Restoring with slog.SetDefault(prev) does not undo that, because prev is a default handler and SetDefault skips the redirect branch for it (Go log/slog/logger.go, SetDefault). After this test, slog default output and log.Printf lines in the rest of the internal/cmd/runner test process go into the orphaned bytes.Buffer, and log flags stay at 0. A later failing test in this package would lose its slog diagnostics.
Fix: in this test capture prevOut, prevFlags := log.Writer(), log.Flags() before line 17 and, in the t.Cleanup at line 18, after slog.SetDefault(prev) also call log.SetOutput(prevOut) and log.SetFlags(prevFlags) (import stdlib log). If the author decides to make loggingContext take a handler (see review body question 2), the test would not need to touch globals at all and this goes away.
Test: go test -race -count=1 -run TestLoggingContext ./internal/cmd/runner/ still passes; add an assertion at the end of the test that log.Writer() is the original writer after cleanup (e.g. a small second subtest/t.Run that checks it after the first has finished).
Constraints: CLAUDE.md test conventions: tests that mutate package-level state must not run t.Parallel, and cleanup goes in t.Cleanup.
There was a problem hiding this comment.
Fixed in 90270ba. The test now captures log.Writer()/log.Flags() up front, restores both in cleanup, and asserts after the subtest that they are back to the originals.
There was a problem hiding this comment.
Verified at 90270bac: internal/cmd/runner/logging_test.go:58-68 restores log.SetOutput/log.SetFlags in cleanup and :82-83 asserts both are back. Removing the SetOutput restore makes the test fail.
| slog.SetDefault(slog.New(slog.NewTextHandler(&buf, nil))) | ||
| t.Cleanup(func() { slog.SetDefault(prev) }) | ||
|
|
||
| logf.FromContext(loggingContext()).Info("fallback notice publish failed", "session", "ns/demo-agent") |
There was a problem hiding this comment.
[minor] Test covers only the loggingContext() helper; reverting the actual wiring keeps it green
Problem: The regression this PR fixes is that runner-side log.FromContext output was dropped. The test calls loggingContext() directly. Replacing context.WithCancel(loggingContext()) at main.go:402 with context.Background(), or deleting ctrl.SetLogger(...) at main.go:265, leaves every test in the package passing.
Fix: extract the two wiring steps into small named helpers in internal/cmd/runner/main.go, for example installLogging(h slog.Handler) (does slog.SetDefault(slog.New(h)) plus ctrl.SetLogger(logr.FromSlogHandler(h)), called from main() at main.go:258-265) and newRootContext() (context.Context, context.CancelFunc) (returns context.WithCancel(loggingContext()), called from run() at main.go:402). Then add tests that (1) after installLogging, logf.Log.Info(...) and a logr.Error(err, ...) call reach the handler with the ERROR level and err text, and (2) logf.FromContext(rootCtx) from newRootContext() reaches the handler. Be aware that a test of helpers still cannot prove run() calls newRootContext; say so in a comment rather than claiming full coverage. Controller-runtime's SetLogger is first-write-wins, so test the helper with a handler you inspect, not by asserting on the global after the first call.
Test: add cases to internal/cmd/runner/logging_test.go, table-driven if three or more; run go test -race -count=1 -run 'TestLoggingContext|TestInstallLogging|TestNewRootContext' ./internal/cmd/runner/.
Constraints: CLAUDE.md test conventions (testify, require for preconditions and assert for facts); tests that mutate globals must not be parallel and must restore all of them (see the comment on line 18).
There was a problem hiding this comment.
Fixed in 90270ba. The wiring is now installLogging(h) and newRootContext(h), called from main() and run(), with tests for both (including the ERROR level and err text through the global logger). newRootContext takes the handler, so its test touches no globals; a comment notes the tests can't prove the two call sites.
There was a problem hiding this comment.
Verified at 90270bac: installLogging (main.go:335, called main.go:258) and newRootContext (main.go:346, called main.go:407) are tested in logging_test.go, including the ERROR level and err text through the global logger. Dropping ctrl.SetLogger or logf.IntoContext fails the matching test. The comment on the unprovable call sites is in place.
| // every log.FromContext call in runner-side code (respond_to_user's | ||
| // fallback-notice failures, the dispatch loop, authz slot promotion) is | ||
| // discarded. SetLogger is first-write-wins and a transitive init() can | ||
| // claim it first (see internal/cmd/channelsd), so run() also carries the |
There was a problem hiding this comment.
[nit] Comment justifies the second logging path with a race not shown to exist in this binary
Problem: The comment says a transitive init() can claim controller-runtime's logger first "(see internal/cmd/channelsd)". In the runner's dependency graph (go list -deps ./internal/cmd/runner) the only candidate named in channelsd, authzed/spicedb/internal/logging, has an init() that only calls SetGlobalLogger(zerolog.Nop()) (zerolog's DefaultContextLogger), not controller-runtime. A grep of all non-stdlib deps finds no other ctrl.SetLogger/logf.SetLogger caller. So the stated rationale for loggingContext() is borrowed, not observed.
Fix: reword lines 259-264 to say only what is true here: the global alone routes log.FromContext on contexts without a logger, and rootCtx carries the same logger explicitly as defence in depth. Do not claim a transitive init() race unless the author can name the package that does it in this binary. Whether to keep both mechanisms is a design decision (review body question 1); do not remove either without that answer.
Test: no test change; go build ./internal/cmd/runner/ confirms the edit compiles.
There was a problem hiding this comment.
Reworded in 90270ba. The channelsd reference and the init() race claim are gone; the comment now says the context logger is defence in depth and that nothing in this binary is known to claim the global first. Both mechanisms are kept.
There was a problem hiding this comment.
Verified at 90270bac: main.go:340-345 no longer cites channelsd or an init() race and describes the context logger as defence in depth. Keeping both mechanisms is fine.
Address review: installLogging and newRootContext are now what main() and run() call, so the tests exercise the real wiring rather than a lookalike. The test restores the stdlib log output slog.SetDefault redirects, and the comments no longer cite an init() race that does not occur in this binary.
samkim
left a comment
There was a problem hiding this comment.
Round 2 re-review of the runner logging fix (commit 90270ba since d8d5cba). The author extracted installLogging(h) and newRootContext(h), added tests for both, restored the stdlib log state in the test cleanup, and reworded the comment. Earlier issues verified: 3 of 3.
Verdict: ready. One optional nit remains, no behaviour is blocked.
Still open:
- nit, latent test-order dependency in
TestInstallLoggingon controller-runtime's first-write-wins global logger; see inline.
Questions still unanswered:
- Question 3 from round 1: was the fallback-notice failure line actually seen in a real pod's
kubectl logsafter this change? Not blocking.
Limit that stays by design: no unit test can prove main() calls installLogging or run() calls newRootContext (the test file says so at internal/cmd/runner/logging_test.go:50-52).
This review was drafted by an automated pipeline and checked before submission.
| }) | ||
| installedOutput.Reset() | ||
|
|
||
| installLogging(installedHandler) |
There was a problem hiding this comment.
[nit] TestInstallLogging can fail for an unrelated reason if controller-runtime's global logger was already claimed
Problem: controller-runtime's SetLogger is first-write-wins. If anything in this test process calls logf.Log / log.FromContext(context.Background()) more than 30s after package init without SetLogger having run, eventuallyFulfillRoot (controller-runtime pkg/log/log.go:54-70) installs a NullLogSink permanently. installLogging then becomes a no-op for the global and the three logf.Log assertions at lines 78-80 fail with no hint why. Not triggered today (package has no TestMain, runs in a few seconds), but it grows with the suite.
Verify: optional. Add a throwaway test in a file that sorts before logging_test.go that sleeps 31s and then calls logf.Log.Info(...); run it together with TestInstallLogging. controller-runtime prints "log.SetLogger(...) was never called" and TestInstallLogging fails. The reviewer reproduced this with go test -overlay.
Fix: add a TestMain in internal/cmd/runner that calls installLogging(installedHandler) once before m.Run(), and keep the assertions in the test (reset installedOutput first). Alternative: leave as is and add a comment here naming the dependency. Either is acceptable; low priority.
Test: go test -race -count=2 -run 'TestInstallLogging|TestNewRootContext' ./internal/cmd/runner/ must still pass.
| var buf bytes.Buffer | ||
| prev := slog.Default() | ||
| slog.SetDefault(slog.New(slog.NewTextHandler(&buf, nil))) | ||
| t.Cleanup(func() { slog.SetDefault(prev) }) |
There was a problem hiding this comment.
Verified at 90270bac: internal/cmd/runner/logging_test.go:58-68 restores log.SetOutput/log.SetFlags in cleanup and :82-83 asserts both are back. Removing the SetOutput restore makes the test fail.
| slog.SetDefault(slog.New(slog.NewTextHandler(&buf, nil))) | ||
| t.Cleanup(func() { slog.SetDefault(prev) }) | ||
|
|
||
| logf.FromContext(loggingContext()).Info("fallback notice publish failed", "session", "ns/demo-agent") |
There was a problem hiding this comment.
Verified at 90270bac: installLogging (main.go:335, called main.go:258) and newRootContext (main.go:346, called main.go:407) are tested in logging_test.go, including the ERROR level and err text through the global logger. Dropping ctrl.SetLogger or logf.IntoContext fails the matching test. The comment on the unprovable call sites is in place.
| // every log.FromContext call in runner-side code (respond_to_user's | ||
| // fallback-notice failures, the dispatch loop, authz slot promotion) is | ||
| // discarded. SetLogger is first-write-wins and a transitive init() can | ||
| // claim it first (see internal/cmd/channelsd), so run() also carries the |
There was a problem hiding this comment.
Verified at 90270bac: main.go:340-345 no longer cites channelsd or an init() race and describes the context logger as defence in depth. Keeping both mechanisms is fine.
Some of the agent runner's log output — including warnings that a reply to the user failed to send — was being discarded instead of appearing in the session pod's logs. Those messages now show up alongside the rest of the runner's output, so these failures can be diagnosed from
kubectl logs.