Skip to content

Stop the runner from silently dropping some of its logs - #53

Merged
josephschorr merged 2 commits into
mainfrom
runner-ctrl-logger
Nov 16, 2026
Merged

josephschorr merged 2 commits into
mainfrom
runner-ctrl-logger

Conversation

@josephschorr

Copy link
Copy Markdown
Member

Some of the agent runner's log output — including warnings that a reply to the user failed to send — was being discarded instead of appearing in the session pod's logs. Those messages now show up alongside the rest of the runner's output, so these failures can be diagnosed from kubectl logs.

The runner never installed a controller-runtime logger, so every
log.FromContext call in runner-side code was discarded - including the
respond_to_user path's report that a fallback notice failed to publish.
Install it globally and carry it on the root context, since a dependency
can claim the global first.
@vercel

vercel Bot commented Nov 5, 2026 •

Copy link
Copy Markdown

The latest updates on your projects. Learn more about Vercel for GitHub.

1 Skipped Deployment
Project Deployment Actions Updated
openagentprimitives Ignored Ignored Preview Nov 12, 2026 8:33pm UTC

Request Review

@samkim samkim left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR routes controller-runtime's logger to the runner's slog handler (a ctrl.SetLogger in main() plus a logr-carrying rootCtx in run()), so log.FromContext lines such as the respond_to_user fallback-notice failure now reach the pod's stderr. I checked the wiring: the call order is right, rootCtx descends into every FromContext user I could find, key/value lists are even, and there is no duplicate output. The change achieves its goal.

Verdict: ready after the inline fixes. Nothing blocks the behaviour; the issues are test hygiene and one inaccurate comment.

Issues, highest first:

  1. (minor) The new test leaves the stdlib log output redirected to a dead buffer after it finishes, so later slog/log output in the internal/cmd/runner test process is lost. See inline on logging_test.go:18.
  2. (minor) The test only covers the loggingContext() helper. Reverting context.WithCancel(loggingContext()) at main.go:402 or deleting ctrl.SetLogger at main.go:265 keeps the suite green. See inline on logging_test.go:20.
  3. (nit) The comment at main.go:259-264 cites a transitive init() race that does not occur in this binary: the spicedb logging init() only touches zerolog, and no other dependency calls SetLogger. See inline on main.go:263.

Questions for the author:

  1. The runner has no observed dependency that claims controller-runtime's logger first. Do you want to keep both the global SetLogger and the per-context logger as defence in depth, or keep just one? If both stay, the comment should say so honestly.
  2. loggingContext() reads the process-global slog.Default(). Do you want it to take the handler from main() instead, which would remove the global mutation in the test? Today run() is only reached via main(), so this is a testability choice, not a bug.
  3. Was the fallback-notice failure line seen in a real session pod's kubectl logs, or only in the unit test?

This review was drafted by an automated pipeline and checked before submission.

Comment thread internal/cmd/runner/logging_test.go Outdated
var buf bytes.Buffer
prev := slog.Default()
slog.SetDefault(slog.New(slog.NewTextHandler(&buf, nil)))
t.Cleanup(func() { slog.SetDefault(prev) })

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] Test cleanup leaves the stdlib log package redirected to a dead buffer

Problem: slog.SetDefault(<non-default handler>) (line 17) also calls log.SetOutput(&handlerWriter{...}) and log.SetFlags(0). Restoring with slog.SetDefault(prev) does not undo that, because prev is a default handler and SetDefault skips the redirect branch for it (Go log/slog/logger.go, SetDefault). After this test, slog default output and log.Printf lines in the rest of the internal/cmd/runner test process go into the orphaned bytes.Buffer, and log flags stay at 0. A later failing test in this package would lose its slog diagnostics.
Fix: in this test capture prevOut, prevFlags := log.Writer(), log.Flags() before line 17 and, in the t.Cleanup at line 18, after slog.SetDefault(prev) also call log.SetOutput(prevOut) and log.SetFlags(prevFlags) (import stdlib log). If the author decides to make loggingContext take a handler (see review body question 2), the test would not need to touch globals at all and this goes away.
Test: go test -race -count=1 -run TestLoggingContext ./internal/cmd/runner/ still passes; add an assertion at the end of the test that log.Writer() is the original writer after cleanup (e.g. a small second subtest/t.Run that checks it after the first has finished).
Constraints: CLAUDE.md test conventions: tests that mutate package-level state must not run t.Parallel, and cleanup goes in t.Cleanup.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 90270ba. The test now captures log.Writer()/log.Flags() up front, restores both in cleanup, and asserts after the subtest that they are back to the originals.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at 90270bac: internal/cmd/runner/logging_test.go:58-68 restores log.SetOutput/log.SetFlags in cleanup and :82-83 asserts both are back. Removing the SetOutput restore makes the test fail.

Comment thread internal/cmd/runner/logging_test.go Outdated
slog.SetDefault(slog.New(slog.NewTextHandler(&buf, nil)))
t.Cleanup(func() { slog.SetDefault(prev) })

logf.FromContext(loggingContext()).Info("fallback notice publish failed", "session", "ns/demo-agent")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[minor] Test covers only the loggingContext() helper; reverting the actual wiring keeps it green

Problem: The regression this PR fixes is that runner-side log.FromContext output was dropped. The test calls loggingContext() directly. Replacing context.WithCancel(loggingContext()) at main.go:402 with context.Background(), or deleting ctrl.SetLogger(...) at main.go:265, leaves every test in the package passing.
Fix: extract the two wiring steps into small named helpers in internal/cmd/runner/main.go, for example installLogging(h slog.Handler) (does slog.SetDefault(slog.New(h)) plus ctrl.SetLogger(logr.FromSlogHandler(h)), called from main() at main.go:258-265) and newRootContext() (context.Context, context.CancelFunc) (returns context.WithCancel(loggingContext()), called from run() at main.go:402). Then add tests that (1) after installLogging, logf.Log.Info(...) and a logr.Error(err, ...) call reach the handler with the ERROR level and err text, and (2) logf.FromContext(rootCtx) from newRootContext() reaches the handler. Be aware that a test of helpers still cannot prove run() calls newRootContext; say so in a comment rather than claiming full coverage. Controller-runtime's SetLogger is first-write-wins, so test the helper with a handler you inspect, not by asserting on the global after the first call.
Test: add cases to internal/cmd/runner/logging_test.go, table-driven if three or more; run go test -race -count=1 -run 'TestLoggingContext|TestInstallLogging|TestNewRootContext' ./internal/cmd/runner/.
Constraints: CLAUDE.md test conventions (testify, require for preconditions and assert for facts); tests that mutate globals must not be parallel and must restore all of them (see the comment on line 18).

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in 90270ba. The wiring is now installLogging(h) and newRootContext(h), called from main() and run(), with tests for both (including the ERROR level and err text through the global logger). newRootContext takes the handler, so its test touches no globals; a comment notes the tests can't prove the two call sites.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at 90270bac: installLogging (main.go:335, called main.go:258) and newRootContext (main.go:346, called main.go:407) are tested in logging_test.go, including the ERROR level and err text through the global logger. Dropping ctrl.SetLogger or logf.IntoContext fails the matching test. The comment on the unprovable call sites is in place.

Comment thread internal/cmd/runner/main.go Outdated
// every log.FromContext call in runner-side code (respond_to_user's
// fallback-notice failures, the dispatch loop, authz slot promotion) is
// discarded. SetLogger is first-write-wins and a transitive init() can
// claim it first (see internal/cmd/channelsd), so run() also carries the

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] Comment justifies the second logging path with a race not shown to exist in this binary

Problem: The comment says a transitive init() can claim controller-runtime's logger first "(see internal/cmd/channelsd)". In the runner's dependency graph (go list -deps ./internal/cmd/runner) the only candidate named in channelsd, authzed/spicedb/internal/logging, has an init() that only calls SetGlobalLogger(zerolog.Nop()) (zerolog's DefaultContextLogger), not controller-runtime. A grep of all non-stdlib deps finds no other ctrl.SetLogger/logf.SetLogger caller. So the stated rationale for loggingContext() is borrowed, not observed.
Fix: reword lines 259-264 to say only what is true here: the global alone routes log.FromContext on contexts without a logger, and rootCtx carries the same logger explicitly as defence in depth. Do not claim a transitive init() race unless the author can name the package that does it in this binary. Whether to keep both mechanisms is a design decision (review body question 1); do not remove either without that answer.
Test: no test change; go build ./internal/cmd/runner/ confirms the edit compiles.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reworded in 90270ba. The channelsd reference and the init() race claim are gone; the comment now says the context logger is defence in depth and that nothing in this binary is known to claim the global first. Both mechanisms are kept.

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at 90270bac: main.go:340-345 no longer cites channelsd or an init() race and describes the context logger as defence in depth. Keeping both mechanisms is fine.

Address review: installLogging and newRootContext are now what main() and
run() call, so the tests exercise the real wiring rather than a lookalike.
The test restores the stdlib log output slog.SetDefault redirects, and the
comments no longer cite an init() race that does not occur in this binary.

@samkim samkim left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Round 2 re-review of the runner logging fix (commit 90270ba since d8d5cba). The author extracted installLogging(h) and newRootContext(h), added tests for both, restored the stdlib log state in the test cleanup, and reworded the comment. Earlier issues verified: 3 of 3.

Verdict: ready. One optional nit remains, no behaviour is blocked.

Still open:

  • nit, latent test-order dependency in TestInstallLogging on controller-runtime's first-write-wins global logger; see inline.

Questions still unanswered:

  1. Question 3 from round 1: was the fallback-notice failure line actually seen in a real pod's kubectl logs after this change? Not blocking.

Limit that stays by design: no unit test can prove main() calls installLogging or run() calls newRootContext (the test file says so at internal/cmd/runner/logging_test.go:50-52).

This review was drafted by an automated pipeline and checked before submission.

})
installedOutput.Reset()

installLogging(installedHandler)

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] TestInstallLogging can fail for an unrelated reason if controller-runtime's global logger was already claimed

Problem: controller-runtime's SetLogger is first-write-wins. If anything in this test process calls logf.Log / log.FromContext(context.Background()) more than 30s after package init without SetLogger having run, eventuallyFulfillRoot (controller-runtime pkg/log/log.go:54-70) installs a NullLogSink permanently. installLogging then becomes a no-op for the global and the three logf.Log assertions at lines 78-80 fail with no hint why. Not triggered today (package has no TestMain, runs in a few seconds), but it grows with the suite.
Verify: optional. Add a throwaway test in a file that sorts before logging_test.go that sleeps 31s and then calls logf.Log.Info(...); run it together with TestInstallLogging. controller-runtime prints "log.SetLogger(...) was never called" and TestInstallLogging fails. The reviewer reproduced this with go test -overlay.
Fix: add a TestMain in internal/cmd/runner that calls installLogging(installedHandler) once before m.Run(), and keep the assertions in the test (reset installedOutput first). Alternative: leave as is and add a comment here naming the dependency. Either is acceptable; low priority.
Test: go test -race -count=2 -run 'TestInstallLogging|TestNewRootContext' ./internal/cmd/runner/ must still pass.

Comment thread internal/cmd/runner/logging_test.go Outdated
var buf bytes.Buffer
prev := slog.Default()
slog.SetDefault(slog.New(slog.NewTextHandler(&buf, nil)))
t.Cleanup(func() { slog.SetDefault(prev) })

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at 90270bac: internal/cmd/runner/logging_test.go:58-68 restores log.SetOutput/log.SetFlags in cleanup and :82-83 asserts both are back. Removing the SetOutput restore makes the test fail.

Comment thread internal/cmd/runner/logging_test.go Outdated
slog.SetDefault(slog.New(slog.NewTextHandler(&buf, nil)))
t.Cleanup(func() { slog.SetDefault(prev) })

logf.FromContext(loggingContext()).Info("fallback notice publish failed", "session", "ns/demo-agent")

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at 90270bac: installLogging (main.go:335, called main.go:258) and newRootContext (main.go:346, called main.go:407) are tested in logging_test.go, including the ERROR level and err text through the global logger. Dropping ctrl.SetLogger or logf.IntoContext fails the matching test. The comment on the unprovable call sites is in place.

Comment thread internal/cmd/runner/main.go Outdated
// every log.FromContext call in runner-side code (respond_to_user's
// fallback-notice failures, the dispatch loop, authz slot promotion) is
// discarded. SetLogger is first-write-wins and a transitive init() can
// claim it first (see internal/cmd/channelsd), so run() also carries the

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Verified at 90270bac: main.go:340-345 no longer cites channelsd or an init() race and describes the context logger as defence in depth. Keeping both mechanisms is fine.

@josephschorr
josephschorr merged commit 79857f3 into main Nov 16, 2026
9 of 10 checks passed
@josephschorr
josephschorr deleted the runner-ctrl-logger branch November 16, 2026 16:57
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants