Skip to content

slicecamd: anchor the fine-acquire settle to the executed TCS move, not a timer - #479

Open
cfremling wants to merge 1 commit into
mainfrom
feat/fineacquire-wait-for-move
Open

slicecamd: anchor the fine-acquire settle to the executed TCS move, not a timer#479
cfremling wants to merge 1 commit into
mainfrom
feat/fineacquire-wait-for-move

Conversation

@cfremling

Copy link
Copy Markdown
Collaborator

THIS FIXES A CRITICAL BUG (measured, UT 2026-08-06 07:50, BD+28d4211)

Fine-acquire corrections do not move the telescope: offset_acam_goal sends
offsetgoal <dRA> <dDEC> fineguiding to acamd, which only shifts the guide
GOAL — the physical pt_offset happens at acamd's next guide solve, up to
one full guide cycle (ACAM exptime + ~few s) later. slicecamd's post-correction
settle is currently a fixed 3 s sleep anchored at the send (FINE_ACQUIRE_SETTLE_SEC,
slicecam_interface.cpp:576-578), followed by 2 skipped frames and 3 fresh
samples. This works fine most of the time, but very short slicecam exposure times in combination with longer exposure times set on acam WILL lead to a situation where all of the slicecam safety and wait logic
completes before the telescope actually moves. This WILL result in a double offset by fineacquire, which can in a very unfortunate case also lead to a fineacquire 'lock' in between these two guide goal changes. Result is that the star is NOT centered on exposure.

On 2026-08-06 the per-frame log shows this end to end:

  • 07:50:25.320 — measurement 1: dRA=−1.035 dDEC=−2.429 (r=2.64″) → goal shift 1 sent
  • 07:50:31.013 — tcsd receives offset -0.983478 -2.485729 40 (move 1, 5.7 s after the send); DONE 31.893
  • 07:50:30.630 / 31.408 / 32.173 — measurement 2's three frames: the first two show the star at the identical pre-move pixel (160,112)
  • 07:50:32.173 — measurement 2: dRA=−0.931 dDEC=−2.388 (r=2.56″) — the same error re-measured from stale sky → goal shift 2 sent
  • 07:50:38.358 — tcsd receives offset -0.902148 -2.568997 40 (move 2 — the double application)
  • 07:50:39.059 — "converged r=0.249″" (star genuinely on the aimpoint after move 1 alone) — 0.13 s before move 2 completes and drags it off
  • During the following three exposures acamd's during-exposure plate solve puts the field at DEC 28.877943 — 0.068″ from the doubly-shifted goal, i.e. the guide loop held the star ~2.6″ off the slit, locked on the wrong spot.
  • All three frames of the V=10.5 standard extracted at SNR −1.5 to −2.5 (expected ~400–500).

Prevalence: 1 confirmed double in 109 fine sessions scanned — rare because the
typical guide cadence (~5 s) beats the ~6 s fineacquire re-measure time, but the odds rise
toward certainty in the combination when ACAM exposures lengthen (poor transparency) while bright
stars keep slicecam frames short.

The fix (add a real check for guide goal change instead of the fixed wait in fineacquire)

  1. acamd tracks the request: offsetgoal marks the goal change
    pending; the guide loop clears it when it executes a correction computed
    from the changed goal (the pending state is captured before the goal is
    read, so a correction already in flight can never clear it — and since
    every correction is goal − actual position, executing one computed from
    the new goal lands the telescope at the new goal). The flag is published
    in the existing status.
  2. slicecamd: do_fineacquire waits until acamd reports the goal change
    applied (pending false, on a status newer than the send) before
    FINE_ACQUIRE_SETTLE_SEC (now a true post-move settle) and fresh
    sampling. A stop interrupts the wait. Timeout FINE_ACQUIRE_MOVE_TIMEOUT
    (default 60 s, cfg.in knob; 0 disables): proceed with a WARNING —
    fail-open, never a hang.
  3. This also removes the converged-while-a-shift-is-in-flight defect for
    free: no sample any decision uses can predate the move it follows.
  4. Ride-along log fix: offset_acam_goal's "requested offsets … arcsec"
    line printed degrees while guiding (the ×3600 sits in the non-guiding
    branch only, slicecam_interface.cpp:2619-2636). Log now prints arcsec on
    both paths; nothing sent to acamd changes.

Notes for the reviewer

  • Config lives in slicecamd.cfg.in (the deployed .cfg is cmake output and
    is regenerated on rebuild — the June 22 aimpoint/gain revert).
  • Not compiled locally (no build environment); build on the instrument.
  • Acceptance after deployment: scrape a night's fine sessions and check every
    post-correction sampling gap sits after its move-DONE; the robot's
    completion-audit race detector stays armed as the regression alarm either
    way.

🤖 Generated with Claude Code

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 837370e902

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread acamd/acam_interface.cpp Outdated
Comment on lines +3700 to +3702
if ( goal_change_at_compute ) {
this->goalshift_pending.store( false, std::memory_order_release );
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Do not acknowledge a newer goal shift

If another offsetgoal request arrives after this guide iteration has read acam_goal but before its blocking pt_offset completes, goal_change_at_compute remains true and this store clears the shared flag for the newer, unexecuted goal. In particular, a fine-acquire correction issued while an earlier GUI/sequencer offset is in flight can then observe the resulting pending=false telemetry and resume sampling before its own move occurs, recreating the double-offset/false-lock failure this change is meant to prevent. Associate completion with a monotonically increasing request generation, or otherwise verify that the goal used for this move is still the latest request before clearing it.

Useful? React with 👍 / 👎.

Comment thread slicecamd/slicecam_interface.cpp Outdated
Comment on lines +586 to +588
return ( this->last_acam_pubtime.load(std::memory_order_acquire) > send_time
&& !this->acam_goalshift_pending.load(std::memory_order_acquire) )
|| !this->is_fineacquire_running.load(std::memory_order_acquire);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Wake the move wait when frame grabbing stops

When framegrab stop, close, or shutdown is requested during this wait, those paths clear should_framegrab_run but do not clear is_fineacquire_running or notify acam_cv, and this predicate ignores the framegrab stop request. Consequently the framegrab thread can remain here for up to the new 60-second timeout, while framegrab("stop") itself times out after roughly five seconds; close() then proceeds to close the cameras even though the framegrab thread has not exited. Include the framegrab stop state in the predicate and notify this condition variable when stopping the loop.

Useful? React with 👍 / 👎.

@cfremling
cfremling force-pushed the feat/fineacquire-wait-for-move branch from 837370e to df8e112 Compare August 14, 2026 06:19
@cfremling

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: df8e1120d5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread slicecamd/slicecam_interface.cpp Outdated
Comment on lines +586 to +587
return ( this->last_acam_pubtime.load(std::memory_order_acquire) > send_time
&& !this->acam_goalshift_pending.load(std::memory_order_acquire) )

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P1 Badge Require acknowledgement of this goal shift before completion

If the guide loop publishes goalshift_pending=false after send_time but before the synchronous offsetgoal command sets the flag, that status may reach slicecamd while the command's subsequent pending=true publication is still queued. The predicate then immediately accepts the newer false status when the command returns, so settling and sampling resume before this correction is executed, recreating the stale-sample/double-offset failure. Track a request generation, or require observing this request's pending state before accepting its completion.

Useful? React with 👍 / 👎.

While ACAM guides, a fine-acquire correction only re-points acamd's
goal; the guide loop applies it at its next executed correction. The
settle was timed from the send, so with short slicecam exposures the
next sample median could be built from pre-move frames and the same
correction applied twice (UT 2026-08-06, BD+28d4211: exposed 2.6 arcsec
off slit).

acamd tracks the request: offsetgoal marks the goal change pending; the
guide loop clears it when it executes a correction computed from the
changed goal (the pending state is captured before the goal is read, so
a correction already in flight can never clear it). The flag is
published; do_fineacquire waits for pending false on a status newer
than its send before settling. A stop interrupts the wait; timeout
FINE_ACQUIRE_MOVE_TIMEOUT (default 60 s, 0 disables) proceeds on the
old timer. Also fixes the offset_acam_goal log line, which printed
degrees labelled arcsec on the guiding path.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@cfremling
cfremling force-pushed the feat/fineacquire-wait-for-move branch from df8e112 to fbf1e77 Compare August 14, 2026 06:38
@cfremling

Copy link
Copy Markdown
Collaborator Author

@codex review

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: fbf1e77ec7

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread acamd/acam_interface.cpp
Comment on lines +3702 to +3705
if ( goal_change_at_compute &&
goal_ra_used == this->acam_goal.ra &&
goal_dec_used == this->acam_goal.dec ) {
this->goalshift_pending.store( false, std::memory_order_release );

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

P2 Badge Reset the guide filter before acknowledging the shifted goal

When offsetperiod is configured above 1, median_filter() can replace the correction calculated from goal_ra_used/goal_dec_used with a mature window accumulated for the previous goal. offset_goal() applies the new coordinates and publishes pending=true before it calls reset_offset_params(), so the guide thread can enter the filter during that interval, send the old median, and then clear the flag here merely because the coordinates still match. Slicecamd consequently treats the new goal as physically applied and resumes sampling even though the TCS received a pre-shift correction; reset the filter atomically before exposing the new goal/pending generation, or associate completion with the correction actually sent.

Useful? React with 👍 / 👎.

@cfremling

Copy link
Copy Markdown
Collaborator Author

Last Codex point (median filter vs goal registration) deliberately not addressed: the offset median filter is pass-through in practice (tcs_offset_period defaults to 1 and no config key sets it), so the scenario needs a configuration that cannot currently exist.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant