Skip PSF convolution for exactly-uniform fields

[astronomyk]

What

PSF.apply_to subtracts the background level before convolving and re-adds it afterwards (psf_base.py). For a spatially uniform field the convolution input is therefore identically zero — the operation provably cannot change the image — yet the kernel interpolation/rescale and the padded-grid FFT were still executed every time.

This PR checks the residual (image − bkg_level) before get_kernel and skips the convolution when it is exactly zero everywhere.

Why it matters (measured, METIS)

Uniform fields are the most expensive frames METIS generates, because a closed light path implies the full thermal waveset and the largest FOV chunks:

• an img_lm dark spends 57 s of a 75 s frame (76%) in PSF.apply_to — kernel rescale + FFT at up to 4639² — convolving a field the operation cannot change;
• the IFU (lms/wcu_lms) pays the same pattern per (n_wave, x, y) cube (observed up to 6383 planes, ~1.6 GB per padded copy), where the FFT copies have also intermittently exhausted memory and killed generation with no traceback;
• for zero-flux IFU darks, skipping the PSF was measured exactly lossless: frames with and without it are identical at rtol 1e-9 on every header card and every data statistic (measured downstream via metis["psf"].include = False, the manual equivalent of this shortcut — see Add --noPsf to skip the PSF convolution for source-free frames METIS_Simulations#212).

Why it is safe

• The check is exact, not a tolerance: any structure at all takes the unchanged full path. When it fires, the output equals the input by the function's own algebra: convolve(0, kernel) + bkg == bkg.
• The FovVolumeList waveset handling earlier in apply_to is untouched, so FOV chunking is identical whether or not the shortcut fires. (This matters: disabling the whole psf effect — the manual workaround — changes FOV chunking and shifts imaging-dark statistics at the 1e-5 level. This shortcut has no such side effect, which is why it is bit-exact where the workaround is only approximately equivalent.)
• The FOV data can be an astropy Quantity; the residual test goes through np.asarray for that reason.
• The only code motion is computing image/bkg_level before get_kernel instead of after; neither depends on the kernel.

Validation

Run downstream through METIS_Simulations with the differential harness (per-pixel comparison of every frame, matched by DO.CATG + MJD-OBS, all 4,194,304 px each):

Case	Wall time (main → patched)	Pixel verdict vs unpatched main
darkLM.yaml (3 frames, H2RG)	162 s → 23 s (7×)	3/3 frames BITWISE IDENTICAL
darkN.yaml (3 frames, GeoSnap)	110 s → 20 s (5.5×)	3/3 frames BITWISE IDENTICAL

The per-FOV fields of METIS imaging darks are exactly uniform, so the shortcut fires on every dark/closed-shutter FOV and the frames are unchanged to the bit. The IFU cube case (where the skipped work is largest, and where the padded-FFT copies have intermittently exhausted memory) gains proportionally more; zero-flux IFU skips were independently measured exactly lossless via the manual route (AstarVienna/METIS_Simulations#212).

(Full harness and measurement log: METIS_Simulations_Deux, findings F-D8/F-D9/F-D24/F-D30/F-D38.)

🤖 Generated with Claude Code

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 75.91%. Comparing base (608adfa) to head (cc19b8f).

Additional details and impacted files

@@            Coverage Diff             @@
##             main     #973      +/-   ##
==========================================
+ Coverage   75.89%   75.91%   +0.01%     
==========================================
  Files          70       70              
  Lines        9030     9037       +7     
==========================================
+ Hits         6853     6860       +7     
  Misses       2177     2177              

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[astronomyk]

Here's a suggestion from the toaster to speed up flat-field and dark simulations

@hugobuddel it's the upstream analogue to AstarVienna/METIS_Simulations#212 as you proposed.

It shouldn't change anything in science-related simulations, but should cut the METIS_Simulations weekly build down from 3+ hours to <1 hour, also meaning everyone on the pipeline team can run things locally, rather than downloading from Zeus, and that we can start branching out from the standard stack of use-cases to also test edge cases for each of the pipeline recipes.

To me the addition looks like it won't hurt ScopeSim, but I'd prefer to have both of your opinions on potential consequences before adding it.

[oczoske]

I've recommended switching off the psf effect(s) where appropriate for a while now but never thought of doing this automatically. This looks robust enough and also deals with cubes correctly.

[hugobuddel]

I suppose the subtraction could maybe lead to floating point errors. Maybe just check whether the values are all the same / constant? That seems simpler, but I suppose this should work too.

Note that if anyone of us would want to get 'toaster suggestions', we can ask them ourselves. We want you to evaluate what the toaster does and then present your human analysis to us. That is, it is your contribution, perhaps helped by an LLM. Feeding us toaster unfiltered output and then let us do the reviewing is rude.

@oczoske I objected to explicitly turning off the PSF effect in METIS_Simulations, because that is hacky. METIS_Simulations should just have some settings to feed to ScopeSim in a declarative way and not then tweak things. The PSF should either be enabled and skipped like this, or turned off in a yaml file in the irdb.

Related, if METIS_Simulations works that way, then we should be able to get to the point where we can reconstruct the ScopeSim input from the headers that ScopeSim outputs. We will never get there we explicitly go modify Effects from METIS_Simulations.

[teutoburg]

I suppose the subtraction could maybe lead to floating point errors. Maybe just check whether the values are all the same / constant? That seems simpler, but I suppose this should work too.

That was my first thought as well.

Note that if anyone of us would want to get 'toaster suggestions', we can ask them ourselves. We want you to evaluate what the toaster does and then present your human analysis to us. That is, it is your contribution, perhaps helped by an LLM. Feeding us toaster unfiltered output and then let us do the reviewing is rude.

Thanks, +1 on that.

[teutoburg]

@oczoske I objected to explicitly turning off the PSF effect in METIS_Simulations, because that is hacky. METIS_Simulations should just have some settings to feed to ScopeSim in a declarative way and not then tweak things. The PSF should either be enabled and skipped like this, or turned off in a yaml file in the irdb.

Disagree somewhat. The exclude_effects is a "top level config" when setting up a Simulation or OpticalTrain. See also the JSON structure on our recent SPIE poster. I do think METIS_Simulations should eventually use that, once it's fully useable and properly schema'd.

[astronomyk]

@oczoske thanks for the approval

@teutoburg METIS_Simulations will probably never be fully useable and properly schema'd. It was only ever meant for us and the AIT team to batch generate specific use cases. So I'm ok with hacking it to do what needs to be done. But this fix also seems like it could be useful beyond just METIS_Simulations.

@hugobuddel see my other rant in #974. I often question my decision to move into management instead of continuing on as a developer. I wish I still was. But alas it's not my job anymore to make the code work properly. So if it gets to the point that I'm feeding you LLM-generated PRs, it's because this should have been done long ago, and not by me. I don't intend it as rudeness, it's simply a means to an end. You all have a much better understanding of the code bases, and so I bring you in here to get your opinions. I vetted the code as doing what I want it do before the PR went up, but I don't have a free 30 mins to write a bespoke PR request. Claude captured the essence and did it in 30 secs. It's purely time management on my behalf. So my apologies for offending you.

....

Grumpiness aside, I do very much appreciate all that you guys do and know about the projects we have going. So, meant fully sincerely, my thanks for all you do - even putting up with my toaster-generated PRs! 👍 :)

[teutoburg]

@teutoburg METIS_Simulations will probably never be fully useable and properly schema'd. It was only ever meant for us and the AIT team to batch generate specific use cases. So I'm ok with hacking it to do what needs to be done. But this fix also seems like it could be useful beyond just METIS_Simulations.

Sorry if I was unclear, the "fully useable and schema'd" never referred to METIS_Simulations itself, but rather to the "JSON-ish-based simulation config" that we presented at SPIE, that is used for "ScopeSim on Zeus" and by extension the A*V Hub in the future. My point was that that syntax would allow for a more declarative and stanardized configuration of ScopeSim simulations, which is something that METIS_Simulations would benefit from using, IMHO. But alas, it (that being the syntax) doesn't really exist yet, so it doesn't matter now anyway...

On LLM'd PRs and your frustration in general: If there is an issue that you're seeing that you think should be addressed by someone in @AstarVienna/scopesim-devs, please do write an issue for that! The issue itself can be LLM'd if you want, because it's just text (plus a MVE if applicable). But if things just get mentioned in a meeting somewhere, they will get lost. If there was an issue for it, which didn't get addressed in time, then the real problem is prioritization. We can talk about that if needed.

[hugobuddel]

Disagree somewhat. The exclude_effects is a "top level config" when setting up a Simulation or OpticalTrain. See also the JSON structure on our recent SPIE poster. I do think METIS_Simulations should eventually use that, once it's fully useable and properly schema'd.

Thanks for the clarification, using exclude_effects makes sense then. #212 simply set metis["psf"].include = False, which it should not do.

Maybe we should add a "do things like this and not like this" document somewhere, so the agents can read it too.

So I'm ok with hacking it to do what needs to be done.

On the other hand, with your management hat on, you are also the one to ensure that we will have a maintainable code base; that is, getting rid of these hacks. That's why I called you out on it in #212, I would be less direct with anyone else (I hope).

(And also here, your human-written message was more useful than the LLM one! Thanks.)

(And don't sell yourself short, you still have a great understanding of the codebase.)