[CodeGen] Suppress tail calls in protected functions by claude[bot] · Pull Request #11 · trailofbits/llvm-project

claude · 2026-08-11T21:02:09Z

Requested by Francesco Bertolaccini · Slack thread

A tail call replaces the caller's frame with the callee's and jumps, so control
never comes back to the caller. A function carrying "zeroize-stack" has
undertaken to clear its frame before it returns, and a tail call takes away the
point at which it would do that: the frame stays live underneath the callee,
and the function reports itself protected while leaving in memory exactly what
the attribute exists to destroy. Tail-call optimization is suppressed in a
protected function. This is decision DD8, which the design records as settled.

The suppression goes where LLVM already decides tail-call eligibility rather
than into a check of its own. isInTailCallPosition in CodeGen/Analysis.cpp is
the target-independent answer to that question, and everything that forms a
tail call out of a call in the IR reaches it: SelectionDAGBuilder through
canTailCall, FastISel, and GlobalISel's CallLowering. Those are the same three
places that each honor "disable-tail-calls" separately, which is what asking
once here avoids. Folding a memcpy, memmove or memset into a tail call to the
library routine goes through it too, and replaces the frame just as thoroughly.

A libcall the legalizer generates has no call in the IR behind it and never
reaches that function. It is asked separately, by the SDNode overload of
TargetLowering::isInTailCallPosition, which carries its own "disable-tail-calls"
check for the same reason, and the second half of the change sits next to it.
The path is not hypothetical: an frem in return position becomes a tail call to
fmod on both x86 and ARM, and was the one remaining way a protected function
still jumped away from its frame.

musttail is diagnosed rather than suppressed. Declining an ordinary tail call
is available because forming one is an optimization; musttail is a requirement
the caller is not allowed to drop. A function that must be replaced at the call
and must clear its frame after it is a function that cannot be generated, so
the combination is rejected instead of being honored in one direction without
saying so.

The rejection is in the Verifier, in verifyMustTailCall, next to "cannot use
musttail call with inline asm". That neighbor has the same shape: not malformed
IR, but musttail combined with something that makes it impossible to honor, and
the Verifier is where that shape already lives. It is also the layer at which
the conflict is fully visible without a target. Leaving it to CodeGen would
surface as the backend's existing "failed to perform tail call elimination on a
call site marked musttail", which is fatal but never names the attribute that
caused it, and which is reached per target and twice over for a call FastISel
starts and SelectionDAG finishes. The frontend diagnostic for the same conflict
written in source is separate work.

Rejecting the combination in the Verifier makes it a bug for a pass to build
one, and one pass did. MergeFunctions rewrites a merged function into a thunk
that tail-calls the body, copies the attributes of the function it replaces
onto that thunk, and uses musttail when both functions are swifttailcc, so two
identical protected swifttailcc functions became a thunk carrying
"zeroize-stack" around a musttail call, aborting the compilation with "Broken
module found" from valid input. Protected functions are excluded from merging,
which is the answer inlining already gives them: a thunk standing in for a
protected function undertakes to clear a frame that no longer holds anything,
and where the convention makes its call musttail it cannot discharge the
undertaking at all.

tailcc and -tailcallopt are covered as well. Both exist to guarantee the
optimization rather than to permit it, and the guarantee is over the frame the
attribute is about, so a protected function does not obtain it by choosing the
convention. The cost is real: a protected tailcc function doing unbounded
mutual recursion now grows the stack. Whether that should be rejected the way
musttail is, rather than quietly losing the guarantee, is left to
trailofbits/vspells-ct-internal-notes#22 rather than settled here.

Where this meets the per-exit register clearing is worth stating, because the
two can look like they overlap. Clearing at a tail-call exit spares the
registers the callee is about to read as outgoing arguments, and a protected
function no longer has a tail-call exit: the exit classification for one now
reports both exits as returns where it used to report a tail call and a return.
That path is unreachable for a protected function. It is not dead. Register
clearing is driven by "zero-call-used-regs", a separate attribute that long
predates this work and that functions carry without "zeroize-stack"; those
functions still tail-call, and the per-exit set is still what makes their
tail-call exits correct. The test covering that case carries only
"zero-call-used-regs" and is untouched here. The two mechanisms answer for
disjoint sets of functions rather than for the same one twice.

No existing test changes, in CodeGen/X86, CodeGen/ARM, or anywhere under
Transforms. Nothing in the tree combines "zeroize-stack" with a tail call,
which is why the suppression arrives without an old test starting to expect
less. The new tests are the contrast in both directions on both targets: a
protected function that would otherwise jump does not, an unprotected one with
the same body still does, and the same pair for a legalizer libcall; the exit
classification changing from a tail call to a return; musttail in a protected
function rejected under two modes with an unprotected musttail untouched; and
the merged pair left unmerged. Each was confirmed load-bearing by breaking the
implementation once and restoring it: dropping either half of the suppression
failed the CodeGen tests at the corresponding check, dropping the Verifier
check failed the musttail test, and dropping the merging exclusion failed the
MergeFunc test.

This is trailofbits/vspells-ct-internal-notes#22, under the umbrella
trailofbits/vspells-ct-internal-notes#17.

AI tool use

This pull request contains AI-generated content. It was prepared with the assistance of Claude Code; the contributor has reviewed the generated code and text, is the author of the contribution, and is accountable for it, per the LLVM AI Tool Use Policy.

Generated by Claude Code

CLAassistant · 2026-08-11T21:02:21Z

Thank you for your submission! We really appreciate it. Like many open source projects, we ask that you sign our Contributor License Agreement before we can accept your contribution.
_{You have signed the CLA already but the status is still pending? Let us recheck it.}

github-actions · 2026-08-11T21:02:32Z

Hello @claude[bot] 👋

Thank you for submitting a Pull Request (PR) to the LLVM Project. Since this is your first PR, here are a few useful links covering our main contribution policies and review practices.

All contributions to LLVM must follow our LLVM AI Tool Use Policy. In particular, if you used AI while working on this PR, remember to add a note to the PR description.
The LLVM Code-Review Policy and Practices document contains practical information about the PR process, including how patches are reviewed and accepted, and who can review a PR.
Our LLVM Developer Policy describes our expectations for code quality, commit summaries and contains notes on our CI system.

Please reply to this message to confirm that you have read these policies, especially the LLVM AI Tool Use Policy, and that any AI tool usage has been noted in the PR description.

Frequently asked questions

How do I add reviewers?

This PR will be automatically labeled, and the relevant teams will be notified. For some parts of the project, reviewers may also be added automatically.

You can also add reviewers manually using the Reviewers section on this page. If you cannot use that section, it is probably because you do not have write permissions for the repository. In that case, you can request a review by tagging reviewers in a comment using @ followed by their GitHub username.

What if there are no comments?

If you have not received any comments on your PR after a week, you can request a review by pinging the PR with a comment such as “Ping”. The common courtesy ping rate is once a week. Please remember that you are asking for volunteer time from other developers.

Are any special GitHub settings required to contribute to LLVM?

We only require contributors to have a public email address associated with their GitHub commits, see this section of LLVM Developer Policy for details.

If you have questions, feel free to leave a comment on this PR, or ask on LLVM Discord or LLVM Discourse.

Thank you,
The LLVM Community

A tail call replaces the caller's frame with the callee's and jumps, so control never comes back to the caller. A function carrying "zeroize-stack" has undertaken to clear its frame before it returns, and a tail call takes away the point at which it would do that: the frame stays live underneath the callee, and the function reports itself protected while leaving in memory exactly what the attribute exists to destroy. Tail-call optimization is suppressed in a protected function. This is decision DD8, which the design records as settled. The suppression goes where LLVM already decides tail-call eligibility rather than into a check of its own. isInTailCallPosition in CodeGen/Analysis.cpp is the target-independent answer to that question, and everything that forms a tail call out of a call in the IR reaches it: SelectionDAGBuilder through canTailCall, FastISel, and GlobalISel's CallLowering. Those are the same three places that each honor "disable-tail-calls" separately, which is what asking once here avoids. Folding a memcpy, memmove or memset into a tail call to the library routine goes through it too, and replaces the frame just as thoroughly. A libcall the legalizer generates has no call in the IR behind it and never reaches that function. It is asked separately, by the SDNode overload of TargetLowering::isInTailCallPosition, which carries its own "disable-tail-calls" check for the same reason, and the second half of the change sits next to it. The path is not hypothetical: an frem in return position becomes a tail call to fmod on both x86 and ARM, and was the one remaining way a protected function still jumped away from its frame. musttail is diagnosed rather than suppressed. Declining an ordinary tail call is available because forming one is an optimization; musttail is a requirement the caller is not allowed to drop. A function that must be replaced at the call and must clear its frame after it is a function that cannot be generated, so the combination is rejected instead of being honored in one direction without saying so. The rejection is in the Verifier, in verifyMustTailCall, next to "cannot use musttail call with inline asm". That neighbor has the same shape: not malformed IR, but musttail combined with something that makes it impossible to honor, and the Verifier is where that shape already lives. It is also the layer at which the conflict is fully visible without a target. Leaving it to CodeGen would surface as the backend's existing "failed to perform tail call elimination on a call site marked musttail", which is fatal but never names the attribute that caused it, and which is reached per target and twice over for a call FastISel starts and SelectionDAG finishes. The frontend diagnostic for the same conflict written in source is separate work. Rejecting the combination in the Verifier makes it a bug for a pass to build one, and one pass did. MergeFunctions rewrites a merged function into a thunk that tail-calls the body, copies the attributes of the function it replaces onto that thunk, and uses musttail when both functions are swifttailcc, so two identical protected swifttailcc functions became a thunk carrying "zeroize-stack" around a musttail call, aborting the compilation with "Broken module found" from valid input. Protected functions are excluded from merging, which is the answer inlining already gives them: a thunk standing in for a protected function undertakes to clear a frame that no longer holds anything, and where the convention makes its call musttail it cannot discharge the undertaking at all. tailcc and -tailcallopt are covered as well. Both exist to guarantee the optimization rather than to permit it, and the guarantee is over the frame the attribute is about, so a protected function does not obtain it by choosing the convention. The cost is real: a protected tailcc function doing unbounded mutual recursion now grows the stack. Whether that should be rejected the way musttail is, rather than quietly losing the guarantee, is left to trailofbits/vspells-ct-internal-notes#22 rather than settled here. Where this meets the per-exit register clearing is worth stating, because the two can look like they overlap. Clearing at a tail-call exit spares the registers the callee is about to read as outgoing arguments, and a protected function no longer has a tail-call exit: the exit classification for one now reports both exits as returns where it used to report a tail call and a return. That path is unreachable for a protected function. It is not dead. Register clearing is driven by "zero-call-used-regs", a separate attribute that long predates this work and that functions carry without "zeroize-stack"; those functions still tail-call, and the per-exit set is still what makes their tail-call exits correct. The test covering that case carries only "zero-call-used-regs" and is untouched here. The two mechanisms answer for disjoint sets of functions rather than for the same one twice. No existing test changes, in CodeGen/X86, CodeGen/ARM, or anywhere under Transforms. Nothing in the tree combines "zeroize-stack" with a tail call, which is why the suppression arrives without an old test starting to expect less. The new tests are the contrast in both directions on both targets: a protected function that would otherwise jump does not, an unprotected one with the same body still does, and the same pair for a legalizer libcall; the exit classification changing from a tail call to a return; musttail in a protected function rejected under two modes with an unprotected musttail untouched; and the merged pair left unmerged. Each was confirmed load-bearing by breaking the implementation once and restoring it: dropping either half of the suppression failed the CodeGen tests at the corresponding check, dropping the Verifier check failed the musttail test, and dropping the merging exclusion failed the MergeFunc test. This is trailofbits/vspells-ct-internal-notes#22, under the umbrella trailofbits/vspells-ct-internal-notes#17.

Follow-up to the tail-call suppression, addressing three review findings. The Verifier rejects a musttail call in a "zeroize-stack" function, but CoroSplit synthesizes one: a presplit coroutine is lowered into resume and destroy clones that hand off with musttail calls (symmetric transfer, and the async coro.end), and the coroutine's function attributes are copied onto those clones. A protected coroutine therefore became a protected function holding a musttail call, aborting with "Broken module found" on input that verified. Reject "zeroize-stack" on a presplit coroutine up front, the same way the musttail combination is rejected, so the split never runs on it. The suppression covered SelectionDAG but not GlobalISel. GlobalISel forms legalizer libcalls on its own path and decides the tail call in the legalizer rather than in TargetLowering::isInTailCallPosition, so a protected function returning frem under -global-isel still tail-branched to fmod with no return path to clear its frame. Add the guard to isLibCallInTailPosition, the shared predicate both the general and the memory libcall paths reach. The guarantee was gated on the bare string "zeroize-stack" repeated across every enforcement site, where a single typo would silently drop the protection. Add Function::hasZeroizeStack() and getZeroizeStackMode(), naming the attribute once, and route CodeGen, the Verifier, the IPO passes and the inline-mode check through them. Tests: reject the protected coroutine in the Verifier, and keep the AArch64 GlobalISel libcall in its own frame. This is trailofbits/vspells-ct-internal-notes#22, under the umbrella trailofbits/vspells-ct-internal-notes#17.

frabert · 2026-09-02T08:53:51Z

+  // rejected in the Verifier (a caller cannot drop it); suppressing it here too
+  // is a backstop for unverified IR, failing closed into the backend's musttail
+  // error rather than a protected function that keeps its frame.
+  if (Call.getCaller()->hasZeroizeStack())


P2: This suppresses every tail-call candidate, including a tail-position tailcc call. tailcc guarantees tail-call optimization, unlike an ordinary tail marker, and the verifier does not reject zeroize-stack combined with tailcc. Consequently the backend silently demotes a call whose calling convention requires a tail call.

Please reject or diagnose zeroize-stack functions containing guaranteed tail-call conventions (including tailcc) rather than overriding the calling-convention contract. A verifier regression for this combination would cover the gap.

Updated to reject zeroize-stack functions containing guaranteed tail calls. Silently demoting such calls would violate the guarantee specified by the LangRef.

frabert · 2026-09-02T08:53:53Z

+  // coroutine would become a protected function holding a musttail call, which
+  // verifyMustTailCall rejects, so reject it here before the split for the same
+  // reason: the frame the attribute must clear is handed off and never cleared.
+  Check(!F.isPresplitCoroutine() || !F.hasZeroizeStack(),


P2: This catches only functions that already carry presplitcoroutine. However, CoroEarlyPass adds that attribute for llvm.coro.id_retcon, llvm.coro.id_retcon_once, and llvm.coro.id_async after initial IR verification. The normal coroutine pipeline then reaches CoroSplitPass without an intervening verifier, so a zeroize-stack async/retcon coroutine can still be split and propagate its attributes onto generated musttail handoff clones.

Please reject this combination when CoroEarly marks the function as presplit, or add an equivalent guard in CoroSplit. An end-to-end coro-early → coro-split regression using initially unmarked async/retcon IR would cover this path.

Updated CoroEarly to reject zeroize-stack async/retcon coroutines before adding presplitcoroutine. This prevents them from reaching CoroSplit and generating incompatiblemusttail clones.

…nctions A tail marker is ordinarily a hint that CodeGen may drop, and it does for a function carrying "zeroize-stack". tailcc and swifttailcc instead guarantee the tail call, so suppressing it silently breaks the convention's contract. Reject the combination in the Verifier, next to the musttail check, when the caller's own convention is tailcc or swifttailcc and the call is marked tail. The Verifier also rejects "zeroize-stack" on a presplit coroutine, but CoroEarly is what marks a retcon, retcon-once, or async coroutine presplit, so protected input verified and then aborted with "Broken module found" after the pass ran. Emit an error in CoroEarly before setting the attribute so the diagnostic names the conflict. Drop the protected_tailcc case from the CodeGen test, since the Verifier now refuses that input, and reword the suppression comment to match.

kumarak · 2026-09-08T17:51:32Z

@claude review this PR

claude Bot force-pushed the zeroize-per-exit-regs branch from c422de1 to 7a8977f Compare August 13, 2026 13:04

claude Bot force-pushed the zeroize-tailcall branch from 78bf8ec to 537628e Compare August 13, 2026 13:04

claude Bot mentioned this pull request Aug 13, 2026

[CodeGen] Fall back to clearing more when the analysis is incomplete #12

Draft

claude Bot force-pushed the zeroize-per-exit-regs branch from 7a8977f to e2d24c8 Compare August 13, 2026 14:29

claude Bot force-pushed the zeroize-tailcall branch from 537628e to 3a8b2b4 Compare August 13, 2026 14:30

claude Bot force-pushed the zeroize-per-exit-regs branch from e2d24c8 to 3e896ab Compare August 14, 2026 12:05

claude Bot force-pushed the zeroize-tailcall branch from 3a8b2b4 to 90e4cce Compare August 14, 2026 12:06

claude Bot force-pushed the zeroize-per-exit-regs branch from 3e896ab to 7283276 Compare August 14, 2026 19:26

claude Bot force-pushed the zeroize-tailcall branch from 90e4cce to 0aa0611 Compare August 14, 2026 19:27

frabert force-pushed the zeroize-tailcall branch from 0aa0611 to 785f0b0 Compare August 18, 2026 10:18

frabert force-pushed the zeroize-per-exit-regs branch from 7283276 to 13fecd7 Compare August 18, 2026 10:18

claude Bot force-pushed the zeroize-per-exit-regs branch from 13fecd7 to 3f3f346 Compare August 18, 2026 11:06

claude Bot force-pushed the zeroize-tailcall branch from 785f0b0 to 9ebb517 Compare August 18, 2026 11:07

claude Bot mentioned this pull request Aug 18, 2026

[IR] Do not inline functions carrying "zeroize-stack" #4

Merged

frabert force-pushed the zeroize-tailcall branch from 9ebb517 to 5078e59 Compare August 19, 2026 09:06

frabert force-pushed the zeroize-per-exit-regs branch from 3f3f346 to 3140df4 Compare August 19, 2026 09:06

frabert force-pushed the zeroize-tailcall branch from 5078e59 to fcac096 Compare August 19, 2026 13:41

frabert force-pushed the zeroize-per-exit-regs branch 2 times, most recently from 2ab5e3a to f1b92b8 Compare August 21, 2026 09:30

frabert force-pushed the zeroize-tailcall branch from fcac096 to ce62d01 Compare August 21, 2026 09:31

frabert force-pushed the zeroize-per-exit-regs branch from f1b92b8 to 476e09e Compare August 24, 2026 14:17

frabert force-pushed the zeroize-tailcall branch from ce62d01 to 731965e Compare August 24, 2026 14:17

frabert force-pushed the zeroize-per-exit-regs branch from 476e09e to 02987b4 Compare August 24, 2026 14:24

frabert force-pushed the zeroize-tailcall branch from 731965e to b26227e Compare August 24, 2026 14:24

frabert force-pushed the zeroize-per-exit-regs branch from 02987b4 to bc7c196 Compare August 25, 2026 14:04

frabert force-pushed the zeroize-tailcall branch from b26227e to 6f9e538 Compare August 25, 2026 14:04

frabert force-pushed the zeroize-per-exit-regs branch from bc7c196 to 459f061 Compare August 26, 2026 11:28

frabert force-pushed the zeroize-tailcall branch from 6f9e538 to f2705d9 Compare August 26, 2026 11:28

frabert force-pushed the zeroize-per-exit-regs branch from 459f061 to c54a7f3 Compare August 26, 2026 12:55

frabert force-pushed the zeroize-tailcall branch from f2705d9 to 8fd3ebc Compare August 26, 2026 12:55

kumarak force-pushed the zeroize-per-exit-regs branch from c54a7f3 to 59b0d5a Compare August 28, 2026 14:41

Base automatically changed from zeroize-per-exit-regs to enforced_secrecy_main August 31, 2026 17:20

kumarak marked this pull request as ready for review August 31, 2026 23:15

kumarak force-pushed the zeroize-tailcall branch from 8fd3ebc to b704392 Compare August 31, 2026 23:22

kumarak force-pushed the zeroize-tailcall branch from b704392 to 217ee96 Compare September 1, 2026 00:03

frabert reviewed Sep 2, 2026

View reviewed changes

kumarak requested a review from frabert September 8, 2026 17:44

kumarak force-pushed the zeroize-tailcall branch from e976d47 to fa8b3a4 Compare September 8, 2026 17:51

Navigation Menu

Uh oh!

[CodeGen] Suppress tail calls in protected functions - #11

[CodeGen] Suppress tail calls in protected functions#11
claude[bot] wants to merge 3 commits into
enforced_secrecy_mainfrom
zeroize-tailcall

claude Bot commented Aug 11, 2026

Uh oh!

CLAassistant commented Aug 11, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Aug 11, 2026

Uh oh!

frabert Sep 2, 2026

Uh oh!

kumarak Sep 8, 2026

Uh oh!

frabert Sep 2, 2026

Uh oh!

kumarak Sep 8, 2026

Uh oh!

kumarak commented Sep 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

Uh oh!

Conversation

claude Bot commented Aug 11, 2026

AI tool use

Uh oh!

CLAassistant commented Aug 11, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Aug 11, 2026

Frequently asked questions

Uh oh!

frabert Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

kumarak Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

frabert Sep 2, 2026

Choose a reason for hiding this comment

Uh oh!

kumarak Sep 8, 2026

Choose a reason for hiding this comment

Uh oh!

kumarak commented Sep 8, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

CLAassistant commented Aug 11, 2026 •

edited

Loading