[v2] Handle broken pipe when command output is piped to an early-exiting reader - #10538
Open
HenrikGharagyozyan wants to merge 1 commit into
Open
[v2] Handle broken pipe when command output is piped to an early-exiting reader#10538HenrikGharagyozyan wants to merge 1 commit into
HenrikGharagyozyan wants to merge 1 commit into
Conversation
Piping a command's output into a reader that exits early, such as "aws s3 ls s3://bucket/ | head -1", printed "[Errno 32] Broken pipe" along with an "Exception ignored" traceback and exited non-zero. BrokenPipeError from writing to the closed pipe reached the blanket except clause in CLIDriver.main() and was handled by GeneralExceptionHandler. The interpreter's final stdout flush then failed on the same pipe, which emitted the traceback and replaced the exit status with Python's own flush failure code. Add a BrokenPipeExceptionHandler to both handler chains, modelled on the existing InterruptExceptionHandler, returning 128 + SIGPIPE to match what standard Unix utilities report for a closed pipe. Point the stdout descriptor at devnull before returning so the shutdown flush cannot fail. Restoring signal.SIG_DFL for SIGPIPE was considered and rejected: Python sets SIGPIPE to SIG_IGN at startup so socket writes raise EPIPE as catchable exceptions, and restoring the default would make broken socket writes terminate the process instead of being retried.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Piping a command's output into a reader that exits early —
aws s3 ls s3://bucket/ | head -1— currently prints[Errno 32] Broken pipe, anException ignored in: <_io.TextIOWrapper ...>traceback, and exits non-zero. This makes it exit quietly with 141.Root cause
s3 lswrites each page through_display_page()→uni_print()→sys.stdout. When the downstream reader exits, the write raisesBrokenPipeError, which reaches the blanketexcept BaseExceptioninCLIDriver.main()and is handled byGeneralExceptionHandler(255). The interpreter's final stdout flush at shutdown then fails on the same closed pipe, emitting the "Exception ignored" line and replacing the exit status with Python's own flush-failure code — which is why #5899 has reports of both 255 and 120.Observed on
v2before this change, against a local stub endpoint:After:
aws_rc=141, empty stderr. An unpiped listing is unchanged at rc 0.Approach
Adds a
BrokenPipeExceptionHandlerto both handler chains, modelled directly on the existingInterruptExceptionHandler(RC = 128 + signal.SIGINT). It returns128 + SIGPIPE(141), the same statusheadandgrepthemselves report for a closed pipe underpipefail. Before returning it points the stdout descriptor at devnull, per the note on SIGPIPE in the Python docs, so the shutdown flush cannot fail.Registered in
construct_cli_error_handlers_chain()as well asconstruct_entry_point_handlers_chain(), sinceCLIDriver.main()catches the exception first.This follows treatment the codebase already applies elsewhere rather than introducing a new convention:
awscli/customizations/logs/ui.pyguards SIGPIPE withif sys.platform != "win32", andAWSCLIPagerManager.get_pager_stream()already swallowsOSErrorbecause "a pager is closed abruptly and causes a broken pipe". The plain-stdout path simply had no equivalent.Alternative considered: restoring
signal.SIG_DFLfor SIGPIPE. Rejected — Python sets SIGPIPE toSIG_IGNat startup so socket writes raiseEPIPEas catchable exceptions; restoring the default would make broken socket writes terminate the process outright instead of being retried by botocore.getattr(signal, 'SIGPIPE', 13)keeps the return code identical on Windows, which has no SIGPIPE but can still raiseBrokenPipeError.Testing
tests/unit/test_errorhandler.py): both chains return 141 with no stderr; the RC matches128 + signal.SIGPIPE; the devnull redirect discards writes and is a no-op for streams with no file descriptor; and unrelatedOSErrors still report 255, sinceBrokenPipeErroris anOSErrorsubclass.tests/functional/s3/test_ls_command.py): end-to-end through the reals3 lspath. Fails onv2today with255 != 141.aws help return-codes.Closes #5899
This change was generated by AI tools, and reviewed by Henrik Gharagyozyan.