[pull] master from google:master - #198
Merged
Merged
Conversation
This is a PR intended to revive the PostgreSQL participation in Google OSS-Fuzz by replacing the current inactive primary_contact with a new active maintainer of the PostgreSQL project. Stephen Frost is no longer active in the PostgreSQL project. Daniel Gustafsson is a long-time PostgreSQL Major Contributor [0] and code committer [1,2]. This change has been approved by the PostgreSQL Core Team [3]. Daniel Gustafsson Email: daniel@yesql.se, dgustafsson@postgresql.org Gmail: dgustafssonpgeu@gmail.com Github: @danielgustafsson Once merged, the plan is to extend the PostgreSQL fuzz test coverage by adding new tests. [0] https://www.postgresql.org/community/contributors/ [1] https://www.postgresql.org/developer/committers/ [2] https://github.com/postgres/postgres/commits?author=danielgustafsson [3] https://www.postgresql.org/developer/core/
## Problem Every `nfstream` OSS-Fuzz build currently fails `bad_build_check`, and has since NumPy 2.3.0 entered the base image. The target is built, then dies before reaching `LLVMFuzzerTestOneInput`: ``` BAD BUILD: /tmp/not-out/.../pcap_fuzzer seems to have either startup crash or exit: Traceback (most recent call last): File "numpy/_core/__init__.py", line 24, in <module> File "numpy/_core/multiarray.py", line 11, in <module> ModuleNotFoundError: No module named 'numpy._core._exceptions' ... ImportError: Unable to import required dependency numpy. [PYI-225:ERROR] Failed to execute script 'pcap_fuzzer' due to unhandled exception! ``` The seed corpus and the packaging command complete successfully; the resulting frozen executable fails during startup because its import graph is incomplete. ## Cause NumPy's `_multiarray_umath` extension imports five classes from `numpy._core._exceptions` during its initialisation, via `IMPORT_GLOBAL` in [`numpy/_core/src/multiarray/npy_static_data.c`](https://github.com/numpy/numpy/blob/v2.4.6/numpy/_core/src/multiarray/npy_static_data.c#L144-L157) — `_ArrayMemoryError`, `_UFuncBinaryResolutionError`, `_UFuncInputCastingError`, `_UFuncNoLoopError` and `_UFuncOutputCastingError`. Those imports live in compiled C, so PyInstaller's static analysis cannot see them and the module is never collected. It used to be collected indirectly because `numpy._core._methods` imported it at Python level. NumPy commit [a51a4f5](numpy/numpy@a51a4f5) removed that import during cleanup. The build image pairs PyInstaller 6.10.0 with NumPy 2.4.6, and PyInstaller selects NumPy's own vendored hook — the build log shows `hook-numpy.py` loaded from `site-packages/numpy/_pyinstaller`, not from `site-packages/PyInstaller/hooks`. That vendored hook declares only: ```python hiddenimports = ['numpy._core._dtype_ctypes', 'numpy._core._multiarray_tests'] ``` PyInstaller fixed its own copy of the hook in [pyinstaller#9162](pyinstaller/pyinstaller#9162), which adds `numpy._core._exceptions` for NumPy >= 2.3.0 and first shipped in PyInstaller 6.14.1. That version also carries the `$PyInstaller-Hook-Priority: 1` marker its hook needs to take precedence over NumPy's vendored one — 6.13.0 already had the marker but not the hidden import, so 6.14.1 is the first release with both, later than the 6.10.0 in the image. ## Fix Declare the module explicitly: ```bash compile_python_fuzzer $fuzzer \ --hidden-import=_cffi_backend \ --hidden-import=numpy._core._exceptions ``` This is preferable to pinning NumPy or overriding PyInstaller: it addresses the observed startup failure directly, works with the image as it stands, and becomes redundant but harmless once the image carries a PyInstaller new enough to supply the hidden import itself. ## Testing `bash -n projects/nfstream/build.sh` passes. Runtime validation will come from OSS-Fuzz's `build_fuzzers` and `check_build` steps; `check_build` currently reproduces the startup failure and will directly verify this fix.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
See Commits and Changes for more details.
Created by
pull[bot] (v2.0.0-alpha.4)
Can you help keep this open source service alive? 💖 Please sponsor : )