[Python][Parquet] Errno 14 Error reading bytes from file with `pyarrow.parquet.ParquetFile()`, but consistently succeeds on second try

[zianliu66-me]

Describe the bug, including details regarding any error messages, version, and platform.

The following bug happens specifically on a Python venv I created on an HPC. Python version is 3.13.5 with pyarrow version 25.0.1. This is a bioinformatics environment aimed to use single-cell packages (such as scanpy/anndata), so possibly other contributing packages also include numpy==2.5.3, pandas==3.0.6, anndata==0.13.4.

The following error reproducibly occurs whenever I run pyarrow.parquet.ParquetFile(filename). However, this error no longer occurs if I 1) run pyarrow.parquet.ParquetFile(filename) directly from the interpreter, or 2) if I specifically do a try/except loop and re-run ParquetFile() on the except (but it doesn't work with a while loop).

Error trace:

Traceback (most recent call last):
  File "/globalscratch/<example>/debug_parquetread.py", line 92, in <module>
    main()
    ~~~~^^
  File "/globalscratch/<example>/debug_parquetread.py", line 86, in main
    pq_example(PQ_SUBSET)
    ~~~~~~~~~~^^^^^^^^^^^
  File "/globalscratch/<example>/debug_parquetread.py", line 51, in pq_example
    parquet_file = pq.ParquetFile(filename)
  File "/globalscratch/<example>/venv/lib/python3.13/site-packages/pyarrow/parquet/core.py", line 331, in __init__
    self.reader.open(
    ~~~~~~~~~~~~~~~~^
        source, use_memory_map=memory_map,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    ...<8 lines>...
        arrow_extensions_enabled=arrow_extensions_enabled,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "pyarrow/_parquet.pyx", line 1676, in pyarrow._parquet.ParquetReader.open
  File "pyarrow/error.pxi", line 92, in pyarrow.lib.check_status
OSError: [Errno 14] Error reading bytes from file. Detail: [errno 14] Bad address

See the following script for reproducing this issue. This is guaranteed to fail on my setup; if I comment out pq_example(), the other two functions behave exactly as described.

import os
import pyarrow.parquet as pq

def pq_example(filename: str):
    # This always fails.
    parquet_file = pq.ParquetFile(filename)
    print("This import worked.")
    return None

def pq_example_2(filename: str):
    # This also always fails, but won't return error due to the try clause.
    i = 0
    while i < 10:
        try:
            parquet_file = pq.ParquetFile(filename)
            print("Loaded successfully.")
            break
        except:
            print(f"Read failed on loop {i}. Retrying...")
            i += 1
            pass
    return None

def pq_example_working(filename: str):
    # This one consistently works on the 'except' read.
    try:
        parquet_file = pq.ParquetFile(filename)
        print("The first import worked.")
    except:
        print("The first read failed. Retrying:")
        parquet_file = pq.ParquetFile(filename)
        print("The second read succeeded.")
    return None

def main():
    pq_example(filename)
    pq_example_working(filename)
    pq_example_2(filename)

if __name__ == '__main__':
    main()

The exact OS information:

NAME="Rocky Linux"
VERSION="8.10 (Green Obsidian)"
ID="rocky"
ID_LIKE="rhel centos fedora"
VERSION_ID="8.10"
PLATFORM_ID="platform:el8"
PRETTY_NAME="Rocky Linux 8.10 (Green Obsidian)"
ANSI_COLOR="0;32"
LOGO="fedora-logo-icon"
CPE_NAME="cpe:/o:rocky:rocky:8:GA"
HOME_URL="https://rockylinux.org/"
BUG_REPORT_URL="https://bugs.rockylinux.org/"
SUPPORT_END="2029-05-31"
ROCKY_SUPPORT_PRODUCT="Rocky-Linux-8"
ROCKY_SUPPORT_PRODUCT_VERSION="8.10"
REDHAT_SUPPORT_PRODUCT="Rocky Linux"
REDHAT_SUPPORT_PRODUCT_VERSION="8.10"

Component(s)

Python

[raulcd]

I really don't understand the minimal case to reproduce this. The example contains three attempts to open a parquet file. One says working, one is on a while loop and is not clear at all what is the problem. Please, could you try to share the minimal reproducer? Is it opening the parquet file on a while loop?
Is it after doing something specific?
Is it for a specific Parquet file or is reproducible with any Parquet file?
Was PyArrow installed from PyPI, from conda, was built from source?

What would be helpful is a snippet with something like this, btw I can't reproduce with the following:

import pyarrow as pa
import pyarrow.parquet as pq

table = pa.table({'a': range(10), 'b': [10] * 10, 'c': [1, 2] * 5})

pq.write_table(table, 'output.parquet')

i = 0
while i < 10:
    parquet_file = pq.ParquetFile('output.parquet')
    i += 1

Please try to use plain languange, AI tends to be overly verbose and it's difficult to grasp the real issue. Thanks!

[zianliu66-me]

Apologies. It's not AI, I just have a tendency to write too verbosely and haven't submitted many bug reports... please definitely let me know if what I wrote below is not clear.

See minimal case below. This is analogous to the code I used to create the problematic parquet file in the first place. This script actually fails earlier, on scanner.to_batches(), with the same error message... But the created parquet file can be read with ParquetFile() just fine:

import pyarrow as pa
import pyarrow.parquet as pq
import pyarrow.dataset as ds

table = pa.table({'a': range(500000), 'b': [10] * 500000, 'c': [1, 2] * 250000})
pq.write_table(table, 'example.parquet')

dataset = ds.dataset(
    "example.parquet",
    format="parquet"
)

scanner = dataset.scanner()
writer = None
try:
    for batch in scanner.to_batches():
        if writer is None:
            writer = pq.ParquetWriter(
                "example2.parquet",
                batch.schema
            )
        writer.write_batch(batch)
finally:
    if writer is not None:
        writer.close()

The script above consistently fails on for batch in scanner.to_batches(): with the same errno 14 error message.

However, it consistently does not fail if I pre-emptively call print(scanner.to_batches()) prior to the while loop. It also does not fail if the parquet is much smaller (table = pa.table({'a': range(10), 'b': [10] * 10, 'c': [1, 2] * 5}) works).

Core issue: errno 14 seems to occur whenever I interact with any sufficiently large parquet file on this system, not necessarily via pq.ParquetFile().

The issue can be bypassed by invoking some type of call to the file... In the above snippet, pre-emptively calling print(scanner.to_batches()) prior to the for loop solves the issue. In my production code, simply calling pq.ParqeutFile() twice bypasses the issue, as the second pq.ParquetFile() command no longer returns an error.

This seems to work with any sufficiently large parquet file, but smaller parquets don't have this issue.

PyArrow was installed via pip onto a venv by first running python -m pip install --upgrade pip setuptools wheel, and then running python -m pip install -r requirements.txt. requirements.txt contains at least the following:

numpy
pandas
pyarrow
dask[array,dataframe,distributed]
zarr
datasets

[raulcd]

Thanks for the response!

I've tried to reproduce you example locally on Debian Forky with Python 3.14.7 and PyArrow 25.0.1 and I have not been able to reproduce. If someone can try to set up a Python 3.13 and Rocky Docker environment and can validate whether the example above is reproducible it would be of great help.

[rok]

From my 🤖 :

On parallel filesystems like Lustre, GPFS or BeeGFS, large reads can bypass the usual kernel copy and pin the destination memory directly. If that memory was just allocated and the program hasn't written to it yet, the pinning can fail with "Bad address" (Errno 14). A normal local filesystem would handle that memory without complaint, and that would explain why only large files fail and why a retry works once the memory has been touched. To test this, copy the same file to a local disk such as /tmp or /dev/shm and run the failing script against that copy in a fresh process: if it never fails there but fails every time on /globalscratch (with df -T naming the filesystem type), the filesystem client is to blame.

So could you please try testing on /globalscratch and say /tmp?

[zianliu66-me]

Thanks a ton! From what I can tell, it is most likely indeed a filesystem issue. Specifically, I ran the same snippet I replied above on a few locations:

• Run on HPC /globalscratch/ which is beegfs: fails, produces behavior as I wrote in the bug report.
• Run on HPC /tmp/ which is xfs: no issue
• Run on HPC home dir /home/<institutional_affiliations>/ which is nfs4: no issue
• Run on my own computer which is btrfs: no issue

[rok]

This is likely due to Beegfs allocating memory differently than pyarrow.

Another suggestion from my 🤖 :

Could you run your failing snippet again on /globalscratch, this time with Arrow's system memory allocator:

ARROW_DEFAULT_MEMORY_POOL=system python your_script.py

To confirm the setting took effect, add print(pyarrow.default_memory_pool().backend_name) at the top of the script. It should print system. Does it still fail with Errno 14?

[zianliu66-me]

Thanks. I tried with both ARROW_DEFAULT_MEMORY_POOL=system and ARROW_DEFAULT_MEMORY_POOL=jemalloc, both produce the exact same Errno 14. The default memory pool on this system is mimalloc.

I've previously tried toggling pq.ParquetFile(memory_map=False) (on the non-snippet code with a parquet that was built, but cannot be read without the workaround), which didn't seem to work either.

[rok]

What happens if memory_map=True?
Perhaps try reporting to https://github.com/ThinkParQ/beegfs/issues to see if they have an idea what could be going wrong?