TL;DR:

My single file Windows app dependent on PyArrow and built with PyInstaller fails to run with a DLL load failure while attempting to import pyarrow.lib.

Version/System Info:

• Python 3.11.9
• PyInstaller 5.13.2
• PyArrow 16.1.0
• Windows 10.0.19045

Details:

Currently developing a small app that depends on the PyArrow Python bindings for Apache Arrow. In bundling the app (data_app.py) into a single Windows .exe with PyInstaller, the app builds in a dedicated Conda environment qt-env using the command

pyinstaller --onefile --clean --debug "all" --add-data "logo.svg;." --hiddenimport "pkg_resources.extern" data_app.py

where logo.svg is just an image loaded by the app and the only relevant warning during the build process is

WARNING: Hidden import "pyarrow.compat" not found!

However, attempting to run the resulting data_app.exe fails with the following stack trace:

# pyarrow.lib not found in PYZ
Traceback (most recent call last):
  File "data_app.py", line 4, in <module>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 690, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 940, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "pyarrow__init__.py", line 65, in <module>
  File "<frozen importlib._bootstrap>", line 1176, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1147, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 676, in _load_unlocked
  File "<frozen importlib._bootstrap>", line 573, in module_from_spec
  File "<frozen importlib._bootstrap_external>", line 1233, in create_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
ImportError: DLL load failed while importing lib: The specified procedure could not be found.

indicating that import of pyarrow.lib is the issue. During the PyInstaller build, I noticed the message

INFO: Loading module hook 'hook-pyarrow.py' from 'C:\Users\user\anaconda3\envs\qt-env\Lib\site-packages\_pyinstaller_hooks_contrib\hooks\stdhooks'

The PyArrow hook file at this location was out of date (from 2020), and PyInstaller would override any attempt to use my own hook file for PyArrow with the --additional-hooks-dir option, so I modified the default hook-pyarrow.py file to reflect the one used in the most recent version of pyinstaller-hooks-contrib, i.e.

from PyInstaller.utils.hooks import collect_submodules, collect_data_files, collect_dynamic_libs

hiddenimports = collect_submodules('pyarrow', filter=lambda x: "tests" not in x)
datas = collect_data_files('pyarrow')
binaries = collect_dynamic_libs('pyarrow')

After modifying the hook-pyarrow.py file, I no longer encountered the WARNING: Hidden import "pyarrow.compat" not found! warning during the PyInstaller build, but attempting to open the app .exe continued to fail with the same stack trace.

To ensure that there was no issue with pyarrow.lib, I ran the following in Python:

import pyarrow.lib as _lib

print(_lib.__file__)

which runs without error and outputs the path of pyarrow.lib as

C:Usersuseranaconda3envsqt-envLibsite-packagespyarrowlib.cp311-win_amd64.pyd

I tried supplying PyInstaller with the pyarrow.lib file directly during the build process using

--hiddenimport "C:Usersuseranaconda3envsqt-envLibsite-packagespyarrowlib.cp311-win_amd64.pyd"

but the resulting app .exe continues to fail with the same stack trace.

If it is relevant, the contents of the builddata_applocalpycspyarrow directory are as follows:

interchange
parquet
vendored
acero.pyc
benchmark.pyc
cffi.pyc
compute.pyc
conftest.pyc
csv.pyc
cuda.pyc
dataset.pyc
feather.pyc
flight.pyc
fs.pyc
ipc.pyc
json.pyc
jvm.pyc
orc.pyc
pandas_compat.pyc
substrait.pyc
types.pyc
util.pyc
_compute_docstrings.pyc
_generated_version.pyc
__init__.pyc

I’d appreciate any help in solving this.