so below are my files under load folder

under test_dp_maneesh_loader i have main.py

from datetime import datetime

import apache_beam as beam


# pylint: disable=too-many-locals
def run(argv=None):
    print("hello world")


if __name__ == "__main__":
    run()

the dockerfile as below

FROM apache/beam_python3.9_sdk:2.61.0
ENV POETRY_VIRTUALENVS_CREATE false 
    POETRY_CACHE_DIR=/var/cache/pypoetry

LABEL maintainer="xyz <[email protected]>"

WORKDIR /load

COPY test_dp_maneesh_loader ./test_dp_maneesh_loader
COPY pyproject.toml poetry.lock ./
RUN python3 -m pip install --upgrade pip
RUN pip3 install "poetry==1.3.2"
RUN poetry install --only main

ENTRYPOINT ["/opt/apache/beam/boot"]

the pyproject.toml

[tool.poetry]
name = "test_issuer"
version = "0.1.0"
description = "testing a small issue"
authors = ["xyz here <[email protected]>"]

[tool.poetry.dependencies]
python = "~3.9"
apache-beam = {extras = ["gcp"], version = "^2.53.0"}
google-auth = "^2.31.0"
#numpy = ">=1.26.4, <2.0"

[tool.poetry.scripts]
load = "test_dp_maneesh_loader.main:run"


[build-system]
requires = ["poetry-core>=1.0.0"]
build-backend = "poetry.core.masonry.api"

now i build the docker image as

docker build -t myapp:1.0 .

and if i run the docker container with the image interactively in the container

docker run -it --entrypoint /bin/bash myapp:1.0

and inside container i do

python main.py

getting the error in the import statement as below

File "/load/test_dp_maneesh_loader/main.py", line 3, in <module>
    import apache_beam as beam
  File "/usr/local/lib/python3.9/site-packages/apache_beam/__init__.py", line 87, in <module>
    ....
ValueError: numpy.dtype size changed, may indicate binary incompatibility. Expected 96 from C header, got 88 from PyObject

i can resolve the issue by downgrading th numpy version to 1.26.4 by mentioning in the toml file. I saw several posts mentioning about the issue on numpy 2 and dependencies.

what is the reason for this error in the import statement itself? Is this the only way to resolve this issue(by downgrading numpy)?

note that i removed many unnecessary lines from the python file.