I’ve a python script /project-name/src/myscript.py that uses SentenceTransformer

from sentence_transformers import SentenceTransformer, util

Pipfile

[[source]]
url = "https://pypi.org/simple"
verify_ssl = true
name = "pypi"

[packages]
async-timeout = "*"
google-cloud-storage = ">=2.5.0"
gcsfs = "*"
sentence-transformers = ">=2.7.0"

[requires]
python_version = "3.10"

I containerize my project and run the docker image locally. It works perfectly fine. But when I push the image to the registry and run it using Airflow (Composer) by spinning up a kubernetes operator

KubernetesPodOperator(
    dag=dag,
    task_id=task_id,
    cmds=cmds,
    env_vars=env_vars,
    container_resources=container_resources,
    image=image_name,
)

The following error occurs

INFO - [base] Traceback (most recent call last):
INFO - [base]   File "/project-name/.local/share/virtualenvs/project-name-IeVDyrtp/lib/python3.10/site-packages/torch/__init__.py", line 178, in _load_global_deps
INFO - [base]     ctypes.CDLL(lib_path, mode=ctypes.RTLD_GLOBAL)
INFO - [base]   File "/usr/local/lib/python3.10/ctypes/__init__.py", line 374, in __init__
INFO - [base]     self._handle = _dlopen(self._name, mode)
INFO - [base] OSError: libcudart.so.12: cannot open shared object file: No such file or directory
INFO - [base] 
INFO - [base] During handling of the above exception, another exception occurred:
INFO - [base] 
INFO - [base] Traceback (most recent call last):
INFO - [base]   File "/project-name/src/myscript.py", line 5, in <module>
INFO - [base]     from sentence_transformers import SentenceTransformer, util
INFO - [base]   File "/project-name/.local/share/virtualenvs/project-name-IeVDyrtp/lib/python3.10/site-packages/sentence_transformers/__init__.py", line 3, in <module>
INFO - [base]     from .datasets import SentencesDataset, ParallelSentencesDataset
INFO - [base]   File "/project-name/.local/share/virtualenvs/project-name-IeVDyrtp/lib/python3.10/site-packages/sentence_transformers/datasets/__init__.py", line 1, in <module>
INFO - [base]     from .DenoisingAutoEncoderDataset import DenoisingAutoEncoderDataset
INFO - [base]   File "/project-name/.local/share/virtualenvs/project-name-IeVDyrtp/lib/python3.10/site-packages/sentence_transformers/datasets/DenoisingAutoEncoderDataset.py", line 1, in <module>
INFO - [base]     from torch.utils.data import Dataset
INFO - [base]   File "/project-name/.local/share/virtualenvs/project-name-IeVDyrtp/lib/python3.10/site-packages/torch/__init__.py", line 238, in <module>
INFO - [base]     _load_global_deps()
INFO - [base]   File "/project-name/.local/share/virtualenvs/project-name-IeVDyrtp/lib/python3.10/site-packages/torch/__init__.py", line 199, in _load_global_deps
INFO - [base]     _preload_cuda_deps(lib_folder, lib_name)
INFO - [base]   File "/project-name/.local/share/virtualenvs/project-name-IeVDyrtp/lib/python3.10/site-packages/torch/__init__.py", line 164, in _preload_cuda_deps
INFO - [base]     raise ValueError(f"{lib_name} not found in the system path {sys.path}")
INFO - [base] ValueError: libcublas.so.*[0-9] not found in the system path ['/project-name/src', '/usr/local/lib/python310.zip', '/usr/local/lib/python3.10', '/usr/local/lib/python3.10/lib-dynload', '/project-name/.local/share/virtualenvs/project-name-IeVDyrtp/lib/python3.10/site-packages']

Note: I try sentence-transformers = "==2.6.0" in Pipfile but it doesn’t help