I am having a very strange issue when trying to use the python databricks-sql-connector in a lambda function on AWS.

Background
My codebase is written in Python and is deployed to AWS Lambda as a docker image.
The docker container runs a FastAPI API using uvicorn.
I manage packages in my project using poetry.

Problem When I add databricks-sql-connector to my project via poetry I.E my pyproject.toml has the following line:

databricks-sql-connector = "^3.1.2"

When deployed to a lambda function and installed via poetry all connections to my SQL warehouse on databricks timeout and I cannot get a connection or a cursor.

However, when I run the docker container locally with the connector installed via poetry I am able to connect to the Databricks SQL warehouse.

After nearly 2 days of tearing my hair out I still have no idea why this problem is occurring.

Weirdly, when I deploy the docker container to a Lambda but install the databricks-sql-connector package via pip3 I am able to connect to the SQL warehouse. I.E install the package via the following line in the Dockerfile

RUN pip3 install databricks-sql-connector

Why is this happening?! I would really like to keep all my dependencies in poetry and I would be very greatful for any suggestions or help.

Thanks in advance

pyproject.toml

[tool.poetry]
version = "0.1.0"
description = ""
authors = [""]
readme = "README.md"

[tool.poetry.dependencies]
python = "^3.11"
pytest = "^6.2.4"
pydantic = "^1.9"
fastapi = "^0.104.1"
uvicorn = "^0.23.2"
dependency-injector = "^4.41.0"
python-json-logger = "^2.0.2"
boto3 = "^1.28.61"
jupyter = "^1.0.0"
py-automapper = "^1.2.3"
dacite = "^1.8.1"
bump-pydantic = "^0.7.0"
qrcode = "^7.4.2"
pycognito = "^2023.5.0"
pulumi-docker = "^4.5.1"
pulumi-aws = "^6.18.2"
pulumi = "^3.102.0"
setuptools = "^69.0.3"
pulumi-awsx = "^2.4.0"
mangum = "^0.17.0"
casbin = "^1.36.0"
awslambdaric = "^2.0.10"
psycopg2-binary = "^2.9.9"
tenacity = "^8.2.3"
httpx = "^0.27.0"
flake8 = "^7.0.0"
black = "^24.4.2"
isort = "^5.13.2"
pyjwt = "^2.8.0"
pytz = "^2024.1"
databricks-sql-connector = "^3.1.2"
sqlalchemy = "^2.0.30"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

Dockerfile

FROM --platform=linux/amd64 public.ecr.aws/docker/library/python:buster as build
ARG GITHUB_SSH_PRIVATE_KEY
ARG GITHUB_SSH_KEY_PASSPHRASE
RUN pip install poetry==1.8.2
# Update aptitude with new repo
RUN apt-get update

# Install software 
RUN apt-get install -y git
ENV POETRY_NO_INTERACTION=1 
    POETRY_VIRTUALENVS_IN_PROJECT=1 
    POETRY_VIRTUALENVS_CREATE=1 
    POETRY_CACHE_DIR=/tmp/poetry_cache

WORKDIR /app

COPY pyproject.toml poetry.lock ./
RUN touch README.md

RUN echo 'hello'

RUN  mkdir /root/.ssh/
RUN echo "${GITHUB_SSH_PRIVATE_KEY}" 
RUN echo "${GITHUB_SSH_PRIVATE_KEY}" > /root/.ssh/id_rsa && 
    chmod 600 /root/.ssh/id_rsa


RUN 
    echo "IdentityFile /root/.ssh/id_rsa" >> /etc/ssh/ssh_config &&   
    echo "StrictHostKeyChecking no" >> /etc/ssh/ssh_config &&  
    eval `ssh-agent -s` && 
    echo "${GITHUB_SSH_KEY_PASSPHRASE}" | ssh-add /root/.ssh/id_rsa && 
    git clone -b dev <REDACTED> 

RUN poetry remove <REDACTED> 

RUN poetry add <REDACTED> 
RUN poetry lock

RUN --mount=type=cache,target=$POETRY_CACHE_DIR poetry install --no-root

FROM --platform=linux/amd64 public.ecr.aws/docker/library/python:3.11-slim-buster as runtime

ARG APP_NAME
ARG AWS_REGION
ARG AWS_ACCESS_KEY_ID
ARG AWS_SECRET_ACCESS_KEY
ARG LOGGER_NAME
ARG COGNITO_USER_POOL_ID
ARG COGNITO_CLIENT_ID
ARG DATABASE_URL
ARG DATABRICKS_HOSTNAME
ARG DATABRICKS_HTTP_PATH
ARG DATABRICKS_ACCESS_TOKEN

# Set environment variables from build arguments
ENV APP_NAME=${APP_NAME}
ENV AWS_DEFAULT_REGION=${AWS_REGION}
ENV AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID}
ENV AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY}
ENV LOGGER_NAME=${LOGGER_NAME}
ENV COGNITO_USER_POOL_ID=${COGNITO_USER_POOL_ID}
ENV COGNITO_CLIENT_ID=${COGNITO_CLIENT_ID}
ENV DATABASE_URL=${DATABASE_URL}
ENV DATABRICKS_HOSTNAME=${DATABRICKS_HOSTNAME}
ENV DATABRICKS_HTTP_PATH=${DATABRICKS_HTTP_PATH}
ENV DATABRICKS_ACCESS_TOKEN=${DATABRICKS_ACCESS_TOKEN}
ENV VIRTUAL_ENV=/app/.venv 
    PATH="/app/.venv/bin:$PATH"

COPY --from=build ${VIRTUAL_ENV} ${VIRTUAL_ENV}
RUN pip3 install databricks-sql-connector
COPY . ./app
WORKDIR /app
ENTRYPOINT [ "python", "-m", "awslambdaric" ]
CMD ["lambda_entrypoint.handler"]