I’m migrating my AWS Lambda Python runtime from 3.8 to 3.11. I’m using AWS Layers to package my dependencies. In my Lambda function, I’m trying to connect to Amazon Redshift using SQLAlchemy. Here are the relevant dependencies specified in my requirements.txt:

redshift-connector==2.0.907
psycopg2-binary==2.9.9
SQLAlchemy==1.4.36
sqlalchemy-redshift==0.8.9

Here’s a snippet of my code:

import sqlalchemy as sq  

url = sq.engine.url.URL.create(
    drivername='redshift+redshift_connector',  # indicate redshift_connector driver and dialect will be used
    host=f"{redshift_endpoint}",  # Amazon Redshift host
    port=5439,  # Amazon Redshift port
    database='dev',  # Amazon Redshift database
    username=f"{redshift_db_user}",  # Amazon Redshift username
    password=f"{redshift_db_password}"  # Amazon Redshift password
)
print('Connection URL is', url)
engine = sq.create_engine(url)
cnn = engine. Connect()

However, I’m encountering the following error:

NoSuchModuleError("Can't load plugin: sqlalchemy.dialects:redshift.redshift_connector")

With the same configuration, the code works perfectly on my local machine (Ubuntu), but it fails on AWS Lambda. I suspect that some packages might be missing in the Lambda environment.

I have tried the suggestions from this Stack Overflow article, but the issue persists. I noticed that sqlalchemy-redshift does not list Python 3.11 as a supported classifier (reference). Could this be the cause of the problem?

Has anyone faced a similar issue, or can anyone suggest a solution to resolve this? Any help would be appreciated. Thanks!