I am trying to build a airflow dag to retrieve the data from Azure SQL Server, generate a txt file, and push it to a sftp destination. I was able to do this from Postgres database but not Azure SQL Server DB.

I found this tutorial: https://www.youtube.com/watch?v=Q7ns-qmHxSI
I append the dockerfile from author’s repo to my dockerfile and copied his dag to my dag folder, updated the user, password, etc variables. When opening airflow UI, it gave me error below:

from airflow.providers.microsoft.mssql.operators.mssql import MsSqlOperator

ModuleNotFoundError: No module named ‘airflow.providers.microsoft.mssql’

Below is what I copied to my dockerfile, which should install both mssql and pyodbc:

USER ROOT

CMD ‘apt update && apt install wget && apt install gnupg2 && wget -qO- https://packages.microsoft.com/keys/microsoft.asc | sudo apt-key add – && apt update && apt install add-apt-repository && cat /etc/apt/sources.list && sudo apt install software-properties-common && sudo su && curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add – && curl https://packages.microsoft.com/config/debian/11/prod.list > /etc/apt/sources.list.d/mssql-release.list && exit && sudo apt-get update && sudo ACCEPT_EULA=Y apt-get install -y msodbcsql18 && Pip install pyodbc && pip install apache-airflow-providers-microsoft-mssql==3.1.0 && python && from airflow.providers.microsoft.mssql.operators.mssql import MsSqlOperator’

USER airflow

I was able to successfully install psycopg2 and have the airflow working for Postgres in Azure, just not Azure SQL Server, I also tried below before commented it out from my dockerfile, same error.

# RUN apt update
# RUN apt install wget
# RUN apt install gnupg2
# RUN apt update

# RUN cat /etc/apt/sources.list
# RUN sudo apt install software-properties-common
# RUN sudo su
# RUN curl https://packages.microsoft.com/keys/microsoft.asc | apt-key add -
# RUN curl https://packages.microsoft.com/config/debian/11/prod.list > /etc/apt/sources.list.d/mssql-release.list
# RUN exit
# RUN sudo apt-get update
# RUN sudo ACCEPT_EULA=Y apt-get install -y msodbcsql18
# RUN pip install pyodbc
# RUN pip install apache-airflow-providers-microsoft-mssql==3.1.0
# RUN pip install 'apache-airflow[azure_blob_storage,azure_data_lake,azure_cosmos,azure_container_instances]'