I have a codebase in an airflow instance which gets updated every 15 min by pulling the latest master from github repo (because we need to some how bring the latest changes in the codebase to airflow instance right).. But there are many tasks running. Tasks will always be running. These tasks uses scripts from that codebase. The problem is when the codebase is getting updated during the time where task is running leading to some errors. How to isolate these two? Any suggestions
My tasks are usually running scripts. I would like to create a docker image which contains all the dependencies to run a script and run the script in the docker container running that image. But don’t have much clarity around how this implementation will be Not good at using docker.
I would like to know how the companies which use airflow sync the codebase.
Anantha Lakshmi Meruva is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.