I’d like to have an Airflow DAG get some status information back after running an AzureContainerInstancesOperator task in Airflow. With the equivalent docker operator, DockerOperator, it is possible to get the last line of output by default inserted into XCOM. With the AzureContainerInstancesOperator, this seems less likely to be possible since the container is run remotely, possibly without network access.

To retrieve the result information, I can think of 3 possibilities, but I’m hoping there are ones I haven’t thought of:

• Retrieve output from the container that is returned to Airflow directly.
• communicate the status back through a file located in a mutually accessible location like an Azure blob, redis, a shared file system, etc.
• use a messaging solution. Our original thought was to use Azure Service Bus, but the out of the box connectors in Airflow seem to just log the message and then delete it. Unless I misunderstand how the out-of-the-box-connectors work, I’d need to write something to create a subscription to a topic which filters by correlation ID and when the correct message arrives, process the contents and branch based on the message contents. This is doable.

Am I missing something obvious here? Can we get back status from the Azure Container Instance that ran? Or is there an easy way to get the contents of an Azure Service Bus message in Airflow?

Note: I have purposefully not tagged this as Azure Containter Instances or Service Bus since I’m asking about the Airflow integration.