I need to trigger my ADF Pipeline when there is an event in Kafka. Is there a way I can do it without Event Hub or it’s the only way?
I will receive a Kafka Message and will trigger pipeline A in subscription 1. Once my pipeline A completes I will post a message in Kafka.
Also what is a good scheduling mechanism where one adf pipeline can be triggered on succession of another pipeline. The dependency is between different subscription as well as same subscription
So in the above case once pipeline A comletes in subscription 1, pipeline B,C,D will trigger in subscription 2,3,4 wheer they will listen to the kafka and trigger their pipelines. Also there will be pipelines X,Y,Z in subscription 2 will run one after another in subscription 2 i.e.B->X->Y->Z. This is one example but there will be multiple jobs.
Is Logic App or Airflow are the solutions for this or any other ways it can be done without storage based event triggers?