I came across the below tutorial on Microsoft learn: https://learn.microsoft.com/en-us/azure/batch/tutorial-run-python-batch-azure-data-factory
It was very helpful to be able to run a python script successfully on ADF and process and persist data on the SQL db.
However, i’m curious as to why we need dedicated nodes for this? Can I not specify an auto-scale formula instead? I am concerned about the cost implications by having a dedicate node/s.