I want to run my azure devops agent pool using kubernetes. I am using this to run a Jupyter notebook which takes quite a bit of time to run. I want to know if it is possible for one job which runs the jupyter notebook to be executed accross multiple pods so that time to run the piepline can be decreased.

I have configure my agent pool as mentioned in Azure DevOps — Self Hosted Agents on Kubernetes — PART-1.

I am able to use hpa to create replicas if the cpu gets high but when the replicas are created they don’t get utilized by the pipeline but just sit idle as shown below.

Is it possible so that the job can be spread across the multiple pods or is there another way to run this job in parallel to reduce the time it takes to run the jupyter notebook.