In DBX I was used to create one ingestion task and then have multiple yml files with different configs use in my deployment.yml like:
tasks:
- task_key: ingest_<source>_<table>
job_cluster_key: job_cluster
python_wheel_task:
package_name: dbx_<source>_<layer>
entry_point: <source>_ingestion_task
parameters:
- --conf-file
- file:fuse://conf/deployments/<source>/tasks/<table>_config.yml
In the python (Task) I could access these parameters in the file like:
value = self.conf["<key>"]["<sub_key>"]
How can I do the same using Databricks Asset Bundles because I really need this. I have multiple generic ingestion functions which I need to create tasks for with separate parameters(files)