I’m trying to run a Datafactory pipeline that has a Databricks notebook that creates a table. However, it always generates the error:

I have a Workflow that will do two things:

Our team has started migrating init scripts from DBFS. Our runtime version is 10.4, so we are considering moving the scripts to either ADLS or Workspace. The second option seems much easier than the first one. The main problem is that our init scripts interact with .whl files,
and according to Microsoft’s instructions, it’s not recommended to use Workspace for storing init scripts in this case. What do you think about this?

I am quite new to databricks workflow.
But my ideia is when an activity (in my case notebooks) fail there is an if/else condition and when the notebooks execute an email is send with a default message.
When an activity fails, I would like to send an email saying which activity failed, but when an activity fails it does not enter the if/else condition and aborts the pipeline.

We have setup Databricks workspace with VNet Injection and enabled private endpoints –
a) Backend private endpoint
b) Frontend private endpoint
c) BrowserAuth private endpoint

Deploying Azure Databricks creates an additional resource group in the background, which includes a data lake. Is it possible to use the data lake that I have already deployed in Azure instead of the one provisioned by Azure Databricks?