I’ve a continuous problem with my ADF pipeline – I’m trying to run a “copy-data” pipeline and encounter access and permission errors with my specific resources.
My source is a Microsoft 365 Table connector (I’m retrieving some columns from my organization mails)
My sink is a storage account.
In my SA access control properties, I’ve given my app appropriate IAM role with the all the necessary permissions so it can access the storage account and write the data successfully.
Now since I don’t want my storage to be public, I chose the public network access to be “Enabled from selected virtual networks and IP addresses”.
-
First try:
Since ADF is a resource instance in my Azure subscription, I’ve specified my instance to have access to my storage account based on its system-assigned managed identity (Microsoft.DataFactory/factories) and configured all propely (see https://roshan-vin4u.medium.com/authenticate-azure-data-factory-with-azure-data-lake-gen-2-using-managed-identities-3663f1449440).
But when I ran the pipeline it failed, claiming I can’t use system-assigned managed identity with Microsoft 365 connector. -
Second try:
I tried to access my storage account with private link using ADF PE, so I’ve created private endpoint and configured it all properly (see https://learn.microsoft.com/en-us/answers/questions/635312/connect-data-factory-to-azure-storage-wiht-private).
First I configured the service endpoint to be the storage’s dfs url and run the pipeline, got the error:
"ErrorCode=UserErrorOffice365DataLoaderError,'Type=Microsoft.DataTransfer.Common.Shared.HybridDeliveryException,Message=Office365 data loading failed to execute. office365LoadErrorType: PermanentError ...Failure happened on 'Sink' side. ErrorCode=AdlsGen2ForbiddenError"
Then I configured it to be the storage’s blob url and got the next error"the remote server return an error: (403) .... Unable to create Azure blob container"
-
Conclusions
I’ve also tested it and got same results when disabling public network access.
My conclusion is that the runtime did access the storage account via the private endpoint but failed (either due to some misconfiguration or some functionality problem).
The weirdest issue is when I tried using a simple ADF copy-data pipeline from one storage account to another using PE for my linked-service and integration runtime, it did ran successfully.
What can be the problem? How do I solve this issue?
Yonatan Shlain is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.