I’m trying to pull some data from Dataverse using the Web API with service principal authentication. Everything works fine when I’m doing it in Python script, but encountered some problems when using Copy Data activity in ADF.
Here is my working code:
BASE_URL = 'https://<org-url>.crm.dynamics.com'
credential = DefaultAzureCredential()
token = credential.get_token(BASE_URL + '/.default/)
headers = {
'Authorization': f'Bearer {token}',
'OData-Version': '4.0',
'OData-MaxVersion': '4.0',
'Accept': 'application/json',
}
url = F'{BASE_URL}/api/data/v9.2/<table>'
response = requests.get(url, headers=headers)
When trying to do something similar in ADF using Copy Data activity, I’m getting one of the errors:
- When I put
https://<org-url>.crm.dynamics.comin the REST linked service as a Base URL, I get the authentication error 401. - When I put
https://<org-url>.crm.dynamics.com/.defaultin the REST linked service as a Base URL, I pass the authentication, but getting incorrect response, because the endpoint ishttps://<org-url>.crm.dynamics.com/.default/api/data/v9.2/<table>instead ofhttps://<org-url>.crm.dynamics.com/api/data/v9.2/<table>
How can I solve this? Why is this /.default part needed in order to achieve the token?
0
I created one app registration to call Dataverse API and added it as application user with proper role like this:
To generate access token using service principal authentication, scope value must end with /.default.
For sample purpose, I tried pulling ‘/WhoAmI’ API response by running below python code where client ID, client secret and tenant ID are added as environment variables:
from azure.identity import DefaultAzureCredential
import requests
BASE_URL = 'https://orgxxxxxx.crm.dynamics.com'
credential = DefaultAzureCredential()
token = credential.get_token(BASE_URL + '/.default').token
headers = {
'Authorization': f'Bearer {token}',
'OData-Version': '4.0',
'OData-MaxVersion': '4.0',
'Accept': 'application/json',
}
url = f'{BASE_URL}/api/data/v9.2/WhoAmI'
response = requests.get(url, headers=headers)
print(response.json())
Response:
To retrieve same data using Copy Data activity in ADF and send response to Azure Blob Storage, you can follow below steps:
Initially, create one REST linked service by selecting Authentication type as service principal with below configuration:
Base URL: https://orgxxxxxx.crm.dynamics.com/api/data/v9.2/
Authentication type: Service principal
Service principal ID: appID
Service principal key: client_secret
Tenant: tenantID
Microsoft Entra ID resource: https://orgxxxxxx.crm.dynamics.com
Later, create one REST dataset in ADF and select above linked service with “Relative URL” as WhoAmI (<table> in your case):
In my case, I tried copying data to Azure Blob Storage for which below linked service need to be created:
Later, create one JSON dataset selecting Azure Blob Storage linked service with file path as below:
Now, create one pipeline and add copy data activity to it with source as REST dataset and sink as JSON dataset:
Source:
Sink:
Response:
To confirm that, I checked the file from Azure Blob Storage where data retrieved successfully as below:
UPDATE:
Alternatively, you can also make use of OAuth 2.0 Client Credential method in the linked service and generate the token with ‘/.default’ as scope.
2