My Azure Machine learning workspace is configured with private endpoint , my aml is configured to a Storage account which is also configured private endpoint and disabled the public access, My Storage Account network will looks like below
The problem:
When I launch the AML workspace from the portal, all are working fine except there is a single API Call which is getting failed with the 403 Forbidden
https://0121921-12012191-12121.workspace.eastus2.api.azureml.ms/notebook/v2.0/subscriptions/<sub-ib>/resourceGroups/<rg>/providers/Microsoft.MachineLearningServices/workspaces/<aml-name>/storage/sasurl?expirationInMinutes=60&containerId=391ff5ac-6576-460f-ba4d-7e03433c68b6
Error Response
{
"error": {
"code": "UserError",
"severity": null,
"message": "Request authorization to storage account failed. Storage account might be behind a VNET.",
"messageFormat": null,
"messageParameters": null,
"referenceCode": null,
"detailsUri": null,
"target": null,
"details": [],
"innerError": {
"code": "ForbiddenError",
"innerError": null
},
"debugInfo": null,
"additionalInfo": null
},
"correlation": {
"operation": "c738a5a6cc2342328058bc3ad77a9520",
"request": "1946a6fcdffd7236"
},
"environment": "eastus2",
"location": "eastus2",
"time": "2024-07-08T07:17:00.8941898+00:00",
"componentName": "notebook-instance",
"statusCode": 403
}
Looking from the error its easy to guess that the AML couldn’t able to reach Storage account to generate the SAS token.
But I dont know why the service needs this SAS token ? Most of the integreation between AML and SA is handled via RBAC I already gave right roles that my computes , notebooks are working fine . But I’m wondering why this particular API getting failed ?
I tried all the below
- Adding the AML Instance direct on SA networks (Resource Instance)
- Adding my public IP on the network from where i access the AML
Nothing seems to be working
Even making this troubleshooting harder is I cannot able to get the proper logs. I tried to enable diagnostic logs on SA (account level) and thought of checking which API is getting failed so that I can look at the IP and may be provide the IP whitelist on SA. But there is no logs specific to SAS token generation.
Only solution works for me is I need to open my SA to full network (Enabled from all network) which i dont want to go with.