We have a job in Azure DataBricks which connects to a Azure Data Lake to fetch the data using below code. Now we have to convert it into MSI Managed Identity authentication instead of Service principal
we have to implement the authentication with out client id and secret
<code>clientId = dbutils.secrets.get(scope = "plsadbsecret", key = "cltid")
clientSecret = dbutils.secrets.get(scope = "plsadbsecret", key = "cltsecret")
storageAccount = "xxxxxxxxxxxxxxxxxx"
folderPath = "xxxxxxxxxxxxxxxxxxxxxxx"
safeLimitFolderPath = "xxxxxxxxxxxxxxxxxxxxxxxxx"
spark.conf.set("fs.azure.account.auth.type." + storageAccount + ".dfs.core.windows.net", "OAuth")
spark.conf.set("fs.azure.account.oauth.provider.type." + storageAccount + ".dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
spark.conf.set("fs.azure.account.oauth2.client.id." + storageAccount + ".dfs.core.windows.net", clientId)
spark.conf.set("fs.azure.account.oauth2.client.secret."+ storageAccount + ".dfs.core.windows.net", clientSecret)
spark.conf.set("fs.azure.account.oauth2.client.endpoint." + storageAccount + ".dfs.core.windows.net", "https://login.microsoftonline.com/xxxxxxx-xxxx-xxxx-xxx- xxxxxxx/oauth2/token")
assetPath =xxxxxxxxxxxxxxx
getEmployees= spark.read.format("delta").load(assetPath)
Expecting to implement the MSI authentication
</code>
<code>clientId = dbutils.secrets.get(scope = "plsadbsecret", key = "cltid")
clientSecret = dbutils.secrets.get(scope = "plsadbsecret", key = "cltsecret")
storageAccount = "xxxxxxxxxxxxxxxxxx"
folderPath = "xxxxxxxxxxxxxxxxxxxxxxx"
safeLimitFolderPath = "xxxxxxxxxxxxxxxxxxxxxxxxx"
spark.conf.set("fs.azure.account.auth.type." + storageAccount + ".dfs.core.windows.net", "OAuth")
spark.conf.set("fs.azure.account.oauth.provider.type." + storageAccount + ".dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
spark.conf.set("fs.azure.account.oauth2.client.id." + storageAccount + ".dfs.core.windows.net", clientId)
spark.conf.set("fs.azure.account.oauth2.client.secret."+ storageAccount + ".dfs.core.windows.net", clientSecret)
spark.conf.set("fs.azure.account.oauth2.client.endpoint." + storageAccount + ".dfs.core.windows.net", "https://login.microsoftonline.com/xxxxxxx-xxxx-xxxx-xxx- xxxxxxx/oauth2/token")
assetPath =xxxxxxxxxxxxxxx
getEmployees= spark.read.format("delta").load(assetPath)
Expecting to implement the MSI authentication
</code>
clientId = dbutils.secrets.get(scope = "plsadbsecret", key = "cltid")
clientSecret = dbutils.secrets.get(scope = "plsadbsecret", key = "cltsecret")
storageAccount = "xxxxxxxxxxxxxxxxxx"
folderPath = "xxxxxxxxxxxxxxxxxxxxxxx"
safeLimitFolderPath = "xxxxxxxxxxxxxxxxxxxxxxxxx"
spark.conf.set("fs.azure.account.auth.type." + storageAccount + ".dfs.core.windows.net", "OAuth")
spark.conf.set("fs.azure.account.oauth.provider.type." + storageAccount + ".dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider")
spark.conf.set("fs.azure.account.oauth2.client.id." + storageAccount + ".dfs.core.windows.net", clientId)
spark.conf.set("fs.azure.account.oauth2.client.secret."+ storageAccount + ".dfs.core.windows.net", clientSecret)
spark.conf.set("fs.azure.account.oauth2.client.endpoint." + storageAccount + ".dfs.core.windows.net", "https://login.microsoftonline.com/xxxxxxx-xxxx-xxxx-xxx- xxxxxxx/oauth2/token")
assetPath =xxxxxxxxxxxxxxx
getEmployees= spark.read.format("delta").load(assetPath)
Expecting to implement the MSI authentication
New contributor
praveena p is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.