Despite the fact that I have checked and tested my blob url path in my Azure storage account I’m still getting a: cannot resolve hostname. It’s like if my secret scopes were not working. Here is what I have done:
- Have my Azure Storage Account and Azure Databricks Service workspace running with a cluster
- I have setup my secrets scopes using databricks cli (the python library) from my terminal:
#Installing the databricks cli
pip install databricks-cli
#created a token:
databricks configure --token
Then on azure databricks: click on my username at top right corner + settings + developer on my left sidebar + clic on manage Access tokens button and generated a new token.
Then it asked me for my Databricks Host: https://adb-151anyothernumersinmyurl3420.0.azuredatabricks.net/ (got this from my url browser bar in the azure databricks workspace)
And hen Asked me for my previous generated token.
Created the scope with: databricks secrets create-scope –scope (my scope name was emailscontainerstorage)
Confirmed that the scope is create with:
databricks secrets list-scopes
Scope Backend KeyVault URL
---------------------- ---------- --------------
emailscontainerstorage DATABRICKS N/A
Then needed to create a key:
databricks secrets put --scope <scop-name> --key <key-name>
In my case: databricks secrets put --scope emailscontainerstorage --key claveaccesso
A text file was opened and pasted my access key (that i got from my azure storage dashboard under the settings + Access keys), then saved and closed the file.
Confirmed my secret scope with secret key:
databricks secres list –scope emailscontainerstorage
databricks secrets list --scope <scope-name> emailscontainerstorage
Key name Last updated
------------ --------------
claveaccesso 171234557520
I can confirm that I have access to the scope with:
databricks secrets list-acls --scope emailscontainerstorage
Principal Permission
----------------------------- ------------
[email protected] MANAGE
After reading guides created a notebook in scala with this code:
val containerstorageaccess = dbutils.secrets.get(scope = "emailstorageaccount", key = "claveaccesso")
val storageaccountname = "submarinobursatil"
val contaninername = "submarinobursatilemails"
spark.conf.set("fs.azure.account.key." + storageaccountname + ".blob.core.windows.net", containerstorageaccess)
val fileName = "sushisommarbella.sxls"
val data = spark.read
.option("header", "true")
.csv("wasbs://" + contaninername + "@" + storageaccountname + ".blob.core.windows.net/" +fileName)
display(data)
But i still keep getting error: IllegalArgumentException: Secret does not exist with scope: emailstorageaccount and key: claveaccesso
Did a second try with python:
spark.conf.set("fs.azure.account.auth.type.submarinobursatil.dfs.core.windows.net", "SAS")
spark.conf.set("fs.azure.sas.token.provider.type.submarinobursatil.dfs.core.windows.net", "org.apache.hadoop.fs.azurebfs.sas.FixedSASTokenProvider")
spark.conf.set("fs.azure.sas.fixed.token.submarinobursatil.dfs.core.windows.net", dbutils.secrets.get(scope="emailscontainerstorage", key="claveaccesso"))```
Ano since i go no errors then I tried to add 2 more lines:
spark.read.load(“abfss://[email protected]/emailsfolder”)
dbutils.fs.ls(“abfss://[email protected]/emailsfolder”)
And now I get the error even when I'm sure that url path is the correct path to my container file: https://submarinobursatil.blob.core.windows.net/submarinobursatilemails/sushisommarbella.xlsx
This is the error:
Py4JJavaError: An error occurred while calling o421.load.
: Status code: -1 error code: null error message: Cannot resolve hostname: submarinobursatil.dfs.core.windows.netjava.net.UnknownHostException
So I'm lost here, can someone point me out in th right direction so I can simply read and display data on dat excel file, or any type of file so I can start treating searches within these files?
Wha am I doing wrong?
I'm using azure datbricks, azure storage account.
Any insights will be appreciated.
I have tried everything I said and got 2 different errors whether using scala or python. And will need some advice since I can only find old tutorials and this is the new version.