I have 3 workspaces in MS Fabric. Workspace A and Workspace B each contain 2 lakehouses. Workspace C, however, only contains 6 notebooks—NB2 and NB4 are for processing data from lakehouses in A, and NB3 and NB5 are for lakehouses in B. I’ve added the corresponding lakehouses to each notebook.
Notebook 1 runs NB2 and NB3 using mssparkutils.notebook.runMultiple(), and NB3 and NB5 are used to create dynamic batches to process the tables. For running the batches, we again use mssparkutils.notebook.runMultiple() in NB2 and NB3. I’ve added all the lakehouses in NB1, making a lakehouse from A the default.
The issue I’m facing is that when I run NB1, it processes data only for the lakehouses in A, but it’s unable to access the tables in the lakehouses from B.
The error message is:
“Table name: sales [TABLE_OR_VIEW_NOT_FOUND] The table or view lhb1.sales cannot be found. Verify the spelling and correctness of the schema and catalog. If you did not qualify the name with a schema, verify the current_schema() output, or qualify the name with the correct schema and catalog. To tolerate the error on drop, use DROP VIEW IF EXISTS or DROP TABLE IF EXISTS.”
My question is:
Why are we getting this error when the lakehouses are already attached to the corresponding notebooks (NB3 and NB5)? I guess that since the default lakehouse for NB1 is from A, it’s unable to locate the lakehouses from B. Is it possible to change the default lakehouse dynamically when processing NB3 and NB5, so that it uses one of the lakehouses from B?
I’d really appreciate your help and suggestions.
I have tried to change the default lakehouses using some configurations but unable to adjust.
4