We’re experiencing extremely slow Databricks SQL queries. I have come across a site that provides a number of Spark SQL optimization tuning techniquess
https://www.linkedin.com/pulse/spark-sql-performance-tuning-configurations-vignesan-saravanan-8hamc/
A number of the recommendations described from the link suggest that the features/functions are already enabled by default. For example, the Spark Cost-Based Optimizer is enabled by default. However, it also mentions that if its not enabled you can enable it by running the following:
spark.conf.set("spark.sql.cbo.enabled", true)
My question are
- How do you determine of the feature/function is enabled
- Will the feature / function work with Databricks SQL notebook as opposed to Databricks Python notebook?