Suppose I have a code:
val spark = SparkSession.builder().master("local").getOrCreate()
spark.conf.set("spark.sql.sources.partitionOverwriteMode", "dynamic")
val partitionMode1 = spark.conf.get("spark.sql.sources.partitionOverwriteMode")
print(s"Partitioning: $partitionMode1")
val partitionMode2 = spark.sparkContext.getConf.get("spark.sql.sources.partitionOverwriteMode")
print(s"Partitioning: $partitionMode2")
This code will fail with error
Exception in thread "main" java.util.NoSuchElementException: spark.sql.sources.partitionOverwriteMode
at org.apache.spark.SparkConf.$anonfun$get$1(SparkConf.scala:245)
at scala.Option.getOrElse(Option.scala:189)
My question is it safe to update such property in runtime using spark.conf.set?