Getting the error while trying to write a delta format file via Azure Databrics to ADLS
AnalysisException: Found invalid character(s) among ' ,;{}()nt=' in the column names of your schema. Please use other characters and try again.
I have semicolon(:) and period(.) in my column names but it’s important that I keep these characters in the names, hence I have to save the file with same column names. I tried saving the CSV and it’s fine but delta is causing the above error.
I am using the following command:
sourcedf.write.mode("overwrite").option("header", "true").option("mergeSchema", "true").option("spark.databricks.delta.properties.defaults.columnMapping.mode", "name").format("delta").save(path)
even tried this option
option("delta.columnMapping.mode", "name")