Our Glue job is writing data into delta format file from data frame. It looks like data frame is fetching correct records however after writing into delta format , the data is in consistent(sometimes writes the correct record and sometimes writes incorrect random record). Need some help to resolve this
Here is the code snippet I used to write into delta format file :
additional_options = {
“path”: fld_path,
“write.parquet.compression-codec”: “snappy”
}
(filtered_df2.write.format(“delta”)
.options(**additional_options)
.mode(“overwrite”)
.save())
Thanks in Advance
Rahul I is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.