I’m trying to write a csv file to a table in databricks, but I’m getting the error in the title. When I researched the issue, it seems that casting the column to another data type will resolve this issue, so I did that and get the following schema.

 |-- fixed_lot_multiplier: string (nullable = true)
 |-- item_category_segment_1: string (nullable = true)
 |-- item_category_segment_2: string (nullable = true)
 |-- item_category_segment_3: string (nullable = true)
 |-- po_category_segment_1: string (nullable = true)
 |-- po_category_segment_2: string (nullable = true)

Clearly, the void data types are gone. So the dataframe is passed back to my primary function, but then I still get above error. I tried printing the schema right after the dataframe is passed back and found that the void data types are somehow back.

 |-- fixed_lot_multiplier: string (nullable = true)
 |-- item_category_segment_1: void (nullable = true)
 |-- item_category_segment_2: void (nullable = true)
 |-- item_category_segment_3: void (nullable = true)
 |-- po_category_segment_1: string (nullable = true)
 |-- po_category_segment_2: void (nullable = true)

Does anyone know why this is happening and/or how I can prevent it?

Thank you!