I have a dataframe and I am trying to do following transformation.

exploded_prod_df = exploded_hits_product_df.withColumn("product_cd", F.explode(F.col("products.customdimensions")))

My issue is products is Null currently, which may not be the case in future. I tried many things but they didn’t work. For example, I tried below code

exploded_prod_df = (
exploded_hits_product_df
.withColumn('products_customdimensions', F.when(F.col('products').isNull(), F.lit(None).cast(T.StructType()))
.otherwise('products.customdimensions'))
.withColumn("product_cd", F.explode_outer(F.col("products_customdimensions")))

)

but this is giving error. Please help me. This is Google Analytics data. Error is: Cannot resolve "CASE WHEN (products IS NULL) THEN NULL ELSE products.customdimensions END" due to data type mismatch: Input to casewhen should all be the same type, but it's ["STRUCT<>", "STRING"].;