I have following array of map and I want to convert it into array of structs to convert all key value pairs to columns of a dataframe

-- DurationPeriod: array (nullable = true)
 |    |-- element: map (containsNull = true)
 |    |    |-- key: string
 |    |    |-- value: string (valueContainsNull = true)

Expected structure

|-- transform_col: array (nullable = true)
 |    |-- element: struct (containsNull = false)
 |    |    |-- key: string (nullable = true)
 |    |    |-- value: string (nullable = true)

Here is the sample data what I will get as array of map. Each map may be empty or we may have more than 2 key value paris. it veries for each occurance.

[{eod -> 2023-06-14, Id -> 123456789}, {eod -> 2028-11-17, Id -> 123456788}]

I am trying to convert map to struct as map is not supporting to convert key, value pairs to columns. Please suggest if there is any solution directly how map key value pairs can be converted to columns on dataframe

I tried with following code, but it is not getting key values and its showing null

df1 = table.select("*",expr("transform(DurationPeriod, x -> struct(x.key as key,x.value as value))").alias("transform_col"))

[{null, null}, {null, null}]

Highly appreciated for your help to resolve this issue. Please let me know what I am missing here.

Thanks,Bab