I have a spark dataframe as follows:

----------------------------------------------------------------------------------------------
| type |   lctNbr   | itmNbr |   lastUpdatedDate   | lctSeqId|  T7797_PRD_LCT_TYP_CD|   FXT_AIL_ID| pmyVbuNbr | upcId   |   vndModId|
____________________________________________________________________________
| prd_lct   145         147       2024-07-22T05:24:14   1          1                         14       126       008236686661    35216

_____________________________________________________________________________

I want to group this data frame by type, lctNbr, itmNbr, and lastUpdatedDate. I jsut want each record to be in the below json format:

  "type": "prd_lct",
  "lctNbr": 145,
  "itmNbr": 147,
  "lastUpdatedDate": "2024-07-22T05:24:14",
  "locations": [
    {
      "lctSeqId": 1,
      "prdLctTypCd": 1,
      "fxtAilId": "14"
    }
  ],
  "itemDetails": [
    {
      "pmyVbuNbr": 126,
      "upcId": "008236686661",
      "vndModId": "35216"
  ]
}

I tried using to_json, collect_list and map_from_entries functions but i just keep getting errors when i do a show command and cant seem to get to the correct format.