What is the maximum number of entries a array can in a Spark column can hold?
I’ve created a struct with the data of some columns combined. Large numbers of these structs now occur for my unique identifier values. I want to combine these structs into an array using collect_list.
Unfortunately, I’m getting errors like