Pyspark – Concatenate DataFrame values only for specific columns listed in one of the columns of the same dataframe
I’m struggling with this problem:
There is a dataframe with couple of columns. One of this columns is a “BKColumns” column of ARRAY type which contains some of columns’ names (not all) of the same dataframe. In every row of the dataframe we can have different values of column names in “BKColumns”.At the very end, I would like to add another column to the dataframe which contains concatenation of values from columns mentioned in “BKColumns”.