I have two spark dfs containing different number of columns with the 1st column being the ID (for both). I want to have the column values for each ID as a dictionary (the visual would make better sense of what I am trying to achieve)
what I have:
enter image description here
The result I am trying to get
enter image description here
What I tried but couldn’t get the result:
=> [row.asDict() for row in df.collect()] to get a list of dictionaries but could not find a way to add each as column value.
PS: I understand my question could be XYProblem but, since, I do not know all the concepts available in pyspark/python, this is the best way I could think of to present the issue.
mohammed shoab is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.