I currently apply basic asc/desc sort to a column in Python like this:
sort_exprs = [(col.asc() if direction == "asc" else col.desc())]
sorted_dataframe = dataframe.orderBy(*sort_exprs)
One of my columns has data in the format
+-----------------+
| columnName|
+-----------------+
| 10 (5)|
| null|
| 15 (20)|
+-----------------+
I’d like to:
- regardless of what type of sort I use, if a column is being sorted, always move the rows with a
null
value for that column to the end - have the option of sorting by the number in the parentheses. In the example above, I’d sort 5 and 20 instead of 10 and 15. The regex to get the number in the parentheses is
r"((d+(.d+)?))"
How would I do this? Thank you in advance