Add column based on other rows in the dataframe

I have a Pyspark dataframe like this

Add column based on other rows in the dataframe

I have a Pyspark dataframe like this

Compare column values between columns having the same suffix but different prefix in the name

Would appreciate some optimization tips here.

Get duplicate rows in a specific column from dataframe

I have a dataframe df: