I would like to write in function pyspark this part

def add_issue_state(df_tr):

I’m new to Pyspark. I come from a SAS background. I’ll try to keep this brief and pretty general.

I currently have a pyspark dataframe like this:

code:
!apt-get install ‘openjdk-19-jre-headless’ -qq > /dev/null

code:
!apt-get install ‘openjdk-19-jre-headless’ -qq > /dev/null

So below will return error because field ‘sex’ doesn’t exist. Is there a way to return nothing/null/empty when the field is not there instead throw an error? I will not use if to check each field because there are many fields.

I have the following transformational need for dataframe:

I have a column “data” in a parquet file.
This is what data contains:”{“”col1″”:123,””col2″”:123,””col3″”:13,””col4″”:565.0, “”col5″”:565.0}”

red hat docker connect yarn:
When I use PySpark to connect to a YARN cluster, an error occurs: Java gateway exited before sending its port number,The error in PySpark appears in the following files: context.py, session.py, and java_gateway.py,My Spark version is 3.1.1, PySpark version is 3.1.1, and Java version is 1.8. The environment variable JAVA_HOME has already been set(use kerberos)