I’m executing a mysql stored procedure (code below) and trying to convert it to a pyspark dataframe.
driver_manager = spark._sc._gateway.jvm.java.sql.DriverManager
connection = driver_manager.getConnection(args["sql_server_jdbc_url"], database_username, database_user_password)
exec_statement = connection.prepareCall("EXEC SP")
exec_statement.execute()
result = exec_statement.getResultSet()
from pyspark.sql import SQLContext, DataFrame
sqlContext = SQLContext(sparkContext=spark.sparkContext, sparkSession=spark)
df = DataFrame(result, sqlContext)
df.printSchema()
df.printSchema doesn’t work and throwing an error Py4JError: An error occurred while calling o83.schema. Trace:
looking for pointers on how to convert the results from stored procedure to a pySpark dataframe.
Tried this Pyspark Jupyter – dataframe created in java code vs python code