I am new to Spark and Kafka world and I am just blindly following a tutorial to create a data engineering project and learn the concepts parallelly.
I have encountered the error mentioned in title for the given code below –
`sel = spark_df.selectExpr(“CAST(value AS STRING)”)
.select(from_json(col(‘value’), schema).alias(‘data’)).select(“data.*”)
print(sel)
return sel`
Note – I am using WSL and VS Code for this project and I am installing libraries when the error exists so I have not setup the proper environment in advance.
Earlier it said that ‘spark-cassandra-connector’ and ‘spark-sql-kafka’ .jar files were missing which led to java errors. So I had manually downloaded and pasted those files into jars sub-folder of PySpark folder.
But its still giving ‘AttributeError’
Mithil Bartakke is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.