I can describe hive table in spark3-shell, spark-shell, beeline and spark-submit job properly. But when I describe the table in spark3-submit job it gives this error:
Unable to get field from serde: org. apache.hadoop.hive.serde2.avro.AvroSerDe java. lang. RuntimeException: MetaException(message:java.lang.NoSuchMethodError org.apache.avro.Schema$Field.‹init>(Ljava/lang/String;Lorg/apache, at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore (Table-java:333) at org.apache.hadoop.hive.ql.metadata.
my build.gradle for spark3:
compileOnly org.apache.hadoop:hadoop-common:2.6.5 compileOnly org.scala-lang:scala-library:2.12.17 implementation org.apache.avro:avro:1.10.2 implementation org.apache.spark:spark-sql_2.12:3.3.2
Tried running spark3 job but it failed.
Pragya Shukla is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.