I have a dataset with array column like that:

My use case is to start a spark job from Dolphin Scheduler or Airflow. This job will read the data from the Apache Pulsar stream for a timespan, process the records, and then shut down or close the application completely.

The normal add a date column would be this

The normal add a date column would be this

The normal add a date column would be this

def apply(plan: LogicalPlan): LogicalPlan = { plan transform { case unresolvedRelation: UnresolvedRelation => val tblSchemaName: Array[String] = unresolvedRelation.tableName.split(“\.”) if (tblSchemaName.length == 1) return plan val schema = tblSchemaName.apply(0) val tblName = tblSchemaName.apply(1) for (ref <- this.refs) { if (tblName == ref.nqName) { return unresolvedRelation.copy(multipartIdentifier = Seq(schema.toUpperCase, tblName.toUpperCase), unresolvedRelation.options, unresolvedRelation.isStreaming) } } unresolvedRelation case unresolvedWith: UnresolvedWith […]

Going off docs/other posts online, you should cache() before checkpoint() because checkpoint() is done afterwards with a different action. However looking at spark query plan, this doesn’t seem to be true for what I’m doing:

I have got characters in feed like ‘NUL’. We are trying a to find a way through SparkSQL to find these records. We tried using x00 etc as part of rlike etc, but spark dataset returned empty dataset

I am trying to process data using pyspark which is set up in to run in AWS ECS service, below are the Transformation I do on data and then write dataframe to parquet.
NOTE: It only fails for big files.

We are using spark to process big data (PB) of events. They are JSON, no fixed schema. We are wondering whether we use dataframe to infer schema, or, we use RDD.