How state management of Spark Structured Streaming queries work when using RocksDB
I’m running a Spark Structured Streaming application that reads from a Kinesis stream, applies some non-stateful transformations and then performs a dropDuplicates
operation using an id, but with no watermark. The state is being stored on RocksDB (Spark 3.5.1)
How state management of Spark Structured Streaming queries work when using RocksDB
I’m running a Spark Structured Streaming application that reads from a Kinesis stream, applies some non-stateful transformations and then performs a dropDuplicates
operation using an id, but with no watermark. The state is being stored on RocksDB (Spark 3.5.1)