I have an ETL flow where I need to combine two Pub/Sub messages on a key and write these into BigQuery. One of the message types is the parent; I am working on payment processing, and this is an order or a payment, for example. The other is the child; this is an update to the payment (“Authorized”, “Paid”, etc).

I am a Data Engineer.
I have used PySpark for a long time and now moving to Apache Beam/Dataflow .

I have a dataflow pipeline that reads messages from kafka, process them, and insert them into bigquery.
I want that the processing / bigquery insertion will happen in time based batches, so that on every (1 minute) interval, all messages that was read from kafka in that interval will be processed into bigquery.

RuntimeError: Pipeline construction environment and pipeline runtime environment are not compatible. If you use a custom container image, check that the Python interpreter minor version and the Apache Beam version in your image match the versions used at pipeline construction time. Submission environment: beam:version:sdk_base:apache/beam_python3.11_sdk:2.54.0. Runtime environment: beam:version:sdk_base:apache/beam_python3.10_sdk:2.56.0.

This is a simplified version of the DoFn I run in Google Cloud Dataflow: