I have a Dataflow pipeline that ingests messages from PubSub. This automatically creates a subscription however the retention duration is 7 days. I like that it creates the subscription so I don’t have to and it’s convenient, however the retention duration has caused some issues when a bug arose and resulted in a huge (and expensive) backlog.
Is there an existing way in the Apache Beam Python SDK to configure this duration without removing the automatic subscription creation?
| "Read from PubSub" >> beam.io.ReadFromPubSub(topic=self.topic_path)
I know I can pass a subscription into this, but then I’d have to create one.