I have a collection in mongo myCollection
which has a Kafka Connector listening to the changes in the collection. There is another collection myCollectionPreview
. This is the collection where I insert all my tested data and if it’s all fine, I rename this collection to myCollection
setting dropTarget=true
.
This is the source mongo connector configuration on both production and my local testing environment.
"config": {
"connector.class":"com.mongodb.kafka.connect.MongoSourceConnector",
"connection.uri":"mongodb://mongo1:27017/?replicaSet=rs0",
"pipeline": "[{"$match":{"$or":[{"ns.coll":{"$in":["myCollection"]},"ns.db":"quickstart"}],"operationType":{"$nin":["drop","rename"]}}}]",
"topic.namespace.map": "{"quickstart.myCollection":"myCollection"}",
"change.stream.full.document":"updateLookup"
}
On production I notice that there are Kafka messages with operationType=replace
when I rename the myCollectionPreview -> myCollection
with dropTarget=true
but on local there is no such message produced on Kafka. I confirmed that the local set up is working as I can see messages on Kafka for other operations types like insert,delete,update
.
What could I be missing?