-
I’ve setup a EventBridge Pipe target to be a FIFO SQS queue
-
The FIFO SQS has a Lambda Trigger that processes the messages in “batches of 1” (for my use-case, I need to process 1 at a time due to some tight constraints)
-
I’ve tweaked the SQS props (visibilityTimeout, delay etc) but have noticed that on multiple occasions, on SQS the messageSent (items being pushed into SQS by the pipe) is correct, but the lambda is not triggered and therefore messageReceive count is not the same as messageSent. Worse part is that the messages are not available in the queue anymore and not in DLQ as well. I’ve read that this is an issue with SQS queues with “low numbers” of events due to the distributed nature of AWS SQS.
-
BUT, one recommendation i’ve got is to make sure that each message generated from the Pipe and sent to SQS should have a unique MessageGroupId. But EventBridge does not seem to allow for unique or dynamic values… you have to hardcode the same static value. (see below)
This was an issue 3 years ago as well, but i’m hoping this is fixed?
- Does anyone know how to make the MessageGroupId in the Pipe dynamic (use a unique value/uuid from the source) / unique?
- If still not possible, what architecture can I use with Pipe’s to add another layer that can process and add unique MessageGroupId’s before sending to the FIFO SQS?
Thanks very much!