I have an SQS FIFO queue that occasionally gets an influx of messages in the order of tens to hundreds. I set up a Lambda processing function and an SQS trigger with the following parameters:
Lambda concurrency: unreserved (1000)
SQS trigger batch size: 1 (The processing is rather long running, involves connecting to S3 and RDS Postgres database, and takes minutes to complete, so I want a single invocation to process one task at a time)
SQS trigger maximum concurrency: 10
SQS trigger batch window: 0
My problem is, I never see concurrent executions go above 1 in lambda monitoring. I have read up about how the SQS trigger works by monitoring how busy the queue is and spinning up more polling agents as necessary and while it’s not ideal for me, I expected to see more than 1 concurrency after some minutes of the queue being busy and seeing backlogged message counts up to a hundred.
Please give some advice on what I’m not seeing here and which other parameters I should check. My ideal set up would be to see the concurrency max out at 10 as per trigger configuration within about a minute of the queue being busy but for now I want to out least understand why concurrency never goes above a single invocation.
Thanks!