I am using the Go programming language for my services and leveraging the AWS SDK for Go and SQL Server.
In our architecture, we have two major services:
- Service A (Consumer)
- Service B (Producer – Many)
We can only process 10 messages at a time due to the limitations of the library.
I have attempted to run the polling and ingestion tasks concurrently, but it is taking a lot of time to poll and ingest to DB.
Currently, I fetch 10 messages at a time and make an ingest call to the database for each message.
In one day, it accumulates around 600,000 events in SQS.
Could someone suggest a better approach to poll and ingest into the database faster?
https://github.com/aws/aws-sdk-go/blob/v1.50.24/service/sqs/api.go#L7357
Architecture:
2