My team has built a log storing mechanism which uses Kafka to store logs in the queue and store it in the database after consumer receives it.
The producer is running on an EC2 instance which is managed by a load balancer, and the consumer is on another instance writing into a database.
Mechanism of sending messages when instance of producer exists
When the traffic increases, multiple producers instances start with the same configuration as existing producer instance which sends data/messages to the pre-existing consumer EC2 instance, where the set condition for the spawning instances is to spawn in the same region but based on availability of zones.
Multiple producer instances sending data to consumer
The problem is when a lot of producer instances are running, there is loss of data in logs, where the consumer is not receiving the messages in the queue.
Pre-conditions have been set such that security group ids match and external ip of consumer instance is provided, so connecting to consumer should not be an issue.
All possible conditions for the instances have been tried.
Not all the logs are going missing but around 15-20% of the total count. Trying to figure out where the issue might be, could it because of AWS or Kafka?
Thank you for your assistance in the following matter.
amogh latlong is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.