I am trying to run performance test script against my test kafka cluster in AWS. It looks like something is restricting the performance to about 2 MB/sec.
I have tried following:
- Increased the type of broker to m7g.2xlarge
- Increased the number of partitions
- Set replication factor to just 1
- Increased storage to more than 10GB (I don’t need that much at this time)
Using the following command
./kafka-producer-perf-test.sh --topic test-topic --num-records 500000 --record-size 1000 --throughput -1 --producer-props bootstrap.servers=b-1:9098,b-2:9098 --producer.config ../kafka.iam.client.properties
In all cases the output of kafka-producer-perf-test.sh script (which is shipped with kafka installation) looks like below:
- 4641 records sent, 927.3 records/sec (0.88 MB/sec), 1588.7 ms avg latency, 3184.0 ms max latency.
- 13120 records sent, 2368.7 records/sec (2.26 MB/sec), 4610.8 ms avg latency, 7805.0 ms max latency.
- 13088 records sent, 2610.8 records/sec (2.49 MB/sec), 10436.8 ms avg latency, 12757.0 ms max latency.
- 11776 records sent, 2314.9 records/sec (2.21 MB/sec), 12943.7 ms avg latency, 13762.0 ms max latency.
- 14080 records sent, 2810.9 records/sec (2.68 MB/sec), 13289.8 ms avg latency, 14007.0 ms max latency.
- 11872 records sent, 2329.2 records/sec (2.22 MB/sec) 12565.5 ms avg latency, 13521.0 ms max latency.
- 14176 records sent, 2832.9 records/sec (2.70 MB/sec), 12937.7 ms avg latency, 13557.0 ms max latency.
- 11808 records sent, 2323.0 records/sec (2.22 MB/sec),12501.0 ms avg latency, 13413.0 ms max latency.
Same script running against a kafka cluster locally results in a speed of over 50 MB/sec. So what am I missing here? Is the high latency because of me running the script from my local machine against a cluster in AWS? If that is the case then why is that this slow because my internet speed is not like that. Why is the latency hovering around a similar number and why is it so high?