I am trying to collect logs from my Kafka which is running on an EKS cluster. My intentions were to use the Bitnami package for Logstash and the Bitnami package for Fluentd. I am having trouble getting this setup and not sure if this is the correct/best way of doing this. How can I get my Kafka metrics (which are in stdout) with fluentd then have them sent to logstash in order to then use Kibana to build a dashboard around these Kafka metrics?
I have tried adding FluentD and Logstash to my EKS cluster but cannot get a connection between my Kafka -> FLuentD -> Logstash. Please help.