I have Kafka running in a Kubernetes cluster and using Promtail to send Kafka messages to Loki. Right now there is about a 5 second delay between when the message is produced and when Loki receives it. I’d like to know if there is any way to get this delay shortened to as close to “real-time” as possible.

I’ve been looking to see if there’s a way to set a scrape interval like there is in Prometheus, but I can’t seem to find any documentation on it.

My scrape config is pretty standard, but here it is if you want it:

- job_name: kafka
        kafka:
          brokers:
          - my-kafka-address:9092
          topics:
          - my-topic
          group_id: promtail-kafka
          labels:
            job: kafka
        relabel_configs:
            - action: replace
              source_labels:
                - __meta_kafka_topic
              target_label: topic
            - action: replace
              source_labels:
                - __meta_kafka_partition
              target_label: partition
            - action: replace
              source_labels:
                - __meta_kafka_group_id
              target_label: group