Context:

• I have a pipeline for getting the metrics from all the hosts system which are spread across different sites.
• Each site has a gateway node (let’s refer it as site gateway) and all the site gateway node are connected to another gateway node (let’s refer it as master gateway).
• Each host have Grafana agent running as a service which collects metrics and remote write it to the Prometheus deployed on site gateway node.
• And Prometheus on site gateway node remote write metrics to Prometheus deployed on master gateway node.
• Then at last the Prometheus on master gateway node write all the metrics to a long term metrics storage (Grafana Mimir).
• Prometheus on site & master gateway is deployed on the local k8s cluster using a custom helm chart.

Pipeline flow:

Host 1(Grafana Agent) -> Site Gateway 1 (Prometheus) —↓

Host 2(Grafana Agent) -> Site Gateway 2 (Prometheus) -> Master Gateway (Prometheus) -> Long term Storage (Grafana Mimir)

Question:

What are the ways on doing an end to end testing of this pipeline ??