I have a metric of type Counter that counts the number of workflows completed. I’m trying to use it to display the total number of workflows completed over the selected period of time.
The application that emits this metric is deployed to multiple regions using Kubernetes. So the following query:
metric=workflow_completed workflow_type=ShopWorkflow
Yields the following results
The sum of all values in the max and latest columns is the right result that I want to produce i.e., 27.
I was hoping that the following query will get me what I want
metric=workflow_completed workflow_type=ShopWorkflow
| sum by workflow_type
And what I want is – which is basically the sum of all values
Instead, what I get is
And I don’t know why. It’s probably got something to do with the automatic quantization. Because if I add quantize to 14d using max (because I’m executing the query over 14 days), then I do get the result that I want. But that can’t be a solution because the consumers of this query will want to change the time range so I can’t just hardcode it to any value. And if I remove 14d from the query then I get the same result that I see approximately the same result that I get without the quantize.
My questions are:
- What explains the values that I’m seeing when I
sum? - How can I change my query to get the actual plain sum of values in the max or latest columns?