Openai llm models gives us the information of how many tokens are consumed, we want to use this data to get how many tokens are consumed in given period of time. So have implemented custom metrics where tokens is set as counter and is sent to Prometheus.
Now i want to know is counter good suitable for this. and what query should i us so that i can get total tokens consumed in given time range.
Currently i have used below query
sum((llm_tokens_total{cluster=~"${cluster}",namespace=~"${namespace}",container="ap"}))
Using a counter for tracking tokens is spot on since counters only increase. To find out the total tokens used in a certain period, use this query:
sum(increase(llm_tokens_total{cluster=~"${cluster}",namespace=~"${namespace}",container="ap"}[time_range]))
Replace time_range with your desired period, like 5m or 1h. This query will show you the total tokens consumed in that time frame.