When querying for Compute Engine costs I get one record with a null workload name.
What services besides k8s consume Compute Engine or maybe there is an unnamed workload in k8s?
How do I further investigate this cost?
SELECT
DATE(usage_start_time) day,
ROUND(SUM(cost),2) AS sum_cost,
labels.value AS workload,
FROM
my_billing_table
LEFT JOIN
UNNEST(labels) AS labels
ON
labels.key = "k8s-workload-name"
WHERE
DATE(usage_start_time) >= DATE_ADD(CURRENT_DATE,INTERVAL -2 DAY)
AND DATE(partition_time) >= DATE_ADD(CURRENT_DATE,INTERVAL -2 DAY)
AND project.name = 'my_project'
AND service.description = 'Compute Engine'
GROUP BY
1,
3
ORDER BY
1 DESC
Please follow the below steps which may help you :
- Check and Enable GKE cost allocation if not yet enabled to get cost breakdown. After enabling it on their cluster, you are now able to get GKE Cost for Clusters and Namespaces, I mean you can get the namespaces names and their associated costs.
As per View GKE cluster costs :
“If GKE cannot determine the cost allocation for a resource, the export includes one of the following values:
-
goog-k8s-unknown: Cloud Billing couldn’t process the SKU. This can occur when a new Compute Engine instance is being provisioned. You can expect some goog-k8s-unknown values during node startup and shutdown, for example when GKE autoscales a cluster.
-
goog-k8s-unsupported-sku: GKE cost allocation does not support this SKU. Treat this the same as <blank>/NULL. There is no guarantee that a SKU is always labeled with this value.
-
<blank>/NULL: GKE cost allocation does not track this resource. This can occur when GKE cost allocation is not enabled, or the resource does not belong to a GKE managed cluster.”
-
As per the Label Propagation : Manually created VMs without K8s are not labeled with a k8s-workload-name, which could be the reason for a null workload name.
-
Check for the Non-Kubernetes Services: Other services, such as Compute Engine VM instances, that do not use Kubernetes may be using resources. You can filter by labels other than “k8s-workload-name” to see if these VMs are consuming resources.
To investigate further :
-
Check the list of instance names, with the help of gcloud compute instances describe command displays data about a particular missed Compute Engine instance and have a look for any custom metadata labels that might indicate the purpose of the instance.
-
With the help of Logging and Monitoring dashboards for any errors around the time of these costs can help to understand better.
-
Query for other Labels: Modify the query to check if other labels are present and identify which workloads don’t have the “k8s-workload-name” label but are still consuming resources.
-
Ensure Workload Labeling: Kubernetes workloads may not have been properly labeled. You could review your Kubernetes cluster configuration to ensure that all workloads are tagged correctly with the “k8s-workload-name” label.
Please refer to the Structure of Detailed data export and also you can use GCP cost allocation feature which can help you to find the origin of the unattributed Compute Engine costs View detailed breakdown of cluster costs for more details.