I’ve tried many ways to update grafna config using values.yaml but when i check pod its keep failing also I’m new to it so trying to learn on it. when i try to change the pod config that also I’m not able to get because its getting deployed using helm.

below are the packages I’m using for deployment using helm

1. grafana/grafana
2. kube-stack-prometheus

once I deploy new change in values.yaml using the helm upgrade then either pod gets an error ErrImagePull or in pod logs i see the error as “ogger=context userId=0 orgId=0 uname= t=2024-04-27T07:24:20.161763369Z level=info msg=”Request Completed” method=GET path=/ status=302 remote_addr=10.11.55.00 time_ms=0 duration=99.836µs size=29 referer= handler=/ status_source=server”

below is the config for values.yaml

grafana:
 grafana.ini:
   smtp:
     enabled: true
     host: 'smtp.office365.com'
     port: 587
     user: "user name"
     password: '''password'''
     fromAddress: "[email protected]"
     fromName: Grafana
     skipVerify: true
     from: "[email protected]"
 
 ingress:
   enabled: true
   annotations:
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: NodePort
    alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:00000000000:certificate/90AAAAAA-00000-MMMM-BBBBB-88888888888
    alb.ingress.kubernetes.io/subnets: "subnet-A,subnet-B,subnet-C" 
    alb.ingress.kubernetes.io/group.name: ms
   rules:
     - host: grafana.dev.domain.com
       http:
        paths:
        - backend:
            service:
              name: kube-stack-prometheus-grafana
              port:
                number: 80
          path: /
          pathType: Prefix

below is the grafana service config

apiVersion: v1
kind: Service
metadata:
  annotations:
    meta.helm.sh/release-name: kube-stack-prometheus
    meta.helm.sh/release-namespace: monitoring
  creationTimestamp: "2024-04-26T15:04:03Z"
  labels:
    app.kubernetes.io/instance: kube-stack-prometheus
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: grafana
    app.kubernetes.io/version: 10.4.1
    helm.sh/chart: grafana-7.3.9
  name: kube-stack-prometheus-grafana
  namespace: monitoring
  resourceVersion: "3141179"
  uid: 30db479f-ec7d-4056-810c-000000000000
spec:
  clusterIP: 171.x.x.x
  clusterIPs:
  - 171.x.x.x
  internalTrafficPolicy: Cluster
  ipFamilies:
  - IPv4
  ipFamilyPolicy: SingleStack
  ports:
  - name: http-web
    port: 80
    protocol: TCP
    targetPort: 3000
  selector:
    app.kubernetes.io/instance: kube-stack-prometheus
    app.kubernetes.io/name: grafana
  sessionAffinity: None
  type: ClusterIP
status:
  loadBalancer: {}

Also when i describe the Ingress it shows the below error:

Default backend: default-http-backend:80 (<error: endpoints “default-http-backend” not found>)

I’m using grafana container exposing on 3000 where a service running on ClusterIP with 80 port and ingress as ip
now please share how to fix these issues?

I have tried to change the ingress type as ip and service type as ClusterIP but in both the cases it didn’t work.
Also tried to fix the container but after redeployment only it started working.
now i have two issues out of three that how to connect ingress with internet while doing curl it says trying to connect.
other is how to update smtp in values.yml so it can update in grafana.ini.
please explain it properly