I’m trying to have a scheduled job in kubernetes to pull the events from the namespace and stream them to logging application. We use splunk for log management and all the job/pod logs are automatically forwarded to the splunk. So, I just need to run ‘kubectl get events -n ‘ to get the events. We are going to run this job within our k8s namespace.

I tried to create a cron job that run on an hourly basis to collect the events. But somehow I get error while running the script.
These are the steps I tried:

1. create a mounted volume for configmap value, this created the shell script file in the pod
2. start the scirpt in initcontainer
   Below is the code:

Deployment.yaml:

apiVersion: batch/v1
kind: CronJob
metadata:
  name: forward-kubernetes-events-job
spec:
  schedule: "0 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          volumes:
            - name: scripts
              emptyDir:
                sizeLimit: "512Mi"
          containers:
            - name: forward-events-container
              image: {{ image-tag }}
              volumeMounts:
                - name: scripts
                  mountPath: /scripts
                  command: ["sh", "-c", {{ .Files.Get "scripts/forward-events.sh" | quote }} ]
              env:
                - name: TARGET_ENV
                  value: "{{ .Values.app.targetEnv }}"
                - name: KUBE_CONFIG
                  value: "{{ .Values.kubeConfig }}"
              securityContext:
                securityContext:
                  runAsUser: 1000
                  allowPrivilegeEscalation: true
                  readOnlyRootFilesystem: false
                  capabilities:
                    drop:
                      - KILL
                      - MKNOD
                      - SYS_CHROOT
              resources:
                resources:
                  requests:
                    memory: "128Mi"
                    cpu: "128m"
                  limits:
                    memory: "500Mi"
                    cpu: "500m"
              readinessProbe:
                exec:
                  command:
                    - /bin/true
              livenessProbe:
                exec:
                  command:
                    - /bin/true
          restartPolicy: Never

Configmap.yaml:

apiVersion: v1
kind: ConfigMap
metadata:
  name: scripts-configmap
data:
  forward-events.sh: |
    #!/bin/bash

    export KUBECONFIG="$KUBE_CONFIG"

    KUBECTL_VERSION=$(curl -L -s https://dl.k8s.io/release/stable.txt)
    curl -LO "https://dl.k8s.io/release/$KUBECTL_VERSION/bin/linux/amd64/kubectl"
    chmod +x kubectl
    mv kubectl /etc/temp/
    
    kubectl config set-context "$KUBE_CONFIG"

    kubectl config get-contexts

    kubectl get events -n core-banking-"$TARGET_ENV" -o json

    echo "Events forwarded to Splunk"

When the job is run, we see this error:

File system is read-only, even when I have set readOnlyRootFilesystem: false.

Is there a better approach to get the events from the namespace? how to make the mounted volume executable?

TIA