Environment:-
AWS managed opensearch service: v2.11
Fluentd deployed on EKS cluster

Fluentd dameonset deployed using below yaml script

apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd
  namespace: kube-system
  labels:
    k8s-app: fluentd-logging
    version: v1
spec:
  selector:
    matchLabels:
      k8s-app: fluentd-logging
      version: v1
  template:
    metadata:
      labels:
        k8s-app: fluentd-logging
        version: v1
    spec:
      containers:
      - name: fluentd
        image: fluent/fluentd:v1.16-debian-1
        imagePullPolicy: IfNotPresent
        env:
        - name: OPENSEARCH_HOST
          value: "vpc-opensearch-xxxxxxxxx.on.aws"
        - name: OPENSEARCH_PORT
          value: "443"
        volumeMounts:
        - name: config
          mountPath: /etc/fluent/
      volumes:
      - name: config
        configMap:
          name: fluentd-config1
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: fluentd-config1
  namespace: kube-system
data:
  fluentd.conf: |

    <source>
      @type tail
      path /var/log/containers/*.log
      pos_file /var/log/fluentd-containers.log.pos
      tag kubernetes.*
      read_from_head true
      <parse>
        @type multi_format
        <pattern>
          format json
          time_key @timestamp
          time_format %Y-%m-%dT%H:%M:%S.%NZ
        </pattern>
        <pattern>
          format /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/
          time_format %Y-%m-%dT%H:%M:%S.%N%:z
        </pattern>
      </parse>
    </source>

    <filter kubernetes.**>
      @type kubernetes_metadata
      @id filter_kube_metadata
    </filter>

    <filter kubernetes.var.log.containers.**>
      @type parser
      <parse>
        @type json
        format json
        time_key time
        time_type string
        time_format "%Y-%m-%dT%H:%M:%S.%NZ"
        keep_time_key false
      </parse>
      key_name log
      replace_invalid_sequence true
      emit_invalid_record_to_error true
      reserve_data true
    </filter>

    <match kubernetes.**>
      @type opensearch
      log_level info
      include_tag_key true
      host "#{ENV['OPENSEARCH_HOST']}"
      port "#{ENV['OPENSEARCH_PORT']}"
      logstash_format true
      logstash_prefix kubernetes
      buffer_chunk_limit 2M
      buffer_queue_limit 32
      flush_interval 5s
      max_retry_wait 30
      disable_retry_limit
      num_threads 8
    </match>

Problem:-
index pattern not visible in the opensearch dashboard, further i can't create index pattern

inside fluentd pod trying to check the fluentd logs but getting errors

fluent@fluentd-gxzc7:/$ fluentd -c /etc/fluent/fluentd.conf --dry-run
2024-05-02 12:11:57 +0000 [info]: init supervisor logger path=nil rotate_age=nil rotate_size=nil
2024-05-02 12:11:57 +0000 [info]: parsing config file is succeeded path="/etc/fluent/fluentd.conf"
2024-05-02 12:11:57 +0000 [info]: gem 'fluentd' version '1.16.3'
2024-05-02 12:11:57 +0000 [info]: starting fluentd-1.16.3 as dry run mode ruby="3.1.4"
2024-05-02 12:11:57 +0000 [error]: config error file="/etc/fluent/fluentd.conf" error_class=Fluent::NotFoundPluginError error="Unknown filter plugin 'kubernetes_metadata'. Run 'gem search -rd fluent-plugin' to find plugins"