I am trying to deploy Airbyte to Amazon EKS by its helm chart.
Following this tutorial, I have created Kubernetes secret production-hm-airbyte-secret
with values
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY
- LOG_S3_BUCKET_NAME
- LOG_S3_BUCKET_REGION
- POSTGRES_USER_NAME
- POSTGRES_PASSWORD
However, after I deploy, I got error inside airbyte-bootloader
Error: couldn’t find key DATABASE_USER in Secret production-hm-airbyte/airbyte-airbyte-secrets
Based on this original values.yaml,
here is my-values.yaml:
global:
state:
storage:
type: S3
database:
type: external
host: production-hm-airbyte-postgres.xxx.us-west-2.rds.amazonaws.com
port: 5432
database: airbyte_db
user: airbyte_user
secretName: production-hm-airbyte-secret
passwordSecretKey: POSTGRES_PASSWORD
logs:
accessKey:
existingSecret: production-hm-airbyte-secret
existingSecretKey: AWS_ACCESS_KEY_ID
secretKey:
existingSecret: production-hm-airbyte-secret
existingSecretKey: AWS_SECRET_ACCESS_KEY
storage:
type: S3
minio:
enabled: false
server:
extraEnv:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: airbyte-logs-secrets
key: AWS_ACCESS_KEY_ID
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: airbyte-logs-secrets
key: AWS_SECRET_ACCESS_KEY
- name: STATE_STORAGE_S3_ACCESS_KEY
valueFrom:
secretKeyRef:
name: airbyte-logs-secrets
key: AWS_ACCESS_KEY_ID
- name: STATE_STORAGE_S3_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: airbyte-logs-secrets
key: AWS_SECRET_ACCESS_KEY
- name: STATE_STORAGE_S3_BUCKET_NAME
valueFrom:
secretKeyRef:
name: airbyte-logs-secrets
key: LOG_S3_BUCKET_NAME
- name: STATE_STORAGE_S3_REGION
valueFrom:
secretKeyRef:
name: airbyte-logs-secrets
key: LOG_S3_BUCKET_REGION
worker:
extraEnv:
- name: AWS_ACCESS_KEY_ID
valueFrom:
secretKeyRef:
name: airbyte-logs-secrets
key: AWS_ACCESS_KEY_ID
- name: AWS_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: airbyte-logs-secrets
key: AWS_SECRET_ACCESS_KEY
- name: STATE_STORAGE_S3_ACCESS_KEY
valueFrom:
secretKeyRef:
name: airbyte-logs-secrets
key: AWS_ACCESS_KEY_ID
- name: STATE_STORAGE_S3_SECRET_ACCESS_KEY
valueFrom:
secretKeyRef:
name: airbyte-logs-secrets
key: AWS_SECRET_ACCESS_KEY
- name: STATE_STORAGE_S3_BUCKET_NAME
valueFrom:
secretKeyRef:
name: airbyte-logs-secrets
key: LOG_S3_BUCKET_NAME
- name: STATE_STORAGE_S3_REGION
valueFrom:
secretKeyRef:
name: airbyte-logs-secrets
key: LOG_S3_BUCKET_REGION
postgresql:
enabled: false
externalDatabase:
host: production-hm-airbyte-postgres.xxx.us-west-2.rds.amazonaws.com
port: 5432
database: airbyte_db
user: airbyte_user
existingSecret: production-hm-airbyte-secret
existingSecretPasswordKey: POSTGRES_PASSWORD
After I append airbyte-bootloader
section:
airbyte-bootloader:
extraEnv:
- name: DATABASE_USER
value: airbyte_user
- name: DATABASE_PASSWORD
valueFrom:
secretKeyRef:
name: production-hm-airbyte-secret
key: POSTGRES_PASSWORD
Now I got error
failed to create typed patch object (production-hm-airbyte/airbyte-airbyte-bootloader; /v1, Kind=Pod): errors: .spec.containers[name=”airbyte-bootloader-container”].env: duplicate entries for key [name=”DATABASE_USER”] .spec.containers[name=”airbyte-bootloader-container”].env: duplicate entries for key [name=”DATABASE_PASSWORD”]
Also, I noticed Airbyte tries to create secret airbyte-airbyte-secrets
with values
- DEFAULT_MINIO_ACCESS_KEY
- DEFAULT_MINIO_SECRET_KEY
- WORKLOAD_API_BEARER_TOKEN
I searched online and found
- https://github.com/airbytehq/airbyte/issues/22094
- https://github.com/airbytehq/airbyte/discussions/33835
But none of them provides a solution.
Any guide would be appreciate, thanks!