I have used the gitlab cloudnative helm chart to self host gitlab and object store backed by s3 on AWS EKS using NGINX ingress. The object store works correctly but I cannot get the registry object store to work with various configurations below. There are two different configurations I’m seeing when it comes to the values.yml. No matter which configuration I’ve tried I receive a blob upload unknown with an error message of:

{"content_type":"application/json","correlation_id":"41ffa7b8b1e69bcd20d10a962cbce957","duration_ms":1,"host":"self-hosted-redacted.com","level":"info","method":"PATCH","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"10.4.73.112:41348","remote_ip":"10.4.73.112","status":404,"system":"http","time":"2024-09-09T03:43:49.489Z","ttfb_ms":0,"uri":"/v2/repo/repo/elx-tutorial/blobs/uploads/c70be522-6bb7-408b-9478-1eade0e45b18?_state=7ls38sWs9i_lrMjKq5Gs8ydBIJxpYHRGVskCcPq0SRN7Ik5hbWUiOiIwNzJfY3MvMDcyLW1vbm9yZXBvL2VseC10dXRvcmlhbCIsIlVVSUQiOiJjNzBiZTUyMi02YmI3LTQwOGItOTQ3OC0xZWFkZTBlNDViMTgiLCJPZmZzZXQiOjAsIlN0YXJ0ZWRBdCI6IjIwMjQtMDktMDlUMDM6NDM6NDkuMTg5NzMwNDgyWiJ9","user_agent":"docker/26.1.4 go/go1.21.11 git-commit/de5c9cf kernel/6.6.31-linuxkit os/linux arch/arm64 UpstreamClient(Docker-Client/26.1.4 \(darwin\))","written_bytes":100}

First is the Registry documentation which reads that it is a subchart for the cloudnative registry.

registry:
  enabled:
  maintenance:
    readonly:
      enabled: false
    uploadpurging:
      enabled: true
      age: 168h
      interval: 24h
      dryrun: false
  image:
    tag: 'v4.8.0-gitlab'
    pullPolicy: IfNotPresent
  annotations:
  service:
    type: ClusterIP
    name: registry
  httpSecret:
    secret:
    key:
  authEndpoint:
  tokenIssuer:
  certificate:
    secret: gitlab-registry
    key: registry-auth.crt
  deployment:
    terminationGracePeriodSeconds: 30
  draintimeout: '0'
  hpa:
    minReplicas: 2
    maxReplicas: 10
    cpu:
      targetAverageUtilization: 75
    behavior:
      scaleDown:
        stabilizationWindowSeconds: 300
  storage:
    secret:
    key: storage
    extraKey:
  validation:
    disabled: true
    manifests:
      referencelimit: 0
      payloadsizelimit: 0
      urls:
        allow: []
        deny: []
  notifications: {}
  tolerations: []
  affinity: {}
  ingress:
    enabled: false
    tls:
      enabled: true
      secretName: redis
    annotations:
    configureCertmanager:
    proxyReadTimeout:
    proxyBodySize:
    proxyBuffering:
  networkpolicy:
    enabled: false
    egress:
      enabled: false
      rules: []
    ingress:
      enabled: false
      rules: []
  serviceAccount:
    create: false
    automountServiceAccountToken: false
  tls:
    enabled: false
    secretName:
    verify: true
    caSecretName:

Second is what is actually in the cloudnative values.yml default chart

[Values](https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/values.yaml?ref_type=heads)

  ## https://docs.gitlab.com/charts/charts/globals#configure-registry-settings
  registry:
    bucket: registry

    certificate: {}
      # secret:
    httpSecret: {}
      # secret:
      # key:
    notificationSecret: {}
      # secret:
      # key:
    tls:
      enabled: false
      # secretName:
    redis:
      cache:
        password: {}
      rateLimiting:
        password: {}
    # https://docs.docker.com/registry/notifications/#configuration
    notifications: {}
      # endpoints:
      #   - name: FooListener
      #     url: https://foolistener.com/event
      #     timeout: 500ms
      #     threshold: 10 # DEPRECATED: use maxretries instead https://gitlab.com/gitlab-org/container-registry/-/issues/1243.
      #     maxretries: 5
      #     backoff: 1s
      #     headers:
      #       FooBar: ['1', '2']
      #       Authorization:
      #         secret: gitlab-registry-authorization-header
      #       SpecificPassword:
      #         secret: gitlab-registry-specific-password
      #         key: password
      # events: {}

    # Settings utilized by other services referencing registry:
    enabled: true
    host:
    # port: 443
    api:
      protocol: http
      serviceName: registry
      port: 5000
    tokenIssuer: gitlab-issuer

And third is basically #1 but just the bucket and storage options [s3 driver](https://gitlab.com/gitlab-org/charts/gitlab/-/blob/master/examples/objectstorage/registry.s3.yaml)

s3:
  bucket: gitlab-registry-storage
  accesskey: AWS_ACCESS_KEY
  secretkey: AWS_SECRET_KEY
  region: us-east-1
  # regionendpoint: "https://minio.example.com:9000"
  v4auth: true

and

  storage:
    secret: gitlab-object-storage
    key: connection

My object store configuration followed the documentation and looks like

provider: AWS
region: region
aws_access_key_id: key
aws_secret_access_key: secret-key

and

    object_store:
      enabled: true
      proxy_download: true
      storage_options: {}
        # server_side_encryption:
      # server_side_encryption_kms_key_id
      connection:
        secret: gitlab-object-storage

After trying various configurations I can get the registry hosted, docker login to work using gitlab authentication, but when I tag and push an image it seems to 404. I’m on us-gov-west-1 if anyone has a good idea.