In my Github Actions workflow, I am trying to dynamically create one K8s Job, and K8s Secret or Configmap per schema, in a list that I’m fetching from Snowflake.

I’m looking for the best way is to accomplish this. Right now, I have a Workflow which fetches the list of schemas in one step, then in another step, I’m iterating over that list, and using the schema name to replace some values in the Secret and Job template, then applying each one to do a data ingestion task.

The problem I’m facing is with replacing the ${SNOWFLAKE_PRIVATE_KEY} and ${SNOWFLAKE_PRIVATE_KEY_PASS} in the stringData section of my Secret. I’m hitting errors like 

”sed: -e expression #4, char 64: unterminated `s’ command”

coming from the sed command in the run section of my Workflow step, which I’m using to try to replace the values:

sed "s#${SNOWFLAKE_PRIVATE_KEY}#${{ env.SNOWFLAKE_PRIVATE_KEY }}#g" "$(pwd)/k8s/sf-to-console-configmap-template.yml" > "$(pwd)/k8s/sf-to-console-configmap-$SANITIZED_SCHEMA_NAME.yml"  

I think this error is caused by Github Actions encoding the snowflake password as “*****” but I am not sure. Also, it seems like there should be a better way to replace these values. Any advice is greatly appreciated. Newbie here btw if you couldn’t tell.

Here is the relevant snippet from the workflow as-is for reference:

- name: Get vault secrets  
        uses: "redacted"  
        with:  
          vault-url: "redacted"  
          role-id: ${{ secrets.VAULT_PROD_ROLE_ID }}  
          secret-id: ${{ secrets.VAULT_PROD_SECRET_ID }}  
          auth-namespace: "redacted"  
          secret-namespace: "redacted"  
          secret-list: |  
            "redacted" 'rsa_key.p8' | SNOWFLAKE_PRIVATE_KEY;  
            "redacted" private_key_pass | SNOWFLAKE_PRIVATE_KEY_PASS;  
  
      - name: Authenticate K8s / Setup kubectl  
        uses: "redacted"  
        with:  
          environment: dev  
          vault-role-id: ${{ secrets.VAULT_PROD_ROLE_ID }}  
          vault-secret-id: ${{ secrets.VAULT_PROD_SECRET_ID }}  
  
      - name: Fetch schema list  
        id: fetch-schemas  
        run: |  
          python3 k8s/fetch-schemas.py  
        env:  
          SNOWFLAKE_PRIVATE_KEY: ${{ env.SNOWFLAKE_PRIVATE_KEY }}  
          SNOWFLAKE_PRIVATE_KEY_PASS: ${{ env.SNOWFLAKE_PRIVATE_KEY_PASS }}  
          SNOWFLAKE_USER: "redacted"  
          SNOWFLAKE_ACCOUNT: "redacted"  
          SNOWFLAKE_ROLE: "redacted"  
          SNOWFLAKE_WAREHOUSE: "redacted"  
  
      - name: Loop through schemas and deploy  
        shell: bash
        run: |  
          SCHEMA_LIST=$(echo '${{ steps.fetch-schemas.outputs.schema_list }}' | jq -r '.[]')  
          for SCHEMA in $SCHEMA_LIST; do  
            echo "Processing schema: $SCHEMA"  
            export SCHEMA_NAME=$SCHEMA  
            SANITIZED_SCHEMA_NAME=$(echo $SCHEMA | sed 's/[_.]/-/g' | tr '[:upper:]' '[:lower:]')  

            # Echo the sanitized schema name  
            echo "Sanitized schema name: $SANITIZED_SCHEMA_NAME"
 
            # Fetch secrets from the vault step  
            SNOWFLAKE_PRIVATE_KEY="${{ env.SNOWFLAKE_PRIVATE_KEY }}"
            SNOWFLAKE_PRIVATE_KEY_PASS="${{ env.SNOWFLAKE_PRIVATE_KEY_PASS }}"

            echo "Current directory:"
            echo $(pwd)
            echo "Contents of k8s/sf-to-console-configmap-template.yml:"
            echo $(cat k8s/sf-to-console-configmap-template.yml)
           
            sed -e 's#${SANITIZED_SCHEMA_NAME}#'"$SANITIZED_SCHEMA_NAME"'#g' -e 's#${SCHEMA_NAME}#'"$SCHEMA_NAME"'#g' -e 's#${SNOWFLAKE_ENV}#'"$ENV_CODE"'#g' "$(pwd)/k8s/sf-to-console-configmap-template.yml" > "$(pwd)/k8s/sf-to-console-configmap-$SANITIZED_SCHEMA_NAME.yml"  

            # Debug: Check the contents of the generated configmap file  
            echo "Contents of $(pwd)/k8s/sf-to-console-configmap-$SANITIZED_SCHEMA_NAME.yml:"  
            cat $(pwd)/k8s/sf-to-console-configmap-$SANITIZED_SCHEMA_NAME.yml  

            # Delete existing Configmap if it exists  
            kubectl delete configmap sf-to-console-configmap-$SANITIZED_SCHEMA_NAME --ignore-not-found  

            # Create Configmap
            kubectl apply -f $(pwd)/k8s/sf-to-console-configmap-$SANITIZED_SCHEMA_NAME.yml

            # Replace placeholder in Job file and apply  
            sed "s#${SCHEMA_NAME}#$SANITIZED_SCHEMA_NAME#g" "$(pwd)/k8s/sf-to-console-job-template.yml" > "$(pwd)/k8s/sf-to-console-job-$SANITIZED_SCHEMA_NAME.yml" 
  
            # Debug: Check the contents of the generated job file  
            echo "Contents of k8s/sf-to-console-job-$SANITIZED_SCHEMA_NAME.yml:"  
            cat k8s/sf-to-console-job-$SANITIZED_SCHEMA_NAME.yml  

            # Delete existing Job if it exists  
            kubectl delete job sf-to-console-job-$SANITIZED_SCHEMA_NAME --ignore-not-found  
            
            kubectl apply -f k8s/sf-to-console-job-$SANITIZED_SCHEMA_NAME.yml  
          done