I have a DBT installation(core) running on Github(GH) Action runners, triggered every 4 hours that refreshes about 100+ models.
Jobs used to complete successfully without any error. However, after recently rotating the Snowflake user password in the GH secrets, it intermittently started throwing password authentication errors. Debug log points to this page – https://community.snowflake.com/s/error-your-user-login-has-been-locked
If I just unlock the user from DB console, its back to running normal using the same password extracted from GH secrets and fails again randomly (but frequently). For eg, it refreshes 80% of the models and fails abruptly with password issues on rest of them.
Error message –
Database error while listing schemas in database “XXXX_SnowDB”
Database Error
250001 (08001): Failed to connect to DB: XXX.snowflakecomputing.com:443. Your user account has been temporarily locked due to too many failed attempts. Try again after 15 minutes or contact your account administrator for assistance. For more information about this error, go to https://community.snowflake.com/s/error-your-user-login-has-been-locked.
Error: Process completed with exit code 2.
1 – Does GH has some kind of cache that needs to be cleaned up? (I am not sure if its the case since workflow is successful for half of the dbt models but fails for rest of them)
2 – How does Snowflake connection behave here. Does it refresh connection with the dbt user for every model, or is it one long connection that refreshes all the models.