I’m trying to use the mariaDB connector/J java client with spark 3.5.0 to read from my database on AWS.

The EMR cluster fails to read from the database with error

Caused by: java.sql.SQLInvalidAuthorizationSpecException: (conn=7486) Access denied for user 'user'@'192.168.1.1' (using password: YES)
at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.createException(ExceptionFactory.java:66)
at org.mariadb.jdbc.internal.util.exceptions.ExceptionFactory.create(ExceptionFactory.java:187)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.authenticationHandler(AbstractConnectProtocol.java:776)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.createConnection(AbstractConnectProtocol.java:553)
at org.mariadb.jdbc.internal.protocol.AbstractConnectProtocol.connectWithoutProxy(AbstractConnectProtocol.java:1387)

But If I connect to one of the core node and I run the same code I can connect to the Database

The code is

mySparkSession
    .read
    .format("jdbc")
    .option("driver", "org.mariadb.jdbc.Driver")
    .option("url", "jdbc:mysql://my-rds-address:my-port/myDB?permitMysqlScheme")
    .option("user", "user")
    .option("password", "password")
    .option("passwordCharacterEncoding", "utf-8")
    .option("sslMode", "verify-ca")
    .option("serverSslCert", "/opt/path/to/cert/my-cert.pem")
    .option("dbtable", "my_table")
    .load()

Spark version is 3.5.0
Scala is 2.12.17
MariaDB connector is 3.4.0
Database is Mysql 8
AWS EMR 7.0.0

I can’t understand why I can connect fine from the shell, but it doesn’t work from the EMR cluster.

The user permissions are correct, username and password are the same, the code is exactly the same.

Any idea that can help me to solve this is appreciated.

Thanks.