I am new to PySpark and I am learning to read a local SQLite db file using PySpark in Jupyter notebook. I am using PySpark 3.5.1. The db file and the notebook were in the same folder. I used the following code
spark = SparkSession.builder
.appName('Ops')
.config("spark.jars", "mysql-connector-j-8.4.0.jar")
.getOrCreate()
df = spark.read.format('jdbc')
.options(url='activities.db', dbtable='activities', driver='com.mysql.jdbc.Driver')
.load()
I had the following error message.
---------------------------------------------------------------------------
IllegalArgumentException Traceback (most recent call last)
Cell In [16], line 3
1 df = spark.read.format('jdbc')
2 .options(url='activities.db', dbtable='activities', driver='com.mysql.jdbc.Driver')
----> 3 .load()
File ~/anaconda3/envs/spark/lib/python3.10/site-packages/pyspark/sql/readwriter.py:314, in DataFrameReader.load(self, path, format, schema, **options)
312 return self._df(self._jreader.load(self._spark._sc._jvm.PythonUtils.toSeq(path)))
313 else:
--> 314 return self._df(self._jreader.load())
File ~/anaconda3/envs/spark/lib/python3.10/site-packages/py4j/java_gateway.py:1322, in JavaMember.__call__(self, *args)
1316 command = proto.CALL_COMMAND_NAME +
1317 self.command_header +
1318 args_command +
1319 proto.END_COMMAND_PART
1321 answer = self.gateway_client.send_command(command)
-> 1322 return_value = get_return_value(
1323 answer, self.gateway_client, self.target_id, self.name)
1325 for temp_arg in temp_args:
1326 if hasattr(temp_arg, "_detach"):
File ~/anaconda3/envs/spark/lib/python3.10/site-packages/pyspark/errors/exceptions/captured.py:185, in capture_sql_exception.<locals>.deco(*a, **kw)
181 converted = convert_exception(e.java_exception)
182 if not isinstance(converted, UnknownException):
183 # Hide where the exception came from that shows a non-Pythonic
184 # JVM exception message.
--> 185 raise converted from None
186 else:
187 raise
IllegalArgumentException: requirement failed: The driver could not open a JDBC connection. Check the URL: activities.db
How should I address the problem? Thank you!