I am encountering an error when trying to use Selenium with Scrapy. The error message is: WebDriver.init() got an unexpected keyword argument ‘executable_path’
I am using a custom middleware in middlewares.py where I only use the executable_path keyword argument in one spot. According to the Selenium documentation, this is a valid argument and should be used as shown below:
class selenium.webdriver.chrome.service.Service(
executable_path=None,
port: int = 0,
service_args: List[str] | None = None,
log_output: int | str | IO[Any] | None = None,
env: Mapping[str, str] | None = None,
**kwargs
)[source]
Here is the relevant section of my middlewares.py:
from scrapy import signals
from scrapy_selenium import SeleniumMiddleware
from selenium import webdriver
from selenium.webdriver.chrome.service import Service as ChromeService
from selenium.webdriver.chrome.options import Options
class CustomSeleniumMiddleware(SeleniumMiddleware):
@classmethod
def from_crawler(cls, crawler):
settings = crawler.settings
driver_name = settings.get('SELENIUM_DRIVER_NAME')
driver_executable_path = settings.get('SELENIUM_DRIVER_EXECUTABLE_PATH')
driver_arguments = settings.get('SELENIUM_DRIVER_ARGUMENTS', [])
browser_executable_path = settings.get('SELENIUM_BROWSER_EXECUTABLE_PATH', None)
options = Options()
for argument in driver_arguments:
options.add_argument(argument)
service = ChromeService(executable_path=driver_executable_path)
driver = webdriver.Chrome(service=service, options=options)
middleware = cls(driver_name, driver_executable_path, driver_arguments, browser_executable_path, driver)
crawler.signals.connect(middleware.spider_closed, signal=signals.spider_closed)
return middleware
def __init__(self, driver_name, driver_executable_path, driver_arguments, browser_executable_path, driver):
self.driver = driver
super().__init__(driver_name, driver_executable_path, driver_arguments, browser_executable_path)
@staticmethod
def create_options(arguments):
chrome_options = Options()
for argument in arguments:
chrome_options.add_argument(argument)
return chrome_options
Based on the Selenium documentation, I expected that using the executable_path keyword argument would correctly set up the Chrome service. However, I am getting the error that it is an unexpected keyword argument.
Here are the steps I have tried so far:
Verified the ChromeService class from Selenium documentation, which indicates that executable_path is a valid argument.
Ensured that the executable_path provided is correct and points to the chromedriver executable.
Updated Selenium to the latest version to ensure compatibility.
Despite these steps, I am still encountering the error. I am not sure if I am misinterpreting the documentation or if there is an issue elsewhere in my code or environment setup.
Any help in resolving this issue would be greatly appreciated.