I was following this tutorial and encountered this error while trying to run the code.
File "/home/nawalnaz/miniconda3/bin/scrapy", line 8, in <module>
sys.exit(execute())
^^^^^^^^^
File "/home/nawalnaz/miniconda3/lib/python3.11/site-packages/scrapy/cmdline.py", line 160, in execute
cmd.crawler_process = CrawlerProcess(settings)
^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/nawalnaz/miniconda3/lib/python3.11/site-packages/scrapy/crawler.py", line 357, in __init__
super().__init__(settings)
File "/home/nawalnaz/miniconda3/lib/python3.11/site-packages/scrapy/crawler.py", line 227, in __init__
self.spider_loader = self._get_spider_loader(settings)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/nawalnaz/miniconda3/lib/python3.11/site-packages/scrapy/crawler.py", line 221, in _get_spider_loader
return loader_cls.from_settings(settings.frozencopy())
^^^^^^^^^^^^^^^^^^^^^
File "/home/nawalnaz/miniconda3/lib/python3.11/site-packages/scrapy/settings/__init__.py", line 470, in frozencopy
copy = self.copy()
^^^^^^^^^^^
File "/home/nawalnaz/miniconda3/lib/python3.11/site-packages/scrapy/settings/__init__.py", line 452, in copy
return copy.deepcopy(self)
^^^^^^^^^^^^^^^^^^^
File "/home/nawalnaz/miniconda3/lib/python3.11/copy.py", line 172, in deepcopy
y = _reconstruct(x, memo, *rv)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/nawalnaz/miniconda3/lib/python3.11/copy.py", line 271, in _reconstruct
state = deepcopy(state, memo)
^^^^^^^^^^^^^^^^^^^^^
File "/home/nawalnaz/miniconda3/lib/python3.11/copy.py", line 146, in deepcopy
y = copier(x, memo)
^^^^^^^^^^^^^^^
File "/home/nawalnaz/miniconda3/lib/python3.11/copy.py", line 231, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
^^^^^^^^^^^^^^^^^^^^^
File "/home/nawalnaz/miniconda3/lib/python3.11/copy.py", line 146, in deepcopy
y = copier(x, memo)
^^^^^^^^^^^^^^^
File "/home/nawalnaz/miniconda3/lib/python3.11/copy.py", line 231, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
^^^^^^^^^^^^^^^^^^^^^
File "/home/nawalnaz/miniconda3/lib/python3.11/copy.py", line 172, in deepcopy
y = _reconstruct(x, memo, *rv)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/nawalnaz/miniconda3/lib/python3.11/copy.py", line 271, in _reconstruct
state = deepcopy(state, memo)
^^^^^^^^^^^^^^^^^^^^^
File "/home/nawalnaz/miniconda3/lib/python3.11/copy.py", line 146, in deepcopy
y = copier(x, memo)
^^^^^^^^^^^^^^^
File "/home/nawalnaz/miniconda3/lib/python3.11/copy.py", line 231, in _deepcopy_dict
y[deepcopy(key, memo)] = deepcopy(value, memo)
^^^^^^^^^^^^^^^^^^^^^
File "/home/nawalnaz/miniconda3/lib/python3.11/copy.py", line 161, in deepcopy
rv = reductor(4)
^^^^^^^^^^^
TypeError: cannot pickle 'module' object
I understand that the module object cannot be deep copied since Python does not allow it, but I am confused on what I should change either within the source code or within scrapy’s files themselves. If anyone could help me, that would be great!
New contributor
kahanahmed3003 is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.