I’m encountering an issue while trying to pass callback keyword arguments (cb_kwargs) in my Scrapy spider. Here’s a simplified version of my code structure:

class MySpider(scrapy.Spider):
    name = 'my_spider'

    def parse(self, response):
        # Extracting product data
        product_data = {
            'name': 'Product Name',
            'price': 100.0,
            'colour': '',
            'size': ['S', 'M', 'L'],
            'reviews_count': 0,
            'reviews_score': 0.0,
        }

        # URLs for API endpoints
        color_api_url = 'https://example.com/color_api'
        reviews_api_url = 'https://example.com/reviews_api'

        # Making request to color API
        yield scrapy.Request(url=color_api_url, callback=self.parse_color, cb_kwargs={'product_data': product_data, 'reviews_api_url': reviews_api_url})

    def parse_color(self, response, **cb_kwargs):
        product_data = cb_kwargs['product_data']
        reviews_api_url = cb_kwargs['reviews_api_url']

        # Parsing color API response
        # Updating product_data['colour']

        # Making request to reviews API
        yield scrapy.Request(url=reviews_api_url, callback=self.parse_reviews, cb_kwargs={'product_data': product_data})

    def parse_reviews(self, response, **cb_kwargs):
        product_data = cb_kwargs['product_data']

        # Parsing reviews API response
        # Updating product_data['reviews_score'] and product_data['reviews_count']

        yield product_data

In this setup, I’m trying to pass product_data and reviews_api_url from parse() to parse_color() using cb_kwargs, and then from parse_color() to parse_reviews(). However, I’m experiencing issues where parse_reviews() doesn’t seem to receive product_data correctly, leading to incorrect or missing data in the final output.

I’ve checked that cb_kwargs are correctly specified when making the requests, and it seems that the code broke when calling the callback functions (parse_color() and parse_reviews()).

Could someone please help me understand the correct approach to pass and utilize cb_kwargs across multiple callback functions in a Scrapy spider? What could be potential pitfalls or best practices to ensure seamless data flow?