I’m using Scrapy in Jupyternotebook to scrape the Yellowpages website and am running into a strange error.
My code scrapes the list view of YellowPages for when a user types in ‘auto’ for a variety of zip codes. While my code is correctly able to scrape almost all of the business listings (around 500) there are 2 that I cannot get my Spider to scrape.
These are 2 businesses: Roger’s Services and Northeastern Bus Rebuilders. (See screenshot below for how they appear on the website). I’ve inspected the website’s html and the structure of the divs containing the information I’m scrapping doesn’t seem markedly different from any of the other divs that were easily scraped. The webpage I’m having issues with can be found here. Note that all businesses on this page and subsequent pages are correctly scraped. Only 2, on this page specifically, seem to not be ‘seen’ by the Spider.