While scraping a website I encountered a strange behaviour, running the script multiple times gives variable results. Sometimes the number of elements obtained by find_all is the expected one, others instead I find duplicates, after using set() I obtain the same elements as before.
soup = BeautifulSoup(driver.page_source, 'html.parser')
cards = soup.find_all("div", {"class": "product-card__body"})
print(len(cards))
print(len(set(cards)))
I have tried looking at the differences in the html source but still I couldn’t figure it out. Using selenium I couldn’t notice differences in the two runs.
Davide Zamblera is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.