I am writing a web scraping script to comb through items from a particular brand on the cvs website, the part I am stuck on is when I drill down to the page of an individual item that has multiple sizes/packs and I need to iterate through all the sizes in the list and get the URLs for each option I can only seem to get the second option only in the list or nothing at all
Above is the section identified as the dropdown block I am trying to iterate though when inspecting the website
from selenium import webdriver
from selenium.webdriver.common.by import By
driver = webdriver.Chrome()
driver.get('https://www.cvs.com/shop/aveeno-daily-moisturizing-lotion-prodid-1013347?skuId=515049')
driver.maximize_window()
driver.implicitly_wait(15)
dropdown = driver.find_element(By.CLASS_NAME, "search_select")
dropdown.click()
options = dropdown.find_elements(By.XPATH, '//div[@class="select-dropdownContainer dswatchesMenuContainer"]')
for option in options:
option.click()
driver.implicitly_wait(2)
print(driver.current_url)
Above is my code that when run I only get the second options URL in the drop down box printed, I have tried it on various other items on the site with more options and still only get the second option. I have also tried replacing the conditions in the options variable with:
//ul[@class="variantsWithoutColorSwatchMobileContainer"]
with By.XPATH
which also resulted in the same output of the url for the second option in the drop down box. and when i replace both arguments in the option variable with either:
By.CSS_SELECTOR, 'li.select-option.select-optionRow',
By.CSS_SELECTOR, 'div.select-dropdownContainer.dswatchesMenuContainer'
Both returned no results whatsoever
How do I get the whole list of URLs in the drop down box? Where am I doing wrong?