I’m scaping booking.com with selenium, it has over 600 results, but selenium is scaping only 51 results. I do not know what I am doing wrong. This is my code.
from selenium import webdriver
from selenium.webdriver.common.by import By
import pandas as pd
url = 'https://www.booking.com/searchresults.html?aid=397645&label=bin859jc-1DCAEoggI46AdIM1gDaKcBiAEBmAExuAEXyAEM2AED6AEB-AECiAIBqAIDuAK-xqeyBsACAdICJGEwYTAzM2QwLTQ1MjEtNGIzNi04NTYyLTcxMjY5NzJlMGNjNtgCBOACAQ&sid=968beab70293559758c39b5f6e59b26e&checkin=2024-05-20&checkout=2024-05-23®ion=5140&'
driver = webdriver.Edge()
driver.get(url)
name = []
location = []
container = driver.find_element(By.CLASS_NAME, "d4924c9e74")
apartment = container.find_elements(By.XPATH, './/div[contains(@role, "group")]')
for x in apartment:
name.append(x.find_element(By.XPATH, './/a[contains(@data-testid, "title")]').text)
location.append(x.find_element(By.XPATH, './/span[contains(@data-testid, "address")]').text)
# print(x.text)
hotels = {
'name' : name,
'location' : location
}
print(hotels)
This is my result
{'name': ['The Mora ZanzibarnOpens in new window', 'ZanzibluenOpens in new window', 'Maisha NungwinOpens in new window', 'Baladin Zanzibar Beach HotelnOpens in new window', 'Be Zanzibar Boutique HotelnOpens in new window', 'Promised Land LodgenOpens in new window', 'Kwanza Resort by SUNRISEnOpens in new window', 'Karafuu Beach Resort & SpanOpens in new window', 'Hekaya ZanzibarnOpens in new window', 'Paradise Beach Resort & SpanOpens in new window', 'Zuri ZanzibarnOpens in new window', 'Tembo House HotelnOpens in new window', 'Sunset Kendwa Beach HotelnOpens in new window', 'Villa Nyota ZanzibarnOpens in new window', 'KizikulanOpens in new window', 'Hotel Riu Jambo - All InclusivenOpens in new window', 'Chumbe Island Coral ParknOpens in new window', 'Gold Zanzibar Beach House & SpanOpens in new window', 'Mtende Beach Bungalow océan viewnOpens in new window', 'Le Mersenne Zanzibar, Autograph CollectionnOpens in new window', 'Pongwe Bay ResortnOpens in new window', 'Aurelia ZanzibarnOpens in new window', 'Aya Beach ResortnOpens in new window', 'Z-Lodge ZanzibarnOpens in new window', 'Mayai Ocean ResortnOpens in new window'], 'location': ['Matemwe', 'Matemwe', 'Nungwi Beach, Nungwi', 'Michamvi', 'Mfumbwi', 'Kizimkazi Beach, Kizimkazi', 'Kizimkazi', 'Pingwe', 'Pingwe', 'Uroa', 'Kendwa Beach, Kendwa', 'Zanzibar City', 'Kendwa Beach, Kendwa', 'Kidenga', 'Kizimkazi', 'Nungwi', 'Mbweni', 'Kendwa Beach, Kendwa', 'Mtende', 'Michamvi', 'Pongwe', 'Ngambo', 'Kizimkazi Beach, Kizimkazi', 'Kiwengwa', 'Bwejuu Beach, Bwejuu']}
0
the results load dynamically as you scroll. You need to use selenium to scroll down the page to load all the results.
See here for example