I am trying to scrape certain odds for a football tournament. To this end I wrote a piece of code which first generates the exact link I want and then loads the corresponding page. The problem is, the page loaded is not the exact page but a ‘neighbor’ (see below). Here is the code:

from selenium import webdriver
from bs4 import BeautifulSoup
import re

driver = webdriver.Chrome()
base_url = 'https://www.oddsportal.com/football/germany/bundesliga/' # next games for bundesliga
base_site = 'https://www.oddsportal.com'
team1 = 'dortmund' 

driver.get(base_url)
soup = BeautifulSoup(driver.page_source, 'html.parser')

# match_tags contains references to dortmund's upcoming games
match_tags = soup.find_all('a', {'href':re.compile('dortmund')})
assert len(match_tags) == 2, 'Unexpected number of tags found' # we expect length 2 here

relevant_tag = match_tags[0]

# having found the relevant information, we now create the second link and try loading the page
game_href = relevant_tag['href']

game_url = base_site + game_href + "#cs;2"
# we should get this link: https://www.oddsportal.com/football/germany/bundesliga/dortmund-heidenheim-zsmd0ggn/#cs;2
assert game_url == 'https://www.oddsportal.com/football/germany/bundesliga/dortmund-heidenheim-zsmd0ggn/#cs;2', 'Unexpected link'

driver.get(game_url)
soup = BeautifulSoup(driver.page_source, 'lxml-xml')
# print(soup.prettify())

# the following command shows that the loaded page is not the correct one. For the correct one it should return a long list, however it returns an empty list upon running
print(soup.find_all("p", {"data-v-962b58dc" : True})) 

This code runs just fine. The problem is, it loads a wrong page! In fact it loads https://www.oddsportal.com/football/germany/bundesliga/dortmund-heidenheim-zsmd0ggn instead of https://www.oddsportal.com/football/germany/bundesliga/dortmund-heidenheim-zsmd0ggn/#cs;2 (notice ‘/#cs;2’ at the end missing). The above code represents a somewhat minimal example: if I directly give the driver the page https://www.oddsportal.com/football/germany/bundesliga/dortmund-heidenheim-zsmd0ggn/#cs;2 to load without previously loading https://www.oddsportal.com/football/germany/bundesliga , it gets it correctly.

Comments. I have tried using webdriver.Firefox() instead of Chrome, with the same result. The code print(driver.current_url) shows the correct link. Fun fact: running the last two command again,

driver.get(game_url)
soup = BeautifulSoup(driver.page_source, 'lxml-xml')

loads the correct page!

Edit This is to answer the question how do we know that the loaded page is not correct. The correct page has the specific structure and print(soup.find_all("p", {"data-v-962b58dc" : True})) returns a long list of tags related to specific results. The wrong page returns empty list upon executing this command (this is just one way, it’s also possible to see that the page is wrong by printing out the source).