I wrote a script in which I defined a ‘next page’ function which scrolls to the bottom of the page, identifies the ‘next’ button and then clicks the button to navigate to the next page. See the following excerpt from my script:

`def next_page():
# Scroll to bottom of page in order for ‘Next’ button to be visible
browser.execute_script(“window.scrollTo(0, document.body.scrollHeight);”)
sleep(1)

try:
    next_button = browser.find_element(By.CLASS_NAME, 'bbsPager_next')
    next_button.click()

except NoSuchElementException:
    print("Next button not found. End of pagination or element not loaded yet.")
    return False

return True`

Later in my script, I call this function as part of a while True loop, but it keeps throwing the NoSuchElement Exception. The odd thing is I know the element exists, and in fact, I am able to successfully navigate to the next page when I manually call the next_page() function. So I know the function works. I just can’t figure out why it is not working when integrated as part of the overall script.

My overall goal was to create a web scraper using Python Selenium that extracts relevant data fields from the current page of the target website, then automatically navigates to the next page and repeats the same process until it has extracted data from all of the pages available. The issue I am having, as discussed above, is that my ‘while True’ loop keeps breaking as a result of the NoSuchElement exception being thrown while executing the next_page() function. The following is an excerpt of the while True loop I am referring to:

while True: extract_page_data() navigation_successful = next_page() if not navigation_successful: print('No more pages left. Exiting...') break

I expected it to extract data from the page, navigate the next page and repeat the process all over again until there are no more pages left. However, the output I receive is as follows:

“Next button not found. End of pagination or element not loaded yet.
No more pages left. Exiting…”

Any assistance would be greatly appreciated! Thank you!!