I’m new to Beautful Soup and have a little experience with Python. I’m using BS for web-scraping, specifically reviews of books on Goodreads. When I use this code, it only gives me the first 20 reviews, never anything beyond that. The page loads at 30 reviews when it’s first opened (not counting the “show more reviews” button)

This is my code –

import requests
from bs4 import BeautifulSoup


url = "https://www.goodreads.com/book/show/26114463/reviews?reviewFilters={%22workId%22:%22kca://work/amzn1.gr.work.v1.lSjSf2r-G6D1advk_kunEg%22,%22after%22:%22NjAwMDEsMTU4Njg4NDczNzY1MQ%22}"

html_text = requests.get(url).text

soup = BeautifulSoup(html_text, 'html.parser')

print(soup)