I am new to Python and trying to web scrape news article from the following webpage: https://www.dr.dk/soeg?query=B%C3%A6redygtighed&sort=Relevance
I am using Python to scrape a Danish news platform for words relating to biodiversity. I want to print all titles relating to biodiversity and then use the BERT model to classify each of the selected biodiversity sentences to determine whether it expresses a positive or negative sentiment. Sentences with positive sentiment get assigned a score of “+1”, negative sentences get as- signed a score of “-1”, and neutral sentences get assigned a score of “0”. I want to get a CSV file printed with all articles and the sentence scores, as well as titles and dates for each of these articles. I plan on compiling the data in STATA.

I think the problem is, that there is a show more button (“+VIS FLERE”) where the articles after the show more button aren’t captured in my Python print. There are in total 3.856 articles, but my code only captures the first 10. Furthermore, I want my code to go into each article (with a new URL) and use the BERT model as mentioned before so capture my wished result.

Does anyone have any experience with doing anything similar and can help?

My code is below.

from selenium import webdriver
from bs4 import BeautifulSoup
import time

driver = webdriver.Chrome()

url = "https://www.dr.dk/soeg?query=biodiversitet"
driver.get(url)

time.sleep(3) 

page_source = driver.page_source

soup = BeautifulSoup(page_source, "html.parser")

div = soup.find("div", {"class": "dre-teaser-list"})  

headlines = div.find_all("a", class_="dre-teaser-title")

for headline in headlines:
    title_text = headline.get_text(strip=True)
    article_url = "https://www.dr.dk" + headline.get('href')
    print(f"Title: {title_text}, URL: {article_url}")

driver.quit()