New to web scraping so forgive the basic question, but I’m trying to scrape film URLs from lists on Letterboxd and having some issues. Using this list as an example, I was able to find the link location in the HTML here:
However, I can’t actually get the link out of this. I’ve tried two methods so far. First I tried grabbing all of the link elements, hoping to then filter out the ones I didn’t need:
library(rvest)
link <- 'https://letterboxd.com/horrorville/list/horrorville-community-80s-video-store-horror/'
page <- read_html(link)
page %>%
html_elements('a') %>%
html_attr('href')
This did return URLs, but none of them were actually for the films from the list. I then tried selecting based on the class name. I’m not very familiar with HTML, but my understanding is that class="frame has-menu"
indicates two separate classes frame
and has-menu
which can be combined in rvest by separating them with periods, so I tried this:
page %>%
html_elements('.frame.has-menu') %>%
html_attr('href')
That didn’t return anything at all.
I saw this other question which sounded similar, so I tried examining the Network tab of my browser’s (Firefox) devtools as the responder suggested. I wasn’t quite sure what to make of it, but it looked like the requests relating to the films were using GET requests, whereas in the other question the responder said the issue is that rvest can’t handle POST requests. What is my issue here?