I am generally familiar with rvest. I know the difference between html_elements() and html_element(). But I can’t get my head around this problem:
Suppose that we have data like the one that is on this webpage. The data is in a hierarchical format and each header has a different number of subheadings.
When I try to scrape, I get 177 headers. But, the subheadings are actually 270. I want to extract the data into a tidy format. But with different vector sizes, I can’t easily combine them into a tibble.
Here is my code with some comments about the results:
page <- read_html("https://postdocs.stanford.edu/about/department-postdoc-admins")
person_departments <- page %>%
html_elements(".item-list") %>%
html_element("h3") %>%
html_text2()
# The above code returns
person_names <- page %>%
html_elements(".item-list li") %>%
html_element("h4") %>%
html_text2()
# This one returns 270 names (some departments have more than 1 admin)
# Using the above codes, I can't get a nice table with two columns, one for the name and one for the person's department.
Here’s somewhat different approach, start with .item-list > ul > li elements, from there you can directly extract name, phone and email. If any of those elements are not available in current li element, you conveniently get an NA value in resulting frame (e.g for Vacant positions and for cases where only Email is listed).
For h4 switch to xpath as it allows to search for ancestors, this way you can get matching h4 for every li in the element list.
library(dplyr, warn.conflicts = FALSE)
library(rvest)
library(purrr)
page <- read_html("https://postdocs.stanford.edu/about/department-postdoc-admins")
page |>
html_elements(".item-list > ul > li") |>
map((li)
list(
h3 = html_element(li, xpath = "./ancestor::div[@class='item-list']/h3"),
h4 = html_element(li, "h4"),
phone = html_element(li, ".views-field-field-s-person-phone-display"),
email = html_element(li, ".views-field-field-s-person-email > a")
) |> map(html_text, trim = TRUE)
) |>
bind_rows() |>
# just to keep complete set of contact details from appearing in a SO post
mutate(across(h4:email, (x) stringr::str_trunc(x, 13, side = "center")))
#> # A tibble: 270 × 4
#> h3 h4 phone email
#> <chr> <chr> <chr> <chr>
#> 1 Advanced Residency Training at Stanford Sofia...zales Phone...-9139 sofia...…
#> 2 Aeronautics and Astronautics Jenny Scholes Phone...-5967 jscho...…
#> 3 African & African-Amer Studies Ashan...hnson Phone...-3969 ashan...…
#> 4 Anesthes, Periop & Pain Med Natal...brera Phone...-0648 ndarl...…
#> 5 Anesthes, Periop & Pain Med Ashle...hnson Phone...-7212 ashle...…
#> 6 Anesthes, Periop & Pain Med Jessi...tinez Phone...-8189 jimen...…
#> 7 Anthropology Julia...itler Phone...-0800 js259...…
#> 8 Archaeology Bilge...dogan Phone...-5731 bilge...…
#> 9 Bill Lane Center for the American West Vacant <NA> <NA>
#> 10 Biochemistry Dan Carino Phone...-6161 dpcar...…
#> # ℹ 260 more rows
In a way it’s a matter of style preference (well.. performance too), but we can also use those same building blocks without map() / lapply() iteration. Using singular html_element() in following context might not sound too intuitive, but there’s this bit in doc:
html_element()returns a nodeset the same length as the input
Must say I personally managed to ignore this until today..
Anyway, we can get identical result to previous example with:
li <- html_elements(page, ".item-list > ul > li")
tibble(
h3 = html_element(li, xpath = "./ancestor::div[@class='item-list']/h3") |> html_text(trim = TRUE),
h4 = html_element(li, "h4") |> html_text(trim = TRUE),
phone = html_element(li, ".views-field-field-s-person-phone-display") |> html_text(trim = TRUE),
email = html_element(li, ".views-field-field-s-person-email > a") |> html_text(trim = TRUE)
)
Becnhmark (note seconds vs milliseconds):
library(dplyr)
library(rvest)
library(purrr)
page <- read_html("https://postdocs.stanford.edu/about/department-postdoc-admins")
bm <- bench::mark(
map_li = page |>
html_elements(".item-list > ul > li") |>
map((li)
list(
h3 = html_element(li, xpath = "./ancestor::div[@class='item-list']/h3"),
h4 = html_element(li, "h4"),
phone = html_element(li, ".views-field-field-s-person-phone-display"),
email = html_element(li, ".views-field-field-s-person-email > a")
) |> map(html_text, trim = TRUE)
) |>
bind_rows(),
vec_li = {
li <- html_elements(page, ".item-list > ul > li")
tibble(
h3 = html_element(li, xpath = "./ancestor::div[@class='item-list']/h3") |> html_text(trim = TRUE),
h4 = html_element(li, "h4") |> html_text(trim = TRUE),
phone = html_element(li, ".views-field-field-s-person-phone-display") |> html_text(trim = TRUE),
email = html_element(li, ".views-field-field-s-person-email > a") |> html_text(trim = TRUE)
)
}, iterations = 10
)
bm
#> # A tibble: 2 × 6
#> expression min median `itr/sec` mem_alloc `gc/sec`
#> <bch:expr> <bch:tm> <bch:tm> <dbl> <bch:byt> <dbl>
#> 1 map_li 1.16s 1.19s 0.842 16.5MB 8.50
#> 2 vec_li 33.63ms 34.95ms 27.5 227.2KB 5.50
ggplot2::autoplot(bm)
2
The main trick is to get the children of the "item-list" elements. Then it’s a matter of processing into a table. Each department is an odd numbered list element, the persons are in the even numbered ones. Care must be taken with vacant department positions.
library(rvest)
link <- "https://postdocs.stanford.edu/about/department-postdoc-admins"
page <- read_html(link)
person_table <- page %>%
html_elements(".item-list") %>%
html_children() |>
html_text2() |>
strsplit("nPhone:|n|Email:") |>
lapply(trimws)
i_odd <- c(TRUE, FALSE)
# get how many persons in each dept
r <- lengths(person_table[!i_odd]) %/% 3L
# if "Vacant", adjust to 1 person (named "Vacant", see below)
r[r == 0L] <- 1L
Department <- person_table[i_odd] |> unlist() |> rep(r)
Person <- lapply(person_table[!i_odd], (x) {
# these are the vacant persons
if( length(x) == 1L) {
matrix(c(x, "", ""), nrow = 1L)
} else {
matrix(x, ncol = 3L, byrow = TRUE)
}
}) |> do.call(rbind, args = _)
result <- cbind(Department, Person) |>
as.data.frame() |>
setNames(c("Department", "Name", "Phone", "Email"))
str(result)
#> 'data.frame': 270 obs. of 4 variables:
#> $ Department: chr "Advanced Residency Training at Stanford" "Aeronautics and Astronautics" "African & African-Amer Studies" "Anesthes, Periop & Pain Med" ...
#> $ Name : chr "Sofia Gonzales" "Jenny Scholes" "Ashante Johnson" "Natalie Darling-Cabrera" ...
#> $ Phone : chr "(650) 724-9139" "(510) 468-5967" "(650) 721-3969" "(650) 497-0648" ...
#> $ Email : chr "[email protected]" "[email protected]" "[email protected]" "[email protected]" ...
head(result)
#> Department Name
#> 1 Advanced Residency Training at Stanford Sofia Gonzales
#> 2 Aeronautics and Astronautics Jenny Scholes
#> 3 African & African-Amer Studies Ashante Johnson
#> 4 Anesthes, Periop & Pain Med Natalie Darling-Cabrera
#> 5 Anesthes, Periop & Pain Med Ashley Johnson
#> 6 Anesthes, Periop & Pain Med Jessica Martinez
#> Phone Email
#> 1 (650) 724-9139 [email protected]
#> 2 (510) 468-5967 [email protected]
#> 3 (650) 721-3969 [email protected]
#> 4 (650) 497-0648 [email protected]
#> 5 (650) 721-7212 [email protected]
#> 6 (650) 497-8189 [email protected]
Created on 2024-09-11 with reprex v2.1.0
2