Thiết kế website giá rẻ

Question

I am looking to scrape publicly available tables from New York’s electricity grid at this url: http://icap.nyiso.com/ucap/public/auc_view_spot_detail.do

I’m able to do so for summer seasons but not for winter seasons. It’s not clear to me what I’m missing so I’m hoping someone smarter can chime in.

Below is my process, starting with a screenshot of the page.

Circled in red above, one must select a combination of Season & Month and hit Display to generate the tables. I have copied the request header info, including the url-encoded payload that I’ve included as the body of the POST request.

# libraries
library(jsonlite) 
library(lubridate)
library(data.table)
library(httr)
library(rvest)

# get session and cookies
initial_url <- "http://icap.nyiso.com/ucap/public/auc_view_spot_detail.do"
initial_response <- GET(initial_url)
cookie_data <- cookies(initial_response)
cookie_string <- paste0(cookie_data$name, "=", cookie_data$value, collapse = "; ")


# Define the POST request headers, including cookies
headers <- c(
  "Accept" = "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7",
  "Accept-Encoding" = "gzip, deflate",
  "Accept-Language" = "en-US,en;q=0.9",
  "Cache-Control" = "max-age=0",
  "Connection" = "keep-alive",
  "Content-Length" = "85",
  "Content-Type" = "application/x-www-form-urlencoded",
  "Cookie" = cookie_string,
  "Host" = "icap.nyiso.com",
  "Origin" = "null",
  "Upgrade-Insecure-Requests" = "1",
  "User-Agent" = "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/127.0.0.0 Safari/537.36"
)

# Define the URL for the POST request
post_url <- "http://icap.nyiso.com/ucap/public/auc_view_spot_detail.do"

# Below is working code for a "Summer" season:
response <- POST(post_url, add_headers(.headers = headers), encode = "form", 
                 body = "seasonId=702793&seasonId=Summer+2024&month=05%2F2024&month=May%2F2024&display=Display")
html_content <- content(response, as = "text")
html <- read_html(html_content)
tables <- html %>% html_nodes("table")
html_table(tables[4]) # print

#[[1]]
## A tibble: 45 × 2
#   X1                           X2            
#   <chr>                        <chr>         
# 1 "05/2024"                    "05/2024"     
# 2 "G-J Locality"               "G-J Locality"
# 3 "Awarded Deficiency (MW)"    "1,888.1"     
# 4 "Awarded Excess (MW)"        "1,694.400"   
# 5 "% Excess Above Requirement" "14.78"       
# 6 "Price ($/kW-M)"             "$4.27"       
# 7 ""                           ""            
# 8 "LI"                         "LI"          
# 9 "Awarded Deficiency (MW)"    "176.9"       
#10 "Awarded Excess (MW)"        "519.200"     
## ℹ 35 more rows
## ℹ Use `print(n = ...)` to see more rows

Strangely enough, this process does not work if I change the body for the winter season, as indicated by inspecting the network. Any idea what I might be missing?

# does not work to generate the data
response <- POST(post_url, add_headers(.headers = headers), encode = "form", 
                 body = "seasonId=702409&seasonId=Winter+2023-2024&month=02%2F2024&month=Feb%2F2024&display=Display")
html_content <- content(response, as = "text")
html <- read_html(html_content)
tables <- html %>% html_nodes("table")
html_table(tables[4]) # there is no such table

A few odd behaviors I’ve noticed:

There are duplicate body parameters, but it will not work if I remove
any one of them.
You can change the seasonId number (seasonId=702793) to any existing ID as long as the string (seasonId=Summer+2024) is correct. Other IDs are inside of here http://icap.nyiso.com/ucap/rest/seasons/public)

I could not locate a specific public rest api for the actual data in the table either.

Thanks for your time and thoughts.

And here are a bunch of body strings I’ve used to determine that this is an issue only for winter seasons:

body_strings <- c("seasonId=700085&seasonId=Winter+2021-2022&month=01%2F2022&month=Jan%2F2022&display=Display", 
"seasonId=700085&seasonId=Winter+2021-2022&month=02%2F2022&month=Feb%2F2022&display=Display", 
"seasonId=700085&seasonId=Winter+2021-2022&month=03%2F2022&month=Mar%2F2022&display=Display", 
"seasonId=700085&seasonId=Winter+2021-2022&month=04%2F2022&month=Apr%2F2022&display=Display", 
"seasonId=700490&seasonId=Summer+2022&month=05%2F2022&month=May%2F2022&display=Display", 
"seasonId=700490&seasonId=Summer+2022&month=06%2F2022&month=Jun%2F2022&display=Display", 
"seasonId=700490&seasonId=Summer+2022&month=07%2F2022&month=Jul%2F2022&display=Display", 
"seasonId=700490&seasonId=Summer+2022&month=08%2F2022&month=Aug%2F2022&display=Display", 
"seasonId=700490&seasonId=Summer+2022&month=09%2F2022&month=Sep%2F2022&display=Display", 
"seasonId=700490&seasonId=Summer+2022&month=10%2F2022&month=Oct%2F2022&display=Display", 
"seasonId=700882&seasonId=Winter+2022-2023&month=11%2F2022&month=Nov%2F2022&display=Display", 
"seasonId=700882&seasonId=Winter+2022-2023&month=12%2F2022&month=Dec%2F2022&display=Display", 
"seasonId=700882&seasonId=Winter+2022-2023&month=01%2F2023&month=Jan%2F2023&display=Display", 
"seasonId=700882&seasonId=Winter+2022-2023&month=02%2F2023&month=Feb%2F2023&display=Display", 
"seasonId=700882&seasonId=Winter+2022-2023&month=03%2F2023&month=Mar%2F2023&display=Display", 
"seasonId=700882&seasonId=Winter+2022-2023&month=04%2F2023&month=Apr%2F2023&display=Display", 
"seasonId=701280&seasonId=Summer+2023&month=05%2F2023&month=May%2F2023&display=Display", 
"seasonId=701280&seasonId=Summer+2023&month=06%2F2023&month=Jun%2F2023&display=Display", 
"seasonId=701280&seasonId=Summer+2023&month=07%2F2023&month=Jul%2F2023&display=Display", 
"seasonId=701280&seasonId=Summer+2023&month=08%2F2023&month=Aug%2F2023&display=Display", 
"seasonId=701280&seasonId=Summer+2023&month=09%2F2023&month=Sep%2F2023&display=Display", 
"seasonId=701280&seasonId=Summer+2023&month=10%2F2023&month=Oct%2F2023&display=Display", 
"seasonId=702409&seasonId=Winter+2023-2024&month=11%2F2023&month=Nov%2F2023&display=Display", 
"seasonId=702409&seasonId=Winter+2023-2024&month=12%2F2023&month=Dec%2F2023&display=Display", 
"seasonId=702409&seasonId=Winter+2023-2024&month=01%2F2024&month=Jan%2F2024&display=Display", 
"seasonId=702409&seasonId=Winter+2023-2024&month=02%2F2024&month=Feb%2F2024&display=Display", 
"seasonId=702409&seasonId=Winter+2023-2024&month=03%2F2024&month=Mar%2F2024&display=Display", 
"seasonId=702409&seasonId=Winter+2023-2024&month=04%2F2024&month=Apr%2F2024&display=Display", 
"seasonId=702793&seasonId=Summer+2024&month=05%2F2024&month=May%2F2024&display=Display", 
"seasonId=702793&seasonId=Summer+2024&month=06%2F2024&month=Jun%2F2024&display=Display", 
"seasonId=702793&seasonId=Summer+2024&month=07%2F2024&month=Jul%2F2024&display=Display", 
"seasonId=702793&seasonId=Summer+2024&month=08%2F2024&month=Aug%2F2024&display=Display"
)

Thiết kế website giá rẻ

Danh mục

Identifying why scraping a website only works for POST request body strings and not others