I have been trying numerous ways but this website is proving very hard to scrape via bs4. Here’s the URL: https://www.nseindia.com/option-chain

I am trying to extract the option value found in the snip below on one of the matches. The aim is to extract all “option” values from the dropdown menu into a list.
enter image description here

I tried using bs4 and I was able to get the parent (that has the id=”select_symbol”) of the all “option” tags I’m looking for, but when digging into what I have fetched, the result I got was an empty list.
Here’s the code that I have set up for this task:

import requests
from bs4 import BeautifulSoup
url = 'https://www.nseindia.com/option-chain'
headers = {
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/119.0.0.0 Safari/537.36"
}

# Send a GET request to the URL

r = requests.get(url, headers=headers)

soup = BeautifulSoup(r.content, 'lxml')

select_ = soup.find("select", {"id":"select_symbol"})

option_content = soup.find("select", {"id":"select_symbol"}).find_all("option")`