I’ve been having trouble with web scraping or sending requests to some dynamic pages for a while now, and it’s because of Virtual-Dom or some loading or human verifications and delay before the page loads completely.

In this case, what I need is to post some data to an URL, that has a pre loading to prevent such this requests.

First of all I used selenium to open the page for pass loading and access the real page content, After that I started collecting some special and random values in page such as token or hash.
Now I have access to values and cookies and all I need to do is Post my payload to URL, but the problem is the page has this delay and loading and every time I’m trying to post data I’ll take the page html code not my desired response.

So the first question is How can we wait on a request to pass this loading and after that post some data?

My second question is does selenium web driver has post method to do this?

import requests
from bs4 import BeautifulSoup
from selenium import webdriver
from selenium.webdriver.chrome.options import Options

session = request.Session()

driver = webdriver.Chrome(options=chrome_options)

reservation_url = 'https://saderat.exchange/reservation'
driver.get(reservation_url)

# this method checks the existence of an element to confirm that the page was fully loaded.
TAW(driver, 'xpath', '//button[contains(text(), "Send")]') 

# Get page source
page = driver.page_source
soup = BeautifulSoup(page, 'html.parser')

# Find values
_token = soup.find('input', attrs={'name': '_token'})['value']
_captcha_data_sitekey = soup.find('div', class_='g-recaptcha')['data-sitekey']

while True:
    __res = driver.execute_script('return grecaptcha.getResponse()')
    if __res != '':
        _captcha = __res
        print("Captcha data:", _captcha)
        break

payload = {
        '_token': _token,
        'fullname': 'jacky chan',
        'mobile': '09393193939',
        'checkbox': 'on',
        'g-recaptcha-response': _captcha
    }

cookies = driver.get_cookies()
for cookie in cookies:
    session.cookies.set(cookie['name'], cookie['value'])

response = session.post(url=reservation_url, data=payload)