I am trying to scrape a website that has APIs. One of the API gives json only if we are logged in.
I got logged in cookies and then send requests, I am successfully sending the request.
But the problem is that, those cookies expire after some time and I have to get new cookies to send the request.
Is there a way, that I can get fresh cookies automatically before scraping the website?
Note: I read somewhere that we can login manually using selenium and save cookies. These cookies are then used for scraping. But I do not know how to get cookies from selenium and use them for request library.
Can anyone help me out?
Link: https://www.arbeitsagentur.de/bewerberboerse/
I used requests library to login but instead of giving 302 response code, it either gave me 200 or 400.
Then I tried to use selenium but the structure of website is very complicated. There are too many redirections and elements are not being fetched successfully.
1
Hm, you can use selenium to log in and then get cookies from selenium, by:
cookies = driver.get_cookies()
then return them and use them in your request, for example:
def convert_cookies_to_requests(cookies):
session_cookies = {}
for cookie in cookies:
session_cookies[cookie['name']] = cookie['value']
return session_cookies