I am trying to scrape a Hertz Rental Car API (URL below) to get pricing options for a specific itinerary. This is not on the disallow list on their robots.txt. I captured the payload from my browser stemming from the initial Hertz (Hertz.com) page populating the trip details (location, dates, pickup times) that is submitted to the request URL below. When I then hit the API below with my payload, headers (token and user agent), cookies, and URL (copied from my network tab), I get a 200 response, but never get my anticipated response as I would expect to see based on what I see in my browser network tab when performing in my browser. I tried making my headers just my token and not having cookies, but that gave me an Incapsula issue. I believe this is a cookie issue based on the response below, and I think my method of taking my Hertz.com response to set my cookie is not working. Need guidance/help.

Request URL: “https://www.hertz.com/rentacar/rest/hertz/v2/itinerary/vehicles”

Code:

import requests
import time

#URL 1 utilized to hit Hertz page and ideally return cookies to do API query with URL 2
url1="https://www.hertz.com"

url2="https://www.hertz.com/rentacar/rest/hertz/v2/itinerary/vehicles"

headers={"dtm_token":"HereIsAfakeTokenforAnExample",
          'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/125.0.0.0 Safari/537.36'}

#Payload copied from browser when performing the request from Hertz's homepage with trip details (location, dates, times)
#This payload is what is submitted by your browser when making a request to book a rental via URL2
params={'lastName': '', 'resSearch': False, 'showRentalAgreement': False, 'showEvRentalAgreement': False, 'goldAnytimeRes': False,
        'checkLIS': False, 'checkFPO': False, 'buttonLIS': False, 'buttonFPO': False, 'showBothElements': False,
        'cdpVerificationFailed': 0, 'forceResHomePage': '', 'href': '/rentacar/rest/home/form', 'confirmationNumber': '',
        'arrivingUpdate': '', 'defaultTab': '', 'militaryClock': 0, 'majorAirport': '', 'returnAtDifferentLocationCheckbox': '',
        'dropoffLocation': '', 'inpPickupAutoFill': 'Y', 'inpPickupStateCode': 'VA', 'inpPickupCountryCode': '', 'inpPickupSearchType': '3',
        'inpDropoffAutoFill': '', 'inpDropoffStateCode': '', 'inpDropoffCountryCode': '', 'inpDropoffSearchType': '', 'pickupHiddenEOAG': 'IADT26',
        'dropoffHiddenEOAG': '', 'memberOtherCdpField': '', 'cdpField': '', 'corporateRate': '', 'officialTravel': '',
        'pcNumber': '', 'typeInRateQuote': 'BEST', 'cvNumber': '', 'itNumber': '', 'originalRqCheckBox': '', 'checkDiscount': '',
        'affiliateMemberJoin': '', 'affiliateMemberID': '', 'affiliateCallCount': 0, 'hertzlinkActive': False, 'companyId': '',
        'pickupDay': '05/22/2024', 'pickupTime': '12:00', 'dropoffDay': '05/23/2024', 'dropoffTime': '12:00', 'no1ClubNumber': '',
        'selectedCarType': 'ACAR', 'ageSelector': '25', 'redeemPoints': '', 'fromLocationSearch': False,
        'recommendationBrowserInfo': {'appCodeName': 'Mozilla', 'appName': 'Netscape', 'appVersion':
                                      '5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36',
                                      'cookieEnabled': True, 'language': 'en-US', 'onLine': True, 'platform': 'MacIntel', 'product': 'Gecko',
                                      'userAgent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/124.0.0.0 Safari/537.36', 'visitorId': ''},
        'pickupLocation': 'Washington Dulles International Airport', 'inpPickupIsServedBy': 'N', 'inpPickupHasSpecialInstruction': 'N',
        'pickupDayStandard': '2024/05/22', 'dropoffDayStandard': '2024/05/23', 'inpPickupOptGoldAnytime': '', 'memberSelectedCdp': '', 'cdpRadioButton': ''}

response1=requests.get(url1, headers=headers)

#introducing sleep to make this less machine-like
time.sleep(2)

#used for debugging
print(response1.status_code)
print(response1.cookies)

#API request to get rental options and pricing
response2=requests.get(url2, headers=headers, params=params, cookies=response1.cookies)

#used for debugging
print(response2.status_code)
print(response2.text)
print(response2.cookies)

Terminal Results:
200
<RequestsCookieJar[]>
200
See response below for snippet of 85-line response
<RequestsCookieJar[]>

Response:

As you were browsing something about your browser made us think you were a bot. There are a few reasons this might happen:

• You’ve disabled JavaScript in your web browser.
• You’re a power user moving through this website with super-human speed.
• You’ve disabled cookies in your web browser.
• A third-party browser plugin, such as Ghostery or NoScript, is preventing JavaScript from running. Additional information is available in this support article.

To regain access, please make sure that cookies and JavaScript are enabled before reloading the page.