The autotrader_scrape from ghardin1314

autotrader_scrape's People

Contributors

Stargazers

Watchers

autotrader_scrape's Issues

How to successfully make request

I absolutely love what you're building here!

I'm having trouble actually making the request and was wondering if you had any suggestions? I can enter the URL in my browser just fine and get the JSON payload back, but when I try to request it in a very simplified format like the following snipped out of get_results.py _get_page:

make_code = 'AUDI'
model_code = 'S4'
firstRecord = 1


url_base = 'https://www.autotrader.com'
path = '/rest/searchresults/base'
url = url_base + path
session = requests.Session()
payload = {'makeCodeList': make_code,
            'searchRadius': '0',
            'listingTypes': 'USED',
            'modelCodeList': model_code,
            'zip': '90250',
            'startYear': '2010',
            'sortBy': 'distanceASC',
            'numRecords': '100',
            'firstRecord': firstRecord,
            }
headers= {'User-Agent': """Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/70.0.3538.77 Safari/537.36"""}

print(f'Requesting {url}')

response = session.get(url, headers=headers, params=payload)

print(f'response: {response}')

, it will either time out, give a 504, or return me error in the HTML saying the page is unavailable.

Are you using some sort of proxy, cookies, other headers or something?

Recommend Projects