As of yesterday, my Camply API calls to recreation.gov are blocked.

Thanks for reporting <a class="user-mention notranslate" data-hovercard-type="user" da

Ahh, I have more info! This is a problem with the <a href="https://ridb.recreation.gov

<a class="user-mention notranslate" data-hovercard-type="user" data-hovercard-url="/us

API calls blocked about camply HOT 7 CLOSED

juftin commented on July 17, 2024

API calls blocked

from camply.

Comments (7)

rboarman commented on July 17, 2024 1

Thank you! I'll send you a little gift. Love your library!

from camply.

juftin commented on July 17, 2024

Thanks for reporting @rboarman. I'll take a look into this - we're currently rotating between a few User-Agent headers:

camply/camply/config/api_config.py

Lines 18 to 31 in 62d1a44

 USER_AGENTS: List[dict] = [ 

 {"User-Agent": ("Mozilla/5.0 (X11; Linux x86_64; rv:10.0) " 

 "Gecko/20100101 Firefox/10.0")}, 

 {"User-Agent": ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 " 

 "(KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36")}, 

 {"User-Agent": ("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 " 

 "(KHTML, like Gecko) Version/14.0.3 Safari/605.1.15")}, 

 {"User-Agent": ("Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_3) AppleWebKit/537.36 " 

 "(KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36")}, 

 {"User-Agent": ("Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) " 

 "Gecko/20100101 Firefox/87.0")}, 

 {"User-Agent": ("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 " 

 "(KHTML, like Gecko) Chrome/89.0.4389.128 Safari/537.36")} 

 ]

Hopefully this is something we'll be able to get around.

from camply.

juftin commented on July 17, 2024

Ahh, I have more info! This is a problem with the RIDB Database API.

❯ export LOG_LEVEL=debug
❯ camply campsites \
    --rec-area 2725 \
    --start-date 2022-07-12 \
    --end-date 2022-07-13 \
    --continuous \
    --notifications pushover \
    --notify-first-try
2022-01-13 10:47:50,836 [  CAMPLY]: camply, the campsite finder ⛺️
2022-01-13 10:47:50,842 [    INFO]: 1 booking nights selected for search, ranging from 2022-07-12 to 2022-07-12
2022-01-13 10:47:50,844 [    INFO]: Retrieving Facility Information for Recreation Area ID: `2725`.
2022-01-13 10:47:50,852 [   DEBUG]: Starting new HTTPS connection (1): ridb.recreation.gov:443
2022-01-13 10:47:50,908 [   DEBUG]: https://ridb.recreation.gov:443 "GET /api/v1/recareas/2725/facilities?full=true&offset=0 HTTP/1.1" 403 919
2022-01-13 10:47:50,910 [    INFO]: {'accept': 'application/json', 'apikey': 'XXXXXXXXXXXXXXXXXXX'}
2022-01-13 10:47:50,910 [   ERROR]: Receiving bad data from Recreation.gov API: <!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<HTML><HEAD><META HTTP-EQUIV="Content-Type" CONTENT="text/html; charset=iso-8859-1">
<TITLE>ERROR: The request could not be satisfied</TITLE>
</HEAD><BODY>
<H1>403 ERROR</H1>
<H2>The request could not be satisfied.</H2>
<HR noshade size="1px">
Request blocked.
We can't connect to the server for this app or website at this time. There might be too much traffic or a configuration error. Try again later, or contact the app or website owner.
<BR clear="all">
If you provide content to customers through CloudFront, you can find steps to troubleshoot and help prevent this error by reviewing the CloudFront documentation.
<BR clear="all">
<HR noshade size="1px">
<PRE>
Generated by cloudfront (CloudFront)
Request ID: uWqHopzPrg1JtSkQPWOfThlGmwJIVPqmmAfRNN9oZMniNvv9v-rxrg==
</PRE>
<ADDRESS>
</ADDRESS>
</BODY></HTML>

camply uses two APIs to talk to recreation.gov:

https://ridb.recreation.gov
- This is the metadata Database / REST API for Recreation.gov - we get campsite, campground, and recreation area metadata from this API
- camply uses an API Key embedded into the software to get this data. You can override this with your RIDB_API_KEY environment variable
https://recreation.gov/api/camps/availability/campground/
- This is the undocumented Campsite Availability API that camply uses
- There is no authorization on this API - this is where we randomly select a User-Agent header to make requests

The strangest thing about this issue is the RIDB Database API Landing Page is displaying the following warning:

Due to high volume of API calls, cache time for RIBD’s camping-availability endpoint will increase from 15 seconds to 1 minute moving forward. Please reach out to the Contact us center for any issues in regards to API disconnecton or downtime.

My understanding though is that the camping-availability endpoint doesn't live on the RIDB 🤷‍♂️. I'll keep digging and see what I can find and follow up here

from camply.

rboarman commented on July 17, 2024

Thank you. My calls were hitting every five minutes which should be fine right?

One thing to consider, is to allow us to pass in an array of user agents via the .camply file. That might help.

Thank you for the assistance!!

from camply.

juftin commented on July 17, 2024

@rboarman you were absolutely right. The RIDB API is kicking back the User-Agent passed from the Python Requests Package - they haven't documented this anywhere but that looks to be the problem. It's probably just a matter of time until it invalidates the attached RIDB API Key as well -- I'll look into having it choose a random API key from a few options soon.

This PR (#45) adds a ton of User-Agents between Chrome, FireFox, and Safari and implements the RIDB API to randomly choose one for each API call.

from camply.

juftin commented on July 17, 2024

I closed this out - tests are passing on my end and the fix has ben released to PyPi and Docker. A simple pip install --upgrade camply should fix everything on your end (once this workflow has finished). Please let me know or re-open this issue if you have any problems. Thanks for reporting @rboarman

from camply.

juftin commented on July 17, 2024

Thanks - so happy you like it. I've tried to build camply to be respectful of the APIs and services it uses - this includes rate limiting and exponential backoff. However this is the first I've seen of recreation.gov pushing back and trying to crack down on tools like camply hitting their API - I wouldn't be surprised to see more bugs like this coming out in the future.

from camply.

API calls blocked about camply HOT 7 CLOSED

Comments (7)

Related Issues (20)

Recommend Projects

React

Vue.js

Typescript

TensorFlow

Django

Laravel

D3

Recommend Topics

javascript

web

server

Machine learning

Visualization

Game

Recommend Org

Facebook

Microsoft

Google

Alibaba

D3

Tencent

	USER_AGENTS: List[dict] = [
	{"User-Agent": ("Mozilla/5.0 (X11; Linux x86_64; rv:10.0) "
	"Gecko/20100101 Firefox/10.0")},
	{"User-Agent": ("Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
	"(KHTML, like Gecko) Chrome/89.0.4389.90 Safari/537.36")},
	{"User-Agent": ("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_6) AppleWebKit/605.1.15 "
	"(KHTML, like Gecko) Version/14.0.3 Safari/605.1.15")},
	{"User-Agent": ("Mozilla/5.0 (Macintosh; Intel Mac OS X 11_2_3) AppleWebKit/537.36 "
	"(KHTML, like Gecko) Chrome/89.0.4389.114 Safari/537.36")},
	{"User-Agent": ("Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:87.0) "
	"Gecko/20100101 Firefox/87.0")},
	{"User-Agent": ("Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 "
	"(KHTML, like Gecko) Chrome/89.0.4389.128 Safari/537.36")}
	]