rmjacobson / recgov_daemon Goto Github PK

Might want to use this for multiple people concurrently. I will always be the one running it, but it would be nice if I could attach different campgrounds/RIDB searches to different emails/phone numbers so those people can be directly notified.

Currently using method described here with ssl create_default context: https://realpython.com/python-send-email/#option-1-using-smtp_ssl

Unfortunately, this requires setting the google account to allow "less secure apps" which isn't ideal. Doing it this way is much easier and preferred for dev, but we should move to actual google OAuth when convenient.

Refs for OAuth: https://developers.google.com/gmail/api/quickstart/python

Beta-tester requested text-notification rather than email notification because they check their email infrequently compared to texts on their phone, and time is of the essence for this project.

This is theoretically possible, but I've never done this before and it may not be free.

Some references:

• https://www.fullstackpython.com/blog/send-sms-text-messages-python.html -- older, but the Twilo service looks like it still exists
• https://gist.github.com/alexle/1294495/39d13f2d4a004a4620c8630d1412738022a4058f -- looks like a possible free option, but would require knowing the carrier of the number we're trying to reach
• https://automatetheboringstuff.com/chapter16/ -- kind of the same thing as the second point above; still need to know which carrier to send it to

README is currently a stub, there are no install instructions to speak of.

• README
  • include refs as necessary!
• install
  • requirements.py
  • environment variables
  • create logs/ directory at the top level of the repo for rolling logs

Python 3 logging across modules/files is a mess and currently doesn't work in this repo. Going to want to actually have logging eventually though because this is a daemon that will run indefinitely and if it errors I want to know exactly where and why/what it was doing when it errored.

Reference Links

• https://www.pylenin.com/blogs/python-logging-guide/
• https://python101.pythonlibrary.org/chapter15_logging.html
• https://stefan.sofa-rockers.org/2020/10/28/raise-from/ -- this is an error handling thing, but is kind of in the same vein

Currently need to ask the user their carrier to make text notifications because we're using the email portal. This is fine, but ideally we'd just be able to automagically find the carrier just with the phone number. See services like twillo or https://freecarrierlookup.com or "textmagic" that might be able to do this with an API.

Some campgrounds on RIDB don't conform to the expected format, usually because that specific campground can't be booked through RIDB. That's fine, but there's no point in continuing to check them if we're only going to get an error.

TODO:

• identify places in the check that have relevant errors we don't want
• implement an error counter for each campground object
• if campground goes over some error threshold, remove it from the list to check

If we want to have multi-user/email support and multi-date range support, we're going to need a better way of inputting that information than the CLI. Right now, it seems like a config/input file is probably the way to go, but formatting might be a mess. JSON is always a good choice, and is already used in this project, but open to other solutions.

Currently we run through all the campgrounds we're searching one by one, then wait for 300s (5 minutes) and then start again. This is fine right now, but depending on how many campgrounds the user enters it can take a long time (i.e. longer than 5 minutes) to check every campground in the loop, which makes waiting 5 minutes afterward kind of pointless.

2 ways to solve this:

• easy way! We check start/end times between the beginning of the loop and the end of the loop through all campgrounds; if diff(start,end) < 5 minutes we sleep the remainder; else we just start the next loop immediately
• medium way! we keep track of a timer for each campground individually because sometimes the site responds faster than others; if we get to a campground and it hasn't been 5 minutes since we last checked it, wait the remainder; else start immediately
  • in theory this would amortize the cost for each wait and ensure that no campground individually is being spammed with requests
• hard way! We abandon the continuous loop and start multithreading. Most of the processing time in this app is waiting for selenium to obtain data for pages, so in theory this would be a good candidate for Python multithreading since we could interleave reading the pages with the loading time for new pages.
  • starting loop would spawn threads for each campground rather than check each one repeatedly
  • each thread keeps track of a timer so that each campground isn't checked more often than once per 5 minutes
  • if threads are started with an offset (e.g. ~10-30 seconds?) we should avoid overloading the recgov site and get us throttled
  • decent resource to get started if we want to go this way: https://testdriven.io/blog/building-a-concurrent-web-scraper-with-python-and-selenium/

Getting the following error: selenium.common.exceptions.NoSuchElementException: Message: no such element: Unable to locate element: {"method":"css selector","selector":".rec-table-overlay"} (Session info: headless chrome=88.0.4324.187)

This is probably just something that happens because either the page has nothing to reload or it happens too quickly. Probably want to catch this error and ignore it.

Want to be able to query the same campground for multiple ranges of days. Ex: July 8 and 9, but also the following weekend.