michaelhball / discogs_alert Goto Github PK

View Code? Open in Web Editor NEW

28.0 6.0 7.0 218 KB

Customised, real-time alerting for your hard-to-find wantlist items.

License: MIT License

Python 96.13% Dockerfile 3.69% Shell 0.18%

discogs wantlist digger

discogs_alert's People

Contributors

Stargazers

Watchers

Forkers

tytiethai alexkalopsia simenflatby ddelange 8o5 leonas2000 fmaass lilo90hui

discogs_alert's Issues

Add package builds for different architectures

Dockerhub image outdated

Would it be possible to update the image on dockerhub?

Pushbullet doesn't work on iOS

Therefore we need to add at least one, ideally many different options for a notification service. Here are some candidate ideas

email
- Gmail, Yagmail, Envelopes, Flanker, usw.
Telegram
Airdroid
Pushline
Push Safer
Mighty Text
Yappy
SMS

Chromedriver not closing properly

          just came back after having my computer off for about eight hours, google chrome has many instances open, so yeah seems to have something to do with that.

Originally posted by @ShiestGuy in #38 (comment)

Hey @ShiestGuy I created a new issue for you here, the place you were posting was for my tracking of a separate issue.

I will investigate this as soon as I can and try to work something out, but in preparation could you please let me know how you're running discogs_alert: which command are you using (docker vs python, etc.), are you running it as a cron job, which interval are you using, how many (roughly) items are in your wantlist etc.

Index and Attribute Error

When a currency of DKK exists in the listings I get the following error:

Show Error

> INFO:discogs_alert.loop:Exception: this might be a real exception, but we're continuing anyway > Traceback (most recent call last): > File "/Users/jonas/Desktop/python_scripts/Discogs/discogs_alert/discogs_alert/scrape.py", line 94, in scrape_listings_from_marketplace > price_currency = re.findall(currency_regex, price_string)[0] > IndexError: list index out of range > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/Users/jonas/Desktop/python_scripts/Discogs/discogs_alert/discogs_alert/loop.py", line 79, in loop > for listing in client_anon.get_marketplace_listings(release.id): > File "/Users/jonas/Desktop/python_scripts/Discogs/discogs_alert/discogs_alert/client.py", line 143, in get_marketplace_listings > return da_scrape.scrape_listings_from_marketplace(self.driver.page_source, release_id) > File "/Users/jonas/Desktop/python_scripts/Discogs/discogs_alert/discogs_alert/scrape.py", line 102, in scrape_listings_from_marketplace > raise ParsingException( > discogs_alert.scrape.ParsingException: Couldn't parse > currency from price_string DKK3633.17 for release 11899856

I saw that something was done for CHF so I tried the same with DKK

try:
  currency_regex = r".*?(?:[\£\$\€\¥]{1})"
  price_currency = re.findall(currency_regex, price_string)[0]
  price_string = price_string.replace(price_currency, "")
except IndexError:
  # TODO: make this part of the Regex, + generalise to other unseen currencies
  if "CHF" in price_string:
      price_currency = "CHF"
      price_string = price_string.replace("CHF", "")
  elif "DKK" in price_string:
      price_currency = "DKK"
      price_string = price_string.replace("DKK", "")
  else:
      raise ParsingException(
          f"Couldn't parse currency from price_string {price_string} for release {release_id}"
      )

However I get an AttibuteError if I do that

Show AttibuteError

> INFO:discogs_alert.loop:AttributeError: will continue looping as usual > Traceback (most recent call last): > File "/Users/jonas/Desktop/python_scripts/Discogs/discogs_alert/discogs_alert/loop.py", line 79, in loop > for listing in client_anon.get_marketplace_listings(release.id): > File "/Users/jonas/Desktop/python_scripts/Discogs/discogs_alert/discogs_alert/client.py", line 143, in get_marketplace_listings > return da_scrape.scrape_listings_from_marketplace(self.driver.page_source, release_id) > File "/Users/jonas/Desktop/python_scripts/Discogs/discogs_alert/discogs_alert/scrape.py", line 35, in scrape_listings_from_marketplace > rows = listings_table.find("tbody").find_all("tr") > AttributeError: 'NoneType' object has no attribute 'find' > INFO:discogs_alert.loop: took 3.9872360229492188

general questions on the script (master listings, already alerted list, brave browser for client)

Hello Michael!

Thank you for the amazing Python script! I cloned the repo and was able to build and install it. Now I want to play around with the code a bit and add some features for myself.

I noticed that If I inputted for the id a master_id 1757517 for the listing of https://www.discogs.com/sell/list?master_id=1757517&ev=mb, I did not get any notifications. Is it how it is programmed?
When I first ran python3 -m discogs_alert -dt ... with my want list, I noticed that sent all the available (old) listing via pushbullet. When uninstalling discogs_alert and reinstalling through git and rerunning the code it did not send those alerts. How does the code work? And it did not notify for all that were for sale. (is maybe the country wrong? and how can I have it use multiple counties to see if shipping is available?)

get the complete list of previous pushes

    pushes_dict = defaultdict(set)
    for p in pushbullet.get_all_pushes(pushbullet_token):
        if "title" in p and "body" in p:
            pushes_dict[p["title"]].add(p["body"])

If I understand correctly, does it check through the push bullet api if it has already sent it before and won't send even if the python script is rerun?

I saw that it uses Chrome, I installed it for now but I was curious if I could make it work with Brave browser. I figured it has something to do with the AnonClient(Client) but I can't figure out how to do it. I would be really happy if you could lead me on the right way!

repeatedly opens up chrome

I love this program, it's insanely useful and I thank you so much for making it. However, keeping it running for a day and opening task manager shows about 200 separate chrome instances running, that all close when I close the program. This over time leads it using up a lot of hardware. Is this happening as a mistake from me, or is it something you can fix?

Use of Discogs list as wantlist doesn't permit per-release configuration

This needs to be fully specked out, but it would be nice if there were a way to make use of per-release constraints when using Discogs lists, not only when using a wantlist.json. The only real lever we have to work with is the comment section, but one could easily come up with a simple schema to enforce such rules.

403 error due to captcha

Hi,

First and foremost, this is a very exciting project!

It appears as though I am hitting a 403 error due to captcha. Has anyone had similar problems?

$ python3 -m discogs_alert -dt redacted -pt redacted

ERROR: status_code: 403, content: b'<!DOCTYPE html>\n<!--[if lt IE 7]> <html class="no-js ie6 oldie" lang="en-US"> <![endif]-->\n<!--[if IE 7]>    <html class="no-js ie7 oldie" lang="en-US"> <![endif]-->\n<!--[if IE 8]>    <html class="no-js ie8 oldie" lang="en-US"> <![endif]-->\n<!--[if gt IE 8]><!--> <html class="no-js" lang="en-US"> <!--<![endif]-->\n<head>\n\n<title>Attention Required! | Cloudflare</title>\n\n<meta name="captcha-bypass" id="captcha-bypass" />\n<meta charset="UTF-8" />\n<meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />\n<meta http-equiv="X-UA-Compatible" content="IE=Edge,chrome=1" />\n<meta name="robots" content="noindex, nofollow" />\n<meta name="viewport" content="width=device-width,initial-scale=1" />\n<link rel="stylesheet" id="cf_styles-css" href="/cdn-cgi/styles/cf.errors.css" type="text/css" media="screen,projection" />\n<!--[if lt IE 9]><link rel="stylesheet" id=\'cf_styles-ie-css\' href="/cdn-cgi/styles/cf.errors.ie.css" type="text/css" media="screen,projection" /><![endif]-->\n<style type="text/css">body{margin:0;padding:0}</style>\n\n\n<!--[if gte IE 10]><!-->\n<script>\n  if (!navigator.cookieEnabled) {\n    window.addEventListener(\'DOMContentLoaded\', function () {\n      var cookieEl = document.getElementById(\'cookie-alert\');\n      cookieEl.style.display = \'block\';\n    })\n  }\n</script>\n<!--<![endif]-->\n\n\n  \n    <script type="text/javascript">\n    //<![CDATA[\n    (function(){\n      window._cf_chl_opt={\n        cvId: "2",\n        cType: "interactive",\n        cNounce: "39725",\n        cRay: "645c94b1ec52efe9",\n        cHash: "25df4da48623f8a",\n        cFPWv: "g",\n        cTTimeMs: "4000",\n        cLt: "n",\n        cRq: {\n          ru: "aHR0cHM6Ly93d3cuZGlzY29ncy5jb20vc2VsbC9yZWxlYXNlLzE0MTg5NzUyP2V2PXJiJnNvcnQ9cHJpY2UlMkNhc2M=",\n          ra: "cHl0aG9uLXJlcXVlc3RzLzIuMjUuMQ==",\n          rm: "R0VU",\n          d: "4oYu1x0vCji6ap7DhGbHNvTNsoZEkpcXfvmbsWTdJBXwAOGcK/7KiuC4n2sCLG1L30S7GP4+gpJ4nuJqjYKVEEiM4fHtv0cOt5XZDGHn5pN1PlEpolHl+Ds4aYZQMPCY0DA7V0wZpFznRd6y/6rNeQVdmBWiGOIi5JK7kedqye34oSSNC9IrfbwV+L/kfpov1Xc3+UfIPULNWIHFP9+iig0Rj/l+D633N2fNL/YIiDT6YDvuuZMMKT+rcs5+CYuO1V7PZlFrcBCmW3UYpcWOBi4q2BIr4J4RxEUqUcwBgURFmKliBkZizb6KWaxKnEPw5MkIrUj8GtrqdpBXOpUi9aDoU2+0lAZZcoyTTUarTpB/WhUo+0wEcJLVfyaJQzyfKAX0P1TBKUh8fM83O7nyhTTsF0ZavEBbLPjq8ggmJoLcuSkwOTHE6UvclFWNNzv7zLKIKJe2fk05ygme1vsUafhEJL2e/cBT7g5shWyAELn19HwJWPHvRC+mAANpIVfz10u+Z73eatcO4VUthGjk0AK537r1hXTQJDk2YcMPHBXxzLJZHM40Nv0hcgWzW2dPTwgl9dY3vbfhpvsd3aZFt+ucP7tL41mfL3F7WWaN7pGROATqT+jdJK1FwDhEcgkmmq44c6FPVQdmDx6qV5gklKc1THhfPCCE914OtY+eHy9Fcxv4UdJ16ae9U40P0MRhIlIcHY5IwmFA4WSmXJTd2ifBmue9AvYlhU9mG0EVTKJPjQVVr/V+pGB4zHB61MtC6RMgjlUv4eywTEXGxcnskVsQSb1xEbsCyl21WrkjkxUiPP0g+edELy+2oW1rlcNXOCy32Ua8aq9ITQmFOHPdlTg4crXgH0PVdZAnPH9qIw2ILT7Y6vht9HDo/szQURJG+GeqwETg5Fs4lb106+rTCyXDoiYVlRUM78Dnl+dvV3Y=",\n          t: "MTYxOTQwNjIzNi40NzMwMDA=",\n          m: "rZTGOEFetolvz13gYuuLh/oVDLL3a0VBmZoBX9P7nmg=",\n          i1: "bz6EMGBFd75vJwOcIqMawg==",\n          i2: "9tQtpPuRyVo0+c4+uHb8Lg==",\n          zh: "y/5gbtqv3ryNnXkEV3okjz0qMwAKw8hrQMiGMrZYIqs=",\n          uh: "5GU+jYv2xJ+bCaE/ARmi/DORbiS/v56CW7E0TH4XWQk=",\n          hh: "aHe4LtJTAXE8cGlMp2YOZtPZttABMyA3bgeN7VeGi3E=",\n        }\n      };\n    }());\n    //]]>\n    </script>\n  \n\n<style type="text/css">\n  #cf-wrapper #spinner {width:69px; margin:  auto;}\n  #cf-wrapper #cf-please-wait{text-align:center}\n  .attribution {margin-top: 32px;}\n  .bubbles { background-color: #f58220; width:20px; height: 20px; margin:2px; border-radius:100%; display:inline-block; }\n  #cf-wrapper #challenge-form { padding-top:25px; padding-bottom:25px; }\n  #cf-hcaptcha-container { text-align:center;}\n  #cf-hcaptcha-container iframe { display: inline-block;}\n  @keyframes fader     { 0% {opacity: 0.2;} 50% {opacity: 1.0;} 100% {opacity: 0.2;} }\n  #cf-wrapper #cf-bubbles { width:69px; }\n  @-webkit-keyframes fader { 0% {opacity: 0.2;} 50% {opacity: 1.0;} 100% {opacity: 0.2;} }\n  #cf-bubbles > .bubbles { animation: fader 1.6s infinite;}\n  #cf-bubbles > .bubbles:nth-child(2) { animation-delay: .2s;}\n  #cf-bubbles > .bubbles:nth-child(3) { animation-delay: .4s;}\n</style>\n</head>\n<body>\n  <div id="cf-wrapper">\n    <div class="cf-alert cf-alert-error cf-cookie-error" id="cookie-alert" data-translate="enable_cookies">Please enable cookies.</div>\n    <div id="cf-error-details" class="cf-error-details-wrapper">\n      <div class="cf-wrapper cf-header cf-error-overview">\n      \n        <h1 data-translate="challenge_headline">One more step</h1>\n        <h2 class="cf-subheadline"><span data-translate="complete_sec_check">Please complete the security check to access</span> www.discogs.com</h2>\n      \n      </div>\n      \n      <div class="cf-section cf-highlight cf-captcha-container">\n        <div class="cf-wrapper">\n          <div class="cf-columns two">\n            <div class="cf-column">\n            \n              <div class="cf-highlight-inverse cf-form-stacked">\n                <form class="challenge-form interactive-form" id="challenge-form" action="/sell/release/14189752?ev=rb&amp;sort=price%2Casc&amp;__cf_chl_captcha_tk__=da8c2960eceefb69ad48a21b6c6468229a36dedf-1619406236-0-AfAfMMaPWc5_2HGJq5PBF9xIqbwBlUHRnnNhFFUvsKDe5Hj_be8ax7owfhVsW286rXP-gn3eTCz3AwL79Bca3dGFeKlPZNx9Kam0QJlaLw51v3GFc1a_xFCOLDRQS0KOZjLBJPd1-xtEZQyCHKEzZO_OLDKjKUltmX1SKtez5b25nM7eRJF-uE7HUCrVaBdmlHfy2_H9oeDE-r41fLZpqKxNAc4T0eUVLQ1Ihi15dAekSuvkxzg3f4_K1Qe1mKpXtPPq2X2EpUHe1t67FknJFiLwCt4iVm5jPZklH95cOfNyGWW2HlleE_pDXv6cFafbW5P0M3enfU4KcbA4yGMHv6DYI2CFiKyeXBiUs4OmVSiPRnqqTTfjssUPl9cL27eMm0TjhSOFtrUOnaD3DXctqDOcBY3wDheV2bV6nAeIIODWkza6wIuuwPa8qiQATsng5OpAOpcb7hTUt5Hvs6poEpOizNynWEdKN3lm9eLtb12IjyFn6_olqz35JSVNWv5154zrs1lrzyVk4cljFd0nPxIUcA2rY-v91RTiZwmkaaOfk0mFW8ScbABkgpEjp8OWi0keUP62kLjuxJPCOztHUCJy3Cp-LspgOsNS0r6uSElRT8au2SvOrs38PizftM-ERo2gV6gkYQDCtxpfGmkfSL-xx6qIolanMn7_cfveD9-7XOxh6AbW5DIeaIHm5SvPCdR2-S8WCNpg0SR-4aJ73ug" method="POST" enctype="application/x-www-form-urlencoded">\n  \n    <div id=\'cf-please-wait\'>\n      <div id=\'spinner\'>\n        <div id="cf-bubbles">\n            <div class="bubbles"></div>\n            <div class="bubbles"></div>\n            <div class="bubbles"></div>\n        </div>\n      </div>\n      <p data-translate="please_wait" id="cf-spinner-please-wait">Please stand by, while we are checking your browser...</p>\n      <p data-translate="redirecting" id="cf-spinner-redirecting" style="display:none">Redirecting...</p>\n      </div>\n  \n  <input type="hidden" name="r" value="e0ba8463616e52a3a85482ade71c756530542cf7-1619406236-0-ASeF10IlcrC0oI1MqDK93hxUpwpHJhnqzrcdGjsQHeXeDBSFU7d3H2Q/addRqB7ouBS3u1JWUERRCsoffJWusGl7N60m24LRwFdlsgr0pk+/M0RDfxqrWNlKt3KHG09BL46Zuw3+tsJRHLWbj/hQ1c0HRnHn6UOgbHc6gS8bAHWqVkeI4lt7sifQC6n7yomWPU+ZPvnA/YsohOALyeLxvDJyRYOxkGHFMTgAf3vzDAXm1+3TW7+3NlbGxNRvpTa9WsIXl9hl2mPXkXB45B0dBhyu7vVvkxwmJUxx7NhIp0s3GJ72CqWzndz3rmTzgzeD5eZImKaIhwVKWkc4O0k4tt0IXNLVvepEEi7DHbVWrFjQ6afNguxk1/C4D4WffRfRwooXsn9AGKw8MPNeNVubkDZhF6UicvhKRQztEfe0M2LmOlQyLIOYWGGoSLvamxkW2+dVHFp7R+eYOi3XxYsnPifKdE9V8TcYJaDplYAB/7ZOs0/ccf1xPMtJuXz7CIGjkBEOZ9Fkxddu1JLqzkelONKQb6A75LYPUqH1F5KmKUeJ0kbOMzOySWkFsA/uYMGEZ+K65EMQlmaEx3cffBV7bEfUsqfT0oKkctnpHn3TBhAyDfNRB+kLbR9EvrFZOh8VNoC6fMFfluP3kypvqMVAqg59AQtgf44oJT+PWW3P5RMwP1xbSSSJUGKCk0nakOyTHozyrq0ZBGJ155NK/LM1a4HyazK+T4Hn15v/bibauzF0B8dYrmqOsVDIOoAjb7onGhh0f/IQqFZz82ysYOmBJnuc2nQxt4lmDSywmkV5o+v5NH/VA6b4mHrT3iCszwC0sbaTyEQLWdNPMuM5Qyt8GBJQTvyhclrHsK0uXDqu0Ujsf+GY+tJXPqZs895yv0oNHdSgzoIknDbLtTtWZF1Xz0G5SfyH9+kWz2fGcUptR6jRH3nB0Uw1fwdNKawfDQBBqTyHH37auq+o0atM3LA2gTD1JKHmNu3ePdbtLxB7/0Jxg94Rpo3PbeY716aSK4nfNsSs6OXMpkb/zCGGvH6kdMHprtC4IgCO91tVlVlX0aQLxq5OibCEiERjVlOGNt8DKjtM+53aboDGkFn64dYvmaWyTm3iUeCQ5BGsaI6ZvfLALuGdHS7P9IToHc9LzkOCfRy2zb/o3fSkk9/4Yq0+cwTFQR0yM8i2GRjnEFXhRuSALXrBMuqWycjwz4eaH8J11pLvr+bkHhsk8jaa9DdFPsk49i1ntoS5v/+8ontxaWzs9JXhd2KP82ncZCH35xSd5CLjQRhl4StOzmormO7co+8eQs+rijJXRY9niQtF1PvzILAHVrgRQLrRZb5VcT90Gg==">\n  <input type="hidden" name="cf_captcha_kind" value="h">\n  <input type="hidden" name="vc" value="f88731207630bd6f32ad444502506af2">\n  \n  <noscript id="cf-captcha-bookmark" class="cf-captcha-info">\n  <h1 data-translate="turn_on_js" style="color:#bd2426;">Please turn JavaScript on and reload the page.</h1>\n  </noscript>\n    <div id="no-cookie-warning" class="cookie-warning" data-translate="turn_on_cookies" style="display:none">\n      <p data-translate="turn_on_cookies" style="color:#bd2426;">Please enable Cookies and reload the page.</p>\n    </div>\n  <script type="text/javascript">\n  //<![CDATA[\n    var a = function() {try{return !!window.addEventListener} catch(e) {return !1} },\n      b = function(b, c) {a() ? document.addEventListener("DOMContentLoaded", b, c) : document.attachEvent("onreadystatechange", b)};\n      b(function(){\n        var cookiesEnabled=(navigator.cookieEnabled)? true : false;\n        if(!cookiesEnabled){\n          var q = document.getElementById(\'no-cookie-warning\');q.style.display = \'block\';\n        }\n      });\n  //]]>\n  </script>\n  <div id="trk_captcha_js" style="background-image:url(\'/cdn-cgi/images/trace/captcha/nojs/h/transparent.gif?ray=645c94b1ec52efe9\')"></div>\n</form>\n  \n  <script type="text/javascript">\n    //<![CDATA[\n    (function(){\n        var isIE = /(MSIE|Trident\\/|Edge\\/)/i.test(window.navigator.userAgent);\n        var trkjs = isIE ? new Image() : document.createElement(\'img\');\n        trkjs.setAttribute("src", "/cdn-cgi/images/trace/captcha/js/transparent.gif?ray=645c94b1ec52efe9");\n        trkjs.id = "trk_captcha_js";\n        trkjs.setAttribute("alt", "");\n        document.body.appendChild(trkjs);\n        var cpo=document.createElement(\'script\');\n        cpo.type=\'text/javascript\';\n        cpo.src="/cdn-cgi/challenge-platform/h/g/orchestrate/captcha/v1?ray=645c94b1ec52efe9";\n        document.getElementsByTagName(\'head\')[0].appendChild(cpo);\n    }());\n    //]]>\n    </script>\n  \n\n\n              </div>\n            </div>\n\n            <div class="cf-column">\n              <div class="cf-screenshot-container">\n              \n                <span class="cf-no-screenshot"></span>\n              \n              </div>\n            </div>\n          </div>\n        </div>\n      </div>\n\n      <div class="cf-section cf-wrapper">\n        <div class="cf-columns two">\n          <div class="cf-column">\n            <h2 data-translate="why_captcha_headline">Why do I have to complete a CAPTCHA?</h2>\n            \n            <p data-translate="why_captcha_detail">Completing the CAPTCHA proves you are a human and gives you temporary access to the web property.</p>\n          </div>\n\n          <div class="cf-column">\n            <h2 data-translate="resolve_captcha_headline">What can I do to prevent this in the future?</h2>\n            \n\n            <p data-translate="resolve_captcha_antivirus">If you are on a personal connection, like at home, you can run an anti-virus scan on your device to make sure it is not infected with malware.</p>\n\n            <p data-translate="resolve_captcha_network">If you are at an office or shared network, you can ask the network administrator to run a scan across the network looking for misconfigured or infected devices.</p>\n            \n              \n            \n          </div>\n        </div>\n      </div>\n      \n\n      <div class="cf-error-footer cf-wrapper w-240 lg:w-full py-10 sm:py-4 sm:px-8 mx-auto text-center sm:text-left border-solid border-0 border-t border-gray-300">\n  <p class="text-13">\n    <span class="cf-footer-item sm:block sm:mb-1">Cloudflare Ray ID: <strong class="font-semibold">645c94b1ec52efe9</strong></span>\n    <span class="cf-footer-separator sm:hidden">&bull;</span>\n    <span class="cf-footer-item sm:block sm:mb-1"><span>Your IP</span>: 76.98.181.121</span>\n    <span class="cf-footer-separator sm:hidden">&bull;</span>\n    <span class="cf-footer-item sm:block sm:mb-1"><span>Performance &amp; security by</span> <a rel="noopener noreferrer" href="https://www.cloudflare.com/5xx-error-landing" id="brand_link" target="_blank">Cloudflare</a></span>\n    \n  </p>\n</div><!-- /.error-footer -->\n\n\n    </div>\n  </div>\n\n  <script type="text/javascript">\n  window._cf_translation = {};\n  \n  \n</script>\n\n\n</body>\n</html>\n'
object of type 'bool' has no len()

AttributeError

Hello im trying to setup discogs-alert on a debian 11 bullseye appliance. I'm getting this error... can you help on how to fix it please? thanks a lot

`
INFO:discogs_alert.loop:

running @#loop

INFO:discogs_alert.loop:AttributeError: will continue looping as usual

Traceback (most recent call last):
File "/usr/local/lib/python3.10/site-packages/discogs_alert/loop.py", line 67, in loop
for listing in client_anon.get_marketplace_listings(release.id):

File "/usr/local/lib/python3.10/site-packages/discogs_alert/client.py", line 143, in get_marketplace_listings
return da_scrape.scrape_listings_from_marketplace(response_content)

File "/usr/local/lib/python3.10/site-packages/discogs_alert/scrape.py", line 51, in scrape_listings_from_marketplace
media_condition = media_condition_tooltips.get("data-condition")

AttributeError: 'NoneType' object has no attribute 'get'

INFO:discogs_alert.loop: took 3.3722472190856934

Add seller country whitelist & blacklist config option

So that one can either include (exclusively) or exclude a list of countries from those considered to be valid releases.

Include shipping price in `price_threshold` filter?

Hi, I was wondering if it was possible to include shipping prices in the filtering? It would really help for the releases that often have to be imported.

Exception: this might be a real exception, but we're continuing anyway

My script keeps failing, with the following errors. Running it on a mac using telegram as the alerter and using a local json wantlist.

Traceback (most recent call last):
  File "<frozen runpy>", line 198, in _run_module_as_main
  File "<frozen runpy>", line 88, in _run_code
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/discogs_alert/__main__.py", line 279, in <module>
    main()
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/discogs_alert/__main__.py", line 274, in main
    schedule.run_pending()
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/schedule/__init__.py", line 822, in run_pending
    default_scheduler.run_pending()
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/schedule/__init__.py", line 100, in run_pending
    self._run_job(job)
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/schedule/__init__.py", line 172, in _run_job
    ret = job.run()
          ^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/schedule/__init__.py", line 693, in run
    ret = self.job_func()
          ^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/discogs_alert/__main__.py", line 272, in <lambda>
    schedule.every(int(60 / frequency)).minutes.do(lambda: da_loop.loop(*args))
                                                           ^^^^^^^^^^^^^^^^^^^
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/discogs_alert/loop.py", line 141, in loop
    client_anon.driver.close()
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/selenium/webdriver/remote/webdriver.py", line 458, in close
    self.execute(Command.CLOSE)
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/selenium/webdriver/remote/webdriver.py", line 347, in execute
    self.error_handler.check_response(response)
  File "/Library/Frameworks/Python.framework/Versions/3.12/lib/python3.12/site-packages/selenium/webdriver/remote/errorhandler.py", line 229, in check_response
    raise exception_class(message, screen, stacktrace)
selenium.common.exceptions.WebDriverException: Message: disconnected: not connected to DevTools
  (failed to check if window was closed: disconnected: not connected to DevTools)
  (Session info: chrome-headless-shell=124.0.6367.78)
Stacktrace:
0   chromedriver                        0x0000000100f239c8 chromedriver + 5216712
1   chromedriver                        0x0000000100f1b603 chromedriver + 5182979
2   chromedriver                        0x0000000100a8d527 chromedriver + 406823
3   chromedriver                        0x0000000100a728b6 chromedriver + 297142
4   chromedriver                        0x0000000100a7300f chromedriver + 299023
5   chromedriver                        0x0000000100a8dab0 chromedriver + 408240
6   chromedriver                        0x0000000100a630be chromedriver + 233662
7   chromedriver                        0x0000000100b0ae05 chromedriver + 921093
8   chromedriver                        0x0000000100afb6c8 chromedriver + 857800
9   chromedriver                        0x0000000100acbfe3 chromedriver + 663523
10  chromedriver                        0x0000000100acc92e chromedriver + 665902
11  chromedriver                        0x0000000100ee18e0 chromedriver + 4946144
12  chromedriver                        0x0000000100ee7994 chromedriver + 4970900
13  chromedriver                        0x0000000100ec23de chromedriver + 4817886
14  chromedriver                        0x0000000100ee84a9 chromedriver + 4973737
15  chromedriver                        0x0000000100eb2664 chromedriver + 4752996
16  chromedriver                        0x0000000100f0ab58 chromedriver + 5114712
17  chromedriver                        0x0000000100f0ad17 chromedriver + 5115159
18  chromedriver                        0x0000000100f1b223 chromedriver + 5181987
19  libsystem_pthread.dylib             0x00007ff815ebb18b _pthread_start + 99
20  libsystem_pthread.dylib             0x00007ff815eb6ae3 thread_start + 15

Stacktrace when launching

I tried to run discogs-alert both on Mac Mojave 10.14.6 and Debian 11, using python 3.9.2 and 3.9.6.
In both cases the stacktrace output is as follows:

python3 -m discogs_alert -dt XXXXXXXXXXXXXXXXXXXXXXXX -pt XXXXXXXXXXXXXXXXXXXXXXXX -wp path

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/urllib3/connection.py", line 169, in _new_conn
conn = connection.create_connection(
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/urllib3/util/connection.py", line 73, in create_connection
for res in socket.getaddrinfo(host, port, family, socket.SOCK_STREAM):
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/socket.py", line 953, in getaddrinfo
for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 382, in _make_request
self._validate_conn(conn)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 1010, in _validate_conn
conn.connect()
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/urllib3/connection.py", line 353, in connect
conn = self._new_conn()
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/urllib3/connection.py", line 181, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPSConnection object at 0x10f1c3610>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/urllib3/connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/urllib3/util/retry.py", line 574, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPSConnectionPool(host='api.ratesapi.io', port=443): Max retries exceeded with url: /api/latest?base=EUR (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x10f1c3610>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 197, in _run_module_as_main
return _run_code(code, main_globals, None,
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/runpy.py", line 87, in _run_code
exec(code, run_globals)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/discogs_alert/main.py", line 70, in
main()
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/click/core.py", line 829, in call
return self.main(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/click/core.py", line 782, in main
rv = self.invoke(ctx)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/click/core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/click/core.py", line 610, in invoke
return callback(*args, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/discogs_alert/main.py", line 65, in main
schedule.run_pending()
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/schedule/init.py", line 563, in run_pending
default_scheduler.run_pending()
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/schedule/init.py", line 94, in run_pending
self._run_job(job)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/schedule/init.py", line 147, in _run_job
ret = job.run()
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/schedule/init.py", line 466, in run
ret = self.job_func()
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/discogs_alert/main.py", line 63, in
schedule.every(int(60 / frequency)).minutes.do(lambda: loop(*args))
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/discogs_alert/loop.py", line 22, in loop
currency_rates = get_currency_rates(currency)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/discogs_alert/utils.py", line 57, in get_currency_rates
return requests.get(f'https://api.ratesapi.io/api/latest?base={base_currency}').json().get('rates')
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/requests/api.py", line 76, in get
return request('get', url, params=params, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/Library/Frameworks/Python.framework/Versions/3.9/lib/python3.9/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPSConnectionPool(host='api.ratesapi.io', port=443): Max retries exceeded with url: /api/latest?base=EUR (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x10f1c3610>: Failed to establish a new connection: [Errno 8] nodename nor servname provided, or not known'))

Error when client is trying to download latest driver_version_to_download, also when installed already

I have Chromedriver & Python 3.10 already installed, still within the exception an attempt is made to download the latest browser for the platform (macOS 14.2.1 (23C71)) fails

INFO:discogs_alert.loop:Exception: this might be a real exception, but we're continuing anyway
Traceback (most recent call last):
  File "/Users/USERNAME/Desktop/code projects/discogs_alert/discogs_alert/discogs_alert/client.py", line 133, in get_driver_path
    return subprocess.check_output(["which", "chromedriver"]).decode().strip()
  File "/Users/USERNAME/.pyenv/versions/3.10.13/lib/python3.10/subprocess.py", line 421, in check_output
    return run(*popenargs, stdout=PIPE, timeout=timeout, check=True,
  File "/Users/USERNAME/.pyenv/versions/3.10.13/lib/python3.10/subprocess.py", line 526, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['which', 'chromedriver']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/USERNAME/Desktop/code projects/discogs_alert/discogs_alert/discogs_alert/loop.py", line 63, in loop
    client_anon = da_client.AnonClient(user_agent)
  File "/Users/USERNAME/Desktop/code projects/discogs_alert/discogs_alert/discogs_alert/client.py", line 111, in __init__
    service = ChromiumService(self.get_driver_path(), log_path=log_path)
  File "/Users/USERNAME/Desktop/code projects/discogs_alert/discogs_alert/discogs_alert/client.py", line 136, in get_driver_path
    return ChromeDriverManager().install()
  File "/Users/USERNAME/Desktop/code projects/discogs_alert/discogs_alert/.venv/lib/python3.10/site-packages/webdriver_manager/chrome.py", line 39, in install
    driver_path = self._get_driver_binary_path(self.driver)
  File "/Users/USERNAME/Desktop/code projects/discogs_alert/discogs_alert/.venv/lib/python3.10/site-packages/webdriver_manager/core/manager.py", line 33, in _get_driver_binary_path
    file = self._download_manager.download_file(driver.get_driver_download_url())
  File "/Users/USERNAME/Desktop/code projects/discogs_alert/discogs_alert/.venv/lib/python3.10/site-packages/webdriver_manager/drivers/chrome.py", line 59, in get_driver_download_url
    modern_version_url = self.get_url_for_version_and_platform(driver_version_to_download, os_type)
  File "/Users/USERNAME/Desktop/code projects/discogs_alert/discogs_alert/.venv/lib/python3.10/site-packages/webdriver_manager/drivers/chrome.py", line 94, in get_url_for_version_and_platform
    raise Exception(f"No such driver version {browser_version} for {platform}")
Exception: No such driver version 120.0.6099.216 for mac-arm64

Chromium zombie processes left behind when ran in Docker

When running discogs_alert in a Docker container, it begins to accrue large amounts of zombie chromium processes over time:

root     1200075  0.0  0.0 1537316 19056 ?       Sl   Jun28   0:07 /usr/bin/containerd-shim-runc-v2 -namespace moby -id <snip> -address /run/containerd/containerd.sock
root     1200102  1.4  0.0  63776 53488 ?        Ss   Jun28  49:04  \_ python -m discogs_alert
root     1200307  0.0  0.0      0     0 ?        Z    Jun28   0:00      \_ [chromium] <defunct>
root     1200308  0.0  0.0      0     0 ?        Z    Jun28   0:00      \_ [chromium] <defunct>
... 10k lines later ...
root     1781396  0.0  0.0      0     0 ?        Z    21:55   0:00      \_ [chromium] <defunct>
root     1781412  0.3  0.0      0     0 ?        Z    21:55   0:00      \_ [chromium] <defunct>
root     1781413  0.5  0.0      0     0 ?        Z    21:55   0:00      \_ [chromium] <defunct>
root     1781436  4.2  0.0      0     0 ?        ZN   21:55   0:04      \_ [chromium] <defunct>

I suspect this is due to an exception being thrown during the loop, which results in the chromium process not being correctly terminated (perhaps related to #40?), combined with the fact that the default docker init doesn't reap zombie processes, but I haven't had the time to look into it further.

michaelhball / discogs_alert Goto Github PK

discogs_alert's People

Contributors

Stargazers

Watchers

Forkers

discogs_alert's Issues

get the complete list of previous pushes

Recommend Projects

Recommend Topics

Recommend Org