foo-git / rewe-discounts Goto Github PK

Followed the instructions to obtain the market ID, but the API seems to have changed:

FAIL: Unknown error while fetching discounts from https://shop.rewe.de/mc/api/markets-stationary/28199. Maybe a typo or the server rejected the request.

Accessing the URL manually ends in a 404.

When trying to receive the current list I get:

(meins) root@dagobert:~/rewe-discounts# python3 rewe_discounts/rewe_discounts.py --market-id 1761063 --output-file "bla.md"
Traceback (most recent call last):
  File "/root/meins/lib/python3.11/site-packages/requests/models.py", line 971, in json
    return complexjson.loads(self.text, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/__init__.py", line 346, in loads
    return _default_decoder.decode(s)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/root/rewe-discounts/rewe_discounts/rewe_discounts.py", line 194, in <module>
    data = scraper.get(url).json()
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/root/meins/lib/python3.11/site-packages/requests/models.py", line 975, in json
    raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
FAIL: Unknown error while fetching discounts from https://mobile-api.rewe.de/api/v3/all-offers?marketCode=1761063, maybe a typo or the server rejected the request.
(meins) root@dagobert:~/rewe-discounts#

At first it looks like the API output has changed and the program cannot deal correctly with the output. After executing the program, it fails with this error message:

 Traceback (most recent call last):
  File "/x/rewe_discounts", line 193, in <module>
    categories = data['_meta']['categories']
 KeyError: '_meta'

However the API url (https://mobile-api.rewe.de/products/offer-search) yields a 500 error, which looks like an internal error of the rewe api. Not sure if I can fix this:

errorReason: "status - 500 by calling Endpoint Service, because of reason: result.data.map is not a function}"

Hi Foo-Git,

kann man dich irgendwie kontaktieren Telegram, Mail oder so?
Hätte da vielleicht was mit Rewe API + Payback und würde dafür auch was zahlen wenn mir bisschen helfen kannst :)

The new API introduced in v2.6 (#17, #18) seems to be broken due to a change by REWE, it now yields a 404 error:

Traceback (most recent call last):
File "[...]/rewe_discounts.py", line 294, in less_elegant_query
data = scraper.get(url).json()
^^^^^^^^^^^^^^^^^^^^^^^
File "[...]/venv/lib/python3.11/site-packages/requests/models.py", line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Extra data: line 1 column 5 (char 4)
FAIL: Unknown error while fetching discounts from https://www.rewe.de/api/all-stationary-offers/[...], maybe a typo or the server rejected the request.

Right now, I have no solution available and the script will not work. If you find the correct API url, please let me know.

Do you know whether there's an API endpoint to get straight up all products available in a given store?

Unfortunally the script stopped working for me few days ago :/
Thats the output from 403 Branch:

File "Python\Python311\Lib\site-packages\requests\models.py", line 971, in json
return complexjson.loads(self.text, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "Python\Python311\Lib\json_init_.py", line 346, in loads
return _default_decoder.decode(s)
^^^^^^^^^^^^^^^^^^^^^^^^^^
File "Python\Python311\Lib\json\decoder.py", line 337, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "Python\Python311\Lib\json\decoder.py", line 355, in raw_decode

raise JSONDecodeError("Expecting value", s, err.value) from None

json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

During handling of the above exception, another exception occurred:

File "rewe_discounts.py", line 194, in
data = scraper.get(url).json()
^^^^^^^^^^^^^^^^^^^^^^^
File "Python\Python311\Lib\site-packages\requests\models.py", line 975, in json
raise RequestsJSONDecodeError(e.msg, e.doc, e.pos)
requests.exceptions.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
FAIL: Unknown error while fetching discounts from https://mobile-api.rewe.de/api/v3/all-offers?marketCode=562286, maybe a typo or the server rejected the request.

Upon opening the link https://mobile-api.rewe.de/api/v3/all-offers?marketCode=562286 manuelly inside my browser it works just fine. Rewe must have improved their IT Skills recently😁
Doesnt necessarily look like an IP block for me.
I've been using the script for a relatively long time, really great work, too bad it doesn't work for me now.

Rewe blocks the IP after few queries. Is there any way around to solve this issue ?