When placekey contains invalid value with more than one @ symbol, placekey_format_is_valid function throws a ValueError

In [2]: import placekey as pk

In [3]: pk.placekey_format_is_valid("5yv-j89-g6k")
Out[3]: True

In [4]: pk.placekey_format_is_valid("@@5yv-j89-g6k")
---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
Cell In [4], line 1
----> 1 pk.placekey_format_is_valid("@@5yv-j89-g6k")

File ~/.virtualenvs/geocode-api/lib/python3.8/site-packages/placekey/placekey.py:338, in placekey_format_is_valid(placekey)
    329 def placekey_format_is_valid(placekey):
    330     """
    331     Boolean for whether or not the format of a Placekey is valid, including
    332     checks for valid encoding of location.
   (...)
    336 
    337     """
--> 338     what, where = _parse_placekey(placekey)
    340     if what:
    341         return _where_part_is_valid(where) and bool(WHAT_REGEX.match(what))

File ~/.virtualenvs/geocode-api/lib/python3.8/site-packages/placekey/placekey.py:369, in _parse_placekey(placekey)
    361 """
    362 Split a Placekey in to what and where parts.
    363 
   (...)
    366 
    367 """
    368 if '@' in placekey:
--> 369     what, where = placekey.split('@')
    370 else:
    371     what, where = None, placekey

ValueError: too many values to unpack (expected 2)

• One of the API tests is failing due to updated placekey including the What part format
• Raised PR #24 to fix the test

First of all, thanks for building such a great tool for referencing the identity and location in space of different entities!

There seems to a mismatch between the requirements.txt file in the current GitHub repo (which works fine when installed to my machine) and the requires.txt file included in the source code of the PlaceKey-py package posted on PyPI.

The GitHub requirements.txt includes requests, ratelimit, backoff, all of which are required to have the main contents of the API submodule work correctly. Thus, when I currently download the package from PyPi and try to call the API submodule, it immediately fails owing to not having downloaded ratelimit as well.

Just wanted to bring this to your attention in case it wasn't intention -- thanks for reading!

Hey, I noticed there is no optional param include_corelogic_clip. It would be useful to have it included as well. Thanks

Our account/apikey was locked out today due to making too many requests in 1 day. We then started seeing some of our applications hang. We traced it down to the line of code referenced above. It seems to go into an infinite loop for as long as 429 response code is returned. If our account is locked out for 1 day, for example, that means this code loops continually making calls to placekey for maybe 1 day (not sure how you guys perform your rate limiting calculations). For example, if we make our entire day's allocations of calls in 1 hour, then we would have to wait 23 hours until our account is unlocked which means this code will loop for 23 hours. A library should not hang for 23 hours so I think this is a bug.

The issue

When a response with an error status code other than 429 is received (like 504), a cryptic error message about parsing the results is printed to the console.

Details

Currently when I use the following code to make a couple hundred requests, I get the following error after about 1 minute of spinning:

pk_api.lookup_placekeys(some_queries)

JSONDecodeError: Expecting value: line 1 column 1 (char 0)

With the following stack trace:

----> pk_api.lookup_placekeys(query)

/.../python3.7/site-packages/placekey/api.py in lookup_placekeys(self, places, strict_address_match, strict_name_match, batch_size, verbose)
    200                     places[i:max_batch_idx],
    201                     strict_address_match=strict_address_match,
--> 202                     strict_name_match=strict_name_match
    203                 )
    204             except RateLimitException:

/.../python3.7/site-packages/placekey/api.py in lookup_batch(self, places, strict_address_match, strict_name_match)
    270                 break
    271 
--> 272         return json.loads(result.text)
    273 
    274     def _validate_query(self, query_dict):

/.../python3.7/json/__init__.py in loads(s, encoding, cls, object_hook, parse_float, parse_int, parse_constant, object_pairs_hook, **kw)
    346             parse_int is None and parse_float is None and
    347             parse_constant is None and object_pairs_hook is None and not kw):
--> 348         return _default_decoder.decode(s)
    349     if cls is None:
    350         cls = JSONDecoder

/.../python3.7/json/decoder.py in decode(self, s, _w)
    335 
    336         """
--> 337         obj, end = self.raw_decode(s, idx=_w(s, 0).end())
    338         end = _w(s, end).end()
    339         if end != len(s):

/.../python3.7/json/decoder.py in raw_decode(self, s, idx)
    353             obj, end = self.scan_once(s, idx)
    354         except StopIteration as err:
--> 355             raise JSONDecodeError("Expecting value", s, err.value) from None
    356         return obj, end

This was quite puzzling, especially since the exact same code worked just 30 minutes prior (with the exact same queries, too).

It seemed obvious that whatever lookup_batch received was not status code 429 (thus broke the while loop), but did not contain any JSON in the response text

I called make_bulk_request directly to see what the real result was and got more helpful output.

pk_api.make_bulk_request({"queries": query[:100]})

<Response [504]>

Suggested solution

Perhaps lookup_batch could raise an error if a non-200 HTTP status is received? Or perhaps just certain 500 codes? Or more of a catch-all error if the result text isn't JSON?

# Make request, and retry if there is a server-side rate limit error
while True: 
    result = self.make_bulk_request(batch_payload)
    if result.status_code != 429: 
        break

if not result.text:
    raise RuntimeError(f"Error: Status code {result.status_code}")

return json.loads(result.text) 

The Placekey python package is capable of querying the single locations and bulk API endpoints, but it is missing the Lineage API endpoint.

Placekey recommends pinging the Lineage endpoint every 3-5 months in order to detect and correct any changes to existing Placekey data - essential to maintaining a Placekey-based dataset. Right now, users must write their own Lineage requests, even though they can use placekey-py for everything else.