Geoparsing don't match some city about geoparsepy HOT 3 CLOSED

geoparsepy will by default remove location names that match common words in WordNet, name lists and stop lists. The whitelist allows you to prevent this removal for specific words you know are OK. this ensures they are in the location cache available for matching.

I suggest you check the location cache to see if it contains 'sciacca' (without the whitelist entry). if the name is missing then its triggering the geo_parse_lib.is_good_place_name() method. this means it might appear on a blacklist, names list of stopword list maybe and is rejected as a 'bad name'.

if the name is in the location cache OK but still does not match, then its being rejected as a viable location match. maybe the token 'sciacca' is being aggregated with other words (e.g. street prefixes) and so removed from the matching pool. try a sent with one word 'sciacca' to check if the word without context matches.

from geoparsepy.

The position "sciacca" is present in cached_locations, but is not present in indexed_locations (and therefore not matched)

This is the cached_locations situation:

grep "'Sciacca'" cache.txt
['europe_places_point_181422', 'Sciacca', (67254001,), (-39152, -39151, -365331), 'POINT(13.0823323 37.5064786)', {'name:ru': 'Шакка'}, ()]

grep "'Asciano'" cache.txt
['europe_places_point_164289', 'Asciano', (61753677,), (-365331, -42172, -41977), 'POINT(11.5594603 43.2348573)', {'gfoss id': '4624'}, ()]
['europe_places_point_164290', 'Asciano', (336080413,), (-42338, -365331, -41977), 'POINT(10.4681697 43.7499307)', {'wikidata': 'Q3624858', 'wikipedia': 'it:Asciano Pisano'}, ()]

This is the indexed_locations situation:

grep "'agrigento'" index.txt
('agrigento',)
('#', 'agrigento')

grep "'sciacca'" index.txt

grep "'asciano'" index.txt

To solve the problem without using the withelist I modified the geoparsepy.geo_parse_lib.is_good_place_name method with this code (I overwrote it):

# other => reject 1g phrases EDIT(only g with len <= 5) and check 2g+ is not only stop words
# since OpenStreetMap has a lot of rubbish 1g phrases like 'station', '24', 'building' ...
if len(list_tokens) == 1 and len(phrase) <= 5:
    return False
if soton_corenlppy.common_parse_lib.is_all_stoplist(list_tokens, dict_geospatial_config):
    return False

This is the indexed_locations situation after modification:

grep "'sciacca'" index.txt
('sciacca',)
('#', 'sciacca')

grep "'asciano'" index.txt
('asciano',)
('#', 'asciano')

A future upgrade for this library could be to insert a blacklist of rubbish OSM names for each language so as not to exclude most of the small cities that do not pass the previous tests.

from geoparsepy.

Thanks CalogeroCantone for your workaround. there is a tradeoff between compiling blacklists (which would contain a lot of names) and the heuristic filters in geoparsepy.geo_parse_lib.is_good_place_name() method. I will consider adding a config hyperparam to allow the phrase length filter to be configured, then users can change that setting to reflect thier appitite for false matches.

The reported problem was actually because you had two languages specified ['it','en']. This means nltk English first names are loaded, and an English name is 'Di' which is short for 'Diana'. This was matched to your sentence with 'di Sciacca' and thus rejected as a probable full name (as 'Di London' would be). I tested with ['it'] and Sciacca was matched OK.

My advice is to load only the language code of the text you will geoparse, and be careful when trying multiple languages. You can (if you have enough memory) load two instances of cached_locations() using two seperate configs, one for each language. This will produce a more precise output.

from geoparsepy.