Filtering the dump about qwikidata HOT 6 CLOSED

For now a hacky way to write a list of entities to disk would be the following,

# entities = a list of instances of WikidataEntity (or its subclasses) 
entity_strings = [json.dumps(ent._entity_dict) for ent in entities]                                  
with open('filtered_entities.json', 'w') as fp:                                                      
    fp.write("[\n")                                                                                  
    fp.write(",\n".join(entity_strings))                                                             
    fp.write("\n]\n")                                                                                
                                                                                                     
wjd = WikidataJsonDump("filtered_entities.json") 

from qwikidata.

If this is worth pursuing, I'm happy to give a go at it, but would need some feedback on the appropriate syntax for entering such restrictions (i.e. for user to specify the restrictions in a flexible-enough way).

from qwikidata.

Yes, its an interesting idea. I think the tricky part is (as you've said) figuring out how to specify the filter. It might be possible to pass in a user defined function that takes in an entity and returns True or False. Another option would be to just provide a function that writes a group of entities to disk so that a user can make their own filtered dumps. Let me prototype a few things and I'll report back.

from qwikidata.

So, after thinking about this a little more, I think the WikidataJsonDump is not the right place for this functionality. I'd like to keep that class doing "one thing well" and handling iteration and chunking of a dump from wikimedia is a nice domain. Also, to really do useful filtering it would be necessary to create instances of WikidataItem and/or WikidataProperty and/or WikidataLexeme and the json dump simply deals with strings and dictionaries.

However, persisting a list of filtered entities to a file is definitely a nice feature. I think the way forward might be to create a utils module that has a function that takes an iterable of entities and writes them to disk in the same format as the json dump. This could be done using the _write_chunk method of WikidataJsonDump except it would take an iterable of entities as opposed to a List of strings. This way the user can implement arbitrary logic to filter and then persist those filtered entities.

from qwikidata.

Tried it just now, works great!

from qwikidata.

PR in to add this functionality to a utils module #22

from qwikidata.