Comments (6)
For now a hacky way to write a list of entities to disk would be the following,
# entities = a list of instances of WikidataEntity (or its subclasses)
entity_strings = [json.dumps(ent._entity_dict) for ent in entities]
with open('filtered_entities.json', 'w') as fp:
fp.write("[\n")
fp.write(",\n".join(entity_strings))
fp.write("\n]\n")
wjd = WikidataJsonDump("filtered_entities.json")
from qwikidata.
If this is worth pursuing, I'm happy to give a go at it, but would need some feedback on the appropriate syntax for entering such restrictions (i.e. for user to specify the restrictions in a flexible-enough way).
from qwikidata.
Yes, its an interesting idea. I think the tricky part is (as you've said) figuring out how to specify the filter. It might be possible to pass in a user defined function that takes in an entity and returns True or False. Another option would be to just provide a function that writes a group of entities to disk so that a user can make their own filtered dumps. Let me prototype a few things and I'll report back.
from qwikidata.
So, after thinking about this a little more, I think the WikidataJsonDump
is not the right place for this functionality. I'd like to keep that class doing "one thing well" and handling iteration and chunking of a dump from wikimedia is a nice domain. Also, to really do useful filtering it would be necessary to create instances of WikidataItem
and/or WikidataProperty
and/or WikidataLexeme
and the json dump simply deals with strings and dictionaries.
However, persisting a list of filtered entities to a file is definitely a nice feature. I think the way forward might be to create a utils
module that has a function that takes an iterable of entities and writes them to disk in the same format as the json dump. This could be done using the _write_chunk
method of WikidataJsonDump
except it would take an iterable of entities as opposed to a List of strings. This way the user can implement arbitrary logic to filter and then persist those filtered entities.
from qwikidata.
Tried it just now, works great!
from qwikidata.
PR in to add this functionality to a utils module #22
from qwikidata.
Related Issues (9)
- Checking value types HOT 3
- Add python 3.5 support
- getting all claims for an items
- dump_entities_to_json fails if entities has only one member HOT 2
- get_truthy_claim_group does not take into consideration qualifiers HOT 2
- How to render Time (datavalue type) HOT 1
- Exception from`return_sparql_query_results` HOT 2
- basic_json_dump.py yields: TypeError: 'list' object is not an iterator HOT 4
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from qwikidata.