This project demonstrates how ConferenceCorpus can be integrated into a project and how the data can be loaded and accessed.
Before running the tests ensure that the raw data of the data sources is downloaded by running the following scripts:
$ scripts/install
$ scripts/getbackup
$ scripts/getofflinedata
To test if the data sources of the ConferenceCorpus are loaded correctly run $scripts/test.
The script runs a Test that loads all data sources from the previously downloaded raw data.
Running the test the first time takes about ~5 min but when running it again it will load the data from its cache ($HOME/.conferenceCorpus/EventCorpus.db) taking ~40 seconds.
To enforce a reloading from the data sources itself, set forceUpdate to True when loading the data sources.
The output of the test sould look like this:
$ bash scripts/test
Starting test test_loadingOfDatasources, debug=False ...
Loading the datasources (this might take some time)
configureCorpusLookup callback called
Warning - using full /home/vmt14/.dblp/dblp.xml dataset ~0.0m records!
Warning - using full /home/vmt14/.dblp/dblp.xml dataset ~0.0m records!
Available datasource ids: ['confref', 'crossref', 'dblp', 'gnd', 'wikidata', 'wikicfp', 'or', 'or-backup', 'orclone', 'orclone-backup']
confref has 37945 events and 4857 series
crossref has 49280 events and 1 series
dblp has 47891 events and 5256 series
gnd has 1000000 events and 1 series
wikidata has 7508 events and 4254 series
wikicfp has 87987 events and 6019 series
or has 9473 events and 1058 series
or-backup has 9231 events and 1028 series
orclone has 9477 events and 1086 series
orclone-backup has 9338 events and 1057 series
test test_loadingOfDatasources, debug=False took 286.1 s
.
----------------------------------------------------------------------
Ran 1 test in 286.119s
OKYou can also check if the data sources at there storage location:
| location | data sources |
|---|---|
$HOME/.conferencecorpus/ |
wikicfp,crossref and confref also the EventCorpus.db is stored here |
$HOME/.dblp/ |
dblp (to get the complete data dump (~4GB) see ConferenceCorpusIntro) |
$HOME/.or/ |
openresearch (original and clone) wiki-Markup files |
The procedure on how to load and access the ConferenceCorpus is described in the class ConferenceCorpusIntro. An overview of the different data sources can be found in the official wiki
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent