mjordan / bagit_indexer Goto Github PK
View Code? Open in Web Editor NEWProof-of-concept tool for extracting data from Bags and indexing it in Elasticsearch
License: The Unlicense
Proof-of-concept tool for extracting data from Bags and indexing it in Elasticsearch
License: The Unlicense
If a bag has been deleted, we should not delete its document in the index. Rather, we should
tombstone
field from false
to 'true
andThat way, the last state of the document, and all of its previous versions, remain in the index.
When Archivematica processes a zipped bag with a bag-info.txt file in it, the information is transferred to a METS file, and can be found in a predictable location in the analog/digital source metadata sub-section of the administrative metadata section . See attached as an example (the original bag-info.txt file and the METS file generated by Archivematica are included).
The tool that extracts data from the bags and indexes them will need to be aware of the location of the data we want to index is (e.g., in a METS file at the analog/digital source metadata sub-section of the administrative metadata section , or if that data doesn't exist, in the bag-info.txt file).
Using something like the proof of concept BagIt Indexer, example logic would be: If there is a file named "METS.xml" at the root of the Bag's /data directory, look for data at /data/METS.xml// and index it so each element is in a searchable field; if "METS.xml" doesn't exist, index the fields in /bag-info.txt. (We'll need a third fallback option here, in case there is a METS.xml file but it doesn't contain /.) Pretty standard stuff. The gotcha here is that if we add a third deposit type (say a non-bag deposit), the indexer script would need to know where to get the relevant data for that deposit type. Given today's software development practices, this sort of extensibility can be handled by using a plugin architecture where a different plugin detects the presence of the desired data and extracts it, and then passes it off to the indexing engine.
Of course, using the same names for the METS elements and bag-info.txt fields will result in better queries. However, plugins can also map source fields to a common field name for indexing purposes.
For example, ./find -q bag_location:bag_01.tgz
will return all records in the index.
ES doesn't do this automatically (_timestamp
was dropped in verion 2). It would be useful to be able to query when the document was added to our index. Maybe something like:
timestamp: {
type: 'date',
format: 'epoch_second'
}
Related to #9.
php bagit_indexer.php
=> ./index
php bagit_search.php
=> ./find
python bagit_watcher.py
=> ./watch
For some reason the extension is missing.
php5-cli, composer, git
Also, remove /home/vagrant/elasticsearch-5.4.1.deb
Currently, the directory specified in --input
needs to be flat. Ability to put Bags in subdirectories would be useful.
Watcher script should be updated to be recursive too.
For example, AND tombstone:false
.
Maybe by providing a command-line option that takes a comma-separated list of paths, like
--files_to_index="foo.txt,dir/bar.xml"
Not sure what the resulting field names should be. Perhaps ES's dynamic mapping is useful here.
To start, make this a dumb indexer, with no preprocessing of XML files, etc.
The watcher script should use the bag's checksum to confirm that it is dealing with the same bag file in its on_moved()
method.
Currently the indexer uses the bag's filename as its ID. For example, for a bag at the location /mnt/storage/bag_567.zip, the ID will be 'bag_567'. This ID is only unique for bags within a given directory, and if we rename a file, any association between its ID and its name is lost.
To work around these limitations, we should use the bag's sha1 checksum as its ID, since the probability of a collision is extremely low. If the probability of collisions is low enough for Git to use sha1 hashes as unique IDs, it's good enough for an index of bags. Perhaps we can even allow users to enter the first x digits of a sha1 hash in find
queries.
The main disadvantage is the bag filenames are much more human-readable than hashes.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.