gem install bundler bundle install rake thief:create_tables
bin/thief bin/server
Then browse to http://localhost:4567
bin/console
Use the interactive console
Thief.sources << Thief::Source::SomeSource.new Thief.fetch
Will go through all assigned sources (must be located in /thief/sources/*.rb) and call their fetch method.
Each Source has its own table for “Person” objects (dapi_people, dapi_wikipedia) and will store all the information in there.
Thief.integrate
Will go through all assigned sources (same as above) and call their integrate method.
Each Integrator will convert its own Person-schema to the integrated schema (Thief::DAPI::Person → Thief::Person)
Thief::Person.all.each { |person| puts person.name }
prints out the stored and integrated people.
Special download mechanism in Thief::Wikipedia::ETL loads the necessary txt file which contains the person data if the file doesn’t exist.
Thief::Wikipedia:ETL.download_file!
This method is called by the fetch method if the necessary file containing the person data doesn’t exist. It will download the zip file, unzip it and rename and move the contained text file to the
wikipedia directory.
You can copy the dapi.rb and dapi-folder to create a new source with ETL, Integrator and Person objects.
The defaults will look under the specific namespace for the etl and integrator classes. (Thief::YourLibrary::ETL)
You can change that by overwriting the etl and integrator methods.
require 'somewhere/over/the/rainbow' module Thief module Sources class YourAwesomeSource def etl ::OneETLToRuleThemAll.new end end end end
Start the app on any rack-compatible webserver with the config.ru file. (in production mode in the examples)
Thin:
thin --rackup config.ru --environment production start
or Unicorn:
unicorn --env production
or WEBrick through rackup:
THIEF_ENV=production rackup --server webrick config.ru
or with shotgun:
shotgun --env production
and so on..
Thief::ETL now has inheritable mechanisms to download and extract files as well as store them in their specific temporary directories.
Thief::Integrator now has inheritable mechanisms for defining a mapping from a source specific schema to the global schema (experimental)