gem install bundler
  bundle install
  rake thief:create_tables

  bin/thief
  bin/server

Then browse to http://localhost:4567

  bin/console

Use the interactive console

  Thief.sources << Thief::Source::SomeSource.new
  Thief.fetch

Will go through all assigned sources (must be located in /thief/sources/*.rb) and call their fetch method.
Each Source has its own table for “Person” objects (dapi_people, dapi_wikipedia) and will store all the information in there.

  Thief.integrate

Will go through all assigned sources (same as above) and call their integrate method.
Each Integrator will convert its own Person-schema to the integrated schema (Thief::DAPI::Person → Thief::Person)

  Thief::Person.all.each { |person| puts person.name }

prints out the stored and integrated people.

Special download mechanism in Thief::Wikipedia::ETL loads the necessary txt file which contains the person data if the file doesn’t exist.

  Thief::Wikipedia:ETL.download_file!

This method is called by the fetch method if the necessary file containing the person data doesn’t exist. It will download the zip file, unzip it and rename and move the contained text file to the
wikipedia directory.

You can copy the dapi.rb and dapi-folder to create a new source with ETL, Integrator and Person objects.
The defaults will look under the specific namespace for the etl and integrator classes. (Thief::YourLibrary::ETL)
You can change that by overwriting the etl and integrator methods.

  require 'somewhere/over/the/rainbow'

  module Thief
    module Sources
      class YourAwesomeSource
        def etl
          ::OneETLToRuleThemAll.new
        end
      end
    end
  end

Start the app on any rack-compatible webserver with the config.ru file. (in production mode in the examples)

Thin:

  thin --rackup config.ru --environment production start

or Unicorn:

  unicorn --env production

or WEBrick through rackup:

  THIEF_ENV=production rackup --server webrick config.ru

or with shotgun:

  shotgun --env production

and so on..

Thief::ETL now has inheritable mechanisms to download and extract files as well as store them in their specific temporary directories.

Thief::Integrator now has inheritable mechanisms for defining a mapping from a source specific schema to the global schema (experimental)