The Webservices project lets you take pretty much any structured data set and turn it into a search API without writing a line of code.

• understands CSV, TSV, JSON, XLS, or XML file formats
• smart guessing of data types and schemas based on heuristics
• versioned APIs by default
• ingest by file upload or URL
• easy polling/refresh of URL-based data sources
• simple YAML-based configuration for each data source
• customize the ETL process via built-in transformations

You can choose either a standard installation or you can use Docker.

If you have Docker and Docker Compose, all you have to do is

$ docker-compose up

And it should automatically bring up the ElasticSearch container and the Rails app, correctly linking them.

Otherwise, follow the standard setup instructions.

You'll need Ruby 2.2.

We use bundler to manage gems. You can install bundler and other required gems like this:

gem install bundler
bundle install

The charlock_holmes gem requires the UCI libraries to be installed. If you are using Homebrew, it's probably as simple as this:

 brew install icu4c

More information about the gem can be found here

We're using ElasticSearch (>= 2.2.0) for fulltext search. On a Mac, it's easy to install with Homebrew.

brew install elasticsearch

Otherwise, follow the instructions to download and run it.

You need to install the delete-by-query plugin and restart elasticsearch if it was already running:

$ plugin install delete-by-query

Webservices can use foreman to start Rails and Sidekiq in development environments.

You'll need to have redis installed on your machine. brew install redis, apt-get install redis-server, etc.

Create the indexes:

bundle exec rake db:create

Generate the default admin user with username [email protected] and password initial_pwd:

bundle exec rake db:devseed    

Fire up a server:

foreman start -f Procfile.dev

This Procfile assumes you already have Elasticsearch and Redis running.

Import some data:
bundle exec rake ita:import[ScreeningList,MarketResearchData]

Admin users can log in and monitor the progress of the Sidekiq import jobs via /sidekiq.

Since v2 of the API, an authentication token is required for every request. Pass it on the query string:

http://localhost:3000/market_research_library/search?api_key=devkey

http://localhost:3000/consolidated_screening_list/search?api_key=devkey&size=5&offset=8

http://localhost:3000/consolidated_screening_list/search?api_key=devkey&q=john

http://localhost:3000/consolidated_screening_list/search?api_key=devkey&sources=SDN,EL

http://localhost:3000/market_research_library/search?api_key=devkey&q=oil

http://localhost:3000/market_research_library/search?api_key=devkey&countries=HU,CA

Or using http headers:

curl -H'Api-Key: devkey' 'http://localhost:3000/v2/market_research_library/search'

Admin users can create and administer search APIs from uploaded files or from URLs. The file formats supported include CSV, TSV, JSON, XLS, and XML. The initial admin user created with the db:devseed task has the admin flag set to true already. To toggle an existing user, you can do this from the Rails console:

email = "[email protected]"
u = User.search(filter: { bool: { must: { term: { email: email } } } }).first
u.update_attribute(:admin, true)

To create an API, click the + next to the Dynamic APIs subnav heading.

To refresh a URL-based api, you can periodically call the rake task to check for updates. Pass the api field from the DataSource as a parameter:

bundle exec rake endpointme:import[business_service_providers]

bundle exec rspec

Elasticsearch must be running. It's easiest to just foreman start -f Procfile.dev.

We track test coverage of the codebase over time to help identify areas where we could write better tests and to see when poorly tested code got introduced.

After running your tests, view the report by opening coverage/index.html.

Click around on the files that have less than 100% coverage to see what lines weren't exercised.

We use Mailcatcher to test emails sent from dev environments. Their advice is not to add the gem to your Gemfile, so in order to use it please do:

gem install mailcatcher
mailcatcher

If you use RVM, you should follow their specific instructions (search for "RVM") on how to install the gem.

If you want to run this app on AWS Opsworks, you may find this guide helpful.