A distributed test run framework built on resque.

Hobson distributes your test suite across N machines aggregating the results live on a beautiful locally run web app.

Hobson can:

• auto balance tests across jobs
• preserve log files and other artifacts to S3
• automatically re-run failed tests
• monitor a git repo reference and auto run new SHAs (a basic CI)

to Change.org for funding this project and to Charles Finkel for solving a heap of bugs

Once you setup N machines running a hobson resque worker all you need to do is run 'hobson test' and Hobson will distribute your test suite across N workers and aggregate the results live into a single web page.

Hobson is completely decentralized apart from it's persistence in a single redis server. There is no central Hobson server process. Hobson is just a collection of resque workers and local sintra app.

0. cd into your project directory and run 'hobson test'
1. this launches the local hobson web application and opens the status page for the test run you've just created and schedules a 'build test run' job for the current git repo and SHA
2. one of your resque workers picks up this job, checks out the given sha and scans it for 'tests'
3. these tests are then balanced across N resque jobs. (N being the number of resque workers you have) and then N 'test run' resque jobs are now scheduled
4. your cluster of resque workers then starts running their subset of your test suite in parallel
5. as each resque worker completes starts a test or completed a test the test run status page is updated
6. Once complete you'll end up with something like the following screenshot
7. Once a worker completes it's task the log files are uploaded to S3

looks something like this…

Hobson looks in ${APP_ROOT}/config/hobson/ for the following hooks. These hooks are evaluated within the hobson process giving you access to the internal API (see code for more details).

• setup.rb
  • run once after bundle install but before tests are run
• before_running_tests.rb
  • run before each test command is executed
• save_artifacts.rb
  • run once after all tests have been run
• teardown.rb
  • run once after test run is complete

The hobson command looks in ${APP_ROOT}/config/hobson.yml for the redis server it should connect to.

Right now a 'test' is an individual feature or spec file. In the near future we'll be increasing cucumber feature granularity to the individual feature level. Unfortunately because specs can be defined dynamically it's non-trivial if not impossible to individually address each spec so we have to stay at the level of the file for rspec.

0. Setup a redis-server (We recommend a dedicated redis-server instance for Hobson lest you run the chance of your data being lost when a projects test run flushes all databases. If anyone knows how to protect an individual db please contact me)

1. Setup an S3 bucket (this is used to upload test run artifacts like logs)

2. Create your Hobson config.yml like so

   ---

   :redis: :host: ec2-0-0-0-0.compute-1.amazonaws.com :port: 6380 :s3: :access_key_id: INTENTIONALLY_LEFT_BLANK :secret_access_key: INTENTIONALLY_LEFT_BLANK 🪣 INTENTIONALLY_LEFT_BLANK

0. Checkout hobson somewhere. For example we'll say you checked it out in ~/hobson_gem
1. Install hobson's gems '(cd ~/hobson_gem/ && bundle)'
2. Create a hobson workspace directory somewhere. For example we'll use ~/hobson
3. Place your config.yml here ~/hobson/config.yml
4. …(here is where you would setup /etc/init.d/hobson and or monit etc.)
5. Start hobson in the the working directory 'cd ~/hobson && ~/hobson_gem/bin/hobson work --daemonize'
6. Rinse and repeat

Hobson persists everything in redis using the following models.

• name (String)
• git url (String)
• test runs (Hobson::Project::TestRun)

• id (String)
• sha (String)
• scheduled_build (Datetime)
• started_building (Datetime)
• scheduled_jobs (Datetime)
• tests (Hobson::Project::TestRun::Tests)

• tests (Hobson::Project::TestRun::Tests::Test)

• name (String)
• state (String) [waiting|started|complete]
• result (String) [PASS|FAIL|PENDING]
• est_runtime (Float) [seconds]
• runtime (Float) [seconds]
• job (Integer) [job index]

• index (Integer)
• scheduled_at (Datetime)
• checking_out_code (Datetime)
• preparing_environment (Datetime)
• running_tests (Datetime)
• saving_artifacts (Datetime)
• tearing_down (Datetime)
• completed_at (Datetime)

• add a requested-by tag to show what engineer created what test_run
• refactor feature tests to mean individual scenarios rather then .feature file
• detect when a sha isn't on origin and error
• update balancing logic to be aware of 0 est runtimes rather then using 0.1 as a hack
• find a fix for empty test files
  • when a test file is empty it's never updated by the hobson status formatter and the whole test run is hung
• refactor away from redis hashes and back to normal keys/namespaces with marshaling
• add hobson & hobson project configuration options
  • min / max jobs per test_run
  • min / max tests per job
  • auto re-run failed ? tests
• improve test_run runtime estimation taking into account avg. setup & teardown
• add custom formatters that dump failures to individual files and upload those artifacts immediately