Make it feel faster

• take prep time into account in ETA
• don't show a text run ETA until all jobs are started

Make it actually faster

• detect not needing to re-prepare
• drop rety from 3 to 2
• dont rety OR only retry once for non-ci test runs

Learn about intermitant failures

• store fail/pass ratio for each test to get a sense of stability
• in change_main exit the process upon first db_charmer error
• buld a gem that allows test run output to files individual for each test
• reserect the cucumber data dump upon failure

failues

• is mysql2 adapter in thread safe mode?
• is redis in thread safe mode?

bugfix

• fix timeline bug

A distributed test run framework built on resque.

Hobson distributes your test suite across N machines aggregating the results live on a beautiful locally run web app.

• auto balance tests across jobs
• preserve log files and other artifacts to the cloud
• automatically re-run failed tests
• monitor a git repo reference and auto run new SHAs (a basic CI)

to Change.org for funding this project and to Charles Finkel for solving a heap of bugs

0. Setup a redis-server (We recommend a dedicated redis-server instance for Hobson lest you run the chance of your data being lost when a projects test run flushes all databases. If anyone knows how to protect an individual db please contact me)

1. Setup a cloud storage like S3 (anything Fog supports) (this is used to upload artifacts like logs)

2. Create your Hobson config.yml like so

   ---

   :redis: :host: 127.0.0.1 :port: 6380 :storage: :provider: AWS :aws_secret_access_key: INTENTIONALLY_LEFT_BLANK :aws_access_key_id: INTENTIONALLY_LEFT_BLANK :directory: BUCKET_NAME_HERE

0. setup N machines capable of running your project's test suite
1. on one of these machines start an isolated redis-server (on another port) * I don't recommend sharing redis with your project(s) * this is the redis you'll reference in your config.yml files later on
2. do this on all of your worker machines: 0. checkout the hobson repo (in /hobson for example) 0. cd ~/hobson && bundle install 0. make a directory for hobson to work within (/hobson_workspace for example) 0. create a ~/hobson_workspace/config.yml file that looks like this 0. …(here is where you would setup /etc/init.d/hobson and or monit etc.) 0. start your workers by running this:
   • cd ~/hobson_workspace && ~/hobson/bin/hobson work

0. DO NOT ADD hobson TO YOUR GEMFILE OR PROJECT
1. checkout the hobson repo on your workstation (in ~/hobson for example)
2. copy the same config.yml file you created earlier to $YOUR_PROJECT/config/hobson.yml
3. add any hooks you might need (see these examples)

just run hobson test from with you project

Something like cd $YOUR_PROJECT && ~/hobson/bin/hobson test

This should start the local Hobson web server opened to your new test run's status page

Hobson is very alpha. The main issue you might face is resque worker death. To get visiblity into this run hobson resque-web to see the status of resque.

Once you setup N machines running a hobson resque worker all you need to do is run 'hobson test' and Hobson will distribute your test suite across N workers and aggregate the results live into a single web page.

Hobson is completely decentralized apart from it's persistence in a single redis server. There is no central Hobson server process. Hobson is just a collection of resque workers and local sintra app.

0. cd into your project directory and run 'hobson test'
1. this launches the local hobson web application and opens the status page for the test run you've just created and schedules a 'build test run' job for the current git repo and SHA
2. one of your resque workers picks up this job, checks out the given sha and scans it for 'tests'
3. these tests are then balanced across N resque jobs. (N being the number of resque workers you have) and then N 'test run' resque jobs are now scheduled
4. your cluster of resque workers then starts running their subset of your test suite in parallel
5. as each resque worker completes starts a test or completed a test the test run status page is updated
6. Once complete you'll end up with something like the following screenshot
7. Once a worker completes it's task the log files are uploaded to S3

looks something like this…

Hobson looks in ${APP_ROOT}/config/hobson/ for the following hooks. These hooks are evaluated within the hobson process giving you access to the internal API (see code for more details).

• setup.rb
  • run once after bundle install but before tests are run
• before_running_tests.rb
  • run before each test command is executed
• save_artifacts.rb
  • run once after all tests have been run
• teardown.rb
  • run once after test run is complete

The hobson command looks in ${APP_ROOT}/config/hobson.yml for the redis server it should connect to.

Right now a 'test' is an individual feature or spec file. In the near future we'll be increasing cucumber feature granularity to the individual feature level. Unfortunately because specs can be defined dynamically it's non-trivial if not impossible to individually address each spec so we have to stay at the level of the file for rspec.

Hobson persists everything in redis using the following models.

    ---
    :redis:
      :host: ec2-0-0-0-0.compute-1.amazonaws.com
      :port: 6380
    :s3:
      :access_key_id: INTENTIONALLY_LEFT_BLANK
      :secret_access_key: INTENTIONALLY_LEFT_BLANK
      :bucket: INTENTIONALLY_LEFT_BLANK

0. Checkout hobson somewhere. For example we'll say you checked it out in ~/hobson_gem
1. Install hobson's gems '(cd ~/hobson_gem/ && bundle)'
2. Create a hobson workspace directory somewhere. For example we'll use ~/hobson
3. Place your config.yml here ~/hobson/config.yml
4. …(here is where you would setup /etc/init.d/hobson and or monit etc.)
5. Start hobson in the the working directory 'cd ~/hobson && ~/hobson_gem/bin/hobson work --daemonize'
6. Rinse and repeat

Hobson persists everything in redis using the following models.

• name (String)
• git_url (String)
• test_runs (Hobson::Project::TestRun)

• project (string)
• id (String)
• sha (String)
• scheduled_build (Datetime)
• started_building (Datetime)
• scheduled_jobs (Datetime)
• tests (Hobson::Project::TestRun::Tests)

• tests (Hobson::Project::TestRun::Tests::Test)

• name (String)
• state (String) [waiting|started|complete]
• result (String) [PASS|FAIL|PENDING]
• est_runtime (Float) [seconds]
• runtime (Float) [seconds]
• job (Integer) [job index]

• index (Integer)
• scheduled_at (Datetime)
• checking_out_code (Datetime)
• preparing_environment (Datetime)
• running_tests (Datetime)
• saving_artifacts (Datetime)
• tearing_down (Datetime)
• completed_at (Datetime)

• add a test run post processing job to do things like compile SimpleCov reports
• detect when a sha isn't on origin and error
• find a fix for empty test files
  • when a test file is empty it's never updated by the hobson status formatter and the whole test run is hung
• add hobson & hobson project configuration options
  • min / max jobs per test_run
  • min / max tests per job
  • auto re-run failed ? tests
• improve test_run runtime estimation taking into account avg. setup & teardown
• add custom formatters that dump failures to individual files and upload those artifacts immediately
• column sorting on test list in web interface