All examples skipped after exception in before: suite hook in queue mode about knapsack_pro-ruby HOT 2 CLOSED

For the future reference.

In order to reproduce the problem, raise an exception outside of a test example.

describe 'abc' do
  raise 'error'
  it do
    expect(true).to be true
  end
end

When tests fail, then new tests are fetched from Queue API. All tests fail so that means a lot of tests will be fetched quickly from Queue API.
Because of that too many test files are assigned to a given CI node index.

If you retry a CI node (and you use KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true) and connection to DB works fine, then the same set of tests that was assigned to the node in the first place would run. Tests will take a long time because there is a lot of them.

Problem:
knapsack_pro does not know how to catch exceptions from inside of the RSpec process because RSpec already caught the exception, so we don't know if the test failure is a regular failure or not (exception outside of a test example). So Knapsack Pro keeps asking Queue API for more tests instead of failing fast and stopping pulling tests.

Possible solution. Ensure the DB is up and running:

Example for Github Actions:

    services:
      postgres:
        image: postgres:10.8
        env:
          POSTGRES_USER: postgres
          POSTGRES_PASSWORD: ""
          POSTGRES_DB: postgres
        ports:
          - 5432:5432
        # needed because the postgres container does not provide a healthcheck
        # tmpfs makes DB faster by using RAM
        options: >-
          --mount type=tmpfs,destination=/var/lib/postgresql/data
          --health-cmd pg_isready
          --health-interval 10s
          --health-timeout 5s
          --health-retries 5

https://docs.github.com/en/actions/using-containerized-services/creating-postgresql-service-containers#running-jobs-in-containers

Maybe this will be helpful:
https://github.com/orgs/community/discussions/25757

If the problem happens again, you can create a new commit and this will start a new queue so that the tests are distributed in new order.

from knapsack_pro-ruby.

Hi @amanfredi

I'm wondering how to reproduce it. I did

# spec_helper.rb
KnapsackPro::Hooks::Queue.before_queue do |queue_id|
  raise 'a'
end

Indeed knapsack pro consumes tests from Queue API and does it fast. It does not execute tests because RSpec could not start tests due to a failure. Tests failed.

CI build should complete work faster when tests are not executed.

I'm not sure why you experienced a timeout from CI? Do you mean you fixed the bug in the code and you started a new build (attempted some kind of retry) and it was split using cached distribution (fixed queue split) so that a single CI node got too many tests (due to them being cached on the API side). If you fixed a bug in the code then I would expect a new git commit hash so that the fixed queue split (cached distribution on API side) won't be used.

Could you give more details on how to reproduce the problem. What you did?
Did you start 1 CI build or maybe 2 because you did a retry? Did you create a new commit?
What would you expect to happen?

from knapsack_pro-ruby.