knapsackpro / knapsack_pro-ruby Goto Github PK

View Code? Open in Web Editor NEW

131.0 7.0 27.0 1.55 MB

Knapsack Pro gem splits tests across parallel CI nodes and makes sure that tests run in optimal time

Home Page: https://knapsackpro.com

License: MIT License

Ruby 99.94% Shell 0.06%

ruby auto-balancing ci-server cucumber minitest rspec spinach turnip parallelism testing-tools

knapsack_pro-ruby's People

Contributors

Stargazers

Watchers

knapsack_pro-ruby's Issues

Doesn't work with feature directories with spaces

Not a big deal, I'll rename our directories but it does yield an error that left me scratching my head for a moment:

I, [2017-03-22T17:01:22.143211 #1642]  INFO -- : [knapsack_pro] Test suite time execution recording enabled.
No such file or directory - features/auction. You can use `cucumber --init` to get started.
I, [2017-03-22T17:01:22.943221 #1642]  INFO -- : [knapsack_pro] No test files were executed on this CI node. When you use knapsack_pro regular mode then probably reason might be very narrowed tests list - you run only tests with specified tag and there are fewer test files with the tag than node total number.
I, [2017-03-22T17:01:23.286539 #1642]  INFO -- : [knapsack_pro] API request UUID: 7257ab85-d79c-469f-865e-22d94487163f
I, [2017-03-22T17:01:23.286757 #1642]  INFO -- : [knapsack_pro] API response:
I, [2017-03-22T17:01:23.286834 #1642]  INFO -- : [knapsack_pro] 
I, [2017-03-22T17:01:23.286864 #1642]  INFO -- : [knapsack_pro] Saved time execution report on API server.
I, [2017-03-22T17:01:23.286950 #1642]  INFO -- : [knapsack_pro] Global time execution for tests: 0s
Exited with code 2

The directory in question is features/auction management, the command being used to run is bundle exec rake "knapsack_pro:cucumber[--format junit --out test-reports/cucumber/junit.xml]"

This is not the same behavior as running with cucumber.

Log output understanding question

I am seeing output from this line in our CI logs, and want to know what it means:

If you need to reproduce a particular subset of tests fetched from API queue then above after each request to Knapsack Pro API you will find example rspec command.

It comes form here:

knapsack_pro-ruby/lib/knapsack_pro/runners/queue/rspec_runner.rb

Line 115 in 3aa4192

 KnapsackPro.logger.info("To retry in development the tests for this CI node please run below command on your machine. It will run all tests in a single run. If you need to reproduce a particular subset of tests fetched from API queue then above after each request to Knapsack Pro API you will find example rspec command.") 

There seems to be something wrong with this sentence, especially here:
... fetched from API queue then above after each request ...

I would like to support, but I don't know what exactly this sentence means.

Is this free?

Pro usually indicates paid. It's worthwhile to clear this out in the README.

NameError: uninitialized constant RSpec::Core::Version

I found an error while setting up a reproduction environment. I did a fresh setup of knapsack pro in an rspec environment. It was running okay:
https://buildkite.com/sj26/knapsack-test/builds/6

Then I added KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES=true and I got an error:
https://buildkite.com/sj26/knapsack-test/builds/8#15887146-f8e4-458a-9255-b6c999cb49d6/79-83

** Invoke knapsack_pro:rspec (first_time)
** Execute knapsack_pro:rspec
rake aborted!
NameError: uninitialized constant RSpec::Core::Version
/opt/rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/knapsack_pro-2.18.0/lib/knapsack_pro/base_allocator_builder.rb:37:in `fast_and_slow_test_files_to_run'
/opt/rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/knapsack_pro-2.18.0/lib/knapsack_pro/allocator_builder.rb:5:in `allocator'
/opt/rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/knapsack_pro-2.18.0/lib/knapsack_pro/runners/base_runner.rb:10:in `initialize'
/opt/rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/knapsack_pro-2.18.0/lib/knapsack_pro/runners/rspec_runner.rb:9:in `new'
/opt/rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/knapsack_pro-2.18.0/lib/knapsack_pro/runners/rspec_runner.rb:9:in `run'
/opt/rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/knapsack_pro-2.18.0/lib/tasks/rspec.rake:5:in `block (2 levels) in <main>'
/opt/rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:279:in `block in execute'
/opt/rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:279:in `each'
/opt/rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:279:in `execute'
/opt/rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/opt/rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/opt/rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/opt/rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/opt/rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/opt/rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/opt/rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/opt/rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/opt/rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/opt/rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:83:in `block in run'
/opt/rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/rbenv/versions/2.7.2/lib/ruby/gems/2.7.0/gems/rake-13.0.6/lib/rake/application.rb:80:in `run'
bin/rake:5:in `<main>'
Tasks: TOP => knapsack_pro:rspec

Here's the full source code at that point:
https://github.com/sj26/knapsack-test/tree/rspec-core-version-error

The command being run is in the pipeline file:
https://github.com/sj26/knapsack-test/blob/rspec-core-version-error/.buildkite/pipeline.yml#L23

There might be a require missing somewhere?

Queue mode is running much slower than regular mode with Knapsack Pro

I have the command as below to run my rspec tests in queue mode

KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true \
KNAPSACK_PRO_PROJECT_DIR=. \
KNAPSACK_PRO_CI_NODE_TOTAL=$CI_NODE_TOTAL \
KNAPSACK_PRO_CI_NODE_INDEX=$(($CI_NODE_INDEX-1)) \
KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=<the token> \
KNAPSACK_PRO_TEST_FILE_LIST_SOURCE_FILE=tmp/crystalball_prediction \
bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format RspecJunitFormatter --out tmp/rspec.xml]"

It's been running fine. I found every time before running, knapsack called this API only once with log

D, [2021-05-20T17:27:44.542380 #134] DEBUG -- : [knapsack_pro] POST https://api.knapsackpro.com/v1/build_distributions/subset

to post all the tests, then each node starts to run what they have accepted.

However, recently I made some changes for how to generate my rspec tests file, (which I don't think it's important to mention here), but I didn't change the command above at all. Then the rspec takes much longer to run. During the investigation, I found that I have log like this

D, [2021-05-20T19:52:55.641851 #134] DEBUG -- : [knapsack_pro] POST https://api.knapsackpro.com/v1/queues/queue

before running each spec. I think this is suspicious, but on the other way, the queue mode suppose to do that, right? But then why before I don't have those queues http calls?

Please help!!!

knapsack_pro Queue Mode for minitest

Basic info

Queue Mode allocates tests dynamically across CI nodes. Tests are fetched from Knapsack Pro API work queue. Thanks to that the test suite split across CI nodes is optimal and you save time execution of your CI build. We would like to add Queue Mode for minitest in knapsack_pro gem.

How Queue Mode works

If you would like to see how Queue Mode works for RSpec see video at https://knapsackpro.com

Example of Queue Mode for RSpec

In case of RSpec we use RSpec::Core::Runner.run that allows providing new test files that will be executed without reloading the whole app environment.
https://relishapp.com/rspec-staging/rspec-core/docs/running-specs-multiple-times-with-different-runner-options-in-the-same-process

Here is an example how Queue Mode is done in knapsack_pro:
https://github.com/KnapsackPro/knapsack_pro-ruby/blob/master/lib/knapsack_pro/runners/queue/rspec_runner.rb

Example of Regular Mode for minitest

Here is example how minitest works in Regular Mode (deterministic split):
https://github.com/KnapsackPro/knapsack_pro-ruby/blob/master/lib/knapsack_pro/runners/minitest_runner.rb

Ideas how to add Queue Mode to minitest

We could do minitest Queue Mode in the simplest way by defining a new rake task for minitest after each test files batch fetched from work queue but this will be slow due to boot time for each rake task run.

The perfect solution would be to provide just a new list of test files to minitest and run them without the need to boot test runner again. As it is done this way in RSpec.

Minitest tips

The overall minitest structure of a run looks like this:
https://github.com/seattlerb/minitest/blob/master/lib/minitest.rb#L108-L118

How to reset state minitest/minitest#549

Possible issues to solve

can other gems related to minitest affect the queue mode?
which version of minitest will support queue mode?
if minitest supports the running new set of tests without reloading the whole environment, will this have any downsides? Like problems with not cleaned some configs from the previous test files batch run? Similar problem happened for RSpec https://github.com/KnapsackPro/knapsack_pro-ruby#why-when-i-use-queue-mode-for-rspec-then-my-tests-fail (there is link to issue in RSpec project)
will reports collected by minitest work in queue mode?

Queue mode seems to have malformed JUnit formatting

We're running our tests using knapsack-pro's queue mode functionality - and it seems to be mangling the output JUnit format files. Here's an example (elided) of what we see in our junit format at the end of a run:

<?xml version="1.0" encoding="UTF-8"?>
<testsuite name="rspec" tests="1561" skipped="0" failures="0" errors="0" time="11.379601" timestamp="2020-11-24T22:52:53+00:00" hostname="57c67ee37a85">
<properties>
<property name="seed" value="64613"/>
</properties>
<testcase classname="spec.controllers.api.devise.sessions_controller_spec" name="Api::Devise::SessionsController DELETE #destroy when unauthenticated user should return 401" file="./spec/controllers/api/devise/sessions_controller_spec.rb" time="0.034123"></testcase>
...
</testsuite>
.services.order_fulfillment_service_spec" name="OrderFulfillmentService.request_order_fulfillment with existing created fulfillment request created recently behaves like does not create a fulfillment request does not create a new one" file="./spec/services/order_fulfillment_service_spec.rb" time="0.331101"></testcase>
...
</testsuite>

Notice that one of the lines is not fully formed - it appears that KnapsackPro is attempting to merge the results of multiple files or some such, but doesn't seem to be doing it successfully.

We are using knapsack-pro 2.8.0, and running the following command:

bundle exec rake "knapsack_pro:queue:rspec[--tag ~unit --tag ~flake --colour --format Fuubar --format RspecJunitFormatter --out test_reports/junit-report-integration-${TEST_NODE_INDEX}.xml]"

We've even tried setting export KNAPSACK_PRO_MODIFY_DEFAULT_RSPEC_FORMATTERS=false in our build scripts prior to running the rake, but to no avail. Any idea what's happening here?

CLI ENV variables not passed by rake task

When I use something like this for local testing:

RAILS_ENV=test CI=true mkdir -p test-reports/cucumber && bundle exec rake "knapsack_pro:cucumber[--format junit --out test-reports/cucumber/junit.xml]"

The RAILS_ENV and CI variable changes aren't passed through to my running code. This is only a trivial bug, but it took me some time to figure out that the rake task wasn't allowing these through as you would see when running the equivalent RAILS_ENV=test CI=true bundle exec rake cucumber.

Include error messages from dry run json report in stdout when knapsack_pro:rspec_test_example_detector fails

I'm trying to enable KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES, but every node in CI fails with the error here. The "actionable error message" above the error suggests running a command on the CI machine to reproduce the problem, e.g.

To reproduce the error triggered by the RSpec, please try to run below command (this way,
you can find out what is causing the error):
bundle exec rspec --format json --dry-run --out \
.knapsack_pro/test_case_detectors/rspec/rspec_dry_run_json_report_node_0.json \
--default-path spec foo.rb bar.rb

When I do, the command completes successfully, every time, so I can't tell why it's failing during automated runs. It's really frustrating to not be able to reproduce the issue. Why not capture the output of the command in knapsack and include it with the error message if that command fails?

Queue Mode Retry

Queue mode is awesome. Being able to distribute a test suite in real time so that nodes finish in roughly equal time is amazing.

We've had one small snag. Running a big integration test suite is bound to result in flakey test failures. On Buildkite we run our rspec integration tests in parallel, so there are many jobs all running a piece of the suite concurrently. Turning on queue mode works great, and results in a very speedy test run. But if one of those parallel jobs has a failure, when it finishes we can re-run just that particular job to retry just that piece of the test suite. Some of the other jobs might still be running. But re-running that same job, with the same rspec seed, doesn't run the same tests.

Is there some way to make sure that re-running the same node in a particular build in queue mode runs the same piece of the test suite again, if it has completed once before?

Split by example => uninitialized constant KnapsackPro::BaseAllocatorBuilder::RSpec

When using the following with Knapsack Pro 2.1.0 I receive an uninitialized constant exception:

KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES=true DISABLE_SPRING=true bundle exec rake knapsack_pro:rspec

Output/Trace:

+ bundle exec rake knapsack_pro:rspec
--
  | .......rake aborted!
  | NameError: uninitialized constant KnapsackPro::BaseAllocatorBuilder::RSpec
  | /bundler_cache/ruby/2.6.0/gems/bootsnap-1.4.5/lib/bootsnap/load_path_cache/core_ext/active_support.rb:79:in `block in load_missing_constant'
  | /bundler_cache/ruby/2.6.0/gems/bootsnap-1.4.5/lib/bootsnap/load_path_cache/core_ext/active_support.rb:8:in `without_bootsnap_cache'
  | /bundler_cache/ruby/2.6.0/gems/bootsnap-1.4.5/lib/bootsnap/load_path_cache/core_ext/active_support.rb:79:in `rescue in load_missing_constant'
  | /bundler_cache/ruby/2.6.0/gems/bootsnap-1.4.5/lib/bootsnap/load_path_cache/core_ext/active_support.rb:58:in `load_missing_constant'
  | /bundler_cache/ruby/2.6.0/gems/knapsack_pro-2.1.0/lib/knapsack_pro/base_allocator_builder.rb:37:in `fast_and_slow_test_files_to_run'
  | /bundler_cache/ruby/2.6.0/gems/knapsack_pro-2.1.0/lib/knapsack_pro/allocator_builder.rb:5:in `allocator'
  | /bundler_cache/ruby/2.6.0/gems/knapsack_pro-2.1.0/lib/knapsack_pro/runners/base_runner.rb:10:in `initialize'
  | /bundler_cache/ruby/2.6.0/gems/knapsack_pro-2.1.0/lib/knapsack_pro/runners/rspec_runner.rb:8:in `new'
  | /bundler_cache/ruby/2.6.0/gems/knapsack_pro-2.1.0/lib/knapsack_pro/runners/rspec_runner.rb:8:in `run'
  | /bundler_cache/ruby/2.6.0/gems/knapsack_pro-2.1.0/lib/tasks/rspec.rake:5:in `block (2 levels) in <main>'
  | /bundler_cache/ruby/2.6.0/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'

I believe the problem persists in Queue mode, though with a different exception:


I, [2020-07-17T17:19:30.163122 #830]  INFO -- : [knapsack_pro] Generating RSpec test examples JSON report for slow test files to prepare it to be split by test examples (by individual 'it's. Thanks to that a single slow test file can be split across parallel CI nodes). Analyzing 33 slow test files.
--
  | ..........rake aborted!
  | There was problem to generate test examples for test suite
  | /bundler_cache/ruby/2.6.0/gems/knapsack_pro-2.1.0/lib/knapsack_pro/test_case_detectors/rspec_test_example_detector.rb:36:in `generate_json_report'

<...snipped logs...>

Tasks: TOP => knapsack_pro:rspec_test_example_detector
--
  | (See full trace by running task with --trace)
  | rake aborted!
  | Could not generate JSON report for RSpec. Rake task failed when running RACK_ENV=test RAILS_ENV=test bundle exec rake knapsack_pro:rspec_test_example_detector
  | /bundler_cache/ruby/2.6.0/gems/knapsack_pro-2.1.0/lib/knapsack_pro/base_allocator_builder.rb:53:in `fast_and_slow_test_files_to_run'
  | /bundler_cache/ruby/2.6.0/gems/knapsack_pro-2.1.0/lib/knapsack_pro/queue_allocator_builder.rb:5:in `allocator'
  | /bundler_cache/ruby/2.6.0/gems/knapsack_pro-2.1.0/lib/knapsack_pro/runners/queue/base_runner.rb:15:in `initialize'
  | /bundler_cache/ruby/2.6.0/gems/knapsack_pro-2.1.0/lib/knapsack_pro/runners/queue/rspec_runner.rb:14:in `new'
  | /bundler_cache/ruby/2.6.0/gems/knapsack_pro-2.1.0/lib/knapsack_pro/runners/queue/rspec_runner.rb:14:in `run'
  | /bundler_cache/ruby/2.6.0/gems/knapsack_pro-2.1.0/lib/tasks/queue/rspec.rake:6:in `block (3 levels) in <main>'
  | /bundler_cache/ruby/2.6.0/gems/rake-13.0.1/exe/rake:27:in `<top (required)>'
  | /usr/local/bundle/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `load'
  | /usr/local/bundle/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:74:in `kernel_load'
  | /usr/local/bundle/gems/bundler-1.17.3/lib/bundler/cli/exec.rb:28:in `run'
  | /usr/local/bundle/gems/bundler-1.17.3/lib/bundler/cli.rb:463:in `exec'
  | /usr/local/bundle/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
  | /usr/local/bundle/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
  | /usr/local/bundle/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
  | /usr/local/bundle/gems/bundler-1.17.3/lib/bundler/cli.rb:27:in `dispatch'
  | /usr/local/bundle/gems/bundler-1.17.3/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
  | /usr/local/bundle/gems/bundler-1.17.3/lib/bundler/cli.rb:18:in `start'
  | /usr/local/bundle/gems/bundler-1.17.3/exe/bundle:30:in `block in <top (required)>'
  | /usr/local/bundle/gems/bundler-1.17.3/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
  | /usr/local/bundle/gems/bundler-1.17.3/exe/bundle:22:in `<top (required)>'
  | /usr/local/bundle/bin/bundle:23:in `load'
  | /usr/local/bundle/bin/bundle:23:in `<main>'
  | Tasks: TOP => knapsack_pro:queue:rspec
  | (See full trace by running task with --trace)

The removal of the KNAPSACK_PRO_RSPEC_SPLIT_BY_TEST_EXAMPLES fixes the problem.

Documentation: Codeship advice for unique builds requires a new API key

Hi,

I followed the instructions to set KNAPSACK_PRO_CI_NODE_BUILD_ID="" in my Codeship environment, but it causes the test suite to never start. (I would get an ArgumentError about a missing node_build_id).

I changed settings several times in an attempt to determine what was going on, and found that I could only start tests successfully with this setting if I had a brand new SHA or I rotated my Knapsack API key. It might be worth mentioning in the README when giving this instruction to advise users to try the builds with a new key.

Unable to run cucumber - missing requires?

I'm not sure what or where I'm missing it, but it seems that I cannot run cucumber on circleci as it breaks in the junit formatter. This doesn't occur locally with the same Gemfile.lock running bundle exec cucumber.

The undefined object is the step itself.

Any thoughts?

bundle exec rake "knapsack_pro:cucumber"
I, [2017-02-20T17:07:38.408884 #23231]  INFO -- : [knapsack_pro] API request UUID: 
I, [2017-02-20T17:07:38.409048 #23231]  INFO -- : [knapsack_pro] API response:
I, [2017-02-20T17:07:38.409147 #23231]  INFO -- : [knapsack_pro] {"build_distribution_id"=>nil, "node_index"=>0, "test_files"=>[{"path"=>"features/public_pages.feature", "time_execution"=>nil}]}
/opt/circleci/ruby/ruby-2.3.1/bin/ruby -S bundle exec cucumber --require features -- features/public_pages.feature
I, [2017-02-20T17:07:44.631094 #23727]  INFO -- : [knapsack_pro] Test suite time execution recording enabled.
Using the default profile...

undefined method `name' for nil:NilClass (NoMethodError)
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-2.4.0/lib/cucumber/formatter/junit.rb:114:in `hook?'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-2.4.0/lib/cucumber/formatter/junit.rb:105:in `create_output_string'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-2.4.0/lib/cucumber/formatter/junit.rb:62:in `on_after_test_case'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-2.4.0/lib/cucumber/events/bus.rb:27:in `block in notify'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-2.4.0/lib/cucumber/events/bus.rb:27:in `each'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-2.4.0/lib/cucumber/events/bus.rb:27:in `notify'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-2.4.0/lib/cucumber/formatter/event_bus_report.rb:28:in `after_test_case'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-2.4.0/lib/cucumber/formatter/fanout.rb:16:in `block in method_missing'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-2.4.0/lib/cucumber/formatter/fanout.rb:15:in `each'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-2.4.0/lib/cucumber/formatter/fanout.rb:15:in `method_missing'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-core-1.5.0/lib/cucumber/core/test/runner.rb:19:in `test_case'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-core-1.5.0/lib/cucumber/core/test/case.rb:23:in `describe_to'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-2.4.0/lib/cucumber/filters/prepare_world.rb:11:in `test_case'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-core-1.5.0/lib/cucumber/core/test/case.rb:23:in `describe_to'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-2.4.0/lib/cucumber/filters/apply_around_hooks.rb:8:in `test_case'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-core-1.5.0/lib/cucumber/core/test/case.rb:23:in `describe_to'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-2.4.0/lib/cucumber/filters/apply_after_hooks.rb:5:in `test_case'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-core-1.5.0/lib/cucumber/core/test/case.rb:23:in `describe_to'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-2.4.0/lib/cucumber/filters/apply_before_hooks.rb:5:in `test_case'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-core-1.5.0/lib/cucumber/core/test/case.rb:23:in `describe_to'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-2.4.0/lib/cucumber/filters/apply_after_step_hooks.rb:8:in `test_case'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-core-1.5.0/lib/cucumber/core/test/case.rb:23:in `describe_to'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-2.4.0/lib/cucumber/filters/activate_steps.rb:11:in `test_case'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-core-1.5.0/lib/cucumber/core/test/case.rb:23:in `describe_to'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-2.4.0/lib/cucumber/filters/quit.rb:11:in `test_case'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-core-1.5.0/lib/cucumber/core/test/case.rb:23:in `describe_to'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-core-1.5.0/lib/cucumber/core/test/filters/locations_filter.rb:17:in `block in done'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-core-1.5.0/lib/cucumber/core/test/filters/locations_filter.rb:16:in `each'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-core-1.5.0/lib/cucumber/core/test/filters/locations_filter.rb:16:in `done'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-core-1.5.0/lib/cucumber/core/filter.rb:61:in `done'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-core-1.5.0/lib/cucumber/core/test/filters/tag_filter.rb:18:in `done'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-core-1.5.0/lib/cucumber/core/compiler.rb:23:in `done'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-core-1.5.0/lib/cucumber/core/gherkin/parser.rb:35:in `done'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-core-1.5.0/lib/cucumber/core.rb:29:in `parse'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-core-1.5.0/lib/cucumber/core.rb:18:in `compile'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-2.4.0/lib/cucumber/runtime.rb:67:in `run!'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-2.4.0/lib/cucumber/cli/main.rb:32:in `execute!'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/gems/cucumber-2.4.0/bin/cucumber:8:in `<top (required)>'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/bin/cucumber:23:in `load'
/home/ubuntu/af/vendor/bundle/ruby/2.3.0/bin/cucumber:23:in `<top (required)>'

Knapsack not running all the cuke scenarios on the nodes assigned

Hi Arthur,

We have been using travis with Knapsack Pro and I see that lately it is not running all the scenarios on the nodes assigned. Besides I see that sometimes its running the duplicate scenarios. Can you please let me know if there has been any change to the gem or do we need to upgrade our knapsack pro gem version.

Regards
Rajani

Deselect tests to run

Hey there!

You can selectively run some tests by specifying their directory, but is there a way to deselect some tests and keep the others using minitest?

Thanks!

Thread.join causes knapsack to hang on circleCI

Hi, I'm encountering a weird behavior.

My pseudo-code looks like this:

Class MyTask
  def order_now
   # Some DB transactions here

   MyModel.transaction do

    Thread.new do
      ActiveRecord::Base.connection_pool.with_connection do
        # Open another DB connection update progress status while in transaction
      end 
    end.join  

 end
end

And my rspec:

context 'it works' do
  it 'works' do 
      service.call
      expect(service).to be_success
  end
end

This works on my local machine. However, Thread.join cause my CI running with Knapsack to hang indefinitely, only to be terminated after 10-minute by CircleCI. There was no error, but here is the log.

{"severity":"ERROR","thread_id":47130853734740,"process_id":362,"message":"/home//my-project/app/services/custom_reports/export.rb:72:in join'\n/home/********/my-project/app/services/custom_reports/export.rb:72:in block (3 levels) in each_rows'\n/home//my-project/app/services/custom_reports/export.rb:47:in loop'\n/home/********/my-project/app/services/custom_reports/export.rb:47:in block (2 levels) in each_rows'\n/home//my-project/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:267:in block in transaction'\n/home/********/my-project/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.8/lib/active_record/connection_adapters/abstract/transaction.rb:239:in block in within_new_transaction'\n/home//my-project/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in block (2 levels) in synchronize'\n/home/********/my-project/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in handle_interrupt'\n/home//my-project/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in block in synchronize'\n/home/********/my-project/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in handle_interrupt'\n/home//my-project/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.8/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in synchronize'\n/home/********/my-project/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.8/lib/active_record/connection_adapters/abstract/transaction.rb:236:in within_new_transaction'\n/home//my-project/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.8/lib/active_record/connection_adapters/abstract/database_statements.rb:267:in transaction'\n/home/********/my-project/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.8/lib/active_record/transactions.rb:212:in transaction'\n/home//my-project/app/services/custom_reports/export.rb:46:in block in each_rows'\n/home/********/my-project/app/services/custom_reports/export.rb:17:in each'"}

CI is using : gem 'knapsack_pro', '~>2.11.0'.

Is it possible to retry failing specs on another worker?

We use rspec-retry to deal with flakey browser specs. Unfortunately we're currently running into an issue where firefox / selenium / geckodriver crashes and is unrecoverable (we believe it's related to docker memory / too many file handles). All the retry attempts of the spec fail, and any further specs that use the browser also fail.

We're working towards solving the underlying issue, but is there a mechanism with knapsack to retry failed specs on another worker?

We've had a few other issues that have caused an entire worker to be inoperable (like postgres crashing and specs not waiting for it to recover), so I think this is applicable to more than just our current issue.

Ci failing with missing file error

I am experimenting with knapsack pro for a project with circleci. My tests are randomly failing with this error. am i missing some configurations ?

Errno::ENOENT: No such file or directory @ rb_sysopen - .knapsack_pro/tracker/prerun_tests_RSpecAdapter_node_1.json

circleci: Running cucumber with test metadata

My build works with knapsack and without creating test metadata. Next, I want to add the test metadata to the builds for better feedback, though I'm running into some issues.

I'm trying to setup our build and obtain circleci test metadata as described:
https://circleci.com/docs/test-metadata/#cucumber

I've tried various ways of specifying parameters (quotes etc), but in the end I run into:

Configure the build00:07cache

modifiers to 'cucumber' must be a map

Action failed: Configure the build

My yaml has been verified with yamllint, here's the relevant snippet:

test:
  override:
    - mkdir -p $CIRCLE_TEST_REPORTS/cucumber:
      parallel: true

    # Step for rspec
    - bundle exec rake knapsack_pro:rspec['--format progress --format RspecJunitFormatter -o $CIRCLE_TEST_REPORTS/rspec/junit.xml']:
        parallel: true

    # Step for cucumber
    - bundle exec rake knapsack_pro:cucumber['--format pretty --format json --out $CIRCLE_TEST_REPORTS/cucumber/tests.cucumber']:
        parallel: true

How should I be specifying these parameters for rspec and cucumber?

KnapsackPro::Hooks::Queue.before_queue not being called when running multiple nodes

We have KnapsackPro running on Heroku CI to balance our Rspec tests.

Using version 2.6 of the knapsack_pro gem, we found that KnapsackPro::Hooks::Queue.before_queue is failing to be called. We upgraded to the latest gem version, and we still are seeing this issue.

It happens when the formation size is 11
It only fails on one or two threads
By skipping that hook an important portion of the test setup is missing causing that most of the tests fail on that thread.

KnapsackPro::Hooks::Queue.before_queue do |_queue_id|
  # This will be called only once before the tests started on the CI node.
  # It will be run inside of the RSpec before(:suite) block only once.
  # It means you will have access to whatever RSpec provides in the context of the before(:suite) block.
  puts '----BEFORE QUEUE----'
  Rake::Task['util:seed_ci'].invoke
end

Ability to set an exit code for RSpec failure

Hi,

We'd like to fail with a particular exit code when RSpec is unsuccessful. I tried putting the argument in our Rake command with rake "knapsack_pro:rspec[--failure-exit-code=3]" but it didn't seem to take. Is there a way to pass that argument on to RSpec?

This is useful for CI providers because we want to retry for infrastructure failures but not for spec failures, and we can control this by using a custom exit code.

Spring preloading and cucumber queue mode

Hi,

I have been updating our cucumber tests to the new queue mode and encountered one issue.
Because it is executing tests in chunks, we are loosing quite a bit of time on initial rails initialisation each time new chunk is started. Having ability to use spring would help with this issue, but doesn't seem it is possible at the moment with the queue runner.

I found there are 2 ways to implement this:

Having some sort of env var like KNAPSACK_PRO_CUCUMBER_QUEUE_SPRING_ENABLED which would transform command to a bundle exec spring cucumber.
Having env var similar to cucumber rake task option binary, which would allow to replace executed cucumber binary with a springified binstub.

WDYT? Maybe there are some other ways?

Run Fallback Mode when Knapsack Pro API returns 5xx error

Related: #67

Ruby 2.6 not supported

Getting this error once I updated to ruby 2.6.0

Randomized with seed 34743
--
  | .............................................................................................................................
  | An error occurred in an `after(:suite)` hook.
  | Failure/Error:
  | def initialize(io, read_timeout: 60, continue_timeout: nil, debug_output: nil)
  | @read_timeout = read_timeout
  | @rbuf = ''.dup
  | @debug_output = debug_output
  |  
  | @io = case io
  | when Socket, OpenSSL::SSL::SSLSocket, IO
  | io
  | when StringIO
  | PatchedStringIO.new(io.string)
  |  
  | ArgumentError:
  | unknown keyword: write_timeout
  | # /usr/local/bundle/gems/webmock-3.4.2/lib/webmock/http_lib_adapters/net_http.rb:259:in `initialize'
  | # /usr/local/bundle/gems/webmock-3.4.2/lib/webmock/http_lib_adapters/net_http.rb:109:in `request'
  | # /usr/local/bundle/gems/knapsack_pro-1.2.0/lib/knapsack_pro/client/connection.rb:91:in `post'
  | # /usr/local/bundle/gems/knapsack_pro-1.2.0/lib/knapsack_pro/client/connection.rb:12:in `call'
  | # /usr/local/bundle/gems/knapsack_pro-1.2.0/lib/knapsack_pro/report.rb:59:in `create_build_subset'
  | # /usr/local/bundle/gems/knapsack_pro-1.2.0/lib/knapsack_pro/report.rb:11:in `save'
  | # /usr/local/bundle/gems/knapsack_pro-1.2.0/lib/knapsack_pro/adapters/rspec_adapter.rb:49:in `block (2 levels) in bind_save_report'
  | D, [2019-01-07T12:40:14.168012 #14] DEBUG -- : [knapsack_pro] Global time execution for tests: 42s

knapsack_pro depends on knapsack

So why not specify that in the .gemspec? That way, I can simply add gem 'knapsack_pro to my Gemfile instead of both?

Can Knapsack_Pro read rspec tests from a file? (a question, not an issue)

What I am doing:

I integrated Crystalball project with Knapsack_Pro. Here is how they work: Crystalball output the list of tests that needs to be ran because they are connected to the ruby code that have been changed by developers, so we only run selected portion of rspec tests. Now Knapsack_Pro need to take this list into KNAPSACK_PRO_TEST_FILE_LIST to start running them in parallel.

The code to run it is something like below:

        KNAPSACK_PRO_FIXED_QUEUE_SPLIT=true \
        KNAPSACK_PRO_PROJECT_DIR=. \
        KNAPSACK_PRO_CI_NODE_TOTAL=$CI_NODE_TOTAL \
        KNAPSACK_PRO_CI_NODE_INDEX=$(($CI_NODE_INDEX-1)) \
        KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=<the token> \
        KNAPSACK_PRO_TEST_FILE_LIST=$(cat tmp/crystalball_prediction) \
        bundle exec rake "knapsack_pro:queue:rspec[--format documentation --format RspecJunitFormatter --out tmp/rspec.xml]"

NOTE: the crystalball_prediction file is the output of Crystalball project.

However I am facing an issue that the number of tests in this file is sometimes too large, so I got argument list too large issue like this: /usr/local/bin/bundle: Argument list too long. I believe the tests being cat out is too large then it is over the ARG_MAX of the nodes. However I didn't get luck to increase the stack to make it work.

Therefor I am wondering if there is a way to ask knapsack_pro read test cases from a file? so I don't have to print it into the parameter. Thanks.

Unable to run with a only a rspec "include" tags specified, they are ignored and all non-matching specs are run

Hello,

The README says you can do this, with the following example:

# run only tests with tagA
KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=api_key_for_tagA bundle exec rake "knapsack_pro:rspec[--tag tagA]"

But we are not able to make it work. When we specify only two single tags to run, they is ignored with the message "All examples were filtered out; ignoring..." and rspec ends up running everything:

I, [2017-05-09T00:22:17.628995 #793]  INFO -- : [knapsack_pro] bundle exec rspec -t uses_extra_session --format progress  --default-path spec "spec/selenium/projects/stories/comments_spec.rb"
D, [2017-05-09T00:22:18.152576 #793] DEBUG -- : [knapsack_pro] Test suite time execution queue recording enabled.
Run options: include {:focus=>true, :uses_extra_session=>true}
 
All examples were filtered out; ignoring {:focus=>true, :uses_extra_session=>true}
...
<Knapsack Pro ends up running all non-matching examples>

We tried setting the following Rspec config:
config.run_all_when_everything_filtered = false

...but this did not help. Plus, this option is going away anyway (see rspec/rspec-core#2266).

This is with Rspec v 3.4.4.

Thanks,
-- Chad

Trying knapsack, but not able to pass arguments to rspec

This is how I have rspec builds configured on semaphore bundle exec rspec --tag ~ci_skip -f p spec but I am not able to pass the arguments to knapsack:

bundle exec rake knapsack_pro:rspec[--tag ~ci_skip -f p]
rake aborted!
No Rakefile found (looking for: p])

(See full trace by running task with --trace)

Error if spec fails before it really begins

knapsack_pro 0.32.0

Potentially affected by: rspec-retry

A preflight error (caused by selenium) triggers an error inside knapsack_pro, presumably because the start time hasn't been set yet.

Backtrace:

Exception: [TypeError] nil can't be coerced into Float
Backtrace:
/bundle/gems/knapsack_pro-0.32.0/lib/knapsack_pro/tracker.rb:21:in `-'
/bundle/gems/knapsack_pro-0.32.0/lib/knapsack_pro/tracker.rb:21:in `stop_timer'
/bundle/gems/knapsack_pro-0.32.0/lib/knapsack_pro/adapters/rspec_adapter.rb:36:in `block (2 levels) in bind_time_tracker'
/bundle/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:443:in `instance_exec'
/bundle/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:443:in `instance_exec'
/bundle/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:357:in `run'
/bundle/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:507:in `block in run_owned_hooks_for'
/bundle/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:506:in `each'
/bundle/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:506:in `run_owned_hooks_for'
/bundle/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:593:in `block in run_example_hooks_for'
/bundle/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:592:in `each'
/bundle/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:592:in `run_example_hooks_for'
/bundle/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:463:in `run'
/bundle/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:503:in `run_after_example'
/bundle/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:269:in `block in run'
/bundle/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:496:in `block in with_around_and_singleton_context_hooks'
/bundle/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:453:in `block in with_around_example_hooks'
/bundle/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:464:in `block in run'
/bundle/gems/rspec-core-3.5.4/lib/rspec/core/hooks.rb:604:in `block in run_around_example_hooks_for'
/bundle/gems/rspec-core-3.5.4/lib/rspec/core/example.rb:338:in `call'
/bundle/bundler/gems/rspec-retry-e2334456bd68/lib/rspec/retry.rb:112:in `block in run'
/bundle/bundler/gems/rspec-retry-e2334456bd68/lib/rspec/retry.rb:101:in `loop'
/bundle/bundler/gems/rspec-retry-e2334456bd68/lib/rspec/retry.rb:101:in `run'
/bundle/bundler/gems/rspec-retry-e2334456bd68/lib/rspec_ext/rspec_ext.rb:12:in `run_with_retry'
/app/spec/spec_helper.rb:345:in `block (2 levels) in <top (required)>'

Add configuration to write to /tmp directory instead of .knapsack_pro

The change in 3.1.0 to use .knapsack_pro directory for temporary files instead of the tmp directory in the user's project directory is causing problems for me.

In the environment where I run tests I have the tmp directory configured to be writable and the root directory configured to be readonly.

Is it possible to add configuration to retain the behaviour prior to the changes in 3.1.0?

I imagine the default behaviour could be to use the .knapsack_pro directory with an optional configuration setting to explicitly set the directory or perhaps a flag to specify that the tmp directory should be used. Of course, adding this configuration makes my life simpler and your code more complicated.

Support for using Queue mode with minitest/spec

We use minitest/spec from https://github.com/seattlerb/minitest together with minitest, and that breaks Queue mode, because context, before, etc aren't defined:

D, [2019-11-15T14:44:55.150743 #9554] DEBUG -- : [knapsack_pro] API request UUID: 9166cd89-c25f-4c42-b103-abb3b2d38822
D, [2019-11-15T14:44:55.150854 #9554] DEBUG -- : [knapsack_pro] API response:
D, [2019-11-15T14:44:55.150921 #9554] DEBUG -- : [knapsack_pro] {"queue_name"=>"987:3924a8d106184efbc955b3890d762fd1", "build_subset_id"=>nil, "test_files"=>[{"path"=>"test/system/subscriber_notifications/subscriber_notifier_test.rb", "time_execution"=>nil}]}
trying to run ["test/system/subscriber_notifications/subscriber_notifier_test.rb"]
D, [2019-11-15T14:45:01.478340 #9554] DEBUG -- : [knapsack_pro] Test suite time execution queue recording enabled.
rake aborted!
NoMethodError: undefined method `context' for SubscriberNotifierTest:Class
Did you mean?  const_set
/home/caspark/src/status-page-web/test/system/subscriber_notifications/subscriber_notifier_test.rb:8:in `<class:SubscriberNotifierTest>'
/home/caspark/src/status-page-web/test/system/subscriber_notifications/subscriber_notifier_test.rb:5:in `<main>'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `require'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:21:in `block in require_with_bootsnap_lfi'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/loaded_features_index.rb:65:in `register'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:20:in `require_with_bootsnap_lfi'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bootsnap-1.3.2/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:29:in `require'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `block in require'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:257:in `load_dependency'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/activesupport-5.2.3/lib/active_support/dependencies.rb:291:in `require'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/knapsack_pro-1.15.0/lib/knapsack_pro/runners/queue/minitest_runner.rb:89:in `block in minitest_run'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/knapsack_pro-1.15.0/lib/knapsack_pro/runners/queue/minitest_runner.rb:88:in `each'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/knapsack_pro-1.15.0/lib/knapsack_pro/runners/queue/minitest_runner.rb:88:in `minitest_run'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/knapsack_pro-1.15.0/lib/knapsack_pro/runners/queue/minitest_runner.rb:67:in `run_tests'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/knapsack_pro-1.15.0/lib/knapsack_pro/runners/queue/minitest_runner.rb:30:in `run'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/knapsack_pro-1.15.0/lib/tasks/queue/minitest.rake:6:in `block (3 levels) in <main>'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bundler-2.0.2/lib/bundler/cli/exec.rb:74:in `load'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bundler-2.0.2/lib/bundler/cli/exec.rb:74:in `kernel_load'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bundler-2.0.2/lib/bundler/cli/exec.rb:28:in `run'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bundler-2.0.2/lib/bundler/cli.rb:465:in `exec'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bundler-2.0.2/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bundler-2.0.2/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bundler-2.0.2/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bundler-2.0.2/lib/bundler/cli.rb:27:in `dispatch'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bundler-2.0.2/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bundler-2.0.2/lib/bundler/cli.rb:18:in `start'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bundler-2.0.2/exe/bundle:30:in `block in <top (required)>'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bundler-2.0.2/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/gems/bundler-2.0.2/exe/bundle:22:in `<top (required)>'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/bin/bundle:23:in `load'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/bin/bundle:23:in `<main>'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/bin/ruby_executable_hooks:24:in `eval'
/home/caspark/.rvm/gems/ruby-2.6.3@status-page/bin/ruby_executable_hooks:24:in `<main>'
Tasks: TOP => knapsack_pro:queue:minitest
(See full trace by running task with --trace)

We do require 'minitest/spec' as one of the first things in our test_helper.

The regular Knapsack Pro runner works fine, but it looks like the Knapsack Pro Queue runner reimplements minitest's test running.

Disable banner and disable knapsack locally

I'm in a team who uses knapsack, and locally we always get a banner telling us about knapsack pro, etc.

We have an (ancient) knapsack pro report committed else knapsack blows up with errors about that file not being a file ENOENT.

Is there a way to keep knapsack loaded but disable it? (I searched for env vars that might do the trick, and found none, and not much in the way of doce)

Or is the preferred solution to avoid loading knapsack all together in environments where it's not used?

Feature requests(adding --seed & --bisect to rspec debug command)

Could you please add the --seed #{seed_number} and --bisect so we don't have to manually add these flags to debug the CI?

timing resets on feature branches

Hi, I've been running Knapsack Pro on Buildkite for a few days now. I just noticed that a feature branch I created directly off our latest master branch seems to have forgotten all about timing info when its specs ran.

Each of the 20 parallel builds showed timing values of nil for every single file they're running:

** Invoke knapsack_pro:rspec (first_time)
** Execute knapsack_pro:rspec
I, [2016-05-10T22:20:04.894834 #52]  INFO -- : [knapsack_pro] API request UUID: 293b1b17-9d09-4798-b1ed-11196faa48e0
I, [2016-05-10T22:20:04.894932 #52]  INFO -- : [knapsack_pro] API response:
I, [2016-05-10T22:20:04.895297 #52]  INFO -- : [knapsack_pro] {"node_index"=>3, "test_files"=>[{"path"=>"spec/features/happy_path_spec.rb", "time_execution"=>nil}, {"path"=>"spec/models/addresses_response_spec.rb", "time_execution"=>nil}, ...truncated...]}
I, [2016-05-10T22:20:05.478307 #57]  INFO -- : [knapsack_pro] Test suite time execution recording enabled.

Is this supposed to happen? I would have expected Knapsack to start with the timing from the latest master build rather than resetting every time. If this is how it's supposed to behave, I don't think it will be very useful to us.

GITHUB_SHA

How can we use GITHUB_SHA when using with Github Actions? Are you going to release an adapter for it?

Queue mode causes rspec_junit_formatter to produce invalid XML when a node runs a spec with all examples commented out.

This looks similar to #40, but we have applied the fix from that thread, and this persists. We've also previously had a similar issue that I filed as #171 that turned out to be caused by specs calling Kernel#exit, but we've since cleaned those up, and are still seeing JUnit XML issues with a new pattern.

This time, we see invalid XML generated when reporting on specs that run in queue mode immediately after another spec that has been entirely commented out. It doesn't always happen, but it has killed 5 builds for us this week. In the example I'm including here, it first runs spec/features/posts/rsvp_spec.rb, which has been temporarily commented out:

D, [2022-08-31T19:55:30.478298 #24812] DEBUG -- : [knapsack_pro] {"queue_name"=>"2796:8c3e59cb04b0ae2e857330d792c31576", "build_subset_id"=>nil, "test_files"=>[{"path"=>"spec/features/posts/rsvp_spec.rb", "time_execution"=>0.0}]}
I, [2022-08-31T19:55:30.478651 #24812]  INFO -- : [knapsack_pro] To retry the last batch of tests fetched from the API Queue, please run the following command on your machine. (If you use the `-- order random` option, remember to add correct `--seed 123` that you can find at the end of the RSpec output.)
I, [2022-08-31T19:55:30.478688 #24812]  INFO -- : [knapsack_pro] bundle exec rspec --format documentation --format RspecJunitFormatter --out tmp/junit/rspec.xml --failure-exit-code 0  --default-path spec "spec/features/posts/rsvp_spec.rb"
No examples found.

...then asks for more tests from the queue, and gets assigned spec/features/phone_sized_spec.rb, which it runs:

D, [2022-08-31T19:55:30.713903 #24812] DEBUG -- : [knapsack_pro] {"queue_name"=>"2796:8c3e59cb04b0ae2e857330d792c31576", "build_subset_id"=>nil, "test_files"=>[{"path"=>"spec/features/phone_sized_spec.rb", "time_execution"=>3138.279317547999}]}
I, [2022-08-31T19:55:30.714145 #24812]  INFO -- : [knapsack_pro] To retry the last batch of tests fetched from the API Queue, please run the following command on your machine. (If you use the `-- order random` option, remember to add correct `--seed 123` that you can find at the end of the RSpec output.)
I, [2022-08-31T19:55:30.714180 #24812]  INFO -- : [knapsack_pro] bundle exec rspec --format documentation --format RspecJunitFormatter --out tmp/junit/rspec.xml --failure-exit-code 0  --default-path spec "spec/features/phone_sized_spec.rb"

...it asks for more tests and then finishes when there are none left:

D, [2022-08-31T20:50:49.743891 #24812] DEBUG -- : [knapsack_pro] {"queue_name"=>"2796:8c3e59cb04b0ae2e857330d792c31576", "build_subset_id"=>nil, "test_files"=>[]}
Knapsack Pro Queue finished!

...then when jenkins tries to parse the JUnit XML, it fails on:

Failed to read test report file /home/ubuntu/workspace/T-453_path-to-pro-styling-issues/tmp/junit/node_2-rspec_final_results.xml
org.dom4j.DocumentException: Error on line 117 of document  : Content is not allowed in trailing section.
	at org.dom4j.io.SAXReader.read(SAXReader.java:511)
	at org.dom4j.io.SAXReader.read(SAXReader.java:392)
	at hudson.tasks.junit.SuiteResult.parse(SuiteResult.java:177)
	at hudson.tasks.junit.TestResult.parse(TestResult.java:384)
	at hudson.tasks.junit.TestResult.parsePossiblyEmpty(TestResult.java:314)
	at hudson.tasks.junit.TestResult.parse(TestResult.java:256)
	at hudson.tasks.junit.TestResult.parse(TestResult.java:242)
	at hudson.tasks.junit.TestResult.parse(TestResult.java:220)
	at hudson.tasks.junit.TestResult.<init>(TestResult.java:174)
	at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:176)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3502)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
	at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.xml.sax.SAXParseException; lineNumber: 117; columnNumber: 1; Content is not allowed in trailing section.
	at java.xml/com.sun.org.apache.xerces.internal.util.ErrorHandlerWrapper.createSAXParseException(ErrorHandlerWrapper.java:204)
...

Attachments:

Ability to ignore some CI nodes

I run multiple tasks on CirrusCI, and I use knapsack_pro only for rspec. I have two other tasks which are unrelated but since they're counted in CI_NODE_TOTAL, knapsack_pro thinks there's no timing data for these nodes.

(only 7 out of 9 nodes are actually run rspec tests)

I'd like to have the ability to ignore these nodes in knapsack_pro somehow.

JUnit files are appended to with RSpec formatters

Hello, found an interesting case while using this gem.

I'm trying to add JUnit formatter to use the results in Jenkins using the junit instruction. When using queue mode with this gem (and service) however, Knapsack might run rspec tests multiple times from the same process. This is causes a problem because --out in RSpec appends to a file. So that means you end up with a file that has something along the lines of:

<xml ...>
<testcase stuff in here />
<xml> # next suite run starts and appends...

Jenkins (and from what I can see, XML parsers in general) don't like when there are multiple <xml> tags in the file, as you can see from this error:

Failed to read test report file /workspace/HCM_PR-7675-355EZNVL3NLOS25MQ7X2QIWUKL7VGXOWVAWXENJHPGTJ4IJEY4FA@5/tmp/rspec_results7c4385c84.xml
org.dom4j.DocumentException: Error on line 2175 of document file:///workspace/HCM_PR-7675-355EZNVL3NLOS25MQ7X2QIWUKL7VGXOWVAWXENJHPGTJ4IJEY4FA@5/tmp/rspec_results7c4385c84.xml : The processing instruction target matching "[xX][mM][lL]" is not allowed.

The more important piece is this:

The processing instruction target matching "[xX][mM][lL]" is not allowed.

Detail on the error is in this stackoverflow: http://stackoverflow.com/questions/19889132/error-the-processing-instruction-target-matching-xxmmll-is-not-allowed

I can see an "after hook" of sorts be helpful here because we could just rename the file afterwards to something that won't be appended to and Jenkins can read the multiple files just fine.

Thoughts?

Filtering knapsackpro records

We were running our tests with a cucumber tag, which was preventing some of our tests from getting noticed and timed by knapsack. This was a good thing.

Accidentally, we ended up running all our tests in travis and knapsackpro timed them all. Subsequent runs with the tag are resulting in knapsackpro returning a list of tests it wants to run that were not tagged, which means, no tests run in some nodes and travis fails for the node.

In order to reset, do we just need to get a new token? Is there way to tell knapsackpro to stop using a particular set of tests?

I should add here that I thought about using KNAPSACK_PRO_TEST_FILE_PATTERN because all the tests we want to run are in a subdir, but we are running both rspec and cucumber tests. I took a look at the code and, as far as I can tell, if I set this for cucumber, rspec will pick it up as well and fail because it won't find rspec files in the directory.

Add test file exclude pattern

Example:

When you would like to run tests only from spec/features directory then run:

KNAPSACK_PRO_TEST_SUITE_TOKEN_RSPEC=$API_TOKEN_FOR_FEATURE_TESTS \
KNAPSACK_PRO_TEST_DIR=spec \
KNAPSACK_PRO_TEST_FILE_PATTERN="spec/features/**{,/*/**}/*_spec.rb" \
bundle exec rake knapsack_pro:queue:rspec

Problem:

What if you would like to run all tests except the spec/features directory?

We could use KNAPSACK_PRO_TEST_FILE_PATTERN and use a pattern that works with Dir.glob but sometimes it's hard to exclude something.

For now users have to specify all directories they want to run tests from.
https://github.com/KnapsackPro/knapsack_pro-ruby#how-to-run-only-rspec-feature-tests-or-non-feature-tests

Idea (todo):

We could add new variable KNAPSACK_PRO_TEST_FILE_EXCLUDE_PATTERN
and then you could do this:

files = Dir.glob(KNAPSACK_PRO_TEST_FILE_PATTERN)
files -= Dir.glob(KNAPSACK_PRO_TEST_FILE_EXCLUDE_PATTERN)

Enhancement for travis: Record TRAVIS_BUILD_NUMBER per TRAVIS_BRANCH

Travis has the ability to rerun nodes. We have sporadic failures and for a while, we have been using the Travis node rerun functionality. However, we figured out the other day that Knapsack doesn't necessarily return the same tests that it returned the last time the node was run. Which is a good thing because then it would never optimize. However, ideally, we'd like knapsack to recognize when we're rerunning a node and return the set of tests it ran for that node.

I'd propose that, in order to make it do this, you could accept the TRAVIS_BUILD_NUMBER and into knapsack and record it for each one and, if you already have that build number, send back the same set of tests. That would make travis re-run behave as expected. To minimize the amount of data you need to archive, you could do it per branch (TRAVIS_BRANCH) and only save the last run per branch and that would still be extremely useful.

What do you say?

VCR issues when running with knapsack

I am facing a strange VCR issue when trying to run knapsack (Regular and queued mode both).

VCR debug logs show that somehow VCR is not picking up/initializing cassettes to replay.

Sample with knapsack: https://gist.github.com/vipulnsward/5d9d616dacb1e0d9b0e5fb4694652f7a
Sample without knapsack: https://gist.github.com/vipulnsward/ab1657cbb5ccfd447f541fb8c58785d1

Notice the difference that with knapsack, this line is missing:

Initialized HTTPInteractionList with request matchers ...

This is called from https://github.com/vcr/vcr/blob/33757dfc6cabc1204e7a209676dfee9b098133d3/lib/vcr/cassette.rb#L103, so I am leaning towards this being some mutex/thread context issue?

This happens randomly. Upto a certain point cassettes work properly, suddenly they don't and it continues to fail after that(probably corruption in some config) till the specs are over.

Wondering if someone has faced or seen any similar issues?

Can't disable Webmock / VCR when running rake task

I'm trying to run the rake knapsack_pro:rspec task on my dockerized Buildkite fleet. However, Webmock is getting in the way:

** Invoke knapsack_pro:rspec (first_time)
** Execute knapsack_pro:rspec
D, [2016-05-06T05:57:53.344852 #55] DEBUG -- : ** [Raven] Event not sent due to excluded environment: test
rake aborted!
WebMock::NetConnectNotAllowedError: Real HTTP connections are disabled. Unregistered request: POST http://api.knapsackpro.com/v1/build_distributions/subset with body 

...redacted

============================================================/usr/local/bundle/gems/webmock-1.21.0/lib/webmock/http_lib_adapters/net_http.rb:114:in `request'
/usr/local/lib/ruby/2.3.0/net/http.rb:1421:in `send_entity'
/usr/local/lib/ruby/2.3.0/net/http.rb:1209:in `post'
/usr/local/bundle/gems/knapsack_pro-0.7.0/lib/knapsack_pro/client/connection.rb:75:in `post'
/usr/local/bundle/gems/knapsack_pro-0.7.0/lib/knapsack_pro/client/connection.rb:11:in `call'
/usr/local/bundle/gems/knapsack_pro-0.7.0/lib/knapsack_pro/allocator.rb:19:in `test_file_paths'
/usr/local/bundle/gems/knapsack_pro-0.7.0/lib/knapsack_pro/runners/base_runner.rb:14:in `test_file_paths'
/usr/local/bundle/gems/knapsack_pro-0.7.0/lib/knapsack_pro/runners/base_runner.rb:18:in `stringify_test_file_paths'
/usr/local/bundle/gems/knapsack_pro-0.7.0/lib/knapsack_pro/runners/rspec_runner.rb:9:in `run'
/usr/local/bundle/gems/knapsack_pro-0.7.0/lib/tasks/rspec.rake:5:in `block (2 levels) in <top (required)>'
/usr/local/bundle/gems/rake-10.5.0/lib/rake/task.rb:240:in `block in execute'
/usr/local/bundle/gems/rake-10.5.0/lib/rake/task.rb:235:in `each'
/usr/local/bundle/gems/rake-10.5.0/lib/rake/task.rb:235:in `execute'
/usr/local/bundle/gems/rake-10.5.0/lib/rake/task.rb:179:in `block in invoke_with_call_chain'
/usr/local/lib/ruby/2.3.0/monitor.rb:214:in `mon_synchronize'
/usr/local/bundle/gems/rake-10.5.0/lib/rake/task.rb:172:in `invoke_with_call_chain'
/usr/local/bundle/gems/rake-10.5.0/lib/rake/task.rb:165:in `invoke'
/usr/local/bundle/gems/rake-10.5.0/lib/rake/application.rb:150:in `invoke_task'
/usr/local/bundle/gems/rake-10.5.0/lib/rake/application.rb:106:in `block (2 levels) in top_level'
/usr/local/bundle/gems/rake-10.5.0/lib/rake/application.rb:106:in `each'
/usr/local/bundle/gems/rake-10.5.0/lib/rake/application.rb:106:in `block in top_level'
/usr/local/bundle/gems/rake-10.5.0/lib/rake/application.rb:115:in `run_with_threads'
/usr/local/bundle/gems/rake-10.5.0/lib/rake/application.rb:100:in `top_level'
/usr/local/bundle/gems/rake-10.5.0/lib/rake/application.rb:78:in `block in run'
/usr/local/bundle/gems/rake-10.5.0/lib/rake/application.rb:176:in `standard_exception_handling'
/usr/local/bundle/gems/rake-10.5.0/lib/rake/application.rb:75:in `run'
/usr/local/bundle/gems/rake-10.5.0/bin/rake:33:in `<top (required)>'
/usr/local/bundle/bin/rake:17:in `load'
/usr/local/bundle/bin/rake:17:in `<main>'
Tasks: TOP => knapsack_pro:rspec

I believe I'm ignoring requests to this hostname correctly in my spec_helper.rb:

# Setup VCR
VCR.configure do |config|
  config.cassette_library_dir = 'spec/fixtures/vcr'
  config.hook_into :webmock
  # allow requests to localhost and knapsack pro:
  config.ignore_hosts 'localhost', '127.0.0.1', '0.0.0.0', 'api.knapsackpro.com'
end

Do I need to do something else to get VCR & webmock out of the way for the knapsack_pro rake task?

Support for Github Actions job summaries

Github Actions has a new features that allows to add arbitrary markdown code for job summaries. It would be great if Knapsack Pro would support that out of the box when running on Github.

https://github.blog/2022-05-09-supercharging-github-actions-with-job-summaries/

This is a library for dotnet that has already support for summaries: https://github.com/Tyrrrz/GitHubActionsTestLogger

Knapsack may not be compatible with Cucumber 1.2.5

I actually got this error with Knapsack as well. I'm hoping you know what it means and can help me resolve it.

undefined method scenario_outline' for #<Cucumber::Ast::Scenario:0x000000073044e8> (NoMethodError) /home/travis/build/<path removed>/vendor/bundle/ruby/2.2.0/gems/knapsack_pro-0.6.1/lib/knapsack_pro/adapters/cucumber_adapter.rb:15:inAround'

rspec_junit_formatter still producing invalid XML after applying suggestion from #40

I'm working on enabling queue mode and splitting tests by examples for my project, and I ran into what looks like the same problem in issue #40. At first, I had invalid XML from every node, with cutoff text outside of XML tags. I found this article and tried applying the suggestion, but I still see invalid XML on 2 out of 26 CI nodes. Looking at the XML, I see duplicate <?xml version="1.0" encoding="UTF-8"?> declarations halfway through the file, and no closing </xml>.

My build script calls the following rake task:

rake "knapsack_pro:queue:rspec[--format documentation --format RspecJunitFormatter --out tmp/junit/rspec.xml --failure-exit-code 0]"

spec_helper.rb has this:

TMP_RSPEC_XML_REPORT = 'tmp/junit/rspec.xml'.freeze
FINAL_RSPEC_XML_REPORT = "tmp/junit/node_#{ENV.fetch('KNAPSACK_PRO_CI_NODE_INDEX')}-rspec_final_results.xml".freeze

KnapsackPro::Hooks::Queue.after_subset_queue do
  if File.exist?(TMP_RSPEC_XML_REPORT)
    FileUtils.mv(TMP_RSPEC_XML_REPORT, FINAL_RSPEC_XML_REPORT)
  end
end

Hooks for queue mode start/finish

We're using Percy.io for our visual regression testing, and they depend on before(:suite) and after(:suite) hooks to notify their backend when a build starts and completes by counting the number of begin and finalise calls that match the total node count. However, the suite hooks fire between queue chunk runs. Is there hooks that let me have these hooks only be run once per node?

Skipping some specs

We use Knapsack Pro with Buildkite and we have currently 8 nodes running our whole test suite, works pretty great. Except for one thing;

We have noticed that when a node fails and you click retry on that specific node another set of specs get assigned.

Is there a way to set a fixed seed value so all nodes get the same specs every time you rebuild for a certain version (maybe based on the git sha)?

Readme syntax

Under the section that briefly mentions "linters" it uses the $CI_NODE_TOTAL variable, where I believe it's supposed to be the index $CI_NODE_INDEX variable provided by Heroku CI.

Happy to open a PR if that's easier to review!

Expand Spring support for Rspec & regular mode

I saw a previous request for spring support to speed up executions, and that it had been implemented specifically for cucumber in queue mode: #98

Is it possible to expand support of this so that it can be utilized for Rspec in non-queue (regular) mode?

Happy to also fork and put up a PR myself