Aha! Saving the artifact gets me this:

{
	"version": "3.8.2",
	"messages": ["\nAn error occurred while loading ./spec/features/spaces/account_creation/onboarding_visit_public_network_course_spec.rb.\nFailure/Error: require_dependency('premailer') # premailer must be manually invoked for batch renders. premailer inlines the email css\n\nLoadError:\n  No such file to load -- premailer\n# ./app/mailers/batch_mailer.rb:2:in `\u003ctop (required)\u003e'\n# ./app/mailers/batch_invite_mailer.rb:1:in `\u003ctop (required)\u003e'\n# ./config/environment.rb:9:in `\u003ctop (required)\u003e'\n# ./spec/spec_helper.rb:2:in `\u003ctop (required)\u003e'\n# ./spec/features/spaces/account_creation/onboarding_visit_public_network_course_spec.rb:1:in `\u003ctop (required)\u003e'\n", "\nAn error occurred while loading ./spec/features/onboarding_visit_public_network_course_spec.rb.\nFailure/Error: require File.expand_path('../../config/environment', __FILE__)\n\nRuntimeError:\n  can't modify frozen Array\n# ./config/environment.rb:9:in `\u003ctop (required)\u003e'\n# ./spec/spec_helper.rb:2:in `\u003ctop (required)\u003e'\n# ./spec/features/onboarding_visit_public_network_course_spec.rb:1:in `\u003ctop (required)\u003e'\n", "No examples found."],
	"examples": [],
	"summary": {
		"duration": 4.3535e-05,
		"example_count": 0,
		"failure_count": 0,
		"pending_count": 0,
		"errors_outside_of_examples_count": 2
	},
	"summary_line": "0 examples, 0 failures, 2 errors occurred outside of examples"
}

edit: Nevermind. I had accidentally commented out too much of the build step while trying to narrow this down and commented out the precompile step.

from knapsack_pro-ruby.

I still think it would be helpful to print those messages out along with the top level error message, rather than encouraging users to repro and then go digging through internal knapsack files.

from knapsack_pro-ruby.

I've prepared the improvement to show the content of the JSON report so that it's easier to debug an error when the error message is printed only inside of the JSON report.
#172

from knapsack_pro-ruby.

@cccCody I've released the knapsack_pro gem 3.3.0 version.

from knapsack_pro-ruby.

What's the content of the .knapsack_pro/test_case_detectors/rspec/rspec_dry_run_json_report_node_0.json file?
Was it generated on the disk when you run the command in the CI environment?

Was the .knapsack_pro directory generated on the CI? Maybe it's a permissions error.

What CI provider do you use?

Why not capture the output of the command in knapsack and include it with the error message if that command fails?

This is something to investigate. I was wondering about printing the error a long time ago, but it was more complex.
There were a few approaches to this and how to make it secure. https://stackoverflow.com/questions/690151/getting-output-of-system-calls-in-ruby

The bundle exec rspec --format json --dry-run --out ... command writes a file on the disk, and most of the time, it does not produce any output, so there was no reason to capture the output.

Please share the full output of the error that you see. You can also email our support at https://knapsackpro.com/contact if the error contains private data (not to share it publicly here).

from knapsack_pro-ruby.

The files do get generated when I run the command manually. I'll send a few samples of them to you by e-mail along with the build log since they might be considered sensitive.

We're using the latest version of Jenkins LTS, and we run our build on single-use EC2 nodes via the Amazon EC2 Jenkins plugin, so if our build script doesn't specifically archive a file as an artifact, it's gone at the end of a build. l'll update our build script to archive .knapsack_pro and report back on that.

I know there are way too many ways to capture stdout in ruby, but this answer in the question you've linked has been my preference: https://stackoverflow.com/a/20001569/1431252

from knapsack_pro-ruby.

I took a look at a couple of nodes and saw the file being generated, even though it was reported as having failed.
I also see that they look like they have the same content on different nodes, even though I'd expect them to list different tests. For example, on the same run, both slow_test_files_node_21.json and slow_test_files_node_26.json have the same content:

[{"path":"spec/features/spaces/account_creation/onboarding_visit_public_network_course_spec.rb","time_execution":2227.537136728004},{"path":"spec/features/onboarding_visit_public_network_course_spec.rb","time_execution":2157.940042964002}]

The error messages show them looking at the same files too:

E, [2022-06-21T22:16:31.209892 #5518] ERROR -- : [knapsack_pro] ---------- START of actionable error message --------------------------------------------------
E, [2022-06-21T22:16:31.209937 #5518] ERROR -- : [knapsack_pro] There was a problem while generating test examples for the slow test files using the RSpec dry-run flag. To reproduce the error triggered by the RSpec, please try to run below command (this way, you can find out what is causing the error):
E, [2022-06-21T22:16:31.209951 #5518] ERROR -- : [knapsack_pro] bundle exec rspec --format json --dry-run --out .knapsack_pro/test_case_detectors/rspec/rspec_dry_run_json_report_node_21.json --default-path spec spec/features/spaces/account_creation/onboarding_visit_public_network_course_spec.rb spec/features/onboarding_visit_public_network_course_spec.rb
E, [2022-06-21T22:16:31.209964 #5518] ERROR -- : [knapsack_pro] ---------- END of actionable error message --------------------------------------------------

E, [2022-06-21T22:16:55.720815 #5596] ERROR -- : [knapsack_pro] ---------- START of actionable error message --------------------------------------------------
E, [2022-06-21T22:16:55.720859 #5596] ERROR -- : [knapsack_pro] There was a problem while generating test examples for the slow test files using the RSpec dry-run flag. To reproduce the error triggered by the RSpec, please try to run below command (this way, you can find out what is causing the error):
E, [2022-06-21T22:16:55.720873 #5596] ERROR -- : [knapsack_pro] bundle exec rspec --format json --dry-run --out .knapsack_pro/test_case_detectors/rspec/rspec_dry_run_json_report_node_26.json --default-path spec spec/features/spaces/account_creation/onboarding_visit_public_network_course_spec.rb spec/features/onboarding_visit_public_network_course_spec.rb
E, [2022-06-21T22:16:55.720885 #5596] ERROR -- : [knapsack_pro] ---------- END of actionable error message --------------------------------------------------

from knapsack_pro-ruby.

I still think it would be helpful to print those messages out along with the top level error message, rather than encouraging users to repro and then go digging through internal knapsack files.

I agree. I'm going to work on improving that.

technical note for future reference

I found out that if the rake knapsack_pro:rspec_test_example_detector command fails when executed by Kernel.system then the error is already printed to the output.
Here is the code:
https://github.com/KnapsackPro/knapsack_pro-ruby/blob/50f969646c8cb92333f70715604bebb8852e72f3/lib/knapsack_pro/base_allocator_builder.rb#L51,L54
I guess this code is fine.

The rake task rake knapsack_pro:rspec_test_example_detector executes the ::RSpec::Core::Runner and when it fails it is supposed to print an error to $stderr.

When ::RSpec::Core::Runner fails then knapsack_pro gem prints actionable error and a command name to reproduce the error manually (this is what happend in case of this issue #170).

https://github.com/KnapsackPro/knapsack_pro-ruby/blob/50f969646c8cb92333f70715604bebb8852e72f3/lib/knapsack_pro/test_case_detectors/rspec_test_example_detector.rb#L38,L43

Something to figure out:

Is ::RSpec::Core::Runner actually printing an error message or not? Maybe sometimes ::RSpec::Core::Runner can fail with no error message to stderr and that's the expected behavior of ::RSpec::Core::Runner. That is why we see no error even the ::RSpec::Core::Runner failed.

The only error message could be inside of a generated file .knapsack_pro/test_case_detectors/rspec/rspec_dry_run_json_report_node_0.json. In such a case, I could detect if the file exists and print its content to the output. This way the user could see what's the error inside of the JSON file (no need for the user to cache it as artifacts on CI).

from knapsack_pro-ruby.