avmnu-sng / rspec-tracer Goto Github PK

When in presence of a flaky test spec-tracer will completely negate the information it gives (it's flakiness) because once the test is a success it will not be run until one of the dependent files is run again.

I know flaky tests is a difficult subject to deal with because if the flaky test just works on the first pass it will remain hidden but at least if it fails on the first run I think it is possible to detect this as a flaky test and keep running it (and maybe even inform the user that there are flaky tests in the test suite)

I wrote this quickly without fully testing my reproduction steps because this is happening in my work project and I cannot simply copy private company code as an example but it should work out the same in theory.

Another way to fix this more simply would be to add the possibility to configure a "always run these tests" option which would allow us to run critical tests every time no matter what the cache says.

Expected behavior

If a test fails and then succeeds on a rerun with no change in the dependencies then the test should always be run.

Actual behavior

Once the flaky test succeeds it will not be rerun

Steps to reproduce the problem

1. The model

class FlakyModel < ApplicationRecord
	def flaky_method
		Random.rand(2)
	end

	def predictable_method
		1
	end
end

2. The spec

require 'rails_helper'

RSpec.describe FlakyModel, type: :model do
	let(:instance) { described_class.new }

	it 'is flaky' do
		expect(instance.flaky_method).to eq(0)
	end

	it 'is not flaky' do
		expect(instance.predictable_method).to eq(1)
	end
end

3. Run specs until failure on flaky test with RSPEC_TRACER_NO_SKIP=true
4. Rerun until flaky test succeeds without RSPEC_TRACER_NO_SKIP=true
5. Rerun again and notice nothing will run

RSpec Tracer version

ruby version 3.0.1
rspec-core version 3.10.1
rspec-tracer version 0.4.0

Is your feature request related to a problem? Please describe.

When running specs in parallel using parallel_tests it gets into a race condition to write the cache file. To understand what happens, download this repo and run parallel_rspec spec. Make sure you don't run the test suite directly with rspec before parallel, because it will cache the result and this problem will not happen.

Describe the solution you'd like

It should gracefully handle multiple instances of rspec-tracer writing the cache.

Describe alternatives you've considered

My original project is 8 years old and it's on Rails 5.2, so running the specs in parallel locally are only possible using parallel_specs. We want to take advantage of the caching system rspec-tracer provides to decrease the time we take running the full test suite.

Codacy detected an issue:

Message: Unnecessary symbol conversion; use :"_#{method_name}" instead.

Occurred on:

• Commit: 1e25d17
• File: lib/rspec_tracer/configuration.rb
• LineNum: 41
• Code: alias_method "_#{method_name}".to_sym, method_name

Currently on:

• Commit: 568c5e7
• File: lib/rspec_tracer/configuration.rb
• LineNum: 41

Is your feature request related to a problem? Please describe.

Problem: A developer modifies a view, runs rspec, if the tests are already cached, no tests are run (with the recommended config in https://github.com/avmnu-sng/rspec-tracer#getting-started).

Describe the solution you'd like

If a erb|slim|jbuilder file is modified, their related specs should be invalidated so tests can run again.

Describe alternatives you've considered

Right now, what some do is rm -rf rspec_tracer_cache/*, and run the tests they want, but they have to remember to do so.

Some other rails projects put rspec-tracer behind a environment flag to control when rspec-tracer runs with rspec, but they have to remember to set it.

# ./spec/spec_helper.rb
require 'simplecov'
SimpleCov.start

if ENV['RSPECACHE']
  require 'rspec_tracer'
  RSpecTracer.start
end

@jwoodrow Tracking it separately.

From #15 (comment):

• Do you mean bundle exec rspec before starting the execution takes around 10 minutes?

Basically it takes ~10 minutes to go from this line to the next

RSpec tracer loaded cache from /home/travis/build/path_to_project/rspec_tracer_cache/6783bc6a970fee584760b32d3498355b

RSpec tracer is running xx examples (actual: xxxx, skipped: xxxx)

but when running things locally it takes much less time to do this same step (3m30 basically) so maybe this is also linked to the limited performance available using CI tools

Is your feature request related to a problem? Please describe.

When RSpecTracer can determine the group examples for a single spec, it "gives up" and outputs the "DO NOT USE RSPEC TRACER" message. I still want to get value from having it on other files, so I want to be able to exclude them until they are fixed. I assumed that this configuration existed when using add_filter but RSpecTracer still runs to that file even when I include them in the filter.

Describe the solution you'd like

Have some sort of:

RSpecTracer.start do
    ignore ['spec/to/ignore_spec.rb']
end

Describe alternatives you've considered

I tried using add_filter as following:

RSpecTracer.start do
    add_filter 'spec/to/ignore_spec.rb'
end

But when I ran rspec spec/to/ignore_spec.rb it would still try to run for that file.

Is your feature request related to a problem? Please describe.

Right now, we don't check if the provided S3 is empty before attempting to download the cache.

Describe the solution you'd like

This was a PR proposing this implementation: #55

Describe alternatives you've considered

I believe this change will increase the performance by checking if the bucket is empty before attempting to download the cache.

Hi @avmnu-sng

I've looked more into my travis issues and noticed that on travis vendor was being added to the examples_coverage.json file which made it take up nearly a whole Gig uncompressed. I've changed the add filters to avoir coverage of these files so hopefully this addresses this situation. Maybe it should be stressed somewhere in the documentation that it is recommended not to exclude vendor from the coverage for rspec-tracer and that behaviour can differ between CI and local when it comes to these folders.

Another thing is the load could be greatly increased I think by introducing a breaking change and moving away from json and using messagepack instead here is an old benchmark I found when it came to sereliazing content and it is also faster when it comes to deserializing. It would also decrease the cache size a significant amount.

Finally would it be possible to add a "single cache history" mode ? Because currently it seems like it acumulates caches for each commit, no ? I think most usages especially when using CI can use a single cache solution since the worst case scenario would be that a few tests would be rerun but even then I don't think this would happen 🤔

Originally posted by @jwoodrow in #15 (comment)

Codacy detected an issue:

Message: Detected non-static command inside system.

Occurred on:

• Commit: 22da8d8
• File: lib/rspec_tracer/remote_cache/aws.rb
• LineNum: 16
• Code: system(

Currently on:

• Commit: 568c5e7
• File: lib/rspec_tracer/remote_cache/aws.rb
• LineNum: 16

@avmnu-sng I had to fix one of your changes in 0.6.0 because you had the following code in the wrong order

seconds, remainder = seconds.divmod(count) # remainder is a Number
remainder = format_duration(remainder) # remainder becomes a String

next if remainder.zero? # String does not have a zero? method

which should actually be

seconds, remainder = seconds.divmod(count)
next if remainder.zero?

remainder = format_duration(remainder)

Originally posted by @jwoodrow in #22 (comment)

Expected behavior

Using the last version of SimpleCov (0.21.2) I should be able to do bundle install successfully.

Actual behavior

It fails because SimpleCov depends on docile ~> 1.1 and rspec-tracer on docile ~> 1.1.0.

Steps to reproduce the problem

Add the last version of SimpleCov and rspec-tracer on the gemfile.

0.2.0

Include Ruby, RSpecTracer, and RSpec version. Also include
Simplecov version if applicable.

Is your feature request related to a problem? Please describe.

When running rspec-tracer with parallel_tests (via turbo_tests to sync outputs) the rspec-tracer logs' output inserts itself in the middle of the output to notify it's done and that's it's generating coverage.

Describe the solution you'd like

Log once at the end of all parallel_specs.

Maybe this could be a good place to look or what turbo_tests implemented to sync the outputs

Describe alternatives you've considered

A config variable that would allow the silencing of all logs from rspec-tracer

Expected behavior

I would like RSpec Tracer to allow me to break from the middle of the suite using ByeBug exit.

Actual behavior

If I exit in the middle of the suite, my caches are crippled and I need to remove them.

Steps to reproduce the problem

Add byebug in any specs. Run suite, exit from buybug console.

(byebug) exit


Pending: (Failures listed here are expected and do not affect your suite's status)
...
RSpec tracer is generating reports
RSpec tracer reports written to <...>/rspec_tracer_cache/9d2558b01c8894c7c5dba273fbdcd589 (took 0.02917 seconds)
Traceback (most recent call last):
	8: from /Users/sergeymoiseev/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/rspec-tracer-0.9.1/lib/rspec_tracer/defaults.rb:15:in `block in <top (required)>'
	7: from /Users/sergeymoiseev/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/rspec-tracer-0.9.1/lib/rspec_tracer.rb:79:in `at_exit_behavior'
	6: from /Users/sergeymoiseev/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/rspec-tracer-0.9.1/lib/rspec_tracer.rb:181:in `run_exit_tasks'
	5: from /Users/sergeymoiseev/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/rspec-tracer-0.9.1/lib/rspec_tracer.rb:193:in `generate_reports'
	4: from /Users/sergeymoiseev/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/rspec-tracer-0.9.1/lib/rspec_tracer.rb:193:in `new'
	3: from /Users/sergeymoiseev/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/rspec-tracer-0.9.1/lib/rspec_tracer/html_reporter/reporter.rb:15:in `initialize'
	2: from /Users/sergeymoiseev/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/rspec-tracer-0.9.1/lib/rspec_tracer/html_reporter/reporter.rb:58:in `format_examples'
	1: from /Users/sergeymoiseev/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/rspec-tracer-0.9.1/lib/rspec_tracer/html_reporter/reporter.rb:58:in `each_pair'
/Users/sergeymoiseev/.rbenv/versions/2.6.6/lib/ruby/gems/2.6.0/gems/rspec-tracer-0.9.1/lib/rspec_tracer/html_reporter/reporter.rb:64:in `block in format_examples': undefined method `[]' for nil:NilClass (NoMethodError)
Coverage report generated for RSpec to <...>/coverage. 2683 / 4858 LOC (55.23%) covered.
Lcov style coverage report generated for RSpec to <...>/coverage/lcov/coverage.lcov
Stopped processing SimpleCov as a previous error not related to SimpleCov has been detected

$ bundle exec rspec
Started RSpec tracer

An error occurred while loading spec_helper.
Failure/Error: RSpecTracer.start

NoMethodError:
  undefined method `transform_keys!' for nil:NilClass
# ./spec/spec_helper.rb:22:in `<top (required)>'

Started RSpec tracer

An error occurred while loading rails_helper.
Failure/Error: RSpecTracer.start

NoMethodError:
  undefined method `transform_keys!' for nil:NilClass
# ./spec/spec_helper.rb:22:in `<top (required)>'
# ./spec/rails_helper.rb:7:in `<top (required)>'
No examples found.


Finished in 0.00008 seconds (files took 6.97 seconds to load)
0 examples, 0 failures, 2 errors occurred outside of examples

Coverage report generated for RSpec to <...>/coverage. 681 / 915 LOC (74.43%) covered.
Lcov style coverage report generated for RSpec to <...>/coverage/lcov/coverage.lcov
Stopped processing SimpleCov as a previous error not related to SimpleCov has been detected

RSpec Tracer version

ruby 2.6.6p146

• rails (6.1.4.1)
• rspec (3.10.0)
• rspec-collection_matchers (1.2.0)
• rspec-core (3.10.1)
• rspec-expectations (3.10.1)
• rspec-mocks (3.10.2)
• rspec-nc (0.3.0)
• rspec-rails (5.0.1)
• rspec-support (3.10.2)
• rspec-tracer (0.9.1)
• rspec_junit_formatter (0.4.1)
• simplecov (0.21.2)
• simplecov-html (0.12.3)
• simplecov-lcov (0.8.0)
• simplecov_json_formatter (0.1.3)

I followed the README and I'm getting this error

LoadError:
  cannot load such file -- rspec-tracer

Caching works fine as long as the previous run commit SHA is part of the git tree. If you remove this (for example, do git commit --amend), there won't be any cache.