danmayer / coverband Goto Github PK

https://bugs.ruby-lang.org/issues/4796

Looks like this issue has been resolved. Has it been resolved adequately to allow using the Coverage class?

I do not have a root cause why it cannot parse json (probably uploaded some invalid baseline to our staging server), but it does not swallow JSON.parse errors, throwing:

JSON::ParserError: 784: unexpected token at ':22,"20":22,"24":22}}'

The error is not catched by coverband and makes webserver fail with 500 error.
It should probably be swallowed and reported in coverband log.

This gem sounds great and I want to use it. I'm just not sure how.

The documentation seems to have most of the parts, but it lacks an overall explanation of work flow.

I've added the gem to my gemfile, made the tmp/coverband_baseline.json file, added the rake tasks and the middleware. Then I wanted to test it locally, but the README kind of stops there.

The coverband_baseline rake task seemed like a good place to start, so I ran it (with --trace). A bunch of json spat out and no errors, so hurray. I did notice, however, that the filenames listed in the json only included my initializers.

With the baseline taken, I assumed the only other thing to do was run the coverband rake task, so I did. When it was finished, my browser popped open to show me a lovely simplecov page. The only problem was that it only listed my initializer files.

Could someone explain to me 1) what the ideal coverband workflow is (add all the stuff from the readme, deploy it, run the baseline, then the coverband tasks?) and 2) any reason why the rest of my production code was missed out?

I'd be happy to submit a PR with more setup instructions in the README once I get my head wrapped around these concepts.

It seems that Coverband.report only works if redis is configured and fails with an exception message if it isn't

/Users/omer.katz/.rvm/gems/ruby-2.2.2@mygemset/gems/coverband-1.1/lib/coverband/reporter.rb:104:in `get_current_scov_data_imp': undefined method `smembers' for nil:NilClass (NoMethodError)
    from /Users/omer.katz/.rvm/gems/ruby-2.2.2@mygemset/gems/coverband-1.1/lib/coverband/reporter.rb:123:in `report_scov'
    from /Users/omer.katz/.rvm/gems/ruby-2.2.2@mygemset/gems/coverband-1.1/lib/coverband/reporter.rb:42:in `report'
    from ruby_file.rb:23:in `<main>

At the very least I think that the method should fail with a proper message.
I can make a pull request for that if you want to.
There's another option though. Maybe we should allow reporting coverage from the in-memory store if redis is not configured.

1. I run rake coverband:clear, coverband:baseline, coverband:coverage
2. I check that Coverband says my method foobar was not executed
3. I go to a page that executes foobar
4. I run coverband:coverage and check that Coverband says foobar has been executed once
5. I go to that page that executes foobar again (with old/new cookies, in the same/different browser, doesn't matter)
6. I run coverband:coverage, and Coverband tells me foobar was still only executed once.

In fact, I can't seem to get any part of a model or view to report an execution count > 1. Have you ever experienced this?

The coverband README warns against running rake coverband:baseline in production. I would like to schedule something like this to run every couple of weeks:

rake coverband:clear 
rake coverband:baseline 

Whats the danger of running coverband:baseline in production?

We are running rails on AWS, and using redis.
Also have a customized baseline that observes the config.ignore setting.
So I have everything working on my localhost, and can execute clear, custom_baseline, and coverage operations generating the expected results.
Yesterday we pushed this to production, but I can't seem to be able to pull coverage from production. On my localhost I have hard coded the config to match our production values for config.redis including the appropriate url, port, and db#. I have logged in to our redis instance, and it seems to be recording activity from production as it reflects changes as I move through the app in production. But on my localhost if I run rake coverband:coverage with these hard coded production settings generates output reflecting files in production. However, it generates a 0% coverage report. As I said I the redis instance has data in it like this:

That changes appropriately as I activate the corresponding code from our production app.
Not exactly sure what I am doing wrong here...

with this code:

# blah.rb
def foo(x)
    if x == 2
        blah = 5 
    else
        blah = 7 
    end 

    return blah
end

# test.rb
...

coverband.start

foo 1
foo 2

coverband.stop
coverband.save
Coverband::Reporter.report

I get

It seems to me like this should be 100% covered, and those lines marked as uncovered are incorrect.

Both the simplecov walk through and Jruby coverage bug links are broken.

As I am sure you are aware coverband_ext is a useful addition to this gem which speeds up sampling https://github.com/danmayer/coverband_ext
They use method which was recently deleted from your codebase as no longer used
They use it here:
https://github.com/danmayer/coverband_ext/blob/master/ext/coverband_ext/coverband_ext.c#L49
It was deleted in this commit:
2f80301
I am not sure whether you are the ones who should fix it, but simply adding it back does solve the problem for me (I monkey patch coverband gem)

I'm using Coverband::Middleware with Redis. Our app is hosted on Heroku. I used smaller Redis instance and when we had bigger traffic we got Redis::CommandError - ERR max number of clients reached - this exception caused that the web request has failed. It was thrown by calling Coverband::Base.instance.report_coverage at vendor/bundle/ruby/2.3.0/gems/coverband-1.5.1/lib/coverband/middleware.rb:13.

I'm afraid that just upgrading the Redis instance is not the solution because when we have a spike in traffic it's possible that even the biggest instance won't help. Is it possible to rescue all these exceptions to ensure that the web request will never be affected? Maybe I missed something in the documentation so please let me know if there is anything I can do. Our priority is to ensure the request will never fail - It's okay to skip the reporting for that request.

I can provide more detailed logs if needed.

I've noticed that actually generating and sharing the coverage report to the rest of the team can be a bit awkward. Code coverage data is stored within our production redis. In order to generate the report, i have to copy that data to my local redis instance. Then i run the coverband:coverage task locally. This is a bit confusing as the path to the files on my local instance is different than in production. It seems to work but looking at the code, i think this may not work for others depending on their coverband settings.

Once i have the coverband report locally generated, now i have to figure out how to share it. The actual coverage data is quite sensitive data given that it has the details of our codebase.

I am not sure of the exact solution to the problem, but i think it would be neat if we could generate and expose the report within production. Since we are on Heroku, we cannot generate it to the local file system. We could generate and store on s3 or another remote file system.

Curious if others have noticed this issue.

thx!

It would be great if this was an option. I'd like to submit a PR with this in it, but am having a hard time slicing in the right spot. Can you a)let me know if this is something you'd be willing to incorporate and b)point me in the right direction to implement it?

from Brian E.

Dan Mayer: wondering if it would be appropriate to ensure HOME is set before this line:

My thoughts:

Basically Coverband always loads SimpleCov which assumes a HOME is set and not all deployed environments have that. Really SimpleCov isn't needed in production at all so we if split the gem and drop the SimpleCov dependancies on the gem, but only have it on the Reporter which isn't loaded in production I think we could reduce the production dependancies.

It would be great if you could use coverband in a way similar to

my_coverband = Coverband.new( configuration ) # or similar

loop do
  my_coverband.sample {
    # code to sample
  }
end

my_coverband.report( io )

Or if might be an issue with the sampling code being in a block --

my_coverband = Coverband.new( configuration ) # or similar

loop do
  my_coverband.start
  # code to sample
  my_coverband.stop
end

my_coverband.report( io )

That way coveband could be used by any application not just a rack based one. 😄

I've got it working for my Rails app, and also for background jobs with a Sidekiq middleware. But for cron jobs that run a Rake task, I'm thinking about how I can wrap those with Coverband.

I have this in my lib/tasks/coverband.rake:

require 'coverband'
Coverband.configure
require 'coverband/tasks'

# Wrap all Rake tasks with Coverband
current_tasks = Rake.application.top_level_tasks
if current_tasks.any? && current_tasks.none? { |t| t.to_s.match?(/^coverband:/) }
  current_tasks.unshift 'coverband:start'
  current_tasks.push 'coverband:stop_and_save'
end

namespace :coverband do
  task :start do
    Coverband::Base.instance.start
  end

  task :stop_and_save do
    Coverband::Base.instance.stop
    Coverband::Base.instance.save
  end
end

This code wraps all Rake tasks with Coverband start/stop/save. Seems to work ok but not sure if that's the best way

Hey dude!

I noticed that this config doesn't ignore app/assets

config.root_paths = ['/app/']
config.ignore = ['vendor', 'spec', 'test', 'app/assets']

I dug in a bit and commenting this line out "fixes" that: https://github.com/danmayer/coverband/blob/master/lib/coverband/reporter.rb#L150

I would've thought the baseline process sets the files that get tracked, no? Do you see any immediate problems removing that line?

Is there an API to read the coverage statistics from Redis?
Some other method?

We're running into an issue where coverband reports data incorrectly. Here's a screenshot that illustrates the problem.

Coverband says that the class is 100% covered, but I know that I didn't make enough calls to exercise the entire controller, and there are many lines that indicate they've never been run, but that don't count as being "missed" for some reason.

This particular controller lives outside of /app/ in a directory that is treated as an "internal gem", so to speak. That directory is pulled into the Gemfile like this:

gemspec path: 'userpages'

To get any coverage data at all for anything in that directory I added this to my coverband config.

config.include_gems = true
config.root_paths = ['/app/','/userpages/']

Could you please shed some light into why creating a baseline is important? The README explains what baseline does but not why.

I understand that rails is restarting the application on each request (so changes can be loaded) -- https://github.com/rails/rails/blob/92703a9ea5d8b96f30e0b706b801c9185ef14f0e/railties/lib/rails/generators/rails/app/templates/config/environments/development.rb.tt#L4-L6.

My solution to trying coverband out in a development environment is to set the startup_delay to 0. Otherwise it never decrements (https://github.com/danmayer/coverband/blob/master/lib/coverband/base.rb#L60) since https://github.com/danmayer/coverband/blob/master/lib/coverband/base.rb#L38-L56 is executed on each request.

Should this be mentioned in the README somewhere? Or should I be doing something different to test this in development?

I am running the rake coverband:coverage task within heroku and updating to s3. I noticed the file path is cut off for anything within the app directory.

I am wondering if coverband/scov is getting confused because heroku deploys the app within a /app directory so the directory ends up being /app/app.

Either way, it's a bit difficult to debug as the issue only happens within heroku. Was wondering if anyone else noticed it.

thx.

See screnshot

I've followed all of the README instructions for installing and configuring coverband and when I run rake coverband:coverage I'm seeing some weird stuff in the coverage report.

What happens is that a browser window will open that shows some amount of coverage, but if I hit refresh, the coverage drops to 0%.

In the terminal where I run the rake task I can see that two reports are being generated.

$ rake coverband:coverage
...
Coverage report generated for RSpec to .../my-app/coverage. 105 / 1666 LOC (6.3%) covered.
Coverage report generated for RSpec to .../my-app/coverage. 0 / 2025 LOC (0.0%) covered.

I've tried following the same process in two different projects, and I'm seeing the same in both.

Readme mentions that it has something to do with initialization of app but it does not help. Why it is json and it's loaded on startup ?

I'm just sitting here wondering what is happening when I run rake coverband:baseline. It never exits. I think there is a problem with my configuration, but I'm not sure what it is. Any ideas? Thanks in advance.

This causes SimpleCov to filter out files on it's reports.
I expected that when I set Coverband's root SimpleCov's root will also be set.

Can you guys check this?

I've tried run here the 1.1 version and doesn't work. Installed v1.0.x and works fine.

error happening:
uninitialized constant Coverband::Base::TracePoint

I've been trying to use coverband on a Rails application running on the Puma webserver using the example configuration, but coverage was always disabled.

After digging around for a while I noticed that Coverband::Base holds a @delay instance variable which gets decreased on each request until it hits zero, at which point coverage is enabled. Puma, however spawns a new thread for each request, so a new instance of Coverband::Base is created with its @delay variable set to the original number specified in the config. This way, it never reaches zero and coverage is always disabled.

I don't know if there's an easy (and threadsafe) fix for this, but I think it should at least be documented in the README. Everything seems to work fine with startup_delay set to 0.

Please let me know if there's anything I'm missing.

I'd be happy to help with either docs or code!

First: Let me be clear - this is not actually the gems fault. The gem has a relatively minor security "flaw" which resulted in an outsize result in our environment because of poor practices / bad luck on our end.

In the baseline task, Coverband loads (and because it's Ruby, executes) every .rb file under both /lib and /app - including potentially destructive scripts that are not loaded by the regular load process.

In our case, we had lib/<submodule/db/schema.rb files from an old submodule that were still present in Capistrano's cached copy. Although we had run Coveralls on staging environments to verify it's operation, since that file was only present on a single production server, when we executed the baseline task on that server, it was loaded and forced schema creation, dropping existing tables.

Although this is definitely one of those "swiss cheese holes align" kind of problems - I think Coverband could benefit from being more intentional about what files it executes while establishing the baseline coverage.

Which of the following approaches would be most compatible with Coverband's overall philosophy?

1. Do not descend into submodule directories, and do not execute Ruby found there.
2. Maintain a 'blacklist' of "dangerous" files that should never be executed.
3. Open the database in "read only" mode while establishing a baseline.
4. Hook into the existing Rails eager load - which should eager load all files referenced by the code base without loading potentially dangerous 'sidecar' scripts.

Hi! How can i use simplecov profile to generate html reports, groups, filters, and so on? For example:

SimpleCov.profiles.define 'test' do
       add_filter '/gems/'
       add_filter '/coverage/'
       add_filter '/reports/'
       add_group 'Features', 'features'
       add_group 'Applications123', 'app'
       add_group 'Core', 'src/core'
end
    
     SimpleCov.start 'test'

I have tried to set up Coverband in my rails project (as a middleware) following the readme's instructions. I managed to create an HTML report for the baseline (using the rake task), but I'm not able to create a report for the requests done while my app is running.

Running rake middleware shows that the middleware is loaded, because I see use Coverband::Middleware in the middleware listing output.

My hypothesis was that Coverband was never started, because I see coverage disabled in my rails server console output for every request. So I tried to force a start with Coverband::Base.instance.start in the config.ru file, but it didn't change anything.

Also, I noticed that rake coverband:clear coverband:coverage gives me the same report as the report for the baseline. But I would have expected no coverage at all (no lines hit whatsoever).

So then I tried to force it another way. I added an around filter in my application_controller:

around_filter :force_coverage

def force_coverage
  Coverband::Base.instance.start
  yield
  Coverband::Base.instance.stop
  Coverband::Base.instance.save
end

And finally, that gave me some coverage at runtime. But from what I understand, it shouldn't be required to add that kind of code.

Can you help me figure out what am I missing?

I was just trying to include Coverband in our app but was wondering why the output shows always empty files and no coverage was detected.

coverband file usage: []

After printing out the files in the TracePoint callback, I saw, that no app files got passed along but some suspicious files with a bootsnap prefix.
I removed bootsnap setup from our config/boot.rb and tried again and all files would be logged. My next idea was to upgrade Bootsnap and everything still works now...

Maybe was related to and old version of bootsnap (Upgraded from 1.1.2 to 1.3.0.) or Bootsnap cache.

I just would like to have this issue for other people that google that problem

It's just an idea I have a long time ago, but may be you've already seen something like does the same coverage, but for the locale strings or files? Sometimes you can find, that in the project, there are a lots of translations that aren't used already for a while. It will be cool to clean them out of the locale files.

baseline = if File.exist?('./tmp/coverband_baseline.json')
JSON.parse(File.read('./tmp/coverband_baseline.json'))
else
{}
end

Make the method take a file path or default to the above... I have been adding this into pretty much all the apps that use cover band.

The main reason why I decided to give coverband a try is to be able to delete useless code located in the views of a rails project. The report generated after 1 or 2 requests on my app shows various coverage for the touched controllers and models, but shows 100% coverage for all the views touched. However, I know this is not the case (I have if/else blocks in some of those views, and I know for sure that both branches of those if/else blocks could not have been touched). My impression is that as long as 1 line of a view is "touched", the report will say the whole view has been 100% covered.

Also, from the readme – although probably not directly related to my issue – I can read:

the line numbers reported for ERB files are often off and aren't considered useful. I recommend filtering out .erb using the config.ignore option.

My question: is coverband an appropriate tool to evaluate line coverage in my rails views?

I'm trying to get coverband working on heroku so I can't just write to the coverage directory. Is it possible to change the output directory to something like /tmp/coverage for now so I can test the report?

I noticed that coverband only shows files that have been hit at least once. This leaves out all of the files that were never hit and the number one candidates for deletion. Was thinking it would be better to go through the full file system to produce the report instead of just relying on the set of files within redis. Here is a hacked branch i used to get it done:

https://github.com/danmayer/coverband/compare/master...kbaum:show_uncovered_files?w=0

Obviously hardcoding the Dir.glob inclusion path is not the right way to go, but i wanted to see if others thought this is the general way to go.

thx!

If I have gem 'coverband', '~> 1.1', in my Rails 4.2.7 Gemfile, and I try to set up s3_bucket as follows:

Coverband.configure do |config|
  config.s3_bucket = ENV['AWS_BUCKET']
end

I get undefined method `s3_bucket' for #Coverband::Configuration...

If I use a forked copy of the master branch of this gem, then the s3_bucket method exists. Has the latest copy of master been pushed up to rubygems?

Can the percentage be changed at runtime? If so, how?

If it can't be changed at runtime, can coverband be disabled/enabled at runtime?

Been getting more familiar with the code and it does not look like Coverband was meant to be used in a multi-threaded environment. For example, Coverband::Base is a singleton keeps state of the current request. In a multi-threaded environment such as Puma or Sidekiq, the worker threads would step on each other.

To be honest, i am not sure how possible it would be to make this library thread safe as each request or worker calls set_trace_func.

Just wanted to start a discussion on this as Sidekiq and Puma have become very popular and more rails installations are relying on a multi-thread model.

thx!
-karl

I noticed a few things, all in lib/coverband.rb.

First, CONFIG_FILE is hard coded to a relative path. This is all well and good until you try to invoke a Rails application from outside Rails.root. From what I can tell, Coverband is looking for the CONFIG_FILE relative to the location where you run your script. Although I can set my working directory before starting the application, this doesn't seem like it should be necessary.

I also noticed that the self.configure method takes in a file argument, but it is unused. It appears like it should allow a caller to, instead of passing in a block, pass in a path to a configuration file instead. This would add a great deal of flexibility into users of Coverband by allowing them to choose a location to put the configuration file, instead of requiring it to be in './config/coverband.rb'.

At one point previously, we used 1.5.4 with success on our code base, and we wanted to reintroduce coverband with the newer version, 2.01...however we are seeing performance issues when coverband is enabled and collecting/sending data to redis... Here is a gist that demonstrates the problem (runnable in a bundle console session) https://gist.github.com/CharlesMcMillan/6bc239429761adc70ef738d05e82de6b

The goal of the gist here is to simulate the cost associated with a single page load in our app, which might include something like 10 requests where the 'report' being sent will include 2394 files with 24 lines of code each (hopefully this demystifies some of the number choices in the gist!)

The results indicate that we are losing between 1-1.5 seconds per request (I got that number by dividing 11.558872 / 10), given the above assumptions/guesstimations

It appears that saving a 'report' to redis is pretty costly here. Did something major change between 1.5.4 and 2.0.1 that would explain this?

coverband should deal with ensuring the file exists or creating it and directories leading up to that file

mkdir -p ./tmp
touch ./tmp/coverband_baseline.json
echo "[]" > ./tmp/coverband_baseline.json

something like that

so rails2 got a good baseline from

require File.expand_path("./config/environment", Dir.pwd)

rails3 needs something better.

Just wondering :-)

rake aborted!
NoMethodError: undefined method track_files' for SimpleCov:Module /Users/ankur/.rvm/gems/ruby-2.2.2/gems/coverband-1.5.0/lib/coverband/reporters/simple_cov_report.rb:22:in report'
/Users/ankur/.rvm/gems/ruby-2.2.2/gems/coverband-1.5.0/lib/coverband/tasks.rb:50:in block (2 levels) in <top (required)>' /Users/ankur/.rvm/gems/ruby-2.2.2/gems/rake-11.2.2/exe/rake:27:in <top (required)>'
/Users/ankur/.rvm/gems/ruby-2.2.2/bin/ruby_executable_hooks:15:in eval' /Users/ankur/.rvm/gems/ruby-2.2.2/bin/ruby_executable_hooks:15:in

Any idea why I am getting this error?

This seems pretty dependent on the file system which is ephemeral on heroku. Does that mean this is not usable with heroku?

thx!

We have some common code in our own gems. I wish there was a way to include certain gems in the coverage while still ignoring all others in /vendor.

The only way I see to do it is to list all the gem paths in ignored except the ones I want to include. That's a lot regex patterns to check when it's checking whether to ignore the file or not.

Not everyone runs Redis, but are running some kind of key/value store. Would it be a huge change to abstract the Redis usage to allow for storage solutions?