Coveralls reporting 0% coverage with Slather/Travis integration about slather HOT 28 OPEN

Hey, thanks for filing an issue. I've been hearing a couple different of people have had this issue today, so I'm going to dig into it tonight a little bit!

from slather.

+1. I set up two projects with Slather + Coveralls yesterday; one is encountering this issue while the other is fine. So while I have no idea what might be causing it, it's not a universal thing.

from slather.

@kylef, I want to say that I vaguely remember something about the Cocoapods post-install hook not consistently working with Travis in the beginning... or at least having some sort of issue?

The reason that I'd believe that it might hae something to do with that plugin is because slather is setup to blow-up if it cannot find any coverage files. I think that each project does generate gcno files for every source file, though, so if a target is generating gcno files that doesn't contain the test suite for said files, then they'll show up with 0 coverage down the boards. Usually slather just circumvents this by finding and using the gcno files with the highest amount of coverage, assuming those to be correct. But maybe those correct ones were never created because the Pods project didn't get configured for instrument program flow and generate coverage files correctly?

I feel that there's a 50/50 chance that paragraph even made any sense 😞

from slather.

The intuit/LocationManager project @smileyborg mentioned isn't using the CP plugin, though. The lib files are included directly in the example project, and its Podfile only exists to add Specta/Expecta.

Poking through the Slather code, it doesn't look like it has a --verbose flag or anything like that? Getting a little more insight into what's actually going on when it runs on Travis could be enlightening.

from slather.

+1 for a --verbose flag; I was wondering if it's possible to use the -s option to get command line output in combination with the Coveralls integration.

from slather.

Yea, I've been thinking about the verbose flag this morning. I was thinking the following things. Let me know what you all think --also, @kylef @ayanonagon @dasmer @eliperkins

(for verbose flag)

• Log when we've successfully configured pods for slather
• Log each GCNO file that is ignored (ones for files that aren't your source files, and also duplicate ones.)
• Log each GCNO file that is processed, along with its
  • created_at timestamp
  • modified_at timestamp
  • Contents? Seems too noisy... but it'd be nice to at least see a couple.
  • Add some sort of #inspect method to Slather::CoverageFile class, and print out this as well.
• Add coverage_service specific logging to each of the services.
  • For example, for coveralls, print out the JSON blob that it'll send to coveralls, and the response from coveralls after it has been posted.

(not related to verbose flag)

• Allow .slather.yml files to use multiple coverage services with coverage_services:, and allow the flags to the command line to stack, ie -s and -x can be used together for cobertura and terminal output.

Is there anything else I should add that would be helpful?

from slather.

@marklarr I agree that contents might be to noisy so maybe leave that out, but everything else would be really helpful

from slather.

@smileyborg You can use -s option in your Travis file, but you will still get 0% coverage.

See:
AlexDenisov/FrameAccessor#17

from slather.

@dasmer That's helpful. So that means that the problem is with Slather picking up the coverage when running on Travis, (probably) not anything with the communication from Slather to Coveralls.

from slather.

@marklarr May I suggest opening a new Issue to discuss adding a --verbose option, and associating commits to that one? Going to sidetrack this one.

from slather.

#71

from slather.

Thanks everyone

from slather.

OK, as far as the original issue here, I had a theory about what might be going on, and I was right.

This was the original .travis.yml that was being used:

sudo: false
language: objective-c
cache: cocoapods
before_install:
  - export LANG=en_US.UTF-8
  - gem install cocoapods xcpretty slather --no-ri --no-rdoc
  - pod install --project-directory=LocationManagerExample

script:
  - set -o pipefail && xcodebuild test -workspace LocationManagerExample/LocationManagerExample.xcworkspace -scheme LocationManagerExample -sdk iphonesimulator ONLY_ACTIVE_ARCH=NO | xcpretty -c

after_success: slather

Note that the tests were being run manually using the script key, instead of letting Travis do its default execution using xctool.

By removing this script key and changing the .travis.yml to:

sudo: false
language: objective-c
cache: cocoapods
xcode_workspace: LocationManagerExample/LocationManagerExample.xcworkspace
xcode_scheme: LocationManagerExample
xcode_sdk: iphonesimulator
podfile: LocationManagerExample/Podfile

before_install:
  - gem install cocoapods --no-ri --no-rdoc
  - gem install slather --no-ri --no-rdoc

after_success: slather

Travis is now in charge of invoking xctool to run the tests, and Slather is now able to correctly get the coverage data and send it over to Coveralls: https://coveralls.io/r/intuit/LocationManager

So, not sure if you'd consider this a bug -- is Slather supposed to support the manual mechanism of test execution that was happening originally?

from slather.

slather should work wtih vanilla xcodebuild, or even running the tests via Xcode. The only consideration that should need to be taken is setting the build_directory in your .slather.yml, as mentioned here. Code.

It's pretty odd that it changes anything, really. I wonder what xcodebuild command travis ends up calling when you had it figure it out on it's own? Does it show it in the travis logs?

from slather.

@smileyborg
https://github.com/lazerwalker/MTLParseAdapter also uses script with xcodebuild and it just works

from slather.

Are we entirely sure that this failure is deterministic?

@dasmer I know that the guys at Travis are switching their OSX providers, maybe they are slowly rolling that out, and slather doesn't play well with the setup on the new machines?@joshk @meatballhat

from slather.

@marklarr Yes, Travis outputs the build commands it's using, just go to this recent build (which successfully slathered coverage data to Coveralls) and search the log for xctool: https://travis-ci.org/intuit/LocationManager/builds/59640143

from slather.

@marklarr And I'm not 100% certain this was deterministic, but a bunch of builds failed to slather correctly before this change, and as soon as I made the change it's working consistently...

from slather.

@dasmer @lazerwalker, I'd say that we should try what @smileyborg has done on IntentKit and FrameAccessor and see what happens. It's not an ideal solution, but it will at least give us more of an idea of what is causing the issue.

from slather.

Yeah, I'm game to give it a shot.

from slather.

That fixed Coveralls for IntentKit, but any of you know how to run things through xcpretty without using script? The ugly output of https://travis-ci.org/intentkit/IntentKit/builds/59646820 is seriously bumming me out.

from slather.

This fixed FrameAccessor too 💃

from slather.

If I had to speculate, I would say it could be that the build output directory is different when using script or something (perhaps Travis recently changed it?) and Slather is unable to find the coverage files where it expects to find them.

from slather.

@smileyborg I agree with your speculation, but the one weird thing about that is that if Slather doesn't find any coverage files, it raises an exception. So, it's definitely finding gcno files, but they're just reporting no coverage, which usually would happen in the case of the build settings for GCC_INSTRUMENT_PROGRAM_FLOW_ARCS and GCC_GENERATE_TEST_COVERAGE_FILES not being enabled.

from slather.

@lazerwalker I think we're SOL on using xcpretty without script. At least we've narrowed it down to script being the issue now, though, and maybe we can figure this out once and for all. I haven't had a lot of time to look at this these last couple days, but the first chance I get to work on anything OS it'll be on this issue.

from slather.

Hey folks! 👋

I'm expiriencing the same behavior, as described earlier. I've tried to setup my project (removing pod linting and ditching xcpretty) to mimic the setup of IntentKit.

Travis will happily build the project but slather will now raise an exception that it can't find the coverage files. Previously I've provided the necessary build options directly in the script. Right now the creation of the coverage file is added to the project file.

Any ideas what I can try next to get the coverage report running again?

from slather.

I am currently trying to integrate slather with coveralls and travis into nui.

We are having a rather large build matrix on travis for that project. Code coverage is generated fine except for a single build with xcode6.1 (travis default) run on a 7.0.3 Simulator.

This bug may be specific to individual iOS versions.

See:
Coveralls
Travis Build

Xcode versions for the matrix and output of slather coverage -s included in travis build

from slather.

I also wasn't getting any coverage on coveralls.io (https://coveralls.io/builds/6001539). On another run it showed list of files, but was missing 1/3 of files: https://coveralls.io/builds/6001574. I was using slather 2.1.0 at that time.

Running slather coverage locally was producing correct results. Locally I've debugged slather gem and it also seemed to send correct data (https://gist.github.com/yas375/6e271111ab382588a0f7b5bba21bbe3b). I haven't inspected the json, but I did check that it has the files I don't see at coveralls.io at all and it passed json lint.

Finally I've tried to use HEAD version of slather

gem 'slather', git: 'https://github.com/SlatherOrg/slather.git', ref: '79d7652'

And now it works! Probably one of these commits has fixed it. Not sure which one :)

from slather.