Comments (6)
hey @mat007 - SpecStatePending
is reserved for tests that are explicitly marked as Pending. To distinguish between specs that were skipped by the user explicitly (via Skip(...)
) and specs skipped for some other reason you can look at spec.FailureMessage()
. If this is non-empty then the spec was skipped explicitly by the user. Otherwise the spec was skipped either because of a filter or a previous timeout or AbortSuite
.
from ginkgo.
Ah, I totally misunderstood what SpecStatePending
meant! 🤦
Thanks for the suggestion about looking at the test failure message. But then, is there now a way to tell the difference between a test that was not run because it was out of focus (suite’s config FocusStrings
, SkipStrings
, LabelFilter
, etc…) and because the suite was aborted before it ran?
from ginkgo.
nor straightforwardly I don’t think - but at this point i should ask: what problem are you trying to solve?
from ginkgo.
Yes, sorry, fair ask!
We run tests nightly on the CI a whole bunch of times in a loop, in batches with various focuses, and record the state of each test.
Then every day we use that data to compute some statistics, for instance the % of passed, failed and skipped tests (a very few actually manually skipped).
When a test fails, sometimes we have to abort the test suite, and as our tests run order is randomized, the remaining tests that do not run are not always the same.
The result is that the % of skipped tests varies a lot, between e.g. ~0.5% and 5%.
We wanted to start categorizing these a bit better, by separating the «real» skipped tests (typically these ~0.5%), the «not focused by design» tests (our tests are split in different «groups» so we can parallelize batch them), and the «unknown» (or «not run», or …) tests. It’s actually important for us to monitor the latter, because when it increases it means the test failure recovery mechanism we have may be starting to get flaky as well, and needs to be looked at.
I hope this is clear enough 😅
I guess I could try to manually re-apply the focus/skip regexps in my after suite report to tell if a test was in our out of focus. 🤔
from ginkgo.
ok got it - thanks. Ginkgo doesn't have super strong first-class support for this right now. It's almost like you want something like "SkipReason" which will allow you to tell if a spec was skipped because it was out of focus, or if the user skipped it, or if an abort/timeout has occured. I could imagine adding that to the codebase though I'm not going to get to it soon :/
If you're up for doing some more work on your end, though, you can add a ReportBeforeSuite
to get a version of the report before the suite runs. This will include all the specs, but specs that are out of focus (because of the various focus/filter flags) will already be skipped. That will give you N_skipped_because_of_focus
. In the ReportAfterSuite
you can measure N_total_skipped
(all specs with SpecStateSkipped
) and N_user_skipped
(all specs with SpecStateSkipped
and non-empty failure message). You can then do N_abort_skipped = N_total_skipped - N_user_skipped - N_skipped_because_of_focus
to get the number that were skipped because of an abort or timeout.
I think that'll get you what you want?
If you want to be more granular and do this on a per-spec level you'll need to be able to connect the specs in the ReportBeforeSuite
to the specs in the ReportAfterSuite
. Ginkgo doesn't give you a unique identifier but if you do something like: id := fmt.Sprintf("%s %s:%d", spec.FullText() , spec.FileName(), spec.LineNumber())
you'll have something that should avoid collisions. That would allow you to compare before and after.
Sorry it's so tedious, though. I'll add SkipReason
to the backlog and see if I can get to it.
from ginkgo.
Thanks mate, that’s a good lead!
I’ll look into implementing it eventually, when time permits 😄
from ginkgo.
Related Issues (20)
- [Question] Can I set a threshold for sum of FlakeAttempts? HOT 2
- Can't set custom GracePeriod on uninterruptible clean up node HOT 2
- Ginkgo v2 CLI breaks when v1 is used HOT 2
- Make it possible to flush output via newline in between tests HOT 2
- cannot support multi version of go HOT 10
- Is there a way to have a single suite that will run tests in sub packages? HOT 2
- Getwd() breaks test caching on build runners like Buildkite HOT 1
- Get coverage from ginkgo
- DeferCleanup() blocks are run after all AfterEach() blocks, regardless of nesting HOT 2
- Is there a way to populate dynamic data in the Tree Construction Phase? HOT 2
- Detail Skipped reason when spec is skipped by cli arguments HOT 6
- Support customizing the SpecContext HOT 4
- ginkgo v2 leaks goroutine HOT 3
- Proposal: RunSpecsWithContext? HOT 14
- Add option to StandardReporter to display things better on github actions HOT 2
- Call dynamic variables through 2 test files. HOT 1
- Skip each test function logs and print in summary HOT 1
- Run a single suite with N instances in parallel with different `BeforeSuite` Setup. HOT 3
- Feeding report back for focusing on failed test-cases HOT 1
- Junit report should have system-out HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from ginkgo.