Comments (7)
Makes sense thanks @agis.
I will create a new issue to track this per your suggestion.
from rspecq.
I will work on this one
from rspecq.
Since this topic is open ended, we can discuss different ideas about it. To answer the questions, I think we need to leverage the way Sentry groups reports. In my opinion, it would be better if we could group reports per file, and each time a flaky job is raised, append it to the event list.
- when was this particular flaky job introduced?
Check the timestamp of the first event
- which file has the most flaky jobs?
Sort by events
- Which are the files currently that contain flaky jobs?
Unresolved events
from rspecq.
It would be nice to also include instructions on how to reproduce the execution order that lead to the error.
from rspecq.
Since this topic is open ended, we can discuss different ideas about it. To answer the questions, I think we need to leverage the way Sentry groups reports. In my opinion, it would be better if we could group reports per file, and each time a flaky job is raised, append it to the event list.
- when was this particular flaky job introduced?
Check the timestamp of the first event
- which file has the most flaky jobs?
Sort by events
- Which are the files currently that contain flaky jobs?
Unresolved events
@kpelelis "Reports per file" is the most straightforward and simple thing to do and I believe provides some immediate benefits (everything you mentioned). Sentry uses what it calls "fingerprint" to decide how to group events. Depending on the default behavior, we might need to explicitly change the fingerprint to achieve this, or the event title change might be sufficient.
It would be nice to also include instructions on how to reproduce the execution order that lead to the error.
@glampr We could do that, but that could potentially be a list of 500 spec files that were executed in the same worker, prior to the flaky one, and that would be a huge payload to submit to Sentry. We could perhaps submit the N (5-10) jobs that run prior to the flaky one, as a best-effort approach. That said, this requires some effort and it's not straightforward to implement, so I suggest to track it in a new issue.
from rspecq.
If I understand correctly, we don't need all possible combinations that lead to the error, for most cases one would be sufficient, to be able to reproduce the error locally and fix it.
from rspecq.
@glampr RSpecQ workers run in a loop, continuously popping tests of the queue and executing them, until the queue is empty. As a result, prior to encountering a flaky test, a worker might have executed a lot of other jobs (i.e. spec files), and you can't know which one is the one that caused the flakiness (if at all).
So we'd have to keep a list of all the jobs that the worker executed prior executing the flaky one, during the current build.
One thing we could also do, but this too is not so straightforward, is to emit the RSpec seed to Sentry. We could do these in next iterations.
from rspecq.
Related Issues (20)
- Automatically handle parallel execution HOT 3
- Performance report HOT 8
- Create an introductory short video
- Provide info on how to reproduce flaky spec
- worker: files_to_example_ids fails if something is printed to stderr
- Allow to connect to Redis via url HOT 4
- RuntimeError: Queue not yet published after 30 seconds HOT 2
- Integrate rubocop in CI
- Reproducing flaky tests HOT 1
- Warn when the Redis instance is not configured to expire keys
- Integrate indirectly using a pub/sub mechanism
- A tool for new releases
- Find a way to get individual worker output HOT 1
- When reproduction flag is passed build/worker ids are not required
- High redis commands/second on parallelization > 1 HOT 2
- sentry-raven is deprecated
- "Formatter ... unknown" error on startup HOT 1
- Using file split to run individual examples as jobs using lots of memory? HOT 2
- Remove sentry as a required dependency HOT 2
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from rspecq.