Comments (8)
@blueyed Can you have a look at #71 , please? Maybe you can test with your code and also have a look at the diff and suggest improvements. :)
from pytest-testmon.
@blueyed I would consider this fixed for now. Maybe there could be heuristics which wouldn't write the data if too many tests fail too quickly. Maybe there could be an option for rollback of the last run, or some other UI which would let you choose if to write or discard the .testmondata changes.
But that's out of scope for not
from pytest-testmon.
The basic idea is that mtimes and file_checksums are just helper/optimization data for large projects. They must not be written after every executed test, because:
- it's a little hard to get right
- it's even more complicated to avoid performance penalty
The dependencies itself, however, should be written after every test, because some test suites take hours and we shouldn't loose all the data on interruption. So far I'm committed to fulfill this requirement.
In other words, current implementation is careful tradeoff between complexity of implementation and performance. And it's unlikely I would like to change it before refactoring the data model (which would fix this, by itself) and refactoring the test suite.
Could you rephrase your requirement in other terms? How do you experience the disadvantage now? Do you have a huge test file which you frequently run with -x and do you have slow start-up? Is there a printout which bothers you?
from pytest-testmon.
@tarpas
The problem is that since mtimes are not saves for a file, this is not considered at all when checking for changed files.
The simple test case is:
- remove .testmondata
- stop your database server
- run pytest with testmon
- abort it after seeing a lot of errors
- start the database server
- run pytest with testmon
- notice that it just replays the failed tests, and won't run the others
This is bad already, but when you explicitly change a file where tests were recorded as failing, testmon will not pick it up: it seems to only look at mtimes for this, which is empty.
What do you think about #67 in this regard?
from pytest-testmon.
Seems like a serious bug. The bug is only when you interrupt the first run (with no .testmondata)?
from pytest-testmon.
At least that is how I'm able to easily reproduce it (and #67 fixes that).
But my colleague and me had both noticed that testmon is not reliable anymore since a while.
I usually just tend to remove .testmondata then (which might trigger this particular issue when aborted again).
Also using -x
is likely to trigger this, since it is an KeyboardInterrupt internally IIRC.
It might be related to b59154b after all.
The problem here is that only parts of the data are saved (for each finished item), but not the additional metadata.
from pytest-testmon.
Daniel, after thinking about it I must say this is by design. When you read the text on http://testmon.org -> Thoughts -> "5. External services (reached by network)" you have to realize database is just that.
A test which ends early with database error is captured in .testmondata as depending only on the first couple of lines. After starting the database and re-running the test, testmon doesn't register any change, so it only re-reports the old exception and doesn't attempt to run the test again.
We could implement a switch/command which re-executes all failures regardless of the .testmondata. That would also mitigate #57 .
What UI do you recommend?
py.test --testmon --tlf
?
tlf as in testmon last failed
from pytest-testmon.
--tlf
sounds good.
from pytest-testmon.
Related Issues (20)
- If test itself changes, should testmon pick it up? HOT 4
- sqlite3.OperationalError: database is locked HOT 26
- Collection seems to be skipped after a run HOT 9
- Testmon db still corrupted with --collect-only HOT 5
- testmon doesn't track sources in venv directory HOT 4
- testmon does not respect test order HOT 2
- Ability to run testmon on an installed package HOT 4
- No source package upload to pypi HOT 4
- fails when git is not present HOT 2
- Missing git version tagsπ€ HOT 2
- Changes > 1.4.5 made testmon more sensitive to differently packaged environments? HOT 13
- Have a command line that outputs all the tests which reach a certain line in the code HOT 1
- Change to Importlib introduces AttributeError for {d.name} if d.name does not exist. HOT 2
- Temporary files not being cleared inside a container HOT 6
- testmon.org appears to be down HOT 4
- testmon does not work with --branch based coverage HOT 4
- Change to less restrictive license HOT 8
- Seeking examples of pytest-testmon in GitHub Actions HOT 3
- Allow using --testmon-noselect and --testmon-nocollect simultaneously HOT 3
- Mark test as 'run-always'
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
π Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. πππ
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google β€οΈ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from pytest-testmon.