Is it possible (and not too much work) to include the test report number from the server in the http response? We could include a link to the report in the "Test report has been sent to tests.obspy.org. Thank you!" message then..

might be nice to have sometimes to filter out tests that don't have a specific module tested. i.e. click "iris" -> only shows tests that have run iris submodule.

advanced version might be: make the pass/fail sign in the first column specific to the selected submodule. i.e. click "iris" -> only shows tests that have run iris + checkmark shows if iris passed or failed (e.g. shows "pass" even if another module produced errors).

can be two very slim columns

I am subscribed to the RSS feed and fetch it once a day. I seem to only ever get 5 items and miss some items in between.

e.g. http://tests.obspy.org/?pr=1444

@barsch Thanks for the latest very fast fixes!

Because the test root dir is the obspy folder now, some code links do not work anymore for new reports.

For example

https://github.com/obspy/obspy/blob/1.3.0a1/clients/fdsn/tests/test_client.py#L1598

should instead be

https://github.com/obspy/obspy/blob/1.3.0a1/obspy/clients/fdsn/tests/test_client.py#L1598

Would be great to be able to filter test results by node name and obspy version string like in #4.

would be nice to see version numbers for flake8 and pyflakes..

We could add an option to set the Travis job ID in the report and have a column with a link to respective Travis build in the report overview page..
https://travis-ci.org/${TRAVIS_REPO_SLUG}/jobs/${TRAVIS_JOB_ID}

There is also an environment variable that stores the pull request number, we could also link that if TRAVIS_PULL_REQUEST is not false..
https://github.com/${TRAVIS_REPO_SLUG}/pull/${TRAVIS_PULL_REQUEST}

https://twitter.com/Mbussonn/status/1525093104835080193

https://github.com/Carreau/pytest-json-report-viewer

If anybody ever feels really bored, we could mess around adding interactive visualization on individual reports.. or even aggregates

See obspy/obspy#1504.

This is for an issue I already addressed on the developer's list, but I create this one to keep other developers in the loop.

Not sure which exactly is the condition to triggers this problem, so here are the test conditions:

• platform: Mac OS X 10.8 system;
• commit a22ebeb075;
• testing against Python 3.3 installed from MacPorts;
• no mock installed, should use unittest.mock instead;
• uses suds-jurko @0.5

Only hypothesis:
All works fine with Python 2.6 and 2.7, so this might be related to Python 3.3 or unittest.mock.

The observed behavior is the following:

petr% obspy-runtests-3.3
..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................ss..F..........................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................

======================================================================
FAIL: test_CatchOutput (obspy.core.tests.test_util_misc.UtilMiscTestCase)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.3/lib/python3.3/site-packages/obspy/core/tests/test_util_misc.py", line 78, in test_CatchOutput
  self.assertEqual(out.stdout, "abc\nghi\njkl\n")
AssertionError: 'abc\n\x1b[?1034hdef\nghi\njkl\n' != 'abc\nghi\njkl\n'
abc
- def
ghi
jkl


----------------------------------------------------------------------
Ran 985 tests in 94.031s

FAILED (failures=1)
Do you want to report this to tests.obspy.org? [n]: y

Error: Could not sent a test report to tests.obspy.org.
BAD REQUEST
petr% 

Regular expression to replace file paths with links to github does not work for windows currently, due to backslash directory separator.

The number of "Tests" shown on the web page should not include the skipped tests, IMHO. The title of the column should probably also be changed to e.g. "Executed". Compare http://tests.obspy.org/18282/ (missed matplotlib, so some station and core tests were skipped) and http://tests.obspy.org/18283/:

Especially in the overview page it's misleading to see the number of all "tried" tests (which is the same for the two above runs) and not the number of actually executed tests (which is different for the two above runs):

Node names in list could be a clickable link that adds a node name filter for this node (like in "selected nodes" filter inlet).

It would be great to be able to filter test results using wildcards or regex..

something like : http://tests.obspy.org/?version=1.0.2&node=docker-deb-*

Would be nice to have, e.g. "18 errors for 0.8.3-312-gc79b on sphinx"

No hurry at all, but might be good to have gh-actions as clickable node filter since we likely switch all CI over to there see obspy/obspy#2591
Didn't see it in the source code here, so I'm guessing it's in the database?

Opening this issue in relation to a discussion in obspy/obspy#2489.

ObsPy's tests have been refactored to use pytest so some modification to the reporter app is needed to support a more pytest friendly output.

Currently I am just using pytest-json-report as it seems to be the best supported json report writer. We could try to stick with xml but it is 2022 after all 😉.

I have also attached an example json report. It is really quite simple to modify so we can easily make changes.
report.zip

@barsch , @trichter

Various people at EuroScipy 2016 seemed pretty interested in the whole test reporting and visualization setup. Might be worth a thought to further generalize/push/advertize this piece of our workflow..

CC @krischer

I've seen CI test reports not getting sent because their payload is too big, likely because of the tracebacks (pytest and decorators blowing tracebacks up massively, sadly). In this case it's "just" 77 test fails.

Error: Could not sent a test report to tests.obspy.org.
Bad Request
b'Request body exceeded settings.DATA_UPLOAD_MAX_MEMORY_SIZE.'

Not sure what to do. I guess aside from allowing larger reports, we could maybe in our obspy reporting code handle a response like that and send the report again without tracebacks maybe?

CC @trichter @barsch

Not sure whats going on but somehow some tests get grouped into "other"..

https://tests.obspy.org/138411/

some "client." modules missing at top..

and got grouped under "other" at bottom

In test reports, the test failure icons (e.g. "#6") are clickable links in the overview at top. In the list of detailed tracebacks for each error, these icons are not clickable links (for convenient copy/paste).