albertziegenhagel / django-testing-framework Goto Github PK

In the tests and test detail views, the checkboxes for statuses that are not present in the current list should be disabled.

Migrated from civviGH/django-testing-framework#13

Values drawn on the history page (e.g. in the tooltip or the sidebar) do not match the style that is used in the test result tables.

E.g. datetime values are printed in the table in a human-friendly format H:MM:SS.SSSSSS, while in the sidebar of the history page the ISO 8601 string representation is used and the tooltip of the chart just prints the numerical value in seconds.

The underlying cause for this issue is that the representation for the results table is generated on the server-side by Django but the representations on the history page are generated on the client-side in Javascript, where we do currently lack support for human friendly representations.

The best approach would probably be to write a bunch of client side functions to generate the desired representations and use those everywhere consistently.

Even though everyone should be very careful before clicking the "Update references" button, it can always happen that someone updates a reference by accident that should actually not have been updated.

Supporting some kind of "revert" operation after a reference operation would be very helpful in such cases.

Migrated from civviGH/django-testing-framework#12

Currently the status of a submission is computed every time it is required (usually only in the front-end).

This can be very expensive, since it requires a lot of DB queries to be executed and leads to noticeable slowdown when loading the web-page for submissions on instances with many submissions/tests.

We should store the status of a submission in the Submission model and update whenever necessary, similar to how it is already done for TestResults. For Submissions this might be more difficult, since a new TestResult instance that is associated with a submission will lead to a change in the submission status. So maybe we need to recompute the Submission status whenever a TestResult is changed, and not when the Submission itself is changed.

We should add support for user management to the application.

This should allow to restrict the access to projects to specific users/groups.

The right side bar on the history page can be collapsed, but it can not be expanded on that page afterwards again, since there is no button to do so.

As a current workaround one would have to visit another page (e.g. a test result details page) to expand the sidebar there and then re-visit the history page.

It would be nice to have some kind of log that collects all reference update events (due to user pressing the "Update References" button, or due to REST API calls).

The log does not need to be infinite. Storing just some limited time into the past (e.g. the logs for the last X months, or just the last N entries) should be sufficient.

Additionally the logs could be used to restore some older state (rolling back the changes applied per log) which might be a possibility to implement #27.

Migrated from civviGH/django-testing-framework#41

Add a comment box to each reference set in the front-end.

e.g. to comment why a reference is updated.

For the test and test detail views it would be handy to have a very general filter text field. Maybe something to how the filters on GitHub issues work.

It is to be discussed whether such a filter should be applied immediately while typing or only after the return key is pressed.

Migrated from civviGH/django-testing-framework#14

In #33 we had to disable the navigation bar due to performance problems.

We should fix those problems and re-enable the navigation buttons.

Currently images are stored within the database as base64 encoded strings.

While this has the benefit that all test data is completely stored in the database and no external files are references, this blows up the database itself A LOT, and thus slowing down all operations on the DB.

We should probably store images as regular files on the disc (in the media folder?!) and just reference those files from the DB.

A nice feature would be that those files should first be hashed (e.g. by an MD5 sum) and only non duplicate images should be stored. This will most likely result in huge disc space gains, since most of the time the test images do actually match the reference image and would not be stored separately.

Currently the webpage is hard to identify in the browser tabs, since it does not have a favicon.

We should add one.

Migrated from civviGH/django-testing-framework#39

Currently, when a user presses the "Update references" button there is no feedback to the user about whether the operation has been executed successfully. Due to the fact the application is running in a browser and requires the network connection, sometimes it happens that it is not immediately obvious to the user whether the references have been updates or not.

A simple message like "Successfully updated X references" showing up after a references update could help a lot here.

Migrated from civviGH/django-testing-framework#20

The documentation for the REST API is still missing some endpoints (e.g. all submission and reference-set endpoints are not documented).

The following endpoints need documentation:

• api/users
• api/users/<str:id>
• api/projects
• api/projects/<str:id>
• api/projects/<str:project_id>/members
• api/projects/<str:project_id>/members/<str:member_id>
• api/projects/<str:project_id>/properties
• api/projects/<str:project_id>/properties/<str:property_id>
• api/projects/<str:project_id>/webhooks
• api/projects/<str:project_id>/webhooks/<str:webhook_id>
• api/projects/<str:project_id>/webhooks/<str:webhook_id>/logs
• api/projects/<str:project_id>/tests
• api/projects/<str:project_id>/tests/<str:test_id>
• api/projects/<str:project_id>/submissions
• api/projects/<str:project_id>/submissions/<str:submission_id>
• api/projects/<str:project_id>/submissions/<str:submission_id>/tests
• api/projects/<str:project_id>/submissions/<str:submission_id>/tests/<str:test_id>
• api/projects/<str:project_id>/submissions/<str:submission_id>/tests/<str:test_id>/history
• api/projects/<str:project_id>/references
• api/projects/<str:project_id>/references/<str:reference_id>
• api/projects/<str:project_id>/references/<str:reference_id>/tests
• api/projects/<str:project_id>/references/<str:reference_id>/tests/<str:test_id>

Currently most of our JavaScript code is written inline into the HTML templates that Django will serve to the client browser.

We should extract that code into own *.js files that served as static files. While this might improve page loading a bit, the main reason to do it would be to allow us to structure our code better, make it more readable and maintainable and would make it easier to add tests for the JavaScript functionality.

Sometimes a user might want to disable triggering webhooks temporarily.

Currently there are two ways to achieve this: The first is to delete the webhook and add it back later. This is far from ideal and additionally comes with the loss of all current log entries for the webhook.
The second is to simply uncheck all events the webhook is triggered on, and check them again when the webhook should be restored. While far better than deleting the webhook, it is still non-ideal since the user needs to remember the events that the webhook was initially triggered on.

We should add an option to set a webhook as "inactive" so that it stays an existing webhook, but will simply not get triggered on any events.

We should only display those properties that are actually marked to influence the reference.

implement a way to open a graph for a given test parameter that shows the development of its value over time. for starters, only > support int/float, not strings, images or lists

maybe have two differently colored lines in the graph to show value/reference combinations.

possible library for drawing graphs: https://www.chartjs.org/

Migrated from civviGH/django-testing-framework#4

Currently, the status of a test is computed from the statuses of all measurement results within the test. This is usually exactly what we want, with one exception: If a test could not be executed successfully (e.g. due to a crash) it might get displayed as successful (because all measurements that could be extracted up to the point of the crash where successful). We should allow to set the status of a test explicitly and thus overwrite the status computed from all test measurements. This would allow to set a test that crashed to broken.

We might limit this to only allow setting a test status that is "worse" than the one computed by the measurements statuses.

E.g. a test with only successful measurements could be set to broken, but a test with at least one failed measurement could not be set to successful.

Migrated from civviGH/django-testing-framework#21

Currently the state of the checked status filters is reset every time one reloads a webpage.

This can be very annoying, e.g. if one wants to have a look on all non successful tests in a submission:

• The user disables displaying successful tests in a submission
• They click on the first test
• They navigate backwards in their browser to go back to the submission overview
• The filter has been reset, and all tests are displayed again.
• The user has to uncheck the "successful" filter