svlad-90 / dlt-message-analyzer Goto Github PK

• Add check, that will NOT allow the user to specify the cache size more, than the actual available amount of RAM on his system.
• Add the implicit check of the cache size setting which will do the same. It should be done in order to cover the relocation of the plugin's settings from one machine to the other.

As a user, I want to be able to automatically build sequence diagrams out of the output of the different applications within the system. I want to be able to normalize data from different applications and put them all into a single unified sequence diagram.

In order to do that we can use plantuml.jar out of here: https://github.com/plantuml/plantuml

The integration point should be an extension of regex name scripting.

User should specify the following groups within the message:

<USID> - optional UML_SEQUENCE_ID
<UCL> - mandatory UML_CLIENT
<URT>|<URS>|<UE> - mandatory one of UML_REQUEST|UML_RESPONSE|UML_EVENT
<US> - mandatory UML_SERVICE
<UM> - mandatory UML_METHOD
<UA> - optional UML_ARGUMENTS

In case if all mandatory groups would be found, the found row would be used to form the single step of the sequence.

Result plantuml formatting would look like this:
<UCL> <URT|URS|UE> <US> : [timestamp] <USID><UM>(<UA>)

• A separate view should be created to visualize the sequence - Done.
• A system call to plantuml.jar should be used to form the svg image - Done.
• QGraphicsSvgItem should be used to visualize the diagram - Done.
• It should be possible to export the diagram to the file system as *.png, *.svg & *.puml - Done.
• The generation of the sequence diagram is an expensive operation. It should be done ONLY by an explicit trigger from the user - Done.
• The generation of the sequence diagram is an expensive operation. There should be some setting, which will specify the maximum number of considered rows in order to avoid "forever stuck" in waiting for the operation to finish - Done.
• The generation of the sequence diagram is an expensive operation. It should be started in a separate worker thread - Not needed.
• It should be possible to instantly cancel the generation of the sequence diagram. Cancelation can be triggered from UI by the user - Done.
• The search view should highlight rows, which will be used for the creation of the sequence diagram - Done.
• The feature will require additional CPU to gather the required data. The search might become slower due to that. There should be a setting to turn on and off the plantuml feature - Done.
• The user should have the possibility to specify, whether arguments should be represented, whether they should be represented with wrapping. - Done.
• Users should be able to turn off and on the consideration of the specific suitable row within the "Search view" for the creation of the sequence diagram. Search view rows should have 3 states from this perspective: suitable_selected, suitable_not_selected, not_suitable - Done.
• It should be possible to clear the diagram from the context menu of the preview tab - Done.
• Delivery of plantuml.jar should be integrated into the plugin - Done.

It is not correct to search for plantuml.jar using the "./plugins/plantuml.jar" path.
When dlt-viewer is installed on the Linux via qmake:

• the executable "dlt-viewer" is placed under the "/usr/bin" path
• the *.so-s of plugins are placed under the "/usr/share/dlt-viewer/plugins/" path
• but plantuml.jar is NOT installed anywhere, while plugin tries to find its location under "./plugins/plantuml.jar", which does not work

The approach should be extended to:

• Try to find the plantuml.jar by its default path. The default path should be somewhere under the same folder, where the settings of the dlt-message-analyzer plugin are located.
• Check the availability of installed plantuml.jar by environment variables. It should be checked whether such an approach is applicable.
• Add the new setting in order to allow the user to specify a custom path to the plantuml.jar
• Add setting which will tell whether the default path, environment variable path, or custom path should have the highest priority

Together with that, it would be great to have similar functionality in order to check the availability of java on the target machine.

Also CMake and make builds should be extended to install plantuml.jar to the user folder.

• Add "visible columns" option to "grouped view"
• Add possibility to reset visible columns to default within the "grouped view"

Once the search is performed with a specific regex, it should be possible to form the list of suggestions for each specified variable.
Once the user is trying to edit a variable's text, suggestions should be proposed to him in the form of the combo-box. A list of suggestions should be formed based on the found content within the specific group. That will allow us to play around with filters in a much more comfortable way.

On Linux, there is an issue - when built dlt-viewer is installed, the plugin can't write to settings files, as they are located nearby the plugin, in the read-only area of the file-system.

This should be changed. Settings should be located in the user folder which is open for read-write operations.

That will allow to:

• avoid the issue with read-only fs area
• have user-specific settings instead of the global ones

P.S. There should also appear a new context menu "Open settings location". That will make life easier for the user if he wants to get an understanding of where the settings are located.

Add the possibility to form an analysis coverage note within the plugin:

• Mark specific lines as the important ones
• Add comments to specific lines within the search result
• Export the coverage note to HTML

Add the possibility to select the font of created diagrams to the UML view

In order to ensure the appropriate quality of our SW, it would be great to create a static-code analysis build session.
The session should be running on ubuntu-latest and use cpp-check tool.
After ( and if ) the session will be established, we will create another task to fix all identified warnings.

Support the possibility to save a python-based trace-analysis class, as a pattern to the patterns view.
Pythion scripts should have access to the logging API of the debug console to fire the messages into it.
Pythion scripts should have access to the search view to add new messages to it.
Pythion scripts should have their own status variables, which stay alive during a single search session.

That will allow implementing the advanced automated trace analysis, which will make conclusions regarding multiple messages and their sequence.

After a static code analysis session was established, it is time to fix the found warnings.

Link to the failed build:
https://github.com/svlad-90/DLT-Message-Analyzer/runs/855833900?check_suite_focus=true

Observed behavior:

The filter view breaks the regex after changing the variable's value.
That happens only after changing the filter view's sorting.
It seems that regex is reconstructed in the wrong order.

Expected behavior:

The regex reconstruction algorithm should not rely on the order of the elements within the view.

Add the possibility to dynamically select, which parts of information to combine into a single string, before sending it to the regex engine. Currently those parts are "APP ID" + " " + "Context ID" + " " + "payload" without the possibility to adjust something. It might be useful to add the possibility of a dynamic selection of columns to be used, e.g. Ecuid, Timestamp, etc. Also, it would be cool to allow us to change the order of the used parts.

In addition to that, the "patterns view" and "regex input" should be extended in order to allow to specify parts of regex, which should be used for each separate column. Some "simplified" & "advanced" view variants. It will allow us to consistently support this feature.

Add the "auto-scroll to the end" option for continuous search in order to have a possibility to stick to the last found message, while the search is running. This option can also be applicable for non-continuous search.

It might happen, that formatting of existing messages in the project does not fit constraint to have all mandatory UML groups. Thus the creation of a sequence diagram is not possible. Still, the client might understand that specific formatting of the message is representing communication with a specific interface and method, and wants to provide this information as part of the regex. We should allow the client to do this.

Let's consider the following example:

2206 2020/08/06 15:27:04.204000 276163.1118 ECU1 SYS JOUR 2020/08/06 15:27:03.556033 cat[49294]: Informational: CAlice: sentName(name: "Alice")
2207 2020/08/06 15:27:04.703000 276163.5468 ECU1 SYS JOUR 2020/08/06 15:27:03.991085 cat[49299]: Informational: CAlice: response(success: 1)

What would I expect to do with the above messages?

• Specify Alex as a CLIENT, despite that string has CAlice
• Specify Bob as a SERVICE, despite it is not presented in the message
• Specify sayMyName as a method, despite it is not presented in the message
• Specify request, response, event, despite they are not presented in the message directly

I want to use the following regex in order to achieve the above mentioned points:
^(SYS).?: Informational: (?<UCL_Alex>CAlice):.?(?<US_Bob>(?<UM_sayMyName>(?sentName)|(?response))).*((?[\w]+))

New parts of behavior:

• UCL_Alex will replace CAlice to Alex in the diagram content ( not supported yet )
• US_Bob will add Bob service, despite, that it is not presented in the message ( not supported yet )
• UM_sayMyName will add sayMyName method, despite that it is not presented in the message ( not supported yet )
• URT & URP are stick to the parts of the message, which only indirectly show whether it is request or response ( supported already )

If supported, above will allow creating sequence diagrams even from non-perfectly formatted messages,
expanding the number of covered use-cases.

P.S. Together with that, it would be good to remove the 32 characters limitation on size of the regex group names. It should be possible to do that via internal mapping of provided group names to some shorter identifiers, e.g. counter-based ones - 1, 2, 3, 4, etc. Short identifiers would be actually used within the regex, while long names would be still known to the plugin's implementation.

Add the possibility to choose font type & font size. The selected settings should be applied to the:

• view itself
• copy-pasted information

The additional settings should be persisted between the plugin's life-cycles

In case if new search results do contain the message with id, which was selected by the user in the previous search result - the search results table should jump to the previously selected row. Otherwise, the memory slot should be reset.

Add the possibility to mark columns, which should be copied during the copy operation. The option should be separated from the column's visualization setting.

Add versioning to each user setting in order to allow us to reset specific settings once it is changed in a non-compatible way.
In case if the version in the file differs from the one specified in the SW - the setting in the file should be discarded and replaced with the default value.

It would be good to remove the 32 characters' limitations on size of the regex group names. It should be possible to do that via internal mapping of provided group names to some shorter identifiers, e.g. counter-based ones - 1, 2, 3, 4, etc. Short identifiers would be actually used within the regex, while long names would be still known to the plugin's implementation.

Add the possibility to unwind selected tree nodes of "Grouped view" back to the "messages table".

Add the possibility to mark columns, which should be copied during the copy operation. The option should be separated from the column's visualization setting.

Currently, we support multiple platforms only on a source-code level.
Thus we doubt, that it is possible to implement the update of precompiled dll-s ( or so-s ) from the cloud.

We will start from less ambitious goals:

• Add notification about the availability of a newer version of the plugin in Git hub
• Add the "check for updates" option

Currently, we have several warnings in clang-tidy build, which are related to the third-party headers:

• antlr
• Qt

Due to the absence of an approach, we've suppressed check of the whole cpp-s, which are using the headers with warnings:

• DLT-Message-Analyzer/dltmessageanalyzerplugin/src/common/PCRE/CMakeLists.txt

  Lines 3 to 19 in c2e2851

  	set(RESTORE_CLANG_TIDY OFF)
  	set(CLANG_TIDY_CACHED_VAL "")
  	if( DMA_CLANG_TIDY_BUILD )
  	set(RESTORE_CLANG_TIDY ON)
  	set(CLANG_TIDY_CACHED_VAL CMAKE_CXX_CLANG_TIDY)
  	set(CMAKE_CXX_CLANG_TIDY "")
  	endif()
  	add_library(DMA_PCRE STATIC
  	PCREHelper.cpp)
  	if( RESTORE_CLANG_TIDY )
  	set(CMAKE_CXX_CLANG_TIDY CLANG_TIDY_CACHED_VAL)
  	set(CLANG_TIDY_CACHED_VAL "")
  	set(RESTORE_CLANG_TIDY OFF)
  	endif()

• DLT-Message-Analyzer/dltmessageanalyzerplugin/src/filtersView/CMakeLists.txt

  Lines 5 to 21 in 6f3e9fd

  	set(RESTORE_CLANG_TIDY OFF)
  	set(CLANG_TIDY_CACHED_VAL "")
  	if( DMA_CLANG_TIDY_BUILD )
  	set(RESTORE_CLANG_TIDY ON)
  	set(CLANG_TIDY_CACHED_VAL CMAKE_CXX_CLANG_TIDY)
  	set(CMAKE_CXX_CLANG_TIDY "")
  	endif()
  	add_library(DMA_filtersView_no_clang_tidy STATIC
  	CFilterItemDelegate.cpp)
  	if( RESTORE_CLANG_TIDY )
  	set(CMAKE_CXX_CLANG_TIDY CLANG_TIDY_CACHED_VAL)
  	set(CLANG_TIDY_CACHED_VAL "")
  	set(RESTORE_CLANG_TIDY OFF)
  	endif()

That is wrong. We should rather exclude failing headers than exclude our own cpp files.

This task is created to:

• Create a rule, which will allow excluding specific cpp from clang-tidy's analysis without copy-pasting a big bunch of CMake directives
• Investigate approach regarding how to move QT's headers to the system headers section
• Investigate approach regarding how to suppress warnings in all third-party headers or at least in specific headers, which have warnings identified

It would be great to mention third-party repositories, which we are using.

Fix mismatch between cache-size consumption measurement inside the plugin and "Task manager"'s RAM consumption

Create the possibility to post-filter already found messages in the "Search view" from an additional text input field. Create the possibility to do a post-filtering inside the post-filtered results.

Add the possibility to mark columns, which should be copied during the copy operation. The option should be separated from the column's visualization setting.

Add the possibility to copy the file name of the specific message.

To plan further scaling of this SW we need to provide the architecture draft regarding its current implementation.
The result of this task should be a set of diagrams describing connections between components and structure of each of the components.

Refactor settings manager to allow its extension with less effort.
A precondition for issues #23, #28, #30.

We need the "Autonumer" setting in the "UML View".
Some users might want to have numbered steps inside the sequence.
That is handy if you've generated sequence with more than several dozens of the steps.

Add the possibility to update the pattern's JSON from the cloud.

The task consists of the following changes:

• Add the possibility to store user authentication data, which consists of the user name and password. It should be possible to store multiple authentication data-sets.
• Add the possibility to add new pattern files from UI. During the addition of the new file, the user should specify the file name and whether the file is synced from the network. If the file is synced from the network, the user should specify the file's URL and the authentication profile.
• Add the possibility to edit file's settings from UI. The same menu, as with the addition of a new file.
• Add the possibility to copy files. The same menu, as with the addition of a new file, but the new file will use the content of
  the original file.
• Add the possibility to delete files from UI.
• JSON file with patterns should start to have a revision, which can be assigned by the user.

Currently, the regex error message contains ONLY a description of the error, without mentioning a specific place at which the error has occurred. Let's have it extended with:

• character number, at which the error has occurred
• possibility to move the cursor to the error within the regex input field

Increase "comments coverage" within the header files. The public section of the API of each class should have a description.

[GENERAL] Adapt to the dlt-viewer's PLUGIN_INTERFACE_VERSION 1.0.1.

• Change "Files view" from text edit to table
• Add the size of the file to "Files view"
• Add possibility to open file's folder from "Files view"

Fix bug with "End & Home buttons works for horizontal, not vertical scrolling within the search view".

Add possibility to safely use one configuration file by multiple instances of dlt-viewer.

• Add the history of used regexp-s within one LC of application
• Add option, which will make a regex history survive multiple LC-s
• Add the possibility to explicitly clear the regex history
• Add possibility to use regex to search through the history

Add "clear" short-cut and context menu to "console view".

Add the shortcut for "Save pattern" operation

It would be great to have an automated build running for:

• each PR created to master
• each push to master

We need to build against dlt-viewer v 2.20.
Build should run on Linux & Windows.

Add representation of tool-tips for main UI elements of the plugin.

Related to the issue - #16
In order to have a check of RAM cache size on each start-up, we need to do a modification of the settings manager. For arithmetic types, it should take a range of allowed values as an additional input parameter and check persisted value against the range.

• If the persisted value is less than the minimum value from the range - persisted value should be implicitly exchanged to the minimum allowed one.
• If the persisted value is more than the maximum value from the range - persisted value should be implicitly exchanged to the maximum allowed one.

Add the possibility to explicitly trigger re-read settings from the JSON file. It helps a lot during work with multiple instances of dlt-viewer at a time.

Filter's view considers the negative, positive look-behind and look-ahead as groups.

That is wrong. The expected behavior is:

• Either interpret groups as groups and all other PCRE elements as text:
  
  OR
• Fully support the whole structuring of PCRE
  

The first approach seems to be more feasible ( as it is more simple ). But, anyway, we first should be able to properly parse the regex. ANTLR can help with that. We can use ANTLR together with the following description of RPCE to generate a parser:
https://github.com/bkiers/pcre-parser

The parser can be used to simplify the hand-written part of the source code.
This change should be applied, as its absence locks the following task - #82

Add the help section with the links to:

• github repo
• web resource of the plugin
• contacts
• etc.

Add the possibility to mark columns, which should be copied during the copy operation. The option should be separated from the column's visualization setting.

In order to increase the level of the support of our users, we need to start to prepare releases with automatically deliverable artifacts.

Currently, we need the following builds to be supported:

• Windows build for msvc-2013 & Qt 5.5.1 for version 2.19 of the dlt-viewer ( to support one of the actual projects )
• Windows build for msvc-2013 & Qt 5.5.1 for version 2.20 of the dlt-viewer ( to support one of the actual projects )
• Windows build for msvc-2019 & Qt 5.15.0 for version 2.19 of the dlt-viewer
• Windows build for msvc-2019 & Qt 5.15.0 for version 2.20 of the dlt-viewer
• Linux build

For Windows, the artifact should be an archive with dlt-viewer + DLT-Message-Analyzer plugin inside.
For Linux, the artifact should be an archive with dlt-viewer's + DLT-Message-Analyzer's source code inside, ready for being compiled with CMake.

For Linux, build is not actually needed, as only the source code is part of the delivery. Still, the corresponding build should compile the code, in order to prevent the delivery of non-compilable source code.