testartool / testar_dev Goto Github PK

1. Start Spy mode in protocol A, SUT x
2. close SUT to get back to TESTAR Settings Dialog
3. Change to another protocol with another SUT and Start Spy-mode
   -> the changes are not taken into account

Right now, the settings object is passed around to all methods that need it.
But passing it to other methods is painful.
Also, this ensures there is only one settings object.

When we execute a Windows 10 store application like Calculator (C:/Windows/System32/calc.exe), the resulting process is not only a calc.exe process (TESTAR prepare a SUT with this name).
Is a combination of several process (the main one is ApplicationFrameHost: https://www.howtogeek.com/325127/what-is-application-frame-host-and-why-is-it-running-on-my-pc/)

So we can't link correctly TESTAR with this kind of applications using the Command_line.

To link to the SUT we must execute manually the calculator and use the SUT_Windows_title mode.

When using this binary zip TESTAR of Ramon (test.settings and protocol included) https://testar.org/alpro/

I tried to filter some of the URLs that are on the right. First I tried with the CAPS_LOCK + CNTRL which does not work. Then I tried filtering with the regular expression that gives some strange behavior.

Here are two videos that shows it:

https://www.dropbox.com/sh/8z71n9ac8o8quaw/AABHCju_igNYLClQkwvqjCbpa?dl=0

The instructions have this:
After installation of these tools the file compile_w10.bat and clean_w10.bat need to be adapted. CALL "C:\2017\BuildTools\Common7\Tools\VsDevCmd.bat" -arch=x64

I can find w7 of both files and I can find w10.cpp but I can't find the files that are supposed to be adapted.

2- If a tab or windows closes, the execution of the webdriver fails:
org.openqa.selenium.NoSuchWindowException: no such window: target window already closed
I dont know if this is a limitation of the webdriver or it can be activated in some way in the protocol,
but sometimes it may be necessary to read the url of new tabs or windows and close them.

3-If SUT closes unexpected the execution of the webdriver fails:
org.openqa.selenium.WebDriverException: chrome not reachable

Is it possible to add some way to detect that the chrome browser is no longer reachable and save that sequence like unexpectedclose? And avoid TESTAR fail.

Suggestion:
If for any reason TESTAR & webdriver fails, sometimes the chromedriver.exe process stays alive.
Would it be possible to add in the beginning of the webdriver protocol, some way to find the existence of chromedriver processes (old executions) and close them to perform a clean invocation of the chromedriver?

SOLVED:

1- The problems seems something into my environment
The position and interaction of the mouse and actions with the widgets seems wrong. However the widgets are represented correctly by the green dots.
It could be something from my personal environment, but with other protocols and applications it doesn't happen, neither with the previous version of webdriver.

4-Casting Objects into Runtime execution error (check if object its Double or Long):
java.util.concurrent.ExecutionException: java.lang.ClassCastException: java.lang.Double cannot be cast to java.lang.Long
Spy Mode on "http:/angularexampleapp.com" and playing with "Heroes List", I try to replicate but cant do it

5- Frameset web containers problem (Finally not Login & Webdriver):
Making a login on a web page that doesn't change the url, it seems that the widget tree doesn't refresh correctly.

When you hover over an element in the GUI you get a help box with useful information.
In some cases there is no help box.
For example in General Settings tab no info for:

• Force to sequence length
• Offline graph conversion
• Sampling interval
• Protocol
  Also in other tabs there is sometime no info.
  Would be great if every field has a help box.

It should be able to listen during testing (ONLINE oracle) and it should stores it all in a file that can be inspected after testing (OFFLINE oracle).

System.out and System.err listens to the StdOut/Err of TESTAR itself. We should somehow catch the StdOUT/ERR of the SUT when it is started up.

HINTS on a possible solution within Java: https://stackoverflow.com/questions/3643939/java-process-with-input-output-stream

HINTS on a possible solution in cpp WinAPI:
https://msdn.microsoft.com/en-us/library/windows/desktop/ms682425(v=vs.85).aspx

https://msdn.microsoft.com/en-us/library/windows/desktop/ms682499(v=vs.85).aspx

When Testar opens a link in a new tab, the webdriver doesn't switch to that tab.
Instead it continues to test the current page.

TESTAR creates an empty sequence in SPY mode and that cannot be executed in Replay / View mode - throws TESTAR exception.

Generate sequence, oracles, and logs with new names when replaying a sequence, so that you can compare the results between different executions of the same test sequence.

Creating a "Wait for TESTAR" icon, when SUT is showing but TESTAR not yet ready

The new Record-mode does not have visualization on (so that it does not cover the SUT for user actions) - now user does not know when to start doing the actions - maybe TESTAR is not yet ready to record.

I have seen this in VLC (but also in the APRO web site). It recovers and starts selecting other actions after a while, but it repeats the same over and over again.

I will try to make a video next time I see it.

Carlos Sengers was playing around with the Graph DB and he found something that looks like a visualization of this behavior.

The protocol_filter.xml is now in the testar/bin directory in stead of the related settings direcory of the used protocol.

In Spy-mode, with a specific TESTAR run-time control, you could add application specific oracles into specific widgets (a bit like click filter is working).

In addition to filtering actions with RegExp, there could be another field for defining "closing actions". Those actions could be automatically used for "graful shutdown" of the SUT, as is possible in stopSystem(), instead of just killing the process.

When starting Spy mode Shift+3 will do nothing, only after Shift+2.

Shift+1 is a toggle key for actions.
Shift+2 is a toggle key for some information.
Suggestion to make Shift+3 a toggle key for detailed information.
OR perhaps better
Shift+1 is a toggle key for actions.
Shits+2 is a toggle key for some information->detailed information->no information-> some information....
Shift+3 no function.

Idea: having a way to define RegExp for titles of closing actions (in a similar way to suspicious titles), and not reporting failing sequence when such an action closes the SUT.

These closing actions could be used for "graceful shutdown" of the SUT in the end of sequence.

This test was executed on pekka dev version, as the master has other issues.
This bug does not happen on Win10.

The HTML report generates the first state and selects an action, but executing action has some bug.

Trace:

Protocol_desktop_generic: selectAction(), (Overriding Urko's Grapher and AbstractProtocol implementation)
AbstractProtocol: saving state into HTML Sequence Report
class nl.ou.testar.GuiStateGraphWithVisitedActions: new state, selecting randomly from 5 available actions
class nl.ou.testar.IdBasedGuiState: addVisitedAction: action removed from the unvisited actions
[END] Running processes (14):
ScreenshotSerialiser finished
TestSerialiser finished
TESTAR sequence graph dump on way ...
Sync XML graph: synced
... finished TESTAR sequence graph dump
TESTAR grapher reset
Exception <Unable to update layered window!> has been caught; Stack trace:
[1] org.fruit.alayer.windows.Windows.UpdateLayeredWindow(Native Method)
[2] org.fruit.alayer.windows.GDIScreenCanvas.end(GDIScreenCanvas.java:180)
[3] org.fruit.monkey.AbstractProtocol.runAction(AbstractProtocol.java:905)
[4] org.fruit.monkey.AbstractProtocol.runTest(AbstractProtocol.java:1196)
[5] org.fruit.monkey.AbstractProtocol.run(AbstractProtocol.java:1512)
[6] org.fruit.monkey.AbstractProtocol.run(AbstractProtocol.java:131)
[7] org.fruit.monkey.Main.main(Main.java:143)

A part of the code (the part that stores the widgets) is implemented in the specific protocol class (for now). I made this choice since it was the only location that new the available widgets which are used to select the next action. I will update the documentation

If we use the Java Swing version, the native elements created at the operating system level are not correctly recognized.

The buttons Close, Minimize, Maximize and others, not belonging to the Java Swing interface and their JVM associated, are not detected by TESTAR.

Sometimes if TESTAR runs with several sequences, it can be seen that the last or last actions that appear as performed inside the log (output / log / sequenceX.log), it doesn't appear as executed in their corresponding sequence file (output / sequence / sequenceX)

This can be compared by opening the log from a text editor and opening the corresponding sequence from the "View" mode offered by TESTAR.

Study the benefits of the Java 9 ,10 and 11 environments in relation to TESTAR.
Prepare an execution environment and perform compilation + execution tests.

UPDATE:

• Branch "testar_new_jdk" allows the compilation "gradlew installdist" and execution of TESTAR into a VM with Java JDK12.
  c7d480a

• [ OK ] Compiles and allow the execution of Notepad + Windows UIAutomation

• [ OK ] Compiles and allow the execution of Calculator.jar + Windows UIAutomation

• [ Swing ERROR Temporally SOLVED] c11ee17
  Access Bridge method:
  Windows.GetHWNDFromAccessibleContext(vmidAC[0],vmidAC[1]);

When TESTAR finds a failure, it should automatically try to find the shortest sequence to reproduce the failure.

• first try if the failure can be reproduced with the same sequence, then try to find a shorter path to it

Happened with IE but not with Chrome:

SUT has a modal popup that is another process, and TESTAR tries to bring original process to foreground and that does not work because of modal popup.

Design a layout of the datamodel which will be used to store the data

The webdriver protcol /webdriver layer designates slide actions incorrectly, resulting in unexecutable actions.
And this also slows down the action visualization.
Example : elements of the lists in the footer on https://www.ou.nl

To remember, an environment running now on Ramons server.

• .NET framework 4.0 should be installed first, otherwise the newer has to be uninstalled
• Windows SDK 7.1 is needed, not available in the latest Visual Studio versions
• the paths in Makefile_w7 has to be modified because ..\bin\something paths dont work

Because they will then get built, so we can spot errors.

Eclipse already adds them when importing with gradle eclipse, but the folder structure is wrong:

Description	Resource	Path	Location	Type
The declared package "" does not match the expected package "settings.desktop_generic"	Protocol_desktop_generic.java	/testar/resources/settings/desktop_generic	line 1	Java Problem

We can:

• Put them in the right package and make Testar understand class names with packages; or
• Keep them where they are but add logic to make Eclipse happy.

In either case it seems to make sense to automatically build the protocols with Gradle.

After build and first start up of TESTAR, the amount of sequences is per default 1, but when doing a GENERATE TEST, this amount is not enforced.

When changing the amount of sequences, this does not solve this.

However, shen adapting the amount of actions, the amount of sequences is also taking into account.

This must be a bug somewhere in saving and/or overwritting the settings file.

The testar.zip file created with .\gradlew distZip contains two times the json-simple-1.1.1.jar file. When unzipping the file you get a ' Confirm File Overwrite' message box (at least with IZArc) asking if you want to overwrite the file. This is a bit annoying.

Problem checked with JEdit and Freemind.

Java applications executed as .exe, dont link correctly with TESTAR using Command_line.

The created SUT is defined as JEdit.exe / Freemind.exe, but the resulting process in the OS is a java.exe / javaw.exe process.

To link to the SUT we must run the .jar application, fortunately JEdit & Freemind contain it.

This is maybe unwanted behaviour instead of a bug, but it messes up the action selection, because random chance of selecting editable widgets is much higher than clickable widgets.

TESTAR generates multiple edit actions with different text for each editable widget, for example text box.

A better behaviour would be generating one edit action for each editable widget, then generating the input for the edit if it is selected, or just generating one input for the action.

If generating many sequences in one TESTAR session, only one HTML sequence report file is created. Should be one HTML file per sequence (preferably in sync with sequence file numbering).

In Record-mode: input text is not captured correctly, sometimes some text is missing on the captured test sequence

Think about the possibility of integrate Jenkins and TESTAR Github, to automate the testing of pull requests.

The main idea would be the execution of a virtual machine that contains the TESTAR version of the pull request, and design a battery of test cases that can be executed from the command line.

Example: testar SUTConnectorValue=C:\Windows\System32\notepad.exe Sequences=20 SequenceLength=100

After each execution check that the result was correct.
(nothing has failed and there are the corresponding logs?)

An idea:

How about providing a list of all the running processes that user can select from, bringing the selected process to foreground and letting user to confirm that is the SUT.

Just a few lines in README.MD to get someone started.

When using the webdriver module, the above exception is thrown.
It originates in sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method).

This way invalid regular expressions are not accepted through the dialog. Now what happens is that no warning whatsoever is given and TESTAR just does not apply it.

it says:

After installation of these tools the file compile_w10.bat and clean_w10.bat need to be adapted. CALL "C:\2017\BuildTools\Common7\Tools\VsDevCmd.bat" -arch=x64

Where are compile_w10.bat and clean_w10.bat?
Itt should be made clearer that you need to edit them.
I have adapted the README.rd in tanja_dev branch. See if you like this better.

By mentioning Close in de Filters tab in the Testar GUI you see no green dot in Spy mode after shift+1.
That is correct.

By using the clickfilter-feature I'am trying to deselect Close, so that it is now a possible action again.
shift_Ctrl.

The result is a green box around Close. Why?

I expected a green dot again and Close deleted from the test.settings file.

The green box around close will not be selected during Generate.

In TESTAR protocol it is possible to define startSUT() and beginSequence() specific actions before starting the normal testing. It would be useful if the user could record a script (as now in GenerateManual mode), and tell TESTAR to run that script (from sequence file) when starting the SUT or a sequence.

Maakt het bouwen misschien nog wat soepeler voor mensen die van Ant komen.

This capture represents the detection of the different widgets when we use the native VLC interface.

VLC can be tested using the native interface, however, if we use the Skins that modify the VLC GUI, the widgets are not recognized. Skins components are not detected by TESTAR or by the Windows OS.

Tools -> preferences -> use custom skin -> Select the downloaded skin
(I have tried the "eDark", "NIce" and "Debian Red Coast" skins on the Windows 10 OS)

For an experiment, to observe execution traces for temporal behavior (e.g. 'eventually'), I created a very simple java application: a single window java application that simulates a stack with a depth of 7.
I deliberately wanted to limit the actions to 'control' TESTAR.

When inspecting the log, i noticed that the abstract ID's of the executed Actions showed up in the output (logs, XML, .DOT)

conforming to the TESTAR_manual 1.3 page 51:

• Concrete
  Action ID: the ID for an action a in state s is calculated as id(a) =f(id(s) + p1 + … + pk) where pi are the action properties.
• Abstract
  Action ID: similar to the concrete action ID, but discarding properties of the action (e.g. texts typed, keys pressed). For example, we can refer to writing in a concrete textbox, no matter of the text used to write

In other words: Abstract ID's for actions are derived from the concrete state id. This functionality is implemented in TESTAR. So, this is by design and as such this is not a code bug.

However, i would have expected that an abstract action-id would be derived from the abstract state-id. Comparable to the way an abstract state is derived from the abstract widget tree.
The current implementation does not condense the number of actions when they are abstracted, which is usually a reason for applying abstraction.
( see
AbstractActions 20180529 CSS.pdf
)

Is this worth changing?

By disabling all the notepad buttons, leaving only the text field and running about one hundred actions, this issue can be observed using the View mode of TESTAR or using the desktop_simple_stategraph protocol (with the html report).

Sometimes the screenshot of the state that transits from an action performed is not shown correctly in these captures.

Current status: we have the text "161541072" in the field
We perform action: Type text "-1408540088"
New state to observe: we have the text "-1408540088" in the field

However, a capture of another different state is shown:

Investigate if it occurs only with the text input fields or also with other widgets

gradlew run tries to read about ./testar/settings, ./testar/output, /testar/windows.dll etc, path that does not exist.

This problem does not occur with installDist because the directory created is correct ./bin/settings, ./bin/output, ./bin/windows.dll, etc.

If you move the files ./testar/target/scripts in ./testar you can work with gradlew run, without having to run installDist every time.

Option 1: Move the commented files (2 folders and 4 files)

Option 2: Modify Main.java code, SettingsDialog.java, Windows.java and the path structure of test.settings,
However, this does not allow working with installDist, since the read path is modified.

Option ?: Can this be solved from some build.gradle file?