konveyor / tackle-test-generator-core Goto Github PK

License: Apache License 2.0

Java 100.00%

tackle-test-generator-core's Introduction

Tackle Test Generator Core

This repository contains the core components of the Tackle-Test tooling for performing automated test generation and differential testing on two application versions (in the context of application modernization). These components are invoked via the Tackle-Test CLI.

The main capabilities enabled by these tools are:

Automated generation of JUnit test cases using different strategies: adaptive random, code-coverage based, and CTD-amplified coverage goals.

This feature leverages two open-source automated unit-test generation tools for Java: Randoop, which performs adaptive-random test generation, and EvoSuite, which applies evolutionary algorithms to generate tests that are directed toward achieving code-coverage goals (e.g., method, statement, branch, and exception coverage).

Additionally, a third test-generation strategy applies combinatorial test design (CTD) to generate test cases that exercise methods using various combinations of subtypes of the declared method parameter types. CTD modeling optimizes the type combinations that computed as coverage goals, while also enabling the selection of different interaction levels.
Automated addition of assertions to the test cases to enable differential testing. This is done by executing test sequences, and observing program states during execution, on the monolith. Based on the observed program states, assertions are generated to check that the same program states occur in the transformed application during test execution. The goal in creating these assertions is to detect meaningful differences in observable program states that indicate transformation bugs or broken functionalities in the transformed application.

Building the Tool

The tool can be built using maven, following these steps:

From the folder tackle-test-generator-unit, download Randoop jars

mvn download:wget@get-randoop-jar download:wget@get-replacecall-jar

Add the following server entry to Maven settings.xml (~/.m2/settings.xml; create one if it doesn't exist), replacing USERNAME with your GitHub username and PASSWORD with your password or a personal access token. This is needed for downloading the specific builds of EvoSuite jars that are hosted in a Maven registry on GitHub Packages.
```
<servers>
  <server>
  <id>github-sinha108</id>
  <username>USERNAME</username>
  <password>PASSWORD</password>
 </server>
</servers>
```
Build test-generator-core jar
```
mvn package
```

CTD modeling and test-plan generation is done using the NIST Automated Combinatorial Testing for Software tool, which is packaged with the test generator (in the lib directory).

Tool Usage

Although the components in this repository provide programming APIs and can also be invoked from the command line, the primary mechanism for usage of the core components is via the Tackle-Test CLI, which provides various commands and configuration options for invoking the core components.

Components of the Toolchain

This following figure illustrates the various components of Tackle Test Generator Core, and the input/output from each component.

The primary components among these are:

The CTD Test-Plan Generator: It takes as input a set of application classes on which it performs static analysis to create a CTD model for each public method in the classes. Each model is then run through CTD optimization, which produces a test plan for a method. The test plan for a method consists of a set of rows, each of which is a coverage goal, specifying a combination of types for method parameters. The output of the test-plan generator is a consolidate test plan (in JSON format) for all public methods of all classes provided as input.

If particular application classes as not provided as input, the test-plan generator analyzes all application classes, which it discovers via user-provided classpath configuration.
The Test-Sequence Initializer: This component takes as input a CTD test plan and leverages Randoop and EvoSuite test cases generators, targeting them at the classes occurring in the test plan, to generate test sequences for public methods of the classes, as well as for constrcutors of parameter types appearing in the CTD test plans. A test sequence for a target method consists of a "setup sequence" that sets up the receiver object state (if the target method is virtual) and actual parameters of the target method, followed by a call to the target method. The output of this component is a set of "initial" test sequences that serve as building blocks for the synthesis of extended test sequences for covering the test-plan rows. The test sequences are stored in a stringified form in a JSON file.
The Test-Sequence Extender: This is the final step in the creation of CTD-amplified tests. The extender takes as input the CTD test plan and the building-block test sequences. It iterates over each row in the test plan and attempts to synthesize a test sequence that covers the row; i.e.., the sequence invokes the target method with the parameter-type vector specified in the row. The extender does this by first creating a pool of sequences from the building-block sequences and then sampling from the pool to create subsequences for types required for a test plan. These subsequences are concatenated, with a call to the target method added at the end, to construct the covering or extended test sequence.

After constructing an extended sequence, the extender runs the sequence in-memory (via reflection) to check whether the sequence executes successfully (i.e., whether the sequence is a valid sequence). Failing test sequences are discarded.

The extender can, optionally, also add assertions to test cases. To do this, the extender executes a sequence in-memory (via reflection), accesses the public states of objects constructed during test execution, and creates assertions on those states. Assertions are added after every statement of the sequence which creates new objects. To handle non-determinism in run-time state, a sequence can be executed multiple times (via a configurable option) and an assertion on an object field is created only if the same value is seen in all executions.

The output from the extender consists of JUnit test classes and summary reports.

Code of Conduct

Refer to Konveyor's Code of Conduct here.

tackle-test-generator-core's People

Contributors

Stargazers

Watchers

tackle-test-generator-core's Issues

Add an option to exclude a list of classes

Is your feature request related to a problem? Please describe.

We currently have an option to target specific classes via the target_class_list config option. We need a similar option for excluded specific classes, in case only a few classes need to be excluded, either because they are not affected by the changes of because of issues encountered in test generation for them.

Describe the solution you'd like

Add an excluded_class_list config option. Any class appearing in them will be excluded from the target list for test generation.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Improve parsing of base test sequences

Is your feature request related to a problem? Please describe.

A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Add Java apps to be used as test inputs

Currently, testgen-core is tested using daytrader and irs only, which is inadequate. Add a few apps from the SF110 benchmark so that the core tools are tested on more variety of test inputs.

Add integration tests for modeling, bb sequence generation, and sequence extension

Add test cases that run CTD modeler, building-block sequence generator, and sequence extender on input apps

Publish testgen-core jar

Create workflow for publishing testgen-core jar (on PR merge to main) to GitHub packages

https://docs.github.com/en/actions/guides/publishing-java-packages-with-maven#publishing-packages-to-the-maven-central-repository-and-github-packages

Modeling fails when sub or super classes not found

Describe the bug

getAllSuperTypes and getAllConcreteTypes fail when they cannot load an implementing/super class. Instead they should log a warning and continue. Example: app 52 of SF110.

Attachments

To help us diagnose the failure, please add these attachments to the issue, as appropriate:

The toml file containing the configuration information used
The log file created by running the CLI with the --verbose and --log-level INFO options
If possible, a zip file containing the application classes, and the file specified as
app_classpath_file option in the toml file together with the library dependencies listed
in the file
If the failure occurred for the generate command, please attach the following files if
they were created during the failing run:
- The CTD test plan JSON file (<app-name>_ctd_models_and_test_plans.json)
- The building-block test sequences JSON files (<app-name>_EvoSuiteTestGenerator_bb_test_sequences.json
  and/or <app-name>_RandoopTestGenerator_bb_test_sequences.json)
If the failure occurred for the execute command, please attach the zip file of the directory
containing the generated test JUnit test cases

Environment information

CLI version
OS: [e.g., Linux, macOS, Windows]

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

Add any other context about the problem here.

Disable stdout messages from acts

Acts prints messages to stdout describing the coverage of created test combinations during test-plan generation; disable these messages by setting System.out to null print stream during the call to the relevant acts API.

Test case instrumentation for measuring interaction coverage

Is your feature request related to a problem? Please describe.

We want to measure interaction coverage of a given test suite (e.g., dev-written tests) in order to exclude that coverage from the space of all n-way type interactions that the ctd-amplified test suite is targeting.

Describe the solution you'd like

Dynamically analyzing the types passed via method interfaces using source-level instrumentation seems the easiest way to achieve this goal. Per method under test, once the passed types are figured out, we can give this type combinations set as an input test suite to ACTS, and generate a test suite in extend mode that augments an existing test suite to reach 100% interaction coverage. The test plan that the modeler creates for test generation will include only the additional combinations that ACTS generated.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Generate test cases at package level of the class under test

Is your feature request related to a problem? Please describe.

Currently the generated test cases is not part of the package of the class under test (CUT). This prohibits from accessing package and protected visibility level CUT methods and fields directly in the test code.

Describe the solution you'd like

Generate the test case as part of the same package as the CUT, so it can access all non-private CUT code elements.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Test generation should target also non-public methods

Is your feature request related to a problem? Please describe.

Currently CTD modeling targets only public methods. This reduces coverage for methods with other visibility levels.

Describe the solution you'd like

CTD should target also methods with package and protected level visibility. As a result it will be also able to target all non-private types. The resulting tests should be included in the same package as the class under test (so that the non-public methods and parameters can be accessed), and the folder hierarchy should follow the package hierarchy (similarly to EvoSuite output tests). The extender will be able to generate concrete values for package and protected level parameters, and assign package and protected level return types to new variables that can be asserted.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Terminate evosuite and randoop after reaching a timeout

Is your feature request related to a problem? Please describe.

Currently randoop may hang if sequence execution invokes program under test non-terminating threads

Describe the solution you'd like

We should limit evosuite and randoop global execution time using a timeout that is based on the number of classes and the time limit per class. If timeout is reached, we can kill the processes and use the sequences they managed to generate while they were running.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Remove obsolete references in core code

Extender is failing to generate any test cases from test generators output

Describe the bug

When running on https://github.com/shashirajraja/onlinebookstore/ extender fails to generate any test cases:


* CTD interaction level: 1
* Total number of classes: 13
* Targeting 10 classes
* Skipping 3 classes with only private methods
* Created a total of 8 test combinations for 8 target methods of 10 target classes
[tkltest|13:38:56.820] Computing test plans with CTD took 2.39 seconds
[tkltest|13:38:56.820] Generating basic block test sequences using CombinedTestGenerator
[tkltest|13:38:56.821] Test generator output will be written to onlinebookstore_CombinedTestGenerator_output.log
[tkltest|13:41:34.357] Generating basic block test sequences with CombinedTestGenerator took 157.54 seconds
[tkltest|13:41:34.357] Extending sequences to reach coverage goals and generating junit tests
* Read test plans for: 8 classes, 8 methods
* Read 2858 base test sequences
* Starting sequence parsing
*   Full:2858  Part:0  Skip:0  Excp:0
* Class sequence pool: 9 classes, 9 sequences
* Method sequence pool: 0 methods, 0 sequences
* Parsed initial test sequences for 8 classes; total parsed sequences (full+partial): 2858
* Generating extended test sequences
* Partition: monolithic
* Processing class servlets.ReceiptServlet
*   1 methods .
*   generated 0 test sequences
*   -- class test-plan coverage rate: 0.00% (0/1)
* Processing class servlets.UserRegisterServlet
*   1 methods .
*   generated 0 test sequences
*   -- class test-plan coverage rate: 0.00% (0/1)
* Processing class servlets.BuyBooksServlet
*   1 methods .
*   generated 0 test sequences
*   -- class test-plan coverage rate: 0.00% (0/1)
* Processing class servlets.RemoveBookServlet
*   1 methods .
*   generated 0 test sequences
*   -- class test-plan coverage rate: 0.00% (0/1)
* Processing class servlets.AdminLoginServlet
*   1 methods .
*   generated 0 test sequences
*   -- class test-plan coverage rate: 0.00% (0/1)
* Processing class servlets.UserLoginServlet
*   1 methods .
*   generated 0 test sequences
*   -- class test-plan coverage rate: 0.00% (0/1)
* Processing class servlets.AddBookServlet
*   1 methods .
*   generated 0 test sequences
*   -- class test-plan coverage rate: 0.00% (0/1)
* Processing class servlets.ViewBookServlet
*   1 methods .
*   generated 0 test sequences
*   -- class test-plan coverage rate: 0.00% (0/1)
* === total CTD test-plan coverage rate: 0.00% (0/8)
* created 0 test classes in total, with 0 test methods and 0 assertions

tkltest-output-unit-onlinebookstore.zip

Attachments

To help us diagnose the failure, please add these attachments to the issue, as appropriate:

The toml file containing the configuration information used
The log file created by running the CLI with the --verbose and --log-level INFO options
If possible, a zip file containing the application classes, and the file specified as
app_classpath_file option in the toml file together with the library dependencies listed
in the file
If the failure occurred for the generate command, please attach the following files if
they were created during the failing run:
- The CTD test plan JSON file (<app-name>_ctd_models_and_test_plans.json)
- The building-block test sequences JSON files (<app-name>_EvoSuiteTestGenerator_bb_test_sequences.json
  and/or <app-name>_RandoopTestGenerator_bb_test_sequences.json)
If the failure occurred for the execute command, please attach the zip file of the directory
containing the generated test JUnit test cases

Environment information

CLI version
OS: [e.g., Linux, macOS, Windows]

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

Add any other context about the problem here.

Investigate coverage gain from JEE support

Enabling JEE support should increases coverage in JEE apps (e.g., DayTrader), but this does not seem to be happening consistently.

NullPointerException in CrawljaxRunner with Clickables Specification

Describe the bug

I created a clickables spec with dont_click items but no click items and got this exception when running the UI generate command with the CLI:

Error creating Crawljax configuration: null 
java.lang.NullPointerException
        at org.konveyor.tackletest.ui.crawljax.CrawljaxRunner.updateClickablesConfiguration(CrawljaxRunner.java:237)
        at org.konveyor.tackletest.ui.crawljax.CrawljaxRunner.createCrawljaxConfiguration(CrawljaxRunner.java:497)
        at org.konveyor.tackletest.ui.crawljax.CrawljaxRunner.main(CrawljaxRunner.java:645)

I think if a user creates a clickables specification file, but does not include any click.element, or dont_click.element items, or both, this exception occurs.

// process click spec
TomlTable[] clickElementSpec = clickableSpec.getArray("click.element")
    .toList()
    .toArray(new TomlTable[0]);

// process don't click element spec
TomlTable[] dontclickElementSpec = clickableSpec.getArray("dont_click.element")
    .toList()
    .toArray(new TomlTable[0]);

Meaningful names for generated test methods

Create more meaningful names for generated test methods (instead of test0, test1, ...). The test name should include the target method name and be of the format test<methodname><int>() where <int> is used when multiple test methods are generated for the same target method.

Add assertions on receiver objects

Is your feature request related to a problem? Please describe.

When an executed statement doesn't return a value, we don't add assertions for it. Instead we should assert the member values of the receiver object.

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Assertions on null values should use assertNull method

Describe the bug

Currently assertions on null values use the default assertEquals method, causing in some cases ambiguous references to assertEquals (see for example app 46 from SF110). In cases of null values we should use instead assertNull and avoid the ambiguity.

Attachments

To help us diagnose the failure, please add these attachments to the issue, as appropriate:

The toml file containing the configuration information used
The log file created by running the CLI with the --verbose and --log-level INFO options
If possible, a zip file containing the application classes, and the file specified as
app_classpath_file option in the toml file together with the library dependencies listed
in the file
If the failure occurred for the generate command, please attach the following files if
they were created during the failing run:
- The CTD test plan JSON file (<app-name>_ctd_models_and_test_plans.json)
- The building-block test sequences JSON files (<app-name>_EvoSuiteTestGenerator_bb_test_sequences.json
  and/or <app-name>_RandoopTestGenerator_bb_test_sequences.json)
If the failure occurred for the execute command, please attach the zip file of the directory
containing the generated test JUnit test cases

Environment information

CLI version
OS: [e.g., Linux, macOS, Windows]

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

Add any other context about the problem here.

Facing issue while generating test cases using tackle-test

Executed below command to generate test case and received following error. Please note I am running the command in windows operating system.

Command: tkltest-unit --config-file ./test/data/irs/tkltest_config.toml generate ctd-amplified

Error: Could not find or load main class org.konveyor.tackle.testgen.model.CTDTestPlanGenerator
Caused by: java.lang.ClassNotFoundException: org.konveyor.tackle.testgen.model.CTDTestPlanGenerator

Please note that I have downloaded all java dependencies and executing the command in python virtual environment

Extend combinatorial coverage measurement to support interfaces with compound types

Is your feature request related to a problem? Please describe.

Currently combinatorial coverage of generated test cases is computed for methods with no compound types in their interfaces.

Describe the solution you'd like

Extend current combinatorial coverage measurement to support compound types. This will require collecting combinatorial coverage of randoop and evosuite generated test cases (probably using dynamic analysis), and creating a cross product of all element combinations for the combinatorial model.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Check why in some cases timestamps are asserted

Describe the bug

Timestamps should not be asserted, but in some cases they are. We should investigate why they escape our repeated execution to eliminate random/changing values

Attachments

To help us diagnose the failure, please add these attachments to the issue, as appropriate:

The toml file containing the configuration information used
The log file created by running the CLI with the --verbose and --log-level INFO options
If possible, a zip file containing the application classes, and the file specified as
app_classpath_file option in the toml file together with the library dependencies listed
in the file
If the failure occurred for the generate command, please attach the following files if
they were created during the failing run:
- The CTD test plan JSON file (<app-name>_ctd_models_and_test_plans.json)
- The building-block test sequences JSON files (<app-name>_EvoSuiteTestGenerator_bb_test_sequences.json
  and/or <app-name>_RandoopTestGenerator_bb_test_sequences.json)
If the failure occurred for the execute command, please attach the zip file of the directory
containing the generated test JUnit test cases

Environment information

CLI version
OS: [e.g., Linux, macOS, Windows]

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

Add any other context about the problem here.

Improvements in object construction

Is your feature request related to a problem? Please describe.

A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Add a log file for slient failures that may reduce coverage

Is your feature request related to a problem? Please describe.

Silent failures for sequence parse errors and classes not found due to missing libraries may result in overall reduced coverage.
Describe the solution you'd like

We should log these failures in a dedicated place for investigation and fixing.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Refine static analysis for Object type hierarchy

Is your feature request related to a problem? Please describe.

In the case of types with a huge hierarchy (like Object type), we get too many types in the hierarchy including infeasible/irrelevant ones. This causes CTD and subsequent test generation to generate many infeasible test cases.

Describe the solution you'd like

We need to refine the analysis.

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Extender failure on missing library dependencies

Describe the bug

Missing library dependencies for the app under test can cause the extender to crash with NoClassDefFound error in some cases (if a type from those libraries occurs in a target method signature or needs to be instantiated during sequence extension).

Attachments

To help us diagnose the failure, please add these attachments to the issue, as appropriate:

The toml file containing the configuration information used
The log file created by running the CLI with the --verbose and --log-level INFO options
If possible, a zip file containing the application classes, and the file specified as
app_classpath_file option in the toml file together with the library dependencies listed
in the file
If the failure occurred for the generate command, please attach the following files if
they were created during the failing run:
- The CTD test plan JSON file (<app-name>_ctd_models_and_test_plans.json)
- The building-block test sequences JSON files (<app-name>_EvoSuiteTestGenerator_bb_test_sequences.json
  and/or <app-name>_RandoopTestGenerator_bb_test_sequences.json)
If the failure occurred for the execute command, please attach the zip file of the directory
containing the generated test JUnit test cases

Environment information

CLI version
OS: [e.g., Linux, macOS, Windows]

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

Add any other context about the problem here.

Wrong assertion created when recorded type is a subclass of formal type

Describe the bug

A wrong assertion is being created when unboxing is used but the formal return type asserted is a super class of the runtime return type recorded

Attachments


java.lang.Object obj5 = SomeClass.someMethod((java.lang.Object)true, 0L);
assertEquals(true, obj5.booleanValue());

To help us diagnose the failure, please add these attachments to the issue, as appropriate:

The toml file containing the configuration information used
The log file created by running the CLI with the --verbose and --log-level INFO options
If possible, a zip file containing the application classes, and the file specified as
app_classpath_file option in the toml file together with the library dependencies listed
in the file
If the failure occurred for the generate command, please attach the following files if
they were created during the failing run:
- The CTD test plan JSON file (<app-name>_ctd_models_and_test_plans.json)
- The building-block test sequences JSON files (<app-name>_EvoSuiteTestGenerator_bb_test_sequences.json
  and/or <app-name>_RandoopTestGenerator_bb_test_sequences.json)
If the failure occurred for the execute command, please attach the zip file of the directory
containing the generated test JUnit test cases

Environment information

CLI version
OS: [e.g., Linux, macOS, Windows]

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

Add any other context about the problem here.

CTD modeling of interfaces with parameterized types

Describe the bug

Handling of interfaces with parameter types fails during CTD modeling:

Error:    CTDModelingTest.testGenerateModelsAndTestPlansForAllClasses:97->compareModels:129 expected:
<[willDecode(Ljava/lang/String;)Z, decode(Ljava/lang/String;)Ljava/lang/Object;]> but was:<[willDecode(Ljava/lang/String;)Z, decode(Ljava/lang/String;)Lcom/ibm/websphere/samples/daytrader/web/websocket/ActionMessage;]>

Remove evosuite jar from repo

Remove evosuite jars and download them from maven registry hosted on github

Make sure unit tests compile upon generating them

Is your feature request related to a problem? Please describe.

Currently the unit tests we are creating might not be compilable. This is detected only when running execute command in cli which is too late.

Describe the solution you'd like

We should attempt to compile the unit tests upon generation. Uncompilable tests should be flagged and removed from the final test set.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Create a junit file for each class as soon as the generation of its test sequences is completed

Is your feature request related to a problem? Please describe.

See konveyor/tackle-test-generator-cli#99

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Handle enum parameter types for target methods

Is your feature request related to a problem? Please describe.

Currently, the test generator does not handle the case where a target method parameter is an enum type. The extender should detect such types and create an assignment of a value from the set of enumerated values of the type.

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

OWNERS.md Missing

OWNERS.md Missing per https://github.com/konveyor/community/blob/main/MAINTAINERS.md guidelines

Upgrade to use newest Java 11 (or newest Java LTS versions)

Is your feature request related to a problem? Please describe.

Java 8 is an older version of Java, still valid as LTS version, but not optimized with the latest features or capabilities of newer versions of Java. Java 11 is still the most extended, also a LTS version, and common in enterprise environments, it makes sense to use this runtime to build the tool.

Describe the solution you'd like

Update build files (e.g.: pom.xml) and documentation to use newer versions of Java (at least Java 11) instead of Java 8.

Additional context

Use a newest LTS version instead of Java 8 will improve the quality and perception of the tool for the users.

Rapid type analysis should consider cases beyond assignments

Describe the bug

For example in SF110 app 6 (jnfe):

transformer.transform(new DOMSource(doc), stringResult);

Attachments

To help us diagnose the failure, please add these attachments to the issue, as appropriate:

The toml file containing the configuration information used
The log file created by running the CLI with the --verbose and --log-level INFO options
If possible, a zip file containing the application classes, and the file specified as
app_classpath_file option in the toml file together with the library dependencies listed
in the file
If the failure occurred for the generate command, please attach the following files if
they were created during the failing run:
- The CTD test plan JSON file (<app-name>_ctd_models_and_test_plans.json)
- The building-block test sequences JSON files (<app-name>_EvoSuiteTestGenerator_bb_test_sequences.json
  and/or <app-name>_RandoopTestGenerator_bb_test_sequences.json)
If the failure occurred for the execute command, please attach the zip file of the directory
containing the generated test JUnit test cases

Environment information

CLI version
OS: [e.g., Linux, macOS, Windows]

Screenshots

If applicable, add screenshots to help explain your problem.

Additional context

Add any other context about the problem here.

Replace FOCUS with ACTS tool

ACTS by NIST (java): https://www.nist.gov/programs-projects/automated-combinatorial-testing-software-acts

Optionally generate test cases from failing sequences

Is your feature request related to a problem? Please describe.

Currently failing sequences are always discarded, however in some cases they may actually reveal issues in the code

Describe the solution you'd like

Optionally generate junit test cases from failing sequences, with exception handling code and assertions that check for the raised exception. We should probably come up with a heuristic in which cases it makes sense to generate such sequences and in which cases these are most probably false failures.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Add a timeout during sequence execution to avoid hanging

Is your feature request related to a problem? Please describe.

Certain sequence executions may hang, e.g., due to non-terminating threads. Test generation might get stuck in such situations.

Describe the solution you'd like

We need to perform synthesized sequence execution from a separate thread and consider it failing if it reaches a certain timeout (e.g., 120 seconds).

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Refine assertion generation on constructors

Is your feature request related to a problem? Please describe.

Currently we assert on all members of an object being constructed, including default initialization. We should not assert when the constructor is argument-less or when the members are statistically initialized or initialized with default values

Describe the solution you'd like

A clear and concise description of what you want to happen.

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

Avoid assertions on simple assignments

Is your feature request related to a problem? Please describe.

Currently we create assertions for any statement in which new values/objects are created. However it doesn't make sense to assert values that are the result of simple primitive assignments.

Describe the solution you'd like

Avoid adding assertions for simple primitive assignments

Describe alternatives you've considered

A clear and concise description of any alternative solutions or features you've considered.

Additional context

Add any other context or screenshots about the feature request here.

dont_click.element exclude all children of all tags

Describe the bug

Instead of don'tClickChildrenOf, the dont_click.element specification item is being used with the under_xpath option. I wanted to exclude all children, no matter the type of tag, so I specified

[[dont_click.element]]
  under_xpath = //div[@id='ibm-masthead']//*

and got the exception:

[2022-03-21 12:21:16] [SEVERE ] [org.konveyor.tackletest.ui.crawljax.CrawljaxRunner main] Error creating Crawljax configuration: Tag name cannot be null 
java.lang.NullPointerException: Tag name cannot be null
        at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:895)
        at com.crawljax.core.configuration.CrawlActionsBuilder.dontClick(CrawlActionsBuilder.java:143)
        at com.crawljax.core.configuration.CrawlRules$CrawlRulesBuilder.dontClick(CrawlRules.java:319)
        at org.konveyor.tackletest.ui.crawljax.CrawljaxRunner.handleClickablesElementSpec(CrawljaxRunner.java:215)
        at org.konveyor.tackletest.ui.crawljax.CrawljaxRunner.updateClickablesConfiguration(CrawljaxRunner.java:247)
        at org.konveyor.tackletest.ui.crawljax.CrawljaxRunner.createCrawljaxConfiguration(CrawljaxRunner.java:497)
        at org.konveyor.tackletest.ui.crawljax.CrawljaxRunner.main(CrawljaxRunner.java:645)

konveyor / tackle-test-generator-core Goto Github PK

tackle-test-generator-core's Introduction

Tackle Test Generator Core

Building the Tool

Tool Usage

Components of the Toolchain

Code of Conduct

tackle-test-generator-core's People

Contributors

Stargazers

Watchers

Forkers

tackle-test-generator-core's Issues

Recommend Projects

Recommend Topics

Recommend Org