finraos / datagenerator Goto Github PK

Ability to set complex objects as values for variables within the dependency model.

Refer to:
https://github.com/FINRAOS/DataGenerator/wiki/DataGenerator-V2-Requirements#r14-common-rules-for-r11-r12-and-r13

Implement the following:
R1.4.3
R1.4.4

DataGenerator Website

• Update website to contain V2 release notice/advertisement banner.
• Quick Start for V2 on landing page. (#50)
• Update User Docs/ API Docs/FAQs
• Update download latest executable to have V1 and V2.

Review Issue #39

Create an interface to allow the user of DG to generate a set amount of rows (like 1 million rows instead of 15 million). Provide a standard implementation, as well as ensure the interface is useful for people using DG across other distributed frameworks #36

Fix all checkstyle issues. Switch checkstyle to break the build on errors.

We have common data types, like:

• Date
• State
• Country
• Currency
• Market symbol
• Short random text
• Long random text (with spaces and so on)
• Phone
• Address
• zip code
• First Name, Last Name
• SSN
  ...

So, if it will be possible to use it inside xml configuration it will be really useful. Now it's necessary to create custom solutions.

Now new user will be frustrated about real power of DG because of lack of knowledge.

Review #39

One customer want to generate json data (to use it as mocks in testing his system)...

Maybe we can provide easy way to:

• export as json
• export as xml
• export to rational DB
• ...

https://docs.sonatype.org/display/Repository/Sonatype+OSS+Maven+Repository+Usage+Guide

1. Prepare a release
2. Deploy release to a staging repository (see 7.a.3 in the docs above)
3. Use the Nexus UI to manually push a staged and “all clear” release to Maven central (See section 8 in docs above) They have several style checks (I think all are related to the POM)

Split into multiple modules:

1. The main DG library
2. Samples depending on the DG library
3. Main pom governing both. Make sure checkstyle still works for the new configuration.

Now we need to analysis / verify results manually.
It's easy to make a mistake in xml script and generate little bit different data set from desire one.

So, if we will have some kind of analysis tool, we will be able to analysis generated data and easy verify it.

Also we can use it for analysis generated data and external/original data to see difference.

Implement ALL EDGES traversal based scenario generation

Ability to generate data such that all edges of a dependency model will be exercised in the final resultant data set.

Features that allow DataGenerator to insert negative scenarios in the generated data set.

Update the pom to reflect DataGenerator name

Having multiple variables with sets that split into different states within a single state does not work properly.

Allow the main job to fire a jetty instance that responds to requests from the tasks and distribute computations during execution.

Implementation of all combinations and pairwise combinations within the dependency model.

Refer to section R1.4.20(1.4.20.1 , 1.4.20.2 and1.4.20.3) in https://github.com/FINRAOS/DataGenerator/wiki/DataGenerator-V2-Requirements#r14-common-rules-for-r11-r12-and-r13

#17

R1.4.8: Ability define expressions and calculations on nodes and edges. Expressions should be able to reference variable types described in R1.4.2 to R1.4.5.

R1.4.8.1: Simple mathematical expressions. (Add, Subtract, Divide, Multiply)
R1.4.8.2: Simple string manipulation. (Concatenate, Replace, Trim, Sub String, RegEx replace)

https://github.com/FINRAOS/DataGenerator/wiki/DataGenerator-V2-Requirements#r14-common-rules-for-r11-r12-and-r13

Research further features possibly required to support transformation of other formats into SCXML.

e.g.
Requirement formats --> SCXML Model XML

http://www.w3.org/TR/scxml/

A locally running jetty can share a GUI (jenkins style) that can be used to edit state charts and save them locally, possibly allowing user collaboration.

BFS hangs sometimes when the number of splits don't exactly match the ranks of the sets in the states.

In simple models, the first SearchWorker finishes its DFS and sets the exit flag before subsequent SearchWorkers can enqueue results.

Add checkstyle & PMD to the pom such that travis build will fail on violations.

Default style is here:
https://java.net/projects/woodstock/sources/svn/content/trunk/webui/tools/checkstyle/sun_checks.xml

We can copy it, and use it as a starting point for checkstyle.

I don't think we've solved this one yet:

Suppose we wanted to generate a set of accounts, which contains some identifying information that is tied to that account (like an owner's name, birthday etc,.).

We now want generate a set of transactions that tie the transactions to each account that we generated above. As it stands right now, we need to generate the set of accounts and generate the transactions using a custom consumer that parses through the account file.

Would like to see if it's possible to eliminate the need for consumers to handle parsing previously generated datasets, and move towards generating the two data sets side by side (if feasable)
#55

http://commons.apache.org/proper/commons-scxml/roadmap.html

We need to check what are the differences between the current library and what's coming next. Also if we can contribute back to them.

Allow the user the execute jobs from the GUI editor.

Ability to generate data such that all nodes of a dependency model will be exercised in the final resultant data set.

https://github.com/FINRAOS/DataGenerator/wiki/DataGenerator-V2-Requirements#r30-have-the-ability-to-learn-a-given-data-set

R1.4.6: Ability to override/reset(R1.4.6.1) and add(R1.4.6.2) variables of types described in R1.4.2 to R1.4.5. (On both nodes and edges)

https://github.com/FINRAOS/DataGenerator/wiki/DataGenerator-V2-Requirements#r14-common-rules-for-r11-r12-and-r13

R1.4.9: Ability define a database queries(R1.4.9.1) and simple flat files(R1.4.9.2) as a way to define variable assignments. Look at the diagram below; the specification on the left is identical to the specification on the right. The only difference is the specification on the right is using a SQL query to define its dataset. Note the SQL can be replaced with a file URL which would contain the table/joined data in a CSV format.

https://github.com/FINRAOS/DataGenerator/wiki/DataGenerator-V2-Requirements#r14-common-rules-for-r11-r12-and-r13

R1.4.12: Ability define conditionals on nodes and edges which would effectively enable selective traversal dependent on the condition evaluating to true.

Note: This means if a conditional fails on a node or an edge there will be no further traversal till the end node. The variables already set in the path traversed will be either considered or not considered as a valid scenario set depending on the global job configuration variable set.