Infrastructure for implementing benchmarx: benchmarks for bidirectional transformation (bx) tools. Also contains a collection of example benchmarx and test runners for various and diverse bx tools.
License: GNU General Public License v3.0
Already covered in another test case
Add Xtend to the installation instructions
The test cases specified in the class IncrementalForward.java obviously have several issues, where some of them are related to the transformation tools used and which are not discussed in this issue.
To sort them out i started running them for eMoflon only. I encountered a strange behavior:
The class comprises 4 methods, which implement test cases according the Issues TC 6a-6d.
When run in isolation (all other tests are commented out), testIncrementalInserts and testIncrementalDeletions seem to work find. They also work if they are run together with testIncrementalRename. However, when all 4 test methods are active (no method is commented), the method "testIncrementalDeletions" fails in the postcondition, as the actual and expected PersonRegisters differ.
My assumption was that tool.initiateSynchronisationDialogue() starts with an empty model each time, so that the results should be comparable, right?
Steps to reproduce: comment out all test methods in IncrementalForward.java and then remove the comment for one method in each step and check the result.
Any suggestion is welcome.
Suggested precondition:
FamilyRegister {
}
PersonRegister {
}
To finalise the precondition, set birthdays of all persons in the person register as an idle target delta.
Then for [e ^ p] propagate the target delta adding the following person:
Skinner, Seymour (male)
(although existing families are to be preferred, there is no suitable family in the family register so a new family Skinner with Seymour as father must be created)
Then for [e ^ p] propagate the target delta adding:
Skinner, Seymour (male)
(there is now a Skinner family so this must be used; as a father already exists the second configuration parameter has to be ignored adding Seymour as a son even though creating parents is preferred if possible)
Then for [!e ^ p] propagate the target delta adding:
Skinner, Seymour (male)
(although there is now a suitable Skinner family a new family must be created with Seymour as father)
Then for [!e ^ !p] propagate the target delta adding:
Skinner, Seymour (male)
(although there are suitable Skinner families a new family must be created with Seymour as son)
Then for [e ^ p] propagate the target delta adding:
Skinner, Seymour (male)
(there is one existing Skinner family where Seymour can be added as a father; this must be used)
Finally, with [e ^ !p] propagate the target delta adding:
Simpson, Bart (male)
Simpson, Lisa (female)
(the existing Simpson family must be used, Bart and Lisa must be added as children)
Final expected postcondition:
FamilyRegister {
}
PersonRegister {
}
Hi @tbuchmann
I propose to close this issue as other tests already check "least change", e.g., the fwd incremental tests that are failing for eMoflon.
Do we really need/have a specifically specially interesting least change test case?
Suggested precondition:
FamilyRegister {
}
PersonRegister {
}
To finalise the precondition, set birthdays of all persons in the person register as an idle target delta.
Then propagate the target delta deleting all persons
Expected post condition:
FamilyRegister {
}
PersonRegister {
}
It took me a while to find out, but the deleteForstSonBart method used in the IncrementalForward actually deletes a Bart that is not necessarily part of the given register. It simply deletes a Bart that is part of the register instance some other helper method has created, but in my case, this is not the same because in the meantime, changes in the FamilyModel were propagated and I have to use a deserialized family register from a previous propagation.
Suggested precondition:
FamilyRegister {
}
PersonRegister {
}
To finalise the precondition, set birthdays of all persons in the person register as an idle target delta.
Then with [e ^ p] propagate the following target edit adding:
Simpson, Marge (female)
Simpson, Lisa (female)
(respecting the order of updates, Marge should be added as a mother in the existing Simpson family, Lisa as a daughter)
Then with [!e ^ p] propagate the following target edit adding:
Simpson, Lisa (female)
Simpson, Marge (female)
(respecting the order of updates, a new family Simpson should be created with Lisa as mother and Marge as daughter)
Expected postcondition:
FamilyRegister {
}
PersonRegister {
}
The following two family registers are asserted as not equal according to the test framework (I have slightly improved the formatting):
Expected
FamilyRegister{
families=[
Family{
familyName="Simpson",
father=Just(FamilyMember{firstName="Homer"}),
mother=Just(FamilyMember{firstName="Maggie"}),
sons=[],
daughters=[]
},
Family{
familyName="Skinner",
father=Just(FamilyMember{firstName="Seymour"}),
mother=Nothing,
sons=[],
daughters=[]
},
Family{
familyName="Skinner",
father=Just(FamilyMember{firstName="Seymour"}),
mother=Nothing,
sons=[
FamilyMember{firstName="Seymour"}
],
daughters=[]
},
Family{
familyName="Skinner",
father=Just(FamilyMember{firstName="Seymour"}),
mother=Nothing,
sons=[
FamilyMember{firstName="Seymour"}
],
daughters=[]
}
]
}
Actual
FamilyRegister{
families=[
Family{
familyName="Simpson",
father=Just(FamilyMember{firstName="Homer"}),
mother=Just(FamilyMember{firstName="Maggie"}),
sons=[],
daughters=[]
},
Family{
familyName="Skinner",
father=Nothing,
mother=Nothing,
sons=[
FamilyMember{firstName="Seymour"}
],
daughters=[]
},
Family{
familyName="Skinner",
father=Just(FamilyMember{firstName="Seymour"}),
mother=Nothing,
sons=[],
daughters=[]
},
Family{
familyName="Skinner",
father=Just(FamilyMember{firstName="Seymour"}),
mother=Nothing,
sons=[
FamilyMember{firstName="Seymour"},
FamilyMember{firstName="Seymour"}
],
daughters=[]
}
]
}
The only difference between them is the order of the family skinner that appears multiple times. I thought that the order of the families in the family register would not be of importance. Is that correct? If so, then perhaps generating these string descriptions has some bug.
A synchroniser is hippocratic if it does not do anything for a pair of models that is already consistent (even if the delta to be propagated is not idle).
Note that this is more than stability, but less than least change.
I think the metamodel for the required "Config" is missing.
You could create a "repository project" for it analogously to Families and Persons using our wizard create new repository project available in the tool bar when you activate the eMoflon perspective (the project symbol with a little black plug). Code should be generated automatically as soon as you place the ecore file in the /model folder of the project (named appropriately).
I then suggest you check in this new project here
/familiestopersons/implementationArtefacts/medini/
Suggested precondition:
FamilyRegister {
}
PersonRegister {
}
Set birthdays of all persons in the person register (this must be propagated as a target delta so the tools' inner state is consistent).
Then propagate the source delta adding the following family members to the family Flanders:
Expected postcondition:
FamilyRegister {
}
PersonRegister {
}
While the paper suggests that the assertion of a post-condition is the last item in every synchronization dialogue, there are actually tests such as IncrementalForward/testStability that have multiple post-condition assertions.
NMF has to use a co-program approach and I therefore, I need to know when a synchronization dialogue is complete and I can exit the program, otherwise it is going to run endlessly. So how can I find out that a synchronization dialogue has ended?
Should contain:
Note: I believe the requirement that the model should not be producible with a batch backward transformation is redundant if it contains an empty family.
ar program in Windows.Suggested input (output of TC 1 #6):
PersonRegister {
}
Expected output:
FamilyRegister {
}
Suggested input (output of TC 1 #6):
PersonRegister {
}
Expected output:
FamilyRegister {
}
The current URI for MediniQVTFamilies2PersonsConfiguration is http://bx2017.f2p.config.ecore
While this works, it is a bit (aesthetically) disturbing as the standard Ecore editor is unable to find the .ecore file and complains about the trace model.
This could easily be avoided by using instead the URI: platform:/resource/MediniQVTFamilies2PersonsConfiguration/model/Configuration.ecore
This way all created models will use references that the editor can resolve automatically.
Hello,
I'm writing to report an issue I encountered while attempting to set up the project by following the steps outlined in the README. After downloading the latest version of the project, I encountered 433 errors during the import process. Some of these errors include messages like "Family cannot be resolved to a type" and "FamilyMember cannot be resolved to a type".
I have a suspicion that this issue might be related to the absence of a "Family.java" file in the codebase. It's possible that this file is generated in some way, possibly through "Families.ecore". However, I'm unsure about the correct steps to resolve these errors and would appreciate any guidance or assistance in rectifying the situation.
Thank you for your help.
Suggested precondition:
FamilyRegister {
}
PersonRegister {
}
Set birthdays of all persons in the person register (this must be propagated as a target delta so the tools' inner state is consistent).
Then propagate the source delta moving Lisa (daughter) to Flanders as Lisa (mother), and Marge (mother) to Skinner as Marge (mother):
Expected postcondition:
FamilyRegister {
}
PersonRegister {
}
Before execution, I should follow this https://github.com/eMoflon/emoflon-ibex-tutorial/releases to resolve the dependence of org.emoflon.ibex.common and org.emoflon.smartemf.
Suggested precondition:
FamilyRegister {
}
PersonRegister {
}
Set birthdays of all persons in the person register (this must be propagated as a target delta so the tools' inner state is consistent).
Then propagate the source delta deleting and re-creating Homer (father).
Expected postcondition:
FamilyRegister {
}
PersonRegister {
}
Hi all,
I just wanted to mention that my SHARE image as linked from the TTC homepage and my paper aren't ready yet. I've requested a mutable clone on Wednesday, 31st, but my request hasn't been approved yet. I'll make it as soon as I get the approval and report back.
That said, since the solution lives in its own GitHub repository and is also integrated in the benchmarx project, you should be easily able to run it anyhow.
Bye, Tassilo
I just started writing my solution and so far I can run it using JUnit. However, 19 test cases fail and it is hard to tell what's the problem. In the "Failure Trace" view I can see some textual representation of the model the test expected and what it actually got. However, that's not too easy to interpret.
Would it be possible to change the framework to call the saveModels(String name) method after each transformation so that we can inspect the results afterwards?
In my case, I save the models there and also print some PDF visualizations. It would also be nice to get a handle on the expected model so that I can print that, too, for comparing it with the actual result model.
I want to setup my SHARE demo as requested in the paper. Unfortunately, there's not even git and maven installed, and I can't find a README with the sudo password so that I can install the dependencies I need for my solution.
Where do I have to look for it?
(Just to be clear, I don't use that exact image but requested a mutable clone of this image which I will update/modify for my solution.)
The paper for the TTC case states that benchmarx can be used with non-JVM tools and quotes the reference implementation for BiGUL as a proof. However, this implementation seems to work state-based. I would like to implement the case in a non-JVM tool that is delta-based. Therefore, would it be possible to export the changes easily in the eMoflon internal change format? From there, I can transform the changes into my own change format and that might possibly work (creating a co-program and communicate via stdin/stdout).
Further, who is measuring time for performance tests? Is that done by benchmarx? In that case, please support a feature that allows non-JVM tools to override this, otherwise the times are not worth anything, as mostly serialization and deserialization will be measured.
Suggested input (output of TC 1 #6):
PersonRegister {
}
Expected output:
FamilyRegister {
}
The families helper class:benchmarx/examples/familiestopersons/metamodels/Families/src/org/benchmarx/families/core/FamilyHelper.java and the persons helper class should be refactored to build upon, e.g., a simple internal DSL for manipulating models.
Here is my review before I'll submit the evaluation sheet this evening.
Sadly, I haven't been able to run the solution because I don't have access to a
Windows machine, so this review is only based on the paper and transformation
source code.
And, the paper doesn't really help in understanding the solution. There's no
"big picture" discussing the overall transformation strategy (like, we're
essentially considering families as collections and have an add operation which
adds to the right role of a family based on temporary stereotypes we gather
beforehand).
Section 2 is much too brief. If you already are an category theory expert, you
might nod your head and go on but if you are not, this section doesn't help at
all. A less formal and more practical explanation of lenses would be more
appropriate.
Then section 3 presents some parts of the actual solution. I've had a hard
time mapping the NMF source code to lenses/synchronization blocks from section
2. So when a synchronization block is an 8-tuple, what are the corresponding
components in the source code of some SynchronizationRule?
And no single rule is explained completely. Just by looking at it, I'd guess
that MemberToMember (why not MemberToPerson?) synchronizes a family member
with a person and enforces that the result of member.GetFullName() must equal
person.Name. (How would you express that multiple values must match?) Well,
and since the type argument IPerson is abstract, I guess this rule is somehow
not instantiable itself.
I somehow grasp how the MemberToMale and MemberToFemale rules work. This
MarkInstantiatingFor() seems to be some kind of rule extension. And in the
method CreateLeftOutput() some extra information is added to the member which
can then be used later on to assign it to the right role of a family.
The paper doesn't list which tests pass (expectedly or unexpectedly) and which
ones fail, just that some incremental tests fail because some changes like
moving a family member are not transcribed as a move operation but as deletion
and insertion.
So all in all, I think the solution is ok although the description could be
improved.
Suggested precondition:
FamilyRegister {
}
PersonRegister {
}
Then propagate the target delta renaming:
Simpson, Bart [2] to Simpson, Bartholomew
Then, with [e ^ p] propagate the target delta renaming:
Simpson, Bart [3] to Skinner, Seymour
Then, with [e ^ !p] propagate the target delta renaming:
Simpson, Bart [1] to Flanders, Todd
Then, with [!e ^ !p] rename:
Simpson, Lisa to Flanders, Lisa
And finally with [!e ^ p] rename:
Simpson, Marge to Flanders, Maude
Expected postcondition:
FamilyRegister {
}
PersonRegister {
}
If the name of a person changes, i.e. "Lisa Simpson" -> "Lisa Flanders" (I am not really into Simpsons to judge how realistic that is), what should be the desired effect?
The problem is that renaming the family Simpson to Flanders also has an effect on the other members of this family, so the result model would be quite different.
Currently, very slow tools can make it difficult to run scalability measurements (they have to be commented out for larger input).
A solution would be to inforce a specified timeout in the BXToolTimer.
persons register is flat so moving is a bit err... boring :)
Suggested input (output of TC 1 #6):
PersonRegister {
}
Expected output:
FamilyRegister {
}
How to generate Table 5, Fig.26, Fig.27, Fig.28, Fig.29 step by step?
to testCreateMalePersonAsFather
Suggested precondition:
After creating the person register, set birthdays of all persons making sure that the Barts (sons) can be distinguished (this must be propagated as a target delta so the tools' inner state is consistent).
FamilyRegister {
}
PersonRegister {
}
Then propagate the source delta deleting the younger Bart (son).
Expected postcondition:
FamilyRegister {
}
PersonRegister {
}
A synchroniser is referred to as stable if it maps idle deltas to idle deltas (i.e., if nothing was changed then nothing is done). Note that hippocraticness is more than this as the delta does not have to be idle.
Repeat TC 4 (#11), and additionally check if exactly the same objects are present in the output model after synchronisation.
While this is not captured by any formal laws, this is of substantial practical relevance (e.g., in an EMF based context with dependent listeners, etc). This test, however, only makes sense for EMF-based tools as arbitrary tools might not even run in a JVM context making it hard to formulate the test.
Suggested precondition:
FamilyRegister {
Skinner
Flanders
Rod (son)
Simpson
Bart (father)
Simpson
Homer (father)
Marge (mother)
Bart (son)
Bart (son)
Lisa (daughter)
Maggie (daughter)
}
PersonRegister {
Flanders, Rod (male)
Simpson, Bart (male)
Simpson, Homer (male)
Simpson, Marge (female)
Simpson, Bart (male)
Simpson, Bart (male)
Simpson, Lisa (female)
Simpson, Maggie (female)
}
Set birthdays of all persons in the person register (this must be propagated as a target delta so the tools' inner state is consistent).
Then propagate the source delta of moving Maggie from the Simpsons family to the Flanders family. In the flanders family, Maggie does not take the role daughter, but the role son.
Expected postcondition:
Expected postcondition:
FamilyRegister {
Skinner
Marge (mother)
Flanders
Rod (son)
Maggie (son)
Simpson
Bart (father)
Simpson
Homer (father)
Bart (son)
Bart (son)
Lisa (daughter)
}
PersonRegister {
Flanders, Rod (male)
Simpson, Bart (male)
Simpson, Homer (male)
Simpson, Marge (female)
Simpson, Bart (male)
Simpson, Bart (male)
Simpson, Lisa (female)
Flanders, Maggie (male) [newly created, but birthdate copied from previous female instance]
}
Do you think we should include this TC @anthonyanjorin ?
Suggested precondition:
FamilyRegister {
}
PersonRegister {
}
Then, with [e ^ !p] propagate the target delta deleting:
Simpson, Homer (male)
and adding:
Simpson, Homer (male)
Expected post condition:
FamilyRegister {
}
PersonRegister {
}
Here is my review before I'll submit the evaluation sheet this evening.
I was able to run the solution with the authors' fork, branch
SDMLibContribution. The solution passes all tests except for these three:
testIncrementalInsertsDynamicConfigtestIncrementalOperationaltestIncrementalRenamingDynamicSince the forked project formats the models in text format all in one line (in
contrast to the main benchmarx project), it is hard to spot what the actual
problem causing the failures is.
Anyway, what I like with this solution is that it is really fast. Only the
BXtend solution is faster (about a factor of 7 for the batch cases and a factor
of 2-3 for the incremental ones), and most importantly, it doesn't seem to have
scalability issues when the model sizes grow.
Another good point is its conciseness. It consists of only about 100 lines of
code when counting just the transformForward() and transformBackward()
methods and the few helper methods called from there.
An obviously weak point of the solution is that it is not bidirectional at all,
i.e., the forward and backward transformation are completely separate. The
only thing they have in common is that they work on the very same kind of
models. That said, this can be a reasonable choice in certain scenarios. In
this very case, in my opinion, the two preference parameters can be a hindrance
for a bidirectional solution because they only have a meaning in the backward
direction but their values must not affect the forward direction. Thus, I also
had to build one relation which explicitly dispatches on the transformation's
direction -- a thing you wouldn't want to have in a truly bidirectional
solution.
The visualizations of the pattern are nice and easy to grasp. However, the
Java code that you actually have to write could be a bit nicer. Do I see it
correctly that some methods like createCondition() get a functional part (the
Condition object) and some text for the visualization which doesn't need to
have anything to with that Condition?
The decision to manifest the correlations between family members and persons by
extending the metamodel with explicit links is a good one which I sadly haven't
considered myself for the FunnyQT solution. However, I think the quite
operational and less declarative approach provided by SDMLib doesn't work for
more complex models. In this case, where we actually just deal with family
members and persons, it is quite feasible to use optional subpatterns and NACs
to distinguish the case where we already have a matching counterpart and the
other case, where we need to create one. But I assume that patterns would
quickly become unwieldy in real-world cases. As a transformation writer, I
don't want to have to encode this "check if exists or create anew" semantics
myself each and every time but want it built into the transformation language.
The solution paper suffices to explain the solution reasonably well, however it
doesn't meet the requirements mentioned in the case description. That is, it
doesn't mention which test cases it passes expectedly or unexpectedly, and
likewise it doesn't mention if the three failing tests fail due to a tool
limitation, a bug, or whatever. (Well, it actually doesn't mention that there
are failing tests at all.)
So all in all, the solution solves almost all test cases, it is concise and it
is really fast. However, it is not bidirectional at all.
I followed the setup guide in the README.MD but after installing the Eclipse Modeling Tools bundle, eMoflon and Xtend, the 6 projects don't build without error automagically. The FamiliesToPersons project complains that there are packages exported which it doesn't actually contain, and because that project won't build, the dependent project BechmarkxFamiliesToPersons doesn't build, too.
Here's a screenshot of the errors I get:
Well, I guess that's because there's nothing in the gen/ directory but how do I get anything generated here?
Suggested precondition:
FamilyRegister {
}
PersonRegister {
}
Set birthdays of all persons in the person register (this must be propagated as a target delta so the tools' inner state is consistent).
Then propagate the source delta renaming the larger Simpson family to Bouvier (Marge's maiden surname):
Expected postcondition:
FamilyRegister {
}
PersonRegister {
}
Is ther a special reason that the Person helper prints out the birthday as year-minute-day? I guess the format should have been year-month-day, is it?
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
Django
The Web framework for perfectionists with deadlines.
Laravel
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft
Open source projects and samples from Microsoft.
Google
Google ❤️ Open Source for everyone.
Alibaba
Alibaba Open Source for everyone
D3
Data-Driven Documents codes.
Tencent
China tencent open source team.