davideuler / diffkit Goto Github PK

vhl summary reports final rowStep as the total number of rows diff'd. However, 
the Engine must overstep the last row in order to discover that there are no 
more rows to be diff'd, so the correct number is rowStep -1. 

column diff summary should include information about clusters of patterns in 
the column diffs, with respect to the rows they occur in. For instance.

3 rows had diffs in column1 
2 rows had diffs in column1 + column2
2 rows had diffs in column1 + column3
etc.

A really nice enhancement to base feature set would be a Spreadsheet DKSource.

Right now, TestCases can only be executed from groovy enabled Eclipse project. 
Users need the ability to run complete TestCase suite from the executable 
application, so that they can validate functionality within their own 
environment (JRE, OS, etc).

put under TCR

by default, user should see no logback logging.

PassthroughPlan allows user to specify Diff.Kind. MagicPlan must also support 
this.

put MySQL under automated TestCaseRunner regression.

When there is no diff, an output file with 0 byte is generated. But I want to 
see a message somewhere that the diff succeeded with 0 rows.

   <bean id="plan" class="org.diffkit.diff.conf.DKMagicPlan">
      <property name="lhsFilePath" value="./test21.lhs.csv" />
      <property name="rhsFilePath" value="./test21.rhs.csv" />
      <property name="diffKind" value="ROW_DIFF" />
      <property name="maxDiffs" value="2" />
      <property name="sinkFilePath" value="./test21.sink.diff" />
   </bean>


leads to this:

     [java]     ... 76 common frames omitted
     [java] Caused by: java.lang.RuntimeException: Could not resolve automatically; need rule for dependency->Dependency[DKPassthroughPlan(DKAutomaticTableComparison,tableComparison_)]
     [java]     at org.diffkit.diff.conf.DKMagicPlanBuilder.resolve(DKMagicPlanBuilder.java:102) [ant.jar:na]
     [java]     ... 78 common frames omitted
     [java] Caused by: java.lang.RuntimeException: Could not resolve automatically; need rule for dependency->Dependency[DKPassthroughPlan(DKAutomaticTableComparison,tableComparison_)]
     [java]     at org.diffkit.diff.conf.DKMagicPlanBuilder.resolve(DKMagicPlanBuilder.java:110) [ant.jar:na]
     [java]     at org.diffkit.diff.conf.DKMagicPlanBuilder.resolve(DKMagicPlanBuilder.java:82) [ant.jar:na]
     [java]     at org.diffkit.diff.conf.DKMagicPlanBuilder.resolve(DKMagicPlanBuilder.java:99) [ant.jar:na]
     [java]     ... 78 common frames omitted
     [java] Caused by: java.lang.RuntimeException: Could not resolve automatically; need rule for dependency->Dependency[DKAutomaticTableComparison(Long,maxDiffs_)]
     [java]     at org.diffkit.diff.conf.DKMagicPlanBuilder.resolve(DKMagicPlanBuilder.java:102) [ant.jar:na]
     [java]     ... 80 common frames omitted
     [java] Caused by: java.lang.RuntimeException: ambiguous rule set; found more than one rule to handle dependency->Dependency[DKAutomaticTableComparison(Long,maxDiffs_)]:[DKMagicPlanRule[maxDiffs], DKMagicPlanRule[defaultMaxDiffs]]
     [java]     at org.diffkit.diff.conf.DKMagicPlanBuilder.getRule(DKMagicPlanBuilder.java:167) [ant.jar:na]
     [java]     at org.diffkit.diff.conf.DKMagicPlanBuilder.resolve(DKMagicPlanBuilder.java:75) [ant.jar:na]
     [java]     at org.diffkit.diff.conf.DKMagicPlanBuilder.resolve(DKMagicPlanBuilder.java:99) [ant.jar:na]
     [java]     ... 80 common frames omitted

What steps will reproduce the problem?
1. Insert text with newlines in a clob in the lhs table
2. Insert the same text without newlines in a clob in the rhs table
3. Use diffkit to compare the columns

What is the expected output? The clobs will be listed as diffs
What do you see instead?  No diffs


What version of the product are you using? 0.6.9
On what operating system? Linux


Please provide any additional information below.

MagicPlan fully reverse engineers a DKTable from the underlying DB table and 
thus requires that the underlying table have a PK.

Need to allow Magic comparison of tables that don't have PKs. Probably allow 
user to specify keys as plan properties. 

What steps will reproduce the problem?
1. This is while Diffing 2 Tables on the same Database, here the tables are in 
DB2 Database. 
2. The Diff Report was sucessfull with No Diffs, but the output on the console 
gives the count of Rows Diffed, to make it more Readable if it would have also 
outputted the Source and Destination Table Names and the Names of Columns 
Compared and the Names of Columns ignored etc.

What is the expected output? What do you see instead?
SOurce and Destination Table Names, and the column Names that are compared

What version of the product are you using? On what operating system?
diffkit-0[1].7.1.zip/LINUX

Please provide any additional information below.

asciidoc

There are two types of errors-- user errors and system (operating) errors.

User errors should not print out java stack traces. Those are not at all usable 
for average user. instead, user errors should print out a crisp, clear, user 
meaningful message.

Only fatal (system) errors should print java stack traces, so that user can 
submit for support.

asciidoc

What steps will reproduce the problem?
1. Either LHS or RHS Table has the Extra Columns WRT to the Other Table
2. It is throwing IllegalARgumnets Exception, A Nice USer Friendly Exception 
would be great
3.

What is the expected output? What do you see instead?
A user Friendly Message

What version of the product are you using? On what operating system?

diffkit-0[1].7.1.zip/LINUX
Please provide any additional information below.

Instead of having to specify both lhsDBTableName and rhsDBTableName, in the 
case where both are the same user should be able to simply specify dbTableName

put under TCR

In below plan, STAR.LHS_REGION and STAR.RHS_REGION do not have PKs defined in 
the DB. Below shows the error message from the app. Instead, it should clearly 
state that the table is missing a PK definition.

ERROR[15:13:19.908]{main}(DKApplication.runPlan:139)-null
java.lang.RuntimeException: java.lang.RuntimeException: Could not resolve automa
tically; need rule for dependency->Dependency[(DKPassthroughPlan,null)]
        at org.diffkit.diff.conf.DKMagicPlan.getBuiltPlan(DKMagicPlan.java:227)
[diffkit-app.jar:na]
        at org.diffkit.diff.conf.DKMagicPlan.getLhsSource(DKMagicPlan.java:203)
[diffkit-app.jar:na]
        at org.diffkit.diff.conf.DKApplication.runPlan(DKApplication.java:134) [
diffkit-app.jar:na]
        at org.diffkit.diff.conf.DKApplication.main(DKApplication.java:80) [diff
kit-app.jar:na]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [na:1.6.0
_10]
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) [na:1.6.0
_10]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) [na:1
.6.0_10]
        at java.lang.reflect.Method.invoke(Unknown Source) [na:1.6.0_10]
        at com.jdotsoft.jarloader.JarClassLoader.invokeMain(JarClassLoader.java:
520) [diffkit-app.jar:na]
        at org.diffkit.diff.conf.DKLauncher.main(DKLauncher.java:31) [diffkit-ap
p.jar:na]
Caused by: java.lang.RuntimeException: Could not resolve automatically; need rul
e for dependency->Dependency[(DKPassthroughPlan,null)]
        at org.diffkit.diff.conf.DKMagicPlanBuilder.resolve(DKMagicPlanBuilder.j
ava:104) [diffkit-app.jar:na]
        at org.diffkit.diff.conf.DKMagicPlanBuilder.build(DKMagicPlanBuilder.jav
a:50) [diffkit-app.jar:na]
        at org.diffkit.diff.conf.DKMagicPlan.getBuiltPlan(DKMagicPlan.java:223)
[diffkit-app.jar:na]
        ... 9 common frames omitted
Caused by: java.lang.RuntimeException: Could not resolve automatically; need rul
e for dependency->Dependency[DKPassthroughPlan(DKDBSource,lhsSource_)]
        at org.diffkit.diff.conf.DKMagicPlanBuilder.resolve(DKMagicPlanBuilder.j
ava:96) [diffkit-app.jar:na]
        ... 11 common frames omitted
Caused by: java.lang.RuntimeException: Could not resolve automatically; need rul
e for dependency->Dependency[DKPassthroughPlan(DKDBSource,lhsSource_)]
        at org.diffkit.diff.conf.DKMagicPlanBuilder.resolve(DKMagicPlanBuilder.j
ava:104) [diffkit-app.jar:na]
        at org.diffkit.diff.conf.DKMagicPlanBuilder.resolve(DKMagicPlanBuilder.j
ava:76) [diffkit-app.jar:na]
        at org.diffkit.diff.conf.DKMagicPlanBuilder.resolve(DKMagicPlanBuilder.j
ava:93) [diffkit-app.jar:na]
        ... 11 common frames omitted
Caused by: java.lang.reflect.InvocationTargetException: null
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
 [na:1.6.0_10]
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(Unknown Source)
 [na:1.6.0_10]
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Unknown Sou
rce) [na:1.6.0_10]
        at java.lang.reflect.Constructor.newInstance(Unknown Source) [na:1.6.0_1
0]
        at org.diffkit.diff.conf.DKMagicDependency.resolve(DKMagicDependency.jav
a:165) [diffkit-app.jar:na]
        at org.diffkit.diff.conf.DKMagicPlanBuilder.resolve(DKMagicPlanBuilder.j
ava:101) [diffkit-app.jar:na]
        ... 13 common frames omitted
Caused by: java.lang.RuntimeException: java.lang.IllegalArgumentException: does
not accept empty args
        at org.diffkit.diff.sns.DKDBSource.getModel(DKDBSource.java:223) [diffki
t-app.jar:na]
        at org.diffkit.diff.sns.DKDBSource.<init>(DKDBSource.java:86) [diffkit-a
pp.jar:na]
        ... 19 common frames omitted
Caused by: java.lang.IllegalArgumentException: does not accept empty args
        at org.diffkit.common.DKValidate.notEmpty(DKValidate.java:43) [diffkit-a
pp.jar:na]
        at org.diffkit.diff.engine.DKTableModel.<init>(DKTableModel.java:48) [di
ffkit-app.jar:na]
        at org.diffkit.diff.sns.DKTableModelUtil.createDefaultTableModel(DKTable
ModelUtil.java:75) [diffkit-app.jar:na]
        at org.diffkit.diff.sns.DKDBSource.getModel(DKDBSource.java:220) [diffki
t-app.jar:na]
        ... 20 common frames omitted

Right now, TestCaseRunner is not executable through ant. We need to make 
execution of full functional TestCase suite a dependency for the dist target.

Should be possible to generate *only* summary, with no detail. By default, if 
nothing specified, you get both summary and detail. If you specify withSummary, 
get summary only. If you specify withDetail, get detail only. Need to specify 
both withs to get full monty.

If you don't pass plan file name as arg to DKTool, you get below stacktrace. 
Instead, we need to present user with more explicitly informative message.


INFO [09:49:54.631]{main}(DKTool.main:36)-context->
org.springframework.beans.factory.NoSuchBeanDefinitionException: No bean named 
'plan' is defined
    at org.springframework.beans.factory.support.DefaultListableBeanFactory.getBeanDefinition(DefaultListableBeanFactory.java:510)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getMergedLocalBeanDefinition(AbstractBeanFactory.java:1056)
    at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:274)
    at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:190)
    at org.springframework.context.support.AbstractApplicationContext.getBean(AbstractApplicationContext.java:1045)
    at org.diffkit.diff.conf.DKTool.main(DKTool.java:37)

build script misses performing asciidoc conversion for compatibility.txt.

Hello all, just started using diffkit to see if it will be useful for 
our company. 
I'm running under windows xp against postgres 8.3.5 
I've been able to get diffkit working but only against a small table 
of 1000 rows. 
When i try to compare 2 identical tables with ~200,000 rows each, I 
get a java.lang.OutOfMemoryError: Java heap space error. 
The tables are defined as follows... 
CREATE TABLE lhs 
( 
  id integer NOT NULL, 
  adclaim integer NOT NULL, 
  adnumber integer NOT NULL, 
  adevent integer, 
  adclaimed numeric(15,2), 
  adsmsapproved numeric(15,2), 
  adapproved numeric(15,2), 
  adpaid numeric(15,2), 
  adbalance numeric(15,2), 
  addatepaid date, 
  adprogram character varying(15), 
  adsize double precision, 
  aduom character varying(15), 
  adunits double precision, 
  adfree1 character varying(20), 
  adfree2 character varying(20), 
  adfree3 character varying(20), 
  adfree4 character varying(20), 
  adfree5 character varying(20), 
  adfree6 double precision, 
  adfree7 double precision, 
  adfree8 date, 
  adfree9 date, 
  admediaseq integer, 
  adinvoice character varying(20), 
  adinsertdate date, 
  addays double precision, 
  adspots double precision, 
  admediaid character varying(15), 
  admedianame character varying(40), 
  admediacode character varying(15), 
  admediazip character varying(10), 
  admediaproduction numeric(15,2), 
  admediaprinting numeric(15,2), 
  admediaapvpct double precision, 
  admediareimpct double precision, 
  admediaadjpct double precision, 
  admediacommentcode character varying(15), 
  admediainvoiceamount numeric(15,2), 
  admediaineligamount numeric(15,2), 
  adprodseq integer, 
  adproduct character varying(15), 
  adprodheight double precision, 
  adprodwidth double precision, 
  adprodsize double precision, 
  adprodapvpct double precision, 
  adprodreimpct double precision, 
  adprodadjpct double precision, 
  adprodpricepoint numeric(15,2), 
  adprodcommentcode character varying(15), 
  adslick character varying(15), 
  adpromotion character varying(22), 
  adfun1 double precision, 
  adfundedby character varying(20), 
  adfundedbyid integer, 
  adonhold character varying(3), 
  adunpaid numeric(15,2), 
  priorapvid integer, 
  CONSTRAINT pkey_ad PRIMARY KEY (id, adclaim, adnumber) 
) 
WITH ( 
  OIDS=FALSE 
); 

plan file 
<?xml version="1.0" encoding="UTF-8"?> 
<beans xmlns="http://www.springframework.org/schema/beans" 
        xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
        xsi:schemaLocation="http://www.springframework.org/schema/beans 
        http://www.springframework.org/schema/beans/spring-beans-3.0.xsd"> 
        <bean id="plan" class="org.diffkit.diff.conf.DKMagicPlan"> 
                <property name="lhsDBTableName" value="lhs" /> 
                <property name="rhsDBTableName" value="rhs" /> 
                <property name="dbConnectionInfo" 
ref="connectionInfo" /> 
                <property name="keyColumnNames"> 
                        <list> 
                                <value>id</value> 
                                <value>adclaim</value> 
                                <value>adnumber</value> 
                        </list> 
                </property> 
                <property name="sinkFilePath" value="./test10.sink.diff" /> 
        </bean> 
</beans> 
command line I'm using (i can't cut and paste from the command line so 
I hope i didnt miss type it here) 
java -jar ..\diffkit-app.jar -planfiles 
test10.plan.xml,dbconnectioninfo.postgres.xml 
and the result (pipped into a file so i know i didnt miss type) 
DiffKit home->C:\Program Files\DiffKit 
planfile(s)->test10.plan.xml 
lhsSource->DKDBSource@3ef810[lhs,jdbc:postgresql://smsdev1:5432/ 
jrctest] 
rhsSource->DKDBSource@100363[rhs,jdbc:postgresql://smsdev1:5432/ 
jrctest] 
sink->DKFileSink@bcda2d[.\test10.sink.diff] 
tableComparison- 
>org.diffkit.diff.conf.DKAutomaticTableComparison@97d01f 

ERROR[14:45:24.278]{main}(DKApplication.main:130)-null 
java.lang.OutOfMemoryError: Java heap space 
        at 
com.jdotsoft.jarloader.JarClassLoader.getJarBytes(JarClassLoader.java: 
681) [diffkit-app.jar:na] 
        at 
com.jdotsoft.jarloader.JarClassLoader.findJarClass(JarClassLoader.java: 
344) [diffkit-app.jar:na] 
        at 
com.jdotsoft.jarloader.JarClassLoader.loadClass(JarClassLoader.java: 
551) [diffkit-app.jar:na] 
        at java.lang.ClassLoader.loadClass(Unknown Source) [na:1.6.0_22] 
        at 
org.postgresql.core.v3.QueryExecutorImpl.processResults(QueryExecutorImpl.j 
ava: 
1817) [diffkit-app.jar:na] 
        at 
org.postgresql.core.v3.QueryExecutorImpl.execute(QueryExecutorImpl.java: 
257) [diffkit-app.jar:na] 
        at 
org.postgresql.jdbc2.AbstractJdbc2Statement.execute(AbstractJdbc2Statement. 
java: 
500) [diffkit-app.jar:na] 
        at 
org.postgresql.jdbc2.AbstractJdbc2Statement.executeWithFlags(AbstractJdbc2S 
tatement.java: 
374) [diffkit-app.jar:na] 
        at 
org.postgresql.jdbc2.AbstractJdbc2Statement.executeQuery(AbstractJdbc2State 
ment.java: 
254) [diffkit-app.jar:na] 
        at org.diffkit.util.DKSqlUtil.executeQuery(DKSqlUtil.java:400) 
[diffkit-app.jar:na] 
        at org.diffkit.diff.sns.DKDBSource.createResultSet(DKDBSource.java: 
219) [diffkit-app.jar:na] 
        at org.diffkit.diff.sns.DKDBSource.open(DKDBSource.java:134) [diffkit- 
app.jar:na] 
        at org.diffkit.diff.engine.DKContext.open(DKContext.java:73) [diffkit- 
app.jar:na] 
        at org.diffkit.diff.engine.DKDiffEngine.diff(DKDiffEngine.java:63) 
[diffkit-app.jar:na] 
        at org.diffkit.diff.engine.DKDiffEngine.diff(DKDiffEngine.java:56) 
[diffkit-app.jar:na] 
        at org.diffkit.diff.conf.DKApplication.doDiff(DKApplication.java:191) 
[diffkit-app.jar:na] 
        at org.diffkit.diff.conf.DKApplication.runPlan(DKApplication.java: 
172) [diffkit-app.jar:na] 
        at org.diffkit.diff.conf.DKApplication.main(DKApplication.java:110) 
[diffkit-app.jar:na] 
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [na: 
1.6.0_22] 
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) [na: 
1.6.0_22] 
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) 
[na:1.6.0_22] 
        at java.lang.reflect.Method.invoke(Unknown Source) [na:1.6.0_22] 
        at 
com.jdotsoft.jarloader.JarClassLoader.invokeMain(JarClassLoader.java: 
520) [diffkit-app.jar:na] 
        at org.diffkit.diff.conf.DKLauncher.main(DKLauncher.java:26) [diffkit- 
app.jar:na] 

   As a user I necessarily do not care about the internals of the diffkit. But I want to know if the command succeeded or not. If the diff failed, then the crisp reason for the failure. Say for example if the QA-INT.FEED_FILE.diff exists then a one line message the file exists. Please delete it and rerun. But I see why you want to see all these messages. It is easy to debug. May be a debug option where it will write elaborate messages to the trace file or something.

put PostgreSQL under full TestCase regression.

Currently, TCR can only test for Diff results. As a functional test harness, it 
should also be able to test for correct handling of exceptional conditions 
(proper exception/failure generation).

User asks for ability to specify more than one plan in a single plan file, with 
the following benefits:

* batch execution of plans
* easier maintenance.

put under TCR

Appears that both Windoze and Mac OS have enough case-insensitivity magic to 
compensate for the error:

In TestCaseRunner.groovy

dbConnectioninfo should be dbConnectionInfo to match the case on the actual 
files.

User should see how long diff took.

Dates are not currently supported

displayColumnNames, specified to the MagicPlan, are not validated. If all of 
the provided displayColumn names are invalid (e.g. because of case or spelling 
issues), then the following exception results:


ERROR[20:13:50.951]{main}(DKApplication.main:107)-null
java.lang.NullPointerException: null
        at org.diffkit.diff.engine.DKStandardTableComparison.getRowDisplayValues
(DKStandardTableComparison.java:223) [diffkit-app.jar:na]
        at org.diffkit.diff.conf.DKAutomaticTableComparison.getRowDisplayValues(
DKAutomaticTableComparison.java:228) [diffkit-app.jar:na]
        at org.diffkit.diff.engine.DKDiffEngine.recordRowDiff(DKDiffEngine.java:
167) [diffkit-app.jar:na]
        at org.diffkit.diff.engine.DKDiffEngine.diff(DKDiffEngine.java:100) [dif
fkit-app.jar:na]
        at org.diffkit.diff.engine.DKDiffEngine.diff(DKDiffEngine.java:51) [diff
kit-app.jar:na]
        at org.diffkit.diff.conf.DKApplication.runPlan(DKApplication.java:168) [
diffkit-app.jar:na]
        at org.diffkit.diff.conf.DKApplication.main(DKApplication.java:89) [diff
kit-app.jar:na]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) [na:1.6.0
_10]
        at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) [na:1.6.0
_10]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) [na:1
.6.0_10]
        at java.lang.reflect.Method.invoke(Unknown Source) [na:1.6.0_10]
        at com.jdotsoft.jarloader.JarClassLoader.invokeMain(JarClassLoader.java:
520) [diffkit-app.jar:na]
        at org.diffkit.diff.conf.DKLauncher.main(DKLauncher.java:31) [diffkit-ap
p.jar:na]

put SQL Server under automated TestCaseRunner regression.

Users should be able to get additional summary view that groups existing row 
and column summary reports by user supplied list of columns. The group by 
column list must all be members of the displayColumns.

Want FileSink to give aggregated summaries, with samples, as well as details. 
E.G.


---
<very high>
diff'd 241 rows in 0:00:01.047, found:
!37 row diffs
@42 column diffs

<high>

---
"column1"-> 15 diffs
"column2"->20 diffs
"column3"->7 diffs

---
<medium>
"column1"->5 diffs
"column1"+"column2"->10 diffs
"column1"+"column3"->5 diffs
"column2"+"column3"->2 diffs

---
<low>

What steps will reproduce the problem?
1. Open test9.lhs.csv and remove the column3-value in one of the row
2. Launch the application
3. RuntimeException is thrown in DKFileSource row 227.

What is the expected output? What do you see instead?

--> Null values  should be accepted

What version of the product are you using? On what operating system?
0.7.2

What steps will reproduce the problem?
1. ~
2. ~
3. ~
What is the expected output? What do you see instead?
~
What version of the product are you using? On what operating system?
~
Please provide any additional information below.:

I have XML data to compare with data in an RDBMS.  It is defined in an .XSD 
(xml schema definition) file.  Any chance of using the.XML data as a source?

asciidoc

Using standalone app for large diffs should give user some indication of 
progress, so she knows the diff as not stalled.

Instead of having to specify both lhsWhereClause and rhsWhereClause, in the 
case where both are the same user should be able to simply specify whereClause.

asciidoc

put Sybase under automated TestCaseRunner regression.

Running embedded TestCase suite from standalone application; OS X passes, XP 
passes, RH linux fails.

Really strange.

put Oracle under automated TestCaseRunner regression.

asciidoc

Right now, JUnit tests are only executable from within Eclipse. Must make them 
a dependency for the dist target.

believe that keyColumnNames in the MagicPlan are not validated up front. If 
invalid (non-existent) column names are specified, believe that DiffEngine 
blows up with cryptic message downstream.

Need to be able to do this:

java -jar diffkit-app.jar -planfiles TABLES.PLAN.XML,CONNECTION_INFO.PLAN.XML

to support modularization and reuse of plan file information.