yudai / gojsondiff Goto Github PK

View Code? Open in Web Editor NEW

522.0 522.0 81.0 2.97 MB

Go JSON Diff

License: Other

Go 99.78% Makefile 0.22%

diff godoc json patch

gojsondiff's People

Contributors

Stargazers

Watchers

gojsondiff's Issues

Identical JSON samples with nested arrays are sometimes identified as different

A Json test in our codebase sometimes fails and sometimes passes. This is probably due to the nested struct arrays root.seatbid[i].bid[j] because the same thing happens with another JSON test in our repo.

╰─➤  go test
--- FAIL: TestJsonSampleRequests (1.94s)
    auction_test.go:123: sample-requests/valid-whole/exemplary/all-ext.json json did not match expected.

         {
           "bidid": "test bid id",
           "id": "some-request-id",
           "nbr": 0,
           "seatbid": [
             0: {
               "bid": [
                 1: {
                   "id": "rubicon-bid",
                   "impid": "",
                   "price": 0
                 },
                 2: {
                   "id": "appnexus-bid",
                   "impid": "",
                   "price": 0
                 },
               ],
               "seat": "seat-id"
             }
           ]
         }
+--  6 lines: E0925 15:09:33.730896   -----------------------------------------------------
FAIL
exit status 1
FAIL    github.com/prebid/prebid-server/endpoints/openrtb2      3.215s

But if we are lucky we get:

╰─➤  go test
+--  5 lines: E0925 15:16:30.083508   -----------------------------------------------------
PASS
ok      github.com/prebid/prebid-server/endpoints/openrtb2      3.318s

In summary, this is the test that I've been using:

func diffJson(t *testing.T, description string, actual []byte, expected []byte) {
	t.Helper()
	diff, err := gojsondiff.New().Compare(actual, expected)
	if err != nil {
		t.Fatalf("%s json diff failed. %v", description, err)
	}

	if diff.Modified() {
		var left interface{}
		if err := json.Unmarshal(actual, &left); err != nil {
			t.Fatalf("%s json did not match, but unmarshalling failed. %v", description, err)
		}
		printer := formatter.NewAsciiFormatter(left, formatter.AsciiFormatterConfig{
			ShowArrayIndex: true,
		})
		output, err := printer.Format(diff)
		if err != nil {
			t.Errorf("%s did not match, but diff formatting failed. %v", description, err)
		} else {
			t.Errorf("%s json did not match expected.\n\n%s", description, output)
		}
	}
}

And these are the actual (left) and expected (right):

 1 {                                    |  1  {
 2   "id": "some-request-id",           |  2   "id": "some-request-id",
 3   "seatbid": [                       |  3   "seatbid": [
 4     {                                |  4     {
 5       "bid": [                       |  5       "bid": [
 6         {                            |  6         {
 7           "id": "appnexus-bid",      |  7           "id": "appnexus-bid",
 8           "impid": "",               |  8           "impid": "",
 9           "price": 0                 |  9           "price": 0
10         }                            | 10         }
11       ],                             | 11       ],
12       "seat": "seat-id"              | 12       "seat": "seat-id"
13     }                                | 13     }
14   ],                                 | 14   ],
15   "bidid": "test bid id",            | 15   "bidid": "test bid id",
16   "nbr": 0                           | 16   "nbr": 0
17 }                                    | 17 }

I'm using the latest version.

Doesn't install

When trying to install as shown in documentation:

$ go get github.com/yudai/gojsondiff/jd

go: downloading github.com/codegangsta/cli v1.22.5
go get: github.com/codegangsta/cli@none updating to
	github.com/codegangsta/[email protected]: parsing go.mod:
	module declares its path as: github.com/urfave/cli
	        but was required as: github.com/codegangsta/cli

Colored output

Is it possible to color the output according to the image in green/red?

Formatter for json arrays diff

In real world there is no only json object (as top level item) but json arrays too. It'd be great to have diff formatter for them too.

Thanks and regards, Alexey Knyshev

New Release: gojsondiff v2.0

Working at https://github.com/yudai/gojsondiff/tree/release-v2.0.

Core changes

Native support for non-object root values
- You can compare strings, arrays and numbers at root (fixes #15 )
- The API will be changed

Formatters

Support JSON Merge Patch (RFC 7396)
Support JSON Patch (RFC 6902)

vulnerability : CVE-2022-29526 present in 1.0.0 release

the whitesource security scan on this repository shows vulnerability CVE-2022-29526 is present.
Is this a real threat from this repo point of view ?
If so, any plan to fix it or fix is already available?

Thanking you so much!!

badcase report

newline delimited json

Would it be possible to have a format option that writes a .delta JSON in one line (i.e. no pretty line breaks). That way you could append deltas to a file and "roll back" changes iterating over lines from bottom to top.

Switch back to k0kubun/pp

This library depends on https://github.com/yudai/pp, but it is outdated from the original version.

The panic on Float is already resolved by your patch upstream k0kubun/pp#15.
Your forked version does no have license file so we have problem clarifying the licenses of the library used in our product.

formatter.v1 imports gojsondiff from master

Hi,

It seems that current master is incompatible with v1, and they should not be used together. However, formatter.v1 imports gojsondiff from master, which causes problems.

Here is an example of a problem import: https://github.com/yudai/gojsondiff/blob/master/formatter/ascii.go#L9.

This example compiles well:

package main

import (
	"github.com/yudai/gojsondiff"
	"github.com/yudai/gojsondiff/formatter"
)

func main() {
	var d gojsondiff.Diff
	c := formatter.AsciiFormatterConfig{}
	f := formatter.NewAsciiFormatter(nil, c)
	f.Format(d)
}

But this one fails to compile:

package main

import (
	"gopkg.in/yudai/gojsondiff.v1"
	"gopkg.in/yudai/gojsondiff.v1/formatter"
)

func main() {
	var d gojsondiff.Diff
	c := formatter.AsciiFormatterConfig{}
	f := formatter.NewAsciiFormatter(nil, c)
	f.Format(d)
}

With the following error:

# command-line-arguments
./gjd.go:14: cannot use d (type "gopkg.in/yudai/gojsondiff.v1".Diff)
    as type "github.com/yudai/gojsondiff".Diff in argument to f.Format: 
      "gopkg.in/yudai/gojsondiff.v1".Diff does not implement
          "github.com/yudai/gojsondiff".Diff (wrong type for Deltas method)
                have Deltas() []"gopkg.in/yudai/gojsondiff.v1".Delta
                want Deltas() []"github.com/yudai/gojsondiff".Delta

Text diff is incorrectly deserialized

The text diff is incorrectly deserialized if diff consists of text insert + text delete. Golang snippet below demonstrates the problem.

package diff

import (
	"encoding/json"
	"os"

	"github.com/yudai/gojsondiff"
	"github.com/yudai/gojsondiff/formatter"
	"golang.org/x/crypto/ssh/terminal"
)

func main() {
	left := make(map[string]interface{})
	right := make(map[string]interface{})
	_ = json.Unmarshal([]byte(`{ "data": { "int": 1, "string": "gcr.io/heptio-images/ks-guestbook-demo:0.1" } }`), &left)
	_ = json.Unmarshal([]byte(`{ "data": { "int": 2, "string": "gcr.io/heptio-images/ks-guestbook-demo:0.2" } }`), &right)
	jsonFmt := formatter.NewDeltaFormatter()

	diff := gojsondiff.New().CompareObjects(left, right)
	jsonDiff, _ := jsonFmt.Format(diff)

	asciiFmt := formatter.NewAsciiFormatter(left, formatter.AsciiFormatterConfig{Coloring: terminal.IsTerminal(int(os.Stdout.Fd()))})
	formatted, _ := asciiFmt.Format(diff)
	println(formatted)

	diffUnmarshaller := gojsondiff.NewUnmarshaller()
	diff, _ = diffUnmarshaller.UnmarshalString(jsonDiff)

	formatted, _ = asciiFmt.Format(diff)
	println(formatted)
}

Program prints ascii formatted diff before and after deserialization. Output below demonstrates that after deserialization string field value is printed as null.

 {
   "data": {
-    "int": 1,
+    "int": 2,
-    "string": "gcr.io/heptio-images/ks-guestbook-demo:0.1"
+    "string": "gcr.io/heptio-images/ks-guestbook-demo:0.2"
   }
 }

 {
   "data": {
-    "int": 1,
+    "int": 2,
-    "string": null
+    "string": null
   }
 }

When swapping the value of textLeft and textRight, I get different results

This is my code:

package main

import (
	"encoding/json"
	"fmt"
	diff "github.com/yudai/gojsondiff"
	"github.com/yudai/gojsondiff/formatter"
)

func main() {
	textLeft := `{
    "name": "text_left",
    "strategy_list": [
        {
            "percent": 100,
            "whitelist": [
                "aabb"
            ],
            "name": "策略1",
            "stra_id": 11
        }
    ]
}`

	textRight := `{
    "business_id": 3,
    "name": "text_right",
    "desc": "text_right desc",
    "strategy_list": [
        {
            "percent": 50,
            "whitelist": null,
            "name": "stra_1",
            "stra_id": 1
        },
        {
            "percent": 50,
            "whitelist": null,
            "name": "stra_2",
            "stra_id": 2
        }
    ]
}`

	// textLeft, textRight = textRight, textLeft
	differ := diff.New()
	d, err := differ.Compare([]byte(textLeft), []byte(textRight))
	if err != nil {
		return
	}

	config := formatter.AsciiFormatterConfig{ShowArrayIndex: true, Coloring: true}

	var aJson map[string]interface{}
	_ = json.Unmarshal([]byte(textLeft), &aJson)
	f := formatter.NewAsciiFormatter(aJson, config)
	result, err := f.Format(d)
	if err != nil {
		return
	}

	println(fmt.Sprintf("result=%s", result))
}

The result is like this

But when I swapped textLeft and textRight, the result became like this

// swap textLeft and textRight before diff.New()
textLeft, textRight = textRight, textLeft

I think the second result is crorrect.
The first result is missing the original content of leftText

Incorrect diff when first item of array changed

I ran into a case where the diff output appears to be incorrect. What follows is a reduced test case.

Contents of left.json:

{
	"Indexes":[
	    2,
	    -1,
	    1
	]
}

Contents of right.json:

{
	"Indexes":[
          -1,
          -1,
          -1
	]
}

Run the following:

jd -f delta left.json right.json > patch.json
jp patch.json left.json > derived_right.json
jd -f delta right.json derived_right.json

(That is, create a diff, then validate that applying the diff to the left input produces the right output.)

Expected output:

{}

(That is, the diff successfully creates right.json)

Actual output:

{
  "Indexes": {
    "_0": [
      -1,
      0,
      0
    ],
    "_t": "a"
  }
}

(That is, derived_right.json is different from right.json)

If you inspect dervied_right.json you see:

{
  "Indexes": [
    -1,
    -1
  ]
}

(That is, derived_right.json is missing one row)

I originally came across this with a much larger json diff, but reduced to this test case with trial and error. Oddly enough, the behavior appears to rely on the values of the integer constants in the left.json. For example, if you flip the position of 2 and 1 in left.json, it works as expected.

Note that I also was seeing similar behavior for an array that was not integer constants, but strings that went from "somevalue" to "" if they were in the first position in the array. I factored them out of the test case because the integer test case still triggered.

I think that the original jsondiffpatch works correctly in this case, at least based on using the webapp version: https://benjamine.github.io/jsondiffpatch/demo/index.html

I'm using the 0.0.2 version of the jd tool

Apologies if I'm missing something obvious or not using this as intended!

how to deal with diffs of JSON where order of array elements isn't important

A lot of JSON that I'm dealing with uses arrays as collections where order isn't important. Is there a workaround for this ? I thought of sorting the arrays and then comparing them, but if the JSON to be compared is available as a string then it has to be first deserialized into Go datastructure to sort. I was wondering if there's a simpler way to do this. Thanks for your work !

confused Similarity func when compare two array

I know it's hard to locate two exactly index which supposed to be compared in an array
and i can understand that the similarity() is a way to make the similarity of two nodes quantifiable in the json, but I don't think your code way to compare two json objects is feasible.
In your code, the complement of the Object delta similarity function is to divide the sum of all the sub deltas similarity by the count of the sum deltas, the formula is like sum(sub deltas similarity)/count(sub deltas). but when compare two objects, the field which is exactly the same will not generate the delta object. so in some way, it's not a fair similarity way.
I think the fair similarity formula is like (sum(sub deltas similarity)+count(same fields))/(count(sub deltas)+count(same fields))
if you can agree with me, I will try to send the pr.
here is some example:
left json: [{"a":1,"b":2,"c":3}] right json:[{"a":2,"b":2,"c":4},{"a":1,"b":2,"c":3,"d":4}]

and the delta result is like
{
"placeholder": [
0: {
- "a": 1,
+ "a": 2,
- "b": 2,
+ "b": 3,
- "c": 3
+ "c": 4
}
+ 0: {
+ "a": 2,
+ "b": 3,
+ "c": 4
+ }
]
}

there are two bugs in the above example I think.
one is the first object {"a":2,"b":2,"c":4} in the right json appear twice in the deltas, which means that object has been compared twice, it may be a bug.
bug code is on below

var x, y int
for x, y = 0, 0; x <= sizeX-2 && y <= sizeY-2; {
	current := dpTable[x][y]
	nextX := dpTable[x+1][y]
	nextY := dpTable[x][y+1]
	xValidLength := len(left) - maxInvalidLength + y
	yValidLength := len(right) - maxInvalidLength + x
	if x+1 < xValidLength && current == nextX {
		freeLeft = append(freeLeft, left[x])
		x++
	} else if y+1 < yValidLength && current == nextY {
		freeRight = append(freeRight, right[y])
		y++
	} else {
		resultDeltas = append(resultDeltas, deltaTable[x][y])
		x++
		y++
	}
}
for ; x < sizeX-1; x++ {
	freeLeft = append(freeLeft, left[x-1]) //here 
}
for ; y < sizeY-1; y++ {
	freeRight = append(freeRight, right[y-1]) //here
}

the x-1 or y-1 index has been appended to the result Deltas, so they are not supposed to be appended into the deleted ary or added ary again , so they should be fixed to freeLeft = append(freeLeft, left[x])
and when i fixed the first bug, the result becomes to
{
"placeholder": [
0: {
- "a": 1,
+ "a": 2,
- "b": 2,
+ "b": 3,
- "c": 3
+ "c": 4
}
+ 1: {
+ "a": 1,
+ "b": 2,
+ "c": 3,
+ "d": 4
+ }
]
}
in this case, I think the first element in the left json array should be compared to the second element in the right json array, cause they're much similar.
the reason is the similarity() way may be not that fair.

CompareObjects hangs on valid input

I found a pair of valid JSON inputs that cause Differ.CompareObjects to hang indefinitely.

[Edit: changing the link to a branch rather than specific commit]

I've added sanitized versions of those files and a hanging unit test over here: https://github.com/pteichman/gojsondiff/tree/infinite-hang

As an API request, would it be useful to be able to disable the substring diffing? For my use case it's enough to know that the values fail string equality checks. Or alternatively, should LCS calculation be part of the display logic rather than in the differ itself?

Delta colors/diff not shown

Using a.json and b.json (formatted and sorted so one could use the 'diff' tool against it also), the jd tool does not show a visual difference using colored ascii mode, and the delta mode isn't very useful either.

The output of diff -u on the files is as follows:

@@ -3,10 +3,6 @@
         "FolderShape": {
             "AdditionalProperties": [
                 {
-                    "FieldURI": "folder:ParentFolderId",
-                    "__type": "PathToUnindexedField:#Exchange"
-                },
-                {
                     "PropertyTag": "0x3004",
                     "PropertyType": "String",
                     "__type": "PathToExtendedField:#Exchange"
@@ -22,6 +18,10 @@
                     "PropertySetId": "A7B529B5-4B75-47A7-A24F-20743D6C55CD",
                     "PropertyType": "Integer",
                     "__type": "PathToExtendedField:#Exchange"
+                },
+                {
+                    "FieldURI": "folder:ParentFolderId",
+                    "__type": "PathToUnindexedField:#Exchange"
                 }
             ],
             "BaseShape": "Default",

Json attached to the following gist: https://gist.github.com/virtuald/71fae5b37d63a2bf9f58ac6933edb46e . Thanks for the tool, it's really useful!

jd format changes integer to scientific notation

Unescaped integer is changed from this:

 {
    "serverId" : "5ba829fa-86d9-4246-b4ce-24cc896180c0",
    "id" : 14253805,
    "typeName" : "Service",
    "typeDisplayName" : "Service",
    "displayName" : "New York Office"
  },

To this:

-    70: {
-      "displayName": "New York Office",
-      "id": 1.4253805e+07,
-      "serverId": "5ba829fa-86d9-4246-b4ce-24cc896180c0",
-      "typeDisplayName": "Service",
-      "typeName": "Service"
-    },

Two non-identical int64 numbers with 17 digits were compared, and the result obtained was the same.

func main() {
json1 := { "custom_id":72211083386779239 }
json2 := { "custom_id":72211083386779230 }

diff, _ := gojsondiff.New().Compare([]byte(json1), []byte(json2))

if diff.Modified() {
	fmt.Println(diff.Deltas())
} else {
	fmt.Println("Two JSONs are the same.")
}

}

gopkg.in compatibility

Hi,

Would you mind to use vN.M.K instead of N.M.K for tag names, e.g. v1.0.0? This way, it would be possible to use gopkg.in to import stable release, like this:

import "gopkg.in/yudai/gojsondiff.v1"

(see http://gopkg.in/yudai/gojsondiff.v1)

You could also add this example to your README.

Notes:

gopkg.in assumes that the first (major) version number should be increased when a breaking change is introduced
gopkg.in also allows to use branches instead of tags

/link gavv/httpexpect#33 gavv/httpexpect#34

Possibly unecessary dependency from yudai/pp on unmarshaler_test.go

Hey!

I'm packaging a go app on Debian (kong/deck) which depends on gojsondiff.
It shouldn't be a big problem, but it introduces an indirect dependency, forcing me to also package your pp fork which doesn't seem like a great idea right now (specially because the upstream is actively maintained, while the for isn't).

Would it be okay for me to open a PR removing the dependency or even importing the upstream pp (which is already distributed on Debian)? :)

Thanks!

Failed to unmarshal valid json

$ echo [1] > 1.json
$ echo [2] > 2.json
$ jd 1.json 2.json

Failed to unmarshal file: json: cannot unmarshal array into Go value of type map[string]interface {}
exit status 3

Dependency issue

This is the output I get when I try to install this module

:!go get github.com/yudai/gojsondiff/jd
go: found github.com/yudai/gojsondiff/jd in github.com/yudai/gojsondiff v1.0.0                                                                                                                    
go: finding module for package github.com/codegangsta/cli
go: found github.com/codegangsta/cli in github.com/codegangsta/cli v1.22.4
go: github.com/yudai/gojsondiff/jd imports
        github.com/codegangsta/cli: github.com/codegangsta/[email protected]: parsing go.mod:
        module declares its path as: github.com/urfave/cli
                but was required as: github.com/codegangsta/cli

shell returned 1

Missing dependencies in vendor/

$ cd $GOPATH/src/github.com/yudai/gojsondiff
$ go test ./...
vendor/github.com/yudai/gojsondiff/deltas.go:5:2: cannot find package "github.com/sergi/go-diff/diffmatchpatch" in any of:
	/Users/ryan/go2/src/foo/vendor/github.com/yudai/gojsondiff/vendor/github.com/sergi/go-diff/diffmatchpatch (vendor tree)
	/Users/ryan/go2/src/foo/vendor/github.com/sergi/go-diff/diffmatchpatch
	/usr/local/Cellar/go/1.8/libexec/src/github.com/sergi/go-diff/diffmatchpatch (from $GOROOT)
	/Users/ryan/go2/src/github.com/sergi/go-diff/diffmatchpatch (from $GOPATH)
vendor/github.com/yudai/gojsondiff/gojsondiff.go:13:2: cannot find package "github.com/yudai/golcs" in any of:
	/Users/ryan/go2/src/foo/vendor/github.com/yudai/gojsondiff/vendor/github.com/yudai/golcs (vendor tree)
	/Users/ryan/go2/src/foo/vendor/github.com/yudai/golcs
	/usr/local/Cellar/go/1.8/libexec/src/github.com/yudai/golcs (from $GOROOT)
	/Users/ryan/go2/src/github.com/yudai/golcs (from $GOPATH)
vendor/github.com/yudai/gojsondiff/jd/main.go:9:2: cannot find package "github.com/codegangsta/cli" in any of:
	/Users/ryan/go2/src/foo/vendor/github.com/yudai/gojsondiff/vendor/github.com/codegangsta/cli (vendor tree)
	/Users/ryan/go2/src/foo/vendor/github.com/codegangsta/cli
	/usr/local/Cellar/go/1.8/libexec/src/github.com/codegangsta/cli (from $GOROOT)
	/Users/ryan/go2/src/github.com/codegangsta/cli (from $GOPATH)
vendor/github.com/yudai/gojsondiff/tests/helper.go:4:2: cannot find package "github.com/onsi/ginkgo" in any of:
	/Users/ryan/go2/src/foo/vendor/github.com/yudai/gojsondiff/vendor/github.com/onsi/ginkgo (vendor tree)
	/Users/ryan/go2/src/foo/vendor/github.com/onsi/ginkgo
	/usr/local/Cellar/go/1.8/libexec/src/github.com/onsi/ginkgo (from $GOROOT)
	/Users/ryan/go2/src/github.com/onsi/ginkgo (from $GOPATH)

This is with a clean $GOPATH (a $GOPATH that contains gojsondiff and nothing else).

The problem

These deps are not in vendor/.

github.com/sergi/go-diff/diffmatchpatch
github.com/yudai/golcs
github.com/codegangsta/cli

This dep appears to be in vendor/, but Go is having trouble finding it for some reason. Perhaps the vendoring of that pkg is incomplete.

github.com/onsi/ginkgo

The real problem

You thought everything was vendored because you have everything in your Godeps.json manifest and you ran godep restore. Unfortunately, godep is a first-gen vendoring tool with all sorts of surprising behavior. I could submit a PR to just add the missing deps, but that still leaves us with the problem of why ginkgo isn't being found, and it doesn't prevent future problems should you decide to add/remove deps.

The real solution

A better solution is to revendor the deps using a better vendoring tool, such as Glide or gvt, so that this problem doesn't happen again. This is what I've done in #17 . In the process of revendoring gojsondiff I discovered some other issues which are detailed in the PR notes.

Who I am

I'm the "vendoring guy" at a company called Apcera. We're using gojsondiff internally (thanks very much for writing it!). I discovered this issue when I tried to vendor gojsondiff itself and it didn't work.

array diff "panic: runtime error: index out of range"

➜ gojsondiff git:(master) ✗ cat a.json && cat b.json
{
"arr":[
"item0",
"item1"
]
}
{
"arr":[
"item0",
"item2",
"item3"
]
}

➜ jd git:(master) ✗ go install && jd -f delta ../a.json ../b.json
panic: runtime error: index out of range

Looks like typo here: https://github.com/yudai/gojsondiff/blob/master/gojsondiff.go#L364

Confusing Array Diff Output

Given two arrays of objects, if the objects appear in a different order, reflect.DeepEqual(expected, actual) will return false, whereas differ.CompareArrays(expected, actual) will contain an unexpected diff.

Reproducer:

package main

import (
  "fmt"

  "github.com/yudai/gojsondiff"
  "github.com/yudai/gojsondiff/formatter"
)

func main() {
  a := map[string]interface{}{"key": "bobby", "val": "tables", "foo": false}
  b := map[string]interface{}{"key": "how", "val": "outage", "foo": true}
  c := map[string]interface{}{"key": "stuffs", "val": "sample", "foo": true}

  expected := []interface{}{a, b, c}
  //actual := []interface{}{a, b, c}
  actual := []interface{}{b, c, a}

  differ := gojsondiff.New()
  diff := differ.CompareArrays(expected, actual)
  //diff := differ.CompareObjects(map[string]interface{}{"_": expected}, map[string]interface{}{"_": actual})
  //formatter := formatter.NewDeltaFormatter()
  formatter := formatter.NewAsciiFormatter(expected, formatter.AsciiFormatterConfig{ShowArrayIndex: true})
  str, err := formatter.Format(diff)

  if err != nil {
    panic(err)
  }
  fmt.Printf("%s\n", str)
}

Diff Results:

$ go run diff.go
 [
   0: {
     "foo": false,
     "key": "bobby",
     "val": "tables"
   },
   1: {
     "foo": true,
     "key": "how",
     "val": "outage"
   },
 ]

Diff Results from jsondiff.com:

Store diff and apply later [feature request]

Hi,

I would like to store the diff in a database (map[string]interface{} output from FormatAsJson) and apply it later. It does not look like its currently possible. Looks like we need n formatFromJson formatter and then n applyJson patch method?

Thank you,
Riaan