yudai / gojsondiff Goto Github PK
View Code? Open in Web Editor NEWGo JSON Diff
License: Other
Go JSON Diff
License: Other
A lot of JSON that I'm dealing with uses arrays as collections where order isn't important. Is there a workaround for this ? I thought of sorting the arrays and then comparing them, but if the JSON to be compared is available as a string then it has to be first deserialized into Go datastructure to sort. I was wondering if there's a simpler way to do this. Thanks for your work !
The text diff is incorrectly deserialized if diff consists of text insert + text delete. Golang snippet below demonstrates the problem.
package diff
import (
"encoding/json"
"os"
"github.com/yudai/gojsondiff"
"github.com/yudai/gojsondiff/formatter"
"golang.org/x/crypto/ssh/terminal"
)
func main() {
left := make(map[string]interface{})
right := make(map[string]interface{})
_ = json.Unmarshal([]byte(`{ "data": { "int": 1, "string": "gcr.io/heptio-images/ks-guestbook-demo:0.1" } }`), &left)
_ = json.Unmarshal([]byte(`{ "data": { "int": 2, "string": "gcr.io/heptio-images/ks-guestbook-demo:0.2" } }`), &right)
jsonFmt := formatter.NewDeltaFormatter()
diff := gojsondiff.New().CompareObjects(left, right)
jsonDiff, _ := jsonFmt.Format(diff)
asciiFmt := formatter.NewAsciiFormatter(left, formatter.AsciiFormatterConfig{Coloring: terminal.IsTerminal(int(os.Stdout.Fd()))})
formatted, _ := asciiFmt.Format(diff)
println(formatted)
diffUnmarshaller := gojsondiff.NewUnmarshaller()
diff, _ = diffUnmarshaller.UnmarshalString(jsonDiff)
formatted, _ = asciiFmt.Format(diff)
println(formatted)
}
Program prints ascii formatted diff before and after deserialization. Output below demonstrates that after deserialization string
field value is printed as null.
{
"data": {
- "int": 1,
+ "int": 2,
- "string": "gcr.io/heptio-images/ks-guestbook-demo:0.1"
+ "string": "gcr.io/heptio-images/ks-guestbook-demo:0.2"
}
}
{
"data": {
- "int": 1,
+ "int": 2,
- "string": null
+ "string": null
}
}
Would it be possible to have a format option that writes a .delta JSON in one line (i.e. no pretty line breaks). That way you could append deltas to a file and "roll back" changes iterating over lines from bottom to top.
$ echo [1] > 1.json
$ echo [2] > 2.json
$ jd 1.json 2.json
Failed to unmarshal file: json: cannot unmarshal array into Go value of type map[string]interface {}
exit status 3
Working at https://github.com/yudai/gojsondiff/tree/release-v2.0.
Using a.json and b.json (formatted and sorted so one could use the 'diff' tool against it also), the jd
tool does not show a visual difference using colored ascii mode, and the delta mode isn't very useful either.
The output of diff -u
on the files is as follows:
@@ -3,10 +3,6 @@
"FolderShape": {
"AdditionalProperties": [
{
- "FieldURI": "folder:ParentFolderId",
- "__type": "PathToUnindexedField:#Exchange"
- },
- {
"PropertyTag": "0x3004",
"PropertyType": "String",
"__type": "PathToExtendedField:#Exchange"
@@ -22,6 +18,10 @@
"PropertySetId": "A7B529B5-4B75-47A7-A24F-20743D6C55CD",
"PropertyType": "Integer",
"__type": "PathToExtendedField:#Exchange"
+ },
+ {
+ "FieldURI": "folder:ParentFolderId",
+ "__type": "PathToUnindexedField:#Exchange"
}
],
"BaseShape": "Default",
Json attached to the following gist: https://gist.github.com/virtuald/71fae5b37d63a2bf9f58ac6933edb46e . Thanks for the tool, it's really useful!
I found a pair of valid JSON inputs that cause Differ.CompareObjects to hang indefinitely.
[Edit: changing the link to a branch rather than specific commit]
I've added sanitized versions of those files and a hanging unit test over here: https://github.com/pteichman/gojsondiff/tree/infinite-hang
As an API request, would it be useful to be able to disable the substring diffing? For my use case it's enough to know that the values fail string equality checks. Or alternatively, should LCS calculation be part of the display logic rather than in the differ itself?
func main() {
json1 := { "custom_id":72211083386779239 }
json2 := { "custom_id":72211083386779230 }
diff, _ := gojsondiff.New().Compare([]byte(json1), []byte(json2))
if diff.Modified() {
fmt.Println(diff.Deltas())
} else {
fmt.Println("Two JSONs are the same.")
}
}
A Json test in our codebase sometimes fails and sometimes passes. This is probably due to the nested struct arrays root.seatbid[i].bid[j]
because the same thing happens with another JSON test in our repo.
╰─➤ go test
--- FAIL: TestJsonSampleRequests (1.94s)
auction_test.go:123: sample-requests/valid-whole/exemplary/all-ext.json json did not match expected.
{
"bidid": "test bid id",
"id": "some-request-id",
"nbr": 0,
"seatbid": [
0: {
"bid": [
1: {
"id": "rubicon-bid",
"impid": "",
"price": 0
},
2: {
"id": "appnexus-bid",
"impid": "",
"price": 0
},
],
"seat": "seat-id"
}
]
}
+-- 6 lines: E0925 15:09:33.730896 -----------------------------------------------------
FAIL
exit status 1
FAIL github.com/prebid/prebid-server/endpoints/openrtb2 3.215s
But if we are lucky we get:
╰─➤ go test
+-- 5 lines: E0925 15:16:30.083508 -----------------------------------------------------
PASS
ok github.com/prebid/prebid-server/endpoints/openrtb2 3.318s
In summary, this is the test that I've been using:
func diffJson(t *testing.T, description string, actual []byte, expected []byte) {
t.Helper()
diff, err := gojsondiff.New().Compare(actual, expected)
if err != nil {
t.Fatalf("%s json diff failed. %v", description, err)
}
if diff.Modified() {
var left interface{}
if err := json.Unmarshal(actual, &left); err != nil {
t.Fatalf("%s json did not match, but unmarshalling failed. %v", description, err)
}
printer := formatter.NewAsciiFormatter(left, formatter.AsciiFormatterConfig{
ShowArrayIndex: true,
})
output, err := printer.Format(diff)
if err != nil {
t.Errorf("%s did not match, but diff formatting failed. %v", description, err)
} else {
t.Errorf("%s json did not match expected.\n\n%s", description, output)
}
}
}
And these are the actual (left) and expected (right):
1 { | 1 {
2 "id": "some-request-id", | 2 "id": "some-request-id",
3 "seatbid": [ | 3 "seatbid": [
4 { | 4 {
5 "bid": [ | 5 "bid": [
6 { | 6 {
7 "id": "appnexus-bid", | 7 "id": "appnexus-bid",
8 "impid": "", | 8 "impid": "",
9 "price": 0 | 9 "price": 0
10 } | 10 }
11 ], | 11 ],
12 "seat": "seat-id" | 12 "seat": "seat-id"
13 } | 13 }
14 ], | 14 ],
15 "bidid": "test bid id", | 15 "bidid": "test bid id",
16 "nbr": 0 | 16 "nbr": 0
17 } | 17 }
I'm using the latest version.
func AnyJsonDiff(left interface{}, right interface{}) (gojsondiff.Diff, error) {
type Container struct {
Payload any
}
leftBs, err := json.Marshal(Container{left})
if err != nil {
return nil, err
}
rightBs, err := json.Marshal(Container{right})
if err != nil {
return nil, err
}
return gojsondiff.New().Compare(leftBs, rightBs)
}
This is my code:
package main
import (
"encoding/json"
"fmt"
diff "github.com/yudai/gojsondiff"
"github.com/yudai/gojsondiff/formatter"
)
func main() {
textLeft := `{
"name": "text_left",
"strategy_list": [
{
"percent": 100,
"whitelist": [
"aabb"
],
"name": "策略1",
"stra_id": 11
}
]
}`
textRight := `{
"business_id": 3,
"name": "text_right",
"desc": "text_right desc",
"strategy_list": [
{
"percent": 50,
"whitelist": null,
"name": "stra_1",
"stra_id": 1
},
{
"percent": 50,
"whitelist": null,
"name": "stra_2",
"stra_id": 2
}
]
}`
// textLeft, textRight = textRight, textLeft
differ := diff.New()
d, err := differ.Compare([]byte(textLeft), []byte(textRight))
if err != nil {
return
}
config := formatter.AsciiFormatterConfig{ShowArrayIndex: true, Coloring: true}
var aJson map[string]interface{}
_ = json.Unmarshal([]byte(textLeft), &aJson)
f := formatter.NewAsciiFormatter(aJson, config)
result, err := f.Format(d)
if err != nil {
return
}
println(fmt.Sprintf("result=%s", result))
}
But when I swapped textLeft and textRight, the result became like this
// swap textLeft and textRight before diff.New()
textLeft, textRight = textRight, textLeft
I think the second result is crorrect.
The first result is missing the original content of leftText
I ran into a case where the diff output appears to be incorrect. What follows is a reduced test case.
Contents of left.json
:
{
"Indexes":[
2,
-1,
1
]
}
Contents of right.json
:
{
"Indexes":[
-1,
-1,
-1
]
}
Run the following:
jd -f delta left.json right.json > patch.json
jp patch.json left.json > derived_right.json
jd -f delta right.json derived_right.json
(That is, create a diff, then validate that applying the diff to the left input produces the right output.)
Expected output:
{}
(That is, the diff successfully creates right.json)
Actual output:
{
"Indexes": {
"_0": [
-1,
0,
0
],
"_t": "a"
}
}
(That is, derived_right.json
is different from right.json
)
If you inspect dervied_right.json
you see:
{
"Indexes": [
-1,
-1
]
}
(That is, derived_right.json
is missing one row)
I originally came across this with a much larger json diff, but reduced to this test case with trial and error. Oddly enough, the behavior appears to rely on the values of the integer constants in the left.json. For example, if you flip the position of 2
and 1
in left.json, it works as expected.
Note that I also was seeing similar behavior for an array that was not integer constants, but strings that went from "somevalue" to "" if they were in the first position in the array. I factored them out of the test case because the integer test case still triggered.
I think that the original jsondiffpatch works correctly in this case, at least based on using the webapp version: https://benjamine.github.io/jsondiffpatch/demo/index.html
I'm using the 0.0.2 version of the jd
tool
Apologies if I'm missing something obvious or not using this as intended!
When trying to install as shown in documentation:
$ go get github.com/yudai/gojsondiff/jd
go: downloading github.com/codegangsta/cli v1.22.5
go get: github.com/codegangsta/cli@none updating to
github.com/codegangsta/[email protected]: parsing go.mod:
module declares its path as: github.com/urfave/cli
but was required as: github.com/codegangsta/cli
I know it's hard to locate two exactly index which supposed to be compared in an array
and i can understand that the similarity() is a way to make the similarity of two nodes quantifiable in the json, but I don't think your code way to compare two json objects is feasible.
In your code, the complement of the Object delta similarity function is to divide the sum of all the sub deltas similarity by the count of the sum deltas, the formula is like sum(sub deltas similarity)/count(sub deltas)
. but when compare two objects, the field which is exactly the same will not generate the delta object. so in some way, it's not a fair similarity way.
I think the fair similarity formula is like (sum(sub deltas similarity)+count(same fields))/(count(sub deltas)+count(same fields))
if you can agree with me, I will try to send the pr.
here is some example:
left json: [{"a":1,"b":2,"c":3}] right json:[{"a":2,"b":2,"c":4},{"a":1,"b":2,"c":3,"d":4}]
and the delta result is like
{
"placeholder": [
0: {
- "a": 1,
+ "a": 2,
- "b": 2,
+ "b": 3,
- "c": 3
+ "c": 4
}
+ 0: {
+ "a": 2,
+ "b": 3,
+ "c": 4
+ }
]
}
there are two bugs in the above example I think.
one is the first object {"a":2,"b":2,"c":4}
in the right json appear twice in the deltas, which means that object has been compared twice, it may be a bug.
bug code is on below
var x, y int
for x, y = 0, 0; x <= sizeX-2 && y <= sizeY-2; {
current := dpTable[x][y]
nextX := dpTable[x+1][y]
nextY := dpTable[x][y+1]
xValidLength := len(left) - maxInvalidLength + y
yValidLength := len(right) - maxInvalidLength + x
if x+1 < xValidLength && current == nextX {
freeLeft = append(freeLeft, left[x])
x++
} else if y+1 < yValidLength && current == nextY {
freeRight = append(freeRight, right[y])
y++
} else {
resultDeltas = append(resultDeltas, deltaTable[x][y])
x++
y++
}
}
for ; x < sizeX-1; x++ {
freeLeft = append(freeLeft, left[x-1]) //here
}
for ; y < sizeY-1; y++ {
freeRight = append(freeRight, right[y-1]) //here
}
the x-1 or y-1 index has been appended to the result Deltas, so they are not supposed to be appended into the deleted ary or added ary again , so they should be fixed to freeLeft = append(freeLeft, left[x])
and when i fixed the first bug, the result becomes to
{
"placeholder": [
0: {
- "a": 1,
+ "a": 2,
- "b": 2,
+ "b": 3,
- "c": 3
+ "c": 4
}
+ 1: {
+ "a": 1,
+ "b": 2,
+ "c": 3,
+ "d": 4
+ }
]
}
in this case, I think the first element in the left json array should be compared to the second element in the right json array, cause they're much similar.
the reason is the similarity() way may be not that fair.
Hey!
I'm packaging a go app on Debian (kong/deck) which depends on gojsondiff
.
It shouldn't be a big problem, but it introduces an indirect dependency, forcing me to also package your pp
fork which doesn't seem like a great idea right now (specially because the upstream is actively maintained, while the for isn't).
Would it be okay for me to open a PR removing the dependency or even importing the upstream pp
(which is already distributed on Debian)? :)
Thanks!
In real world there is no only json object (as top level item) but json arrays too. It'd be great to have diff formatter for them too.
Thanks and regards, Alexey Knyshev
Unescaped integer is changed from this:
{
"serverId" : "5ba829fa-86d9-4246-b4ce-24cc896180c0",
"id" : 14253805,
"typeName" : "Service",
"typeDisplayName" : "Service",
"displayName" : "New York Office"
},
To this:
- 70: {
- "displayName": "New York Office",
- "id": 1.4253805e+07,
- "serverId": "5ba829fa-86d9-4246-b4ce-24cc896180c0",
- "typeDisplayName": "Service",
- "typeName": "Service"
- },
the whitesource security scan on this repository shows vulnerability CVE-2022-29526 is present.
Is this a real threat from this repo point of view ?
If so, any plan to fix it or fix is already available?
Thanking you so much!!
Given two arrays of objects, if the objects appear in a different order, reflect.DeepEqual(expected, actual)
will return false
, whereas differ.CompareArrays(expected, actual)
will contain an unexpected diff.
Reproducer:
package main
import (
"fmt"
"github.com/yudai/gojsondiff"
"github.com/yudai/gojsondiff/formatter"
)
func main() {
a := map[string]interface{}{"key": "bobby", "val": "tables", "foo": false}
b := map[string]interface{}{"key": "how", "val": "outage", "foo": true}
c := map[string]interface{}{"key": "stuffs", "val": "sample", "foo": true}
expected := []interface{}{a, b, c}
//actual := []interface{}{a, b, c}
actual := []interface{}{b, c, a}
differ := gojsondiff.New()
diff := differ.CompareArrays(expected, actual)
//diff := differ.CompareObjects(map[string]interface{}{"_": expected}, map[string]interface{}{"_": actual})
//formatter := formatter.NewDeltaFormatter()
formatter := formatter.NewAsciiFormatter(expected, formatter.AsciiFormatterConfig{ShowArrayIndex: true})
str, err := formatter.Format(diff)
if err != nil {
panic(err)
}
fmt.Printf("%s\n", str)
}
Diff Results:
$ go run diff.go
[
0: {
"foo": false,
"key": "bobby",
"val": "tables"
},
1: {
"foo": true,
"key": "how",
"val": "outage"
},
]
Hi,
I would like to store the diff in a database (map[string]interface{} output from FormatAsJson) and apply it later. It does not look like its currently possible. Looks like we need n formatFromJson formatter and then n applyJson patch method?
Thank you,
Riaan
This library depends on https://github.com/yudai/pp, but it is outdated from the original version.
Hi,
Would you mind to use vN.M.K
instead of N.M.K
for tag names, e.g. v1.0.0
? This way, it would be possible to use gopkg.in to import stable release, like this:
import "gopkg.in/yudai/gojsondiff.v1"
(see http://gopkg.in/yudai/gojsondiff.v1)
You could also add this example to your README.
Notes:
Hope to support ignoring the differences caused by different array orders when diffing specific keys
➜ gojsondiff git:(master) ✗ cat a.json && cat b.json
{
"arr":[
"item0",
"item1"
]
}
{
"arr":[
"item0",
"item2",
"item3"
]
}
➜ jd git:(master) ✗ go install && jd -f delta ../a.json ../b.json
panic: runtime error: index out of range
Looks like typo here: https://github.com/yudai/gojsondiff/blob/master/gojsondiff.go#L364
This is the output I get when I try to install this module
:!go get github.com/yudai/gojsondiff/jd
go: found github.com/yudai/gojsondiff/jd in github.com/yudai/gojsondiff v1.0.0
go: finding module for package github.com/codegangsta/cli
go: found github.com/codegangsta/cli in github.com/codegangsta/cli v1.22.4
go: github.com/yudai/gojsondiff/jd imports
github.com/codegangsta/cli: github.com/codegangsta/[email protected]: parsing go.mod:
module declares its path as: github.com/urfave/cli
but was required as: github.com/codegangsta/cli
shell returned 1
Is it possible to color the output according to the image in green/red?
$ cd $GOPATH/src/github.com/yudai/gojsondiff
$ go test ./...
vendor/github.com/yudai/gojsondiff/deltas.go:5:2: cannot find package "github.com/sergi/go-diff/diffmatchpatch" in any of:
/Users/ryan/go2/src/foo/vendor/github.com/yudai/gojsondiff/vendor/github.com/sergi/go-diff/diffmatchpatch (vendor tree)
/Users/ryan/go2/src/foo/vendor/github.com/sergi/go-diff/diffmatchpatch
/usr/local/Cellar/go/1.8/libexec/src/github.com/sergi/go-diff/diffmatchpatch (from $GOROOT)
/Users/ryan/go2/src/github.com/sergi/go-diff/diffmatchpatch (from $GOPATH)
vendor/github.com/yudai/gojsondiff/gojsondiff.go:13:2: cannot find package "github.com/yudai/golcs" in any of:
/Users/ryan/go2/src/foo/vendor/github.com/yudai/gojsondiff/vendor/github.com/yudai/golcs (vendor tree)
/Users/ryan/go2/src/foo/vendor/github.com/yudai/golcs
/usr/local/Cellar/go/1.8/libexec/src/github.com/yudai/golcs (from $GOROOT)
/Users/ryan/go2/src/github.com/yudai/golcs (from $GOPATH)
vendor/github.com/yudai/gojsondiff/jd/main.go:9:2: cannot find package "github.com/codegangsta/cli" in any of:
/Users/ryan/go2/src/foo/vendor/github.com/yudai/gojsondiff/vendor/github.com/codegangsta/cli (vendor tree)
/Users/ryan/go2/src/foo/vendor/github.com/codegangsta/cli
/usr/local/Cellar/go/1.8/libexec/src/github.com/codegangsta/cli (from $GOROOT)
/Users/ryan/go2/src/github.com/codegangsta/cli (from $GOPATH)
vendor/github.com/yudai/gojsondiff/tests/helper.go:4:2: cannot find package "github.com/onsi/ginkgo" in any of:
/Users/ryan/go2/src/foo/vendor/github.com/yudai/gojsondiff/vendor/github.com/onsi/ginkgo (vendor tree)
/Users/ryan/go2/src/foo/vendor/github.com/onsi/ginkgo
/usr/local/Cellar/go/1.8/libexec/src/github.com/onsi/ginkgo (from $GOROOT)
/Users/ryan/go2/src/github.com/onsi/ginkgo (from $GOPATH)
This is with a clean $GOPATH (a $GOPATH that contains gojsondiff
and nothing else).
These deps are not in vendor/
.
github.com/sergi/go-diff/diffmatchpatch
github.com/yudai/golcs
github.com/codegangsta/cli
This dep appears to be in vendor/
, but Go is having trouble finding it for some reason. Perhaps the vendoring of that pkg is incomplete.
github.com/onsi/ginkgo
You thought everything was vendored because you have everything in your Godeps.json
manifest and you ran godep restore
. Unfortunately, godep
is a first-gen vendoring tool with all sorts of surprising behavior. I could submit a PR to just add the missing deps, but that still leaves us with the problem of why ginkgo isn't being found, and it doesn't prevent future problems should you decide to add/remove deps.
A better solution is to revendor the deps using a better vendoring tool, such as Glide or gvt, so that this problem doesn't happen again. This is what I've done in #17 . In the process of revendoring gojsondiff
I discovered some other issues which are detailed in the PR notes.
I'm the "vendoring guy" at a company called Apcera. We're using gojsondiff
internally (thanks very much for writing it!). I discovered this issue when I tried to vendor gojsondiff
itself and it didn't work.
Hi,
It seems that current master is incompatible with v1, and they should not be used together. However, formatter.v1
imports gojsondiff
from master, which causes problems.
Here is an example of a problem import: https://github.com/yudai/gojsondiff/blob/master/formatter/ascii.go#L9.
This example compiles well:
package main
import (
"github.com/yudai/gojsondiff"
"github.com/yudai/gojsondiff/formatter"
)
func main() {
var d gojsondiff.Diff
c := formatter.AsciiFormatterConfig{}
f := formatter.NewAsciiFormatter(nil, c)
f.Format(d)
}
But this one fails to compile:
package main
import (
"gopkg.in/yudai/gojsondiff.v1"
"gopkg.in/yudai/gojsondiff.v1/formatter"
)
func main() {
var d gojsondiff.Diff
c := formatter.AsciiFormatterConfig{}
f := formatter.NewAsciiFormatter(nil, c)
f.Format(d)
}
With the following error:
# command-line-arguments
./gjd.go:14: cannot use d (type "gopkg.in/yudai/gojsondiff.v1".Diff)
as type "github.com/yudai/gojsondiff".Diff in argument to f.Format:
"gopkg.in/yudai/gojsondiff.v1".Diff does not implement
"github.com/yudai/gojsondiff".Diff (wrong type for Deltas method)
have Deltas() []"gopkg.in/yudai/gojsondiff.v1".Delta
want Deltas() []"github.com/yudai/gojsondiff".Delta
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.