bashtools / jsonpath.sh Goto Github PK
View Code? Open in Web Editor NEWJSONPath implementation in Bash for filtering, merging and modifying JSON
Home Page: https://bashtools.github.io/JSONPath.sh/
License: Other
JSONPath implementation in Bash for filtering, merging and modifying JSON
Home Page: https://bashtools.github.io/JSONPath.sh/
License: Other
$ echo '{"a": 1}' | ./JSONPath.sh -j '$.a'
{
0000"a":
,
"a":
,
{
0000"a":1
}
$ uname -sr
Darwin 17.7.0
$ bash --version | grep 'GNU bash'
GNU bash, version 5.0.7(1)-release (x86_64-apple-darwin17.7.0)
$ echo $LANG
en_US.UTF-8
$ echo $LC_CTYPE
en_US.UTF-8
Somewhat better with coreutils:
# brew install coreutils
$ export PATH="/usr/local/opt/coreutils/libexec/gnubin:$PATH"
$ echo '{"a": 1}' | ./JSONPath.sh -j '$.a'
{
0000"a":1
}
The following didn't help
brew install grep
export PATH="/usr/local/opt/grep/libexec/gnubin:$PATH"
brew install gnu-sed
export PATH="/usr/local/opt/gnu-sed/libexec/gnubin:$PATH"
brew install gnu-indent
export PATH="/usr/local/opt/gnu-indent/libexec/gnubin:$PATH"
brew install gawk
alias awk=gawk
The following tests fail against Ubuntu 18.04:
TEST: test/flatten-test.sh
1..57
... snip ...
ok 14 - flatten/goessner.net.expanded.argp2
--- - 2019-09-29 17:41:37.492358300 +0000
+++ flatten/goessner.net.expanded_argp3.flattened 2019-09-29 12:01:19.000000000 +0000
@@ -1,4 +1,3 @@
-[0,"title"] "Sayings of the Century"
[1,"title"] "Sword of Honour"
[2,"title"] "Moby Dick"
[3,"title"] "The Lord of the Rings"
not ok 15 - flatten/goessner.net.expanded.argp3
... snip ...
--- - 2019-09-29 17:41:39.326693000 +0000
+++ flatten/goessner.net.expanded_argp9.flattened 2019-09-29 12:01:19.000000000 +0000
@@ -0,0 +1,5 @@
+[0,"title"] "Sayings of the Century"
+[1,"title"] "Sword of Honour"
+[2,"title"] "Moby Dick"
+[3,"title"] "The Lord of the Rings"
+[4,"title"] "Planet Urth"
not ok 21 - flatten/goessner.net.expanded.argp9
... snip ...
--- - 2019-09-29 17:41:41.321667200 +0000
+++ flatten/jsonpath.com_argp1.flattened 2019-09-29 12:01:19.000000000 +0000
@@ -0,0 +1 @@
+["type"] "iPhone"
not ok 32 - flatten/jsonpath.com.argp1
... snip ...
--- - 2019-09-29 17:46:12.407275000 +0000
+++ flatten/swapi.co_argp6.flattened 2019-09-29 12:01:19.000000000 +0000
@@ -1,4 +1,3 @@
-[0] "http://swapi.co/api/films/2/"
[1] "http://swapi.co/api/films/5/"
[2] "http://swapi.co/api/films/4/"
[3] "http://swapi.co/api/films/6/"
not ok 56 - flatten/swapi.co.argp6
ok 57 - flatten/swapi.co.argp7
4 test(s) failed
FAIL: test/flatten-test.sh 0
TEST: test/json-encoding-test.sh
1..73
... snip ...
No JSON object could be decoded
not ok 31 - valid/goessner.net.expanded.argp9
... snip ...
No JSON object could be decoded
not ok 42 - valid/jsonpath.com.argp1
... snip ...
3 test(s) failed
FAIL: test/json-encoding-test.sh 4
TEST: test/passthrough-test.sh
1..130
... snip ...
0 test(s) failed
OK: ---- test/passthrough-test.sh
TEST: test/valid-test.sh
1..73
... snip ...
--- - 2019-09-29 18:21:56.090888400 +0000
+++ valid/goessner.net.expanded_argp3.parsed 2019-09-29 12:01:19.000000000 +0000
@@ -1,4 +1,3 @@
-["store","book",0,"title"] "Sayings of the Century"
["store","book",1,"title"] "Sword of Honour"
["store","book",2,"title"] "Moby Dick"
["store","book",3,"title"] "The Lord of the Rings"
not ok 25 - valid/goessner.net.expanded.argp3
... snip ...
--- - 2019-09-29 18:21:57.751861400 +0000
+++ valid/goessner.net.expanded_argp9.parsed 2019-09-29 12:01:19.000000000 +0000
@@ -0,0 +1,5 @@
+["store","book",0,"title"] "Sayings of the Century"
+["store","book",1,"title"] "Sword of Honour"
+["store","book",2,"title"] "Moby Dick"
+["store","book",3,"title"] "The Lord of the Rings"
+["store","book",4,"title"] "Planet Urth"
not ok 31 - valid/goessner.net.expanded.argp9
... snip ...
--- - 2019-09-29 18:21:58.878464300 +0000
+++ valid/jsonpath.com_argp1.parsed 2019-09-29 12:01:19.000000000 +0000
@@ -0,0 +1 @@
+["phoneNumbers",0,"type"] "iPhone"
not ok 42 - valid/jsonpath.com.argp1
... snip ...
--- - 2019-09-29 18:22:26.283757700 +0000
+++ valid/swapi.co_argp6.parsed 2019-09-29 12:01:19.000000000 +0000
@@ -1,4 +1,3 @@
-["results",9,"films",0] "http://swapi.co/api/films/2/"
["results",9,"films",1] "http://swapi.co/api/films/5/"
["results",9,"films",2] "http://swapi.co/api/films/4/"
["results",9,"films",3] "http://swapi.co/api/films/6/"
not ok 71 - valid/swapi.co.argp6
ok 72 - valid/swapi.co.argp7
ok 73 - valid/terraform.argp1
6 test(s) failed
FAIL: test/valid-test.sh 7
FAILURE 1 / 4
I've excluded the failing test cases already mentioned in #8 .
To reproduce, I ran ./wrap_in_docker.sh ./all-tests.sh
using the following script wrap_in_docker.sh
:
#!/usr/bin/env bash
set -euo pipefail
readonly script_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
readonly RUN_ARGS="$@"
readonly target_image=json-path-bash
docker build -t "$target_image" "$script_dir"
docker run --rm -v "$(pwd):/localdir" -i "$target_image" sh -c "cd /localdir && $RUN_ARGS"
and the Dockerfile
FROM ubuntu
RUN apt-get update && apt-get -y install python
JSON file:
{
"type": "FeatureCollection",
"name": "Loop_Trail",
"features": [{
"type": "Feature",
"properties": {
"type": "Improved",
"tident": "loop trail",
"ident": "T1",
"Latitude": 29.817366931975631,
"Longitude": -82.398078610345138,
"y_proj": 3458853.014648342,
"x_proj": -9172512.1532477476,
"comment": "",
"new_trk": null,
"new_seg": null,
"display": "",
"color": "",
"altitude": 0.0,
"depth": 0.0,
"temp": 0.0,
"time": "",
"model": "",
"filename": "",
"ltime": "",
"desc": "",
"link": ""
},
"geometry": {
"type": "MultiLineString",
"coordinates": [
[
[-9172512.153247747570276, 3458853.014648342039436],
[-9172516.276892628520727, 3458853.380998854525387],
[-9172527.76284097880125, 3458852.102625732310116],
[-9172553.879514154046774, 3458848.130447461735457],
[-9172580.67746327072382, 3458859.028638950083405],
[-9172616.376520942896605, 3458853.466818191576749],
[-9172650.237693645060062, 3458859.092683436814696],
[-9172656.787734545767307, 3458863.902595946099609],
[-9172668.544189484789968, 3458867.509711052756757],
[-9172677.165886519476771, 3458867.102373105473816],
[-9172685.451398713514209, 3458870.011378903407604],
[-9172702.079195905476809, 3458871.042532142717391],
[-9172716.728845156729221, 3458871.858488189987838],
[-9172737.918516477569938, 3458874.896869246847928],
[-9172750.40300078690052, 3458868.487062633037567],
[-9172799.155385768041015, 3458864.794122570194304],
[-9172811.462871868163347, 3458853.273390656337142],
[-9172822.929895531386137, 3458841.194182069040835],
[-9172829.060260597616434, 3458808.147421804722399],
[-9172826.886188978329301, 3458767.283252045512199],
[-9172822.733970277011395, 3458751.994127082638443],
[-9172828.482510069385171, 3458740.506800198927522],
[-9172829.03242801502347, 3458724.745046642143279],
[-9172817.09897468239069, 3458710.562687636353076],
[-9172809.718490494415164, 3458716.252537921071053],
[-9172793.622800694778562, 3458717.852408944163471],
[-9172788.276123931631446, 3458715.458367603830993],
[-9172773.860245944932103, 3458722.963286225218326],
[-9172770.604149956256151, 3458724.691249841824174],
[-9172766.386253135278821, 3458721.900121493730694],
[-9172759.146031018346548, 3458711.63610094320029],
[-9172755.264319203794003, 3458709.972185332328081],
[-9172749.777380080893636, 3458715.081778103020042],
[-9172745.345749905332923, 3458716.112919636070728],
[-9172731.087945682927966, 3458725.293284447863698],
[-9172725.359443029388785, 3458724.862894857302308],
[-9172715.403024978935719, 3458728.052391129080206],
[-9172699.496902368962765, 3458730.442288412712514]
]
]
}
},
{
"type": "Feature",
"properties": {
"type": "Improved",
"tident": "loop trail",
"ident": "T39",
"Latitude": 29.816406762108624,
"Longitude": -82.399761547028433,
"y_proj": 3458730.4422884127,
"x_proj": -9172699.496902369,
"comment": "",
"new_trk": null,
"new_seg": null,
"display": "",
"color": "",
"altitude": 0.0,
"depth": 0.0,
"temp": 0.0,
"time": "",
"model": "",
"filename": "",
"ltime": "",
"desc": "",
"link": ""
},
"geometry": {
"type": "MultiLineString",
"coordinates": [
[
[-9172699.496902368962765, 3458730.442288412712514],
[-9172693.455310769379139, 3458732.608350321650505]
]
]
}
},
{
"type": "Feature",
"properties": {
"type": "Improved",
"tident": "loop trail",
"ident": "T41",
"Latitude": 29.816423730022812,
"Longitude": -82.399707274487682,
"y_proj": 3458732.6083503217,
"x_proj": -9172693.4553107694,
"comment": "",
"new_trk": null,
"new_seg": null,
"display": "",
"color": "",
"altitude": 0.0,
"depth": 0.0,
"temp": 0.0,
"time": "",
"model": "",
"filename": "",
"ltime": "",
"desc": "",
"link": ""
},
"geometry": {
"type": "MultiLineString",
"coordinates": [
[
[-9172693.455310769379139, 3458732.608350321650505],
[-9172689.781723748892546, 3458733.925421689171344],
[-9172673.330925136804581, 3458749.428440262097865],
[-9172668.105586804449558, 3458751.844263973645866],
[-9172659.90467806905508, 3458768.217052695341408],
[-9172658.355110624805093, 3458777.654920535627753],
[-9172655.481954025104642, 3458786.909622615668923],
[-9172655.052261060103774, 3458798.398271592799574],
[-9172648.380882352590561, 3458808.329319821670651],
[-9172633.862591058015823, 3458835.805302318651229],
[-9172626.714764591306448, 3458839.30481178779155],
[-9172616.376520942896605, 3458853.466818191576749]
]
]
}
}
]
}
I want to add the following segment in to the above JSON file (in to ["features",0,"properties"]
"properties": {
"name": "Some Other Trailname",
"filters": [
"Cycling",
"dogAccess",
"kidFriendly"
],
"price": "Free",
"hours": {
"mon": "Sunrise - Sundown",
"tue": "Sunrise - Sundown",
"wed": "Sunrise - Sundown",
"thu": "Sunrise - Sundown",
"fri": "Sunrise - Sundown",
"sat": "Sunrise - Sundown",
"sun": "Sunrise - Sundown"
}
},
How do I do that using JSONPath.sh ?
IETF RFC 9535 defines a proposed standard for JSONPath. Other resources:
(See here for updates to this list of resources.)
Valid approaches for this project are to (a) implement the RFC in whole or in part, (b) document its behaviour relative to the RFC, and (c) do nothing.
I can reproduce the following two queries failing when running ./all-tests.sh
:
$ cat test/valid/goessner.net.expanded.argp20
$.store.book[?(@.title<"j")].title
$ cat test/valid/goessner.net.expanded.argp21
$.store.book[?(@.title>"j")].title
It seems all the entries missing in the latter query (i.e. greater than j
) turn up in the previous one instead (i.e. lighter than j
). This would be correct if case sensitivity is respected.
Output:
TEST: test/valid-test.sh
1..73
ok 1 - valid/api.github.com.argp1
... snip ...
ok 22 - valid/goessner.net.expanded.argp2
--- - 2019-09-29 20:51:08.000000000 +0200
+++ valid/goessner.net.expanded_argp20.parsed 2019-09-29 14:01:19.000000000 +0200
@@ -1,25 +1,11 @@
-["store","book",0,"title"] "Sayings of the Century"
-["store","book",1,"title"] "Sword of Honour"
-["store","book",2,"title"] "Moby Dick"
-["store","book",3,"title"] "The Lord of the Rings"
-["store","book",4,"title"] "Planet Urth"
["store","book",5,"title"] "Girl From Above: Betrayal (The 1000 Revolution)"
["store","book",6,"title"] "Girl From Above: Escape (The 1000 Revolution Book 2)"
["store","book",7,"title"] "Girl From Above: Trapped (The 1000 Revolution Book 3)"
["store","book",8,"title"] "Girl From Above: Trust (The 1000 Revolution Book 4)"
-["store","book",9,"title"] "Leviathan Wakes: Book 1 of the Expanse"
["store","book",10,"title"] "Caliban's War: Book 2 of the Expanse"
["store","book",11,"title"] "Abaddon's Gate: Book 3 of the Expanse"
-["store","book",12,"title"] "The Reality Dysfunction (Nights Dawn Book 1)"
-["store","book",13,"title"] "The Naked God (Nights Dawn Book 3)"
-["store","book",14,"title"] "The Neutronium Alchemist (Nights Dawn Book 2)"
-["store","book",15,"title"] "The Abyss Beyond Dreams (Chronicle of the Fallers Book 1)"
-["store","book",16,"title"] "Night Without Stars (Chronicle of the Fallers Book 2)"
["store","book",17,"title"] "Dark Space: The Original Trilogy (Books 1-3) (Dark Space Trilogies)"
["store","book",18,"title"] "Dark Space (Book 4): Revenge"
["store","book",19,"title"] "Dark Space (Book 5): Avilon"
["store","book",20,"title"] "Dark Space (Book 6): Armageddon"
-["store","book",21,"title"] "The Honour of the Knights (Battle for the Solar System, #1)"
-["store","book",22,"title"] "The Third Side (Battle for the Solar System, #2)"
-["store","book",23,"title"] "Split Second"
["store","book",24,"title"] "BrainWeb"
not ok 23 - valid/goessner.net.expanded.argp20
--- - 2019-09-29 20:51:09.000000000 +0200
+++ valid/goessner.net.expanded_argp21.parsed 2019-09-29 14:01:19.000000000 +0200
@@ -0,0 +1,14 @@
+["store","book",0,"title"] "Sayings of the Century"
+["store","book",1,"title"] "Sword of Honour"
+["store","book",2,"title"] "Moby Dick"
+["store","book",3,"title"] "The Lord of the Rings"
+["store","book",4,"title"] "Planet Urth"
+["store","book",9,"title"] "Leviathan Wakes: Book 1 of the Expanse"
+["store","book",12,"title"] "The Reality Dysfunction (Nights Dawn Book 1)"
+["store","book",13,"title"] "The Naked God (Nights Dawn Book 3)"
+["store","book",14,"title"] "The Neutronium Alchemist (Nights Dawn Book 2)"
+["store","book",15,"title"] "The Abyss Beyond Dreams (Chronicle of the Fallers Book 1)"
+["store","book",16,"title"] "Night Without Stars (Chronicle of the Fallers Book 2)"
+["store","book",21,"title"] "The Honour of the Knights (Battle for the Solar System, #1)"
+["store","book",22,"title"] "The Third Side (Battle for the Solar System, #2)"
+["store","book",23,"title"] "Split Second"
not ok 24 - valid/goessner.net.expanded.argp21
ok 25 - valid/goessner.net.expanded.argp3
... snip ...
2 test(s) failed
FAIL: test/valid-test.sh 1
I can reproduce this consistently on macOS, Ubuntu (Docker's latest) and Alpine 3.10.
It would be great to have the option to get the line number where the match was found.
@cburgmer's JSONPath comparison project is currently discussing some issues relating to a proposed implementation of JSONPath known as "Proposal A". May I encourage you to get involved if you are interested in influencing the direction or, indeed, if you want to make your own proposals.
Similarly, please join us in slack (invitation) for informal discussions on the comparison project and potential JSONPath standardisation.
The following queries provide results that do not match those of other implementations of JSONPath
(compare https://cburgmer.github.io/json-path-comparison/):
$[1:10]
Input:
["first", "second", "third"]
Expected output:
["second", "third"]
Error:
sed: bad regex '\[(([0-9]+|"[^"]+")[],]){9999}(.*)': Invalid contents of {}
No JSON object could be decoded
$[-1:]
Input:
["first", "second", "third"]
Expected output:
["third"]
Actual output:
["first", "second", "third"]
$[1:]
Input:
["first", "second", "third", "forth", "fifth"]
Expected output:
["second", "third", "forth", "fifth"]
Error:
sed: bad regex '\[(([0-9]+|"[^"]+")[],]){9999}(.*)': Invalid contents of {}
No JSON object could be decoded
$[0:3:2]
Input:
["first", "second", "third", "forth", "fifth"]
Expected output:
["first", "third"]
Actual output:
["first", "second", "third", "forth", "fifth"]
$[0:3:1]
Input:
["first", "second", "third", "forth", "fifth"]
Expected output:
["first", "second", "third"]
Actual output:
["first", "second", "third", "forth", "fifth"]
$['key']
Input:
{"key": "value"}
Expected output:
["value"]
Actual output:
{"key": "value"}
$['key','another']
Input:
{"key": "value", "another": "entry"}
Expected output:
["value", "entry"]
Actual output:
{"another": "entry", "key": "value"}
$['0']
Input:
{"0": "value"}
Expected output:
["value"]
Error:
sed: bad regex '\[(([0-9]+|"[^"]+")[],]){9999}(.*)': Invalid contents of {}
No JSON object could be decoded
$['special:"chars']
Input:
{"special:\"chars": "value"}
Expected output:
["value"]
Error:
sed: bad regex '\[(([0-9]+|"[^"]+")[],]){9999}(.*)': Invalid contents of {}
No JSON object could be decoded
$['*']
Input:
{"*": "value"}
Expected output:
["value"]
Actual output:
{"*": "value"}
$.key
Input:
{"key": "value"}
Expected output:
["value"]
Actual output:
{"key": "value"}
$.key
Input:
{"key": ["first", "second"]}
Expected output:
[["first", "second"]]
Actual output:
["first", "second"]
$.key
Input:
{"key": null}
Expected output:
[null]
Actual output:
{"key": null}
$..key
Input:
{"object": {"key": "value", "array": [{"key": "something"}, {"key": {"key": "russian dolls"}}]}, "key": "top"}
Expected output:
["top", "value", "something", {"key": "russian dolls"}, "russian dolls"]
Actual output:
{"key": "top", "object": {"array": [{"key": "something"}, {"key": {"key": "russian dolls"}}], "key": "value"}}
$.store..price
Input:
{"store": {"book": [{"category": "reference", "author": "Nigel Rees", "title": "Sayings of the Century", "price": 8.95}, {"category": "fiction", "author": "Evelyn Waugh", "title": "Sword of Honour", "price": 12.99}, {"category": "fiction", "author": "Herman Melville", "title": "Moby Dick", "isbn": "0-553-21311-3", "price": 8.99}, {"category": "fiction", "author": "J. R. R. Tolkien", "title": "The Lord of the Rings", "isbn": "0-395-19395-8", "price": 22.99}], "bicycle": {"color": "red", "price": 19.95}}}
Expected output:
[8.95, 12.99, 8.99, 22.99, 19.95]
Actual output:
{"bicycle": {"price": 19.95}, "book": [{"price": 8.95}, {"price": 12.99}, {"price": 8.99}, {"price": 22.99}]}
$
Input:
{"key": "value", "another key": {"complex": ["a", 1]}}
Expected output:
[{"another key": {"complex": ["a", 1]}, "key": "value"}]
Error:
sed: bad regex '\[(([0-9]+|"[^"]+")[],]){9999}(.*)': Invalid contents of {}
No JSON object could be decoded
$[*]
Input:
["string", 42, {"key": "value"}, [0, 1]]
Expected output:
["string", 42, {"key": "value"}, [0, 1]]
Error:
Extra data: line 4 column 6 - line 12 column 1 (char 28 - 94)
$[*]
Input:
{"some": "string", "int": 42, "object": {"key": "value"}, "array": [0, 1]}
Expected output:
["string", 42, {"key": "value"}, [0, 1]]
Actual output:
{"array": [0, 1], "int": 42, "object": {"key": "value"}, "some": "string"}
$.*
Input:
["string", 42, {"key": "value"}, [0, 1]]
Expected output:
["string", 42, {"key": "value"}, [0, 1]]
Error:
Extra data: line 4 column 6 - line 12 column 1 (char 28 - 94)
$.*
Input:
{"some": "string", "int": 42, "object": {"key": "value"}, "array": [0, 1]}
Expected output:
["string", 42, {"key": "value"}, [0, 1]]
Actual output:
{"array": [0, 1], "int": 42, "object": {"key": "value"}, "some": "string"}
For reference, the output was generated by the program in https://github.com/cburgmer/json-path-comparison/tree/Bash_JSONPath.sh/implementations/Bash_JSONPath.sh.
It seems that the path used in https://github.com/bashtools/JSONPath.sh/blob/ce89800d1f0d4e265fccf69e007ad5ea827edc8a/ensure_deps.sh#L34C22-L34C61 is not valid anymore for the current Homebrew installation. Instead the binaries currently seem to be found under /opt/homebrew/opt/coreutils/libexec/gnubin
.
Can file a PR when I find more time.
When I try to use JSONPath on Mac OS it outputs the following error:
line 250: readarray: command not found
A declarative, efficient, and flexible JavaScript library for building user interfaces.
๐ Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. ๐๐๐
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google โค๏ธ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.