bashtools / jsonpath.sh Goto Github PK
View Code? Open in Web Editor NEWJSONPath implementation in Bash for filtering, merging and modifying JSON
Home Page: http://jsonpath.obdi.io
License: Other
jsonpath.sh's People
Contributors
Stargazers
Watchers
Forkers
chvostek qinghai5060 adminstratorhosing ianldgs 0xflotus cburgmer sbonefas c0psrul3 sambacha yavko bash-bastion ankurvaishley dohoudanieljsonpath.sh's Issues
JSON output fails on Mac OS
$ echo '{"a": 1}' | ./JSONPath.sh -j '$.a'
{
0000"a":
,
"a":
,
{
0000"a":1
}
$ uname -sr
Darwin 17.7.0
$ bash --version | grep 'GNU bash'
GNU bash, version 5.0.7(1)-release (x86_64-apple-darwin17.7.0)
$ echo $LANG
en_US.UTF-8
$ echo $LC_CTYPE
en_US.UTF-8
Somewhat better with coreutils:
# brew install coreutils
$ export PATH="/usr/local/opt/coreutils/libexec/gnubin:$PATH"
$ echo '{"a": 1}' | ./JSONPath.sh -j '$.a'
{
0000"a":1
}
The following didn't help
brew install grep
export PATH="/usr/local/opt/grep/libexec/gnubin:$PATH"
brew install gnu-sed
export PATH="/usr/local/opt/gnu-sed/libexec/gnubin:$PATH"
brew install gnu-indent
export PATH="/usr/local/opt/gnu-indent/libexec/gnubin:$PATH"
brew install gawk
alias awk=gawk
Failing test cases on Ubuntu 18.04
The following tests fail against Ubuntu 18.04:
TEST: test/flatten-test.sh
1..57
... snip ...
ok 14 - flatten/goessner.net.expanded.argp2
--- - 2019-09-29 17:41:37.492358300 +0000
+++ flatten/goessner.net.expanded_argp3.flattened 2019-09-29 12:01:19.000000000 +0000
@@ -1,4 +1,3 @@
-[0,"title"] "Sayings of the Century"
[1,"title"] "Sword of Honour"
[2,"title"] "Moby Dick"
[3,"title"] "The Lord of the Rings"
not ok 15 - flatten/goessner.net.expanded.argp3
... snip ...
--- - 2019-09-29 17:41:39.326693000 +0000
+++ flatten/goessner.net.expanded_argp9.flattened 2019-09-29 12:01:19.000000000 +0000
@@ -0,0 +1,5 @@
+[0,"title"] "Sayings of the Century"
+[1,"title"] "Sword of Honour"
+[2,"title"] "Moby Dick"
+[3,"title"] "The Lord of the Rings"
+[4,"title"] "Planet Urth"
not ok 21 - flatten/goessner.net.expanded.argp9
... snip ...
--- - 2019-09-29 17:41:41.321667200 +0000
+++ flatten/jsonpath.com_argp1.flattened 2019-09-29 12:01:19.000000000 +0000
@@ -0,0 +1 @@
+["type"] "iPhone"
not ok 32 - flatten/jsonpath.com.argp1
... snip ...
--- - 2019-09-29 17:46:12.407275000 +0000
+++ flatten/swapi.co_argp6.flattened 2019-09-29 12:01:19.000000000 +0000
@@ -1,4 +1,3 @@
-[0] "http://swapi.co/api/films/2/"
[1] "http://swapi.co/api/films/5/"
[2] "http://swapi.co/api/films/4/"
[3] "http://swapi.co/api/films/6/"
not ok 56 - flatten/swapi.co.argp6
ok 57 - flatten/swapi.co.argp7
4 test(s) failed
FAIL: test/flatten-test.sh 0
TEST: test/json-encoding-test.sh
1..73
... snip ...
No JSON object could be decoded
not ok 31 - valid/goessner.net.expanded.argp9
... snip ...
No JSON object could be decoded
not ok 42 - valid/jsonpath.com.argp1
... snip ...
3 test(s) failed
FAIL: test/json-encoding-test.sh 4
TEST: test/passthrough-test.sh
1..130
... snip ...
0 test(s) failed
OK: ---- test/passthrough-test.sh
TEST: test/valid-test.sh
1..73
... snip ...
--- - 2019-09-29 18:21:56.090888400 +0000
+++ valid/goessner.net.expanded_argp3.parsed 2019-09-29 12:01:19.000000000 +0000
@@ -1,4 +1,3 @@
-["store","book",0,"title"] "Sayings of the Century"
["store","book",1,"title"] "Sword of Honour"
["store","book",2,"title"] "Moby Dick"
["store","book",3,"title"] "The Lord of the Rings"
not ok 25 - valid/goessner.net.expanded.argp3
... snip ...
--- - 2019-09-29 18:21:57.751861400 +0000
+++ valid/goessner.net.expanded_argp9.parsed 2019-09-29 12:01:19.000000000 +0000
@@ -0,0 +1,5 @@
+["store","book",0,"title"] "Sayings of the Century"
+["store","book",1,"title"] "Sword of Honour"
+["store","book",2,"title"] "Moby Dick"
+["store","book",3,"title"] "The Lord of the Rings"
+["store","book",4,"title"] "Planet Urth"
not ok 31 - valid/goessner.net.expanded.argp9
... snip ...
--- - 2019-09-29 18:21:58.878464300 +0000
+++ valid/jsonpath.com_argp1.parsed 2019-09-29 12:01:19.000000000 +0000
@@ -0,0 +1 @@
+["phoneNumbers",0,"type"] "iPhone"
not ok 42 - valid/jsonpath.com.argp1
... snip ...
--- - 2019-09-29 18:22:26.283757700 +0000
+++ valid/swapi.co_argp6.parsed 2019-09-29 12:01:19.000000000 +0000
@@ -1,4 +1,3 @@
-["results",9,"films",0] "http://swapi.co/api/films/2/"
["results",9,"films",1] "http://swapi.co/api/films/5/"
["results",9,"films",2] "http://swapi.co/api/films/4/"
["results",9,"films",3] "http://swapi.co/api/films/6/"
not ok 71 - valid/swapi.co.argp6
ok 72 - valid/swapi.co.argp7
ok 73 - valid/terraform.argp1
6 test(s) failed
FAIL: test/valid-test.sh 7
FAILURE 1 / 4
I've excluded the failing test cases already mentioned in #8 .
To reproduce, I ran ./wrap_in_docker.sh ./all-tests.sh using the following script wrap_in_docker.sh:
#!/usr/bin/env bash
set -euo pipefail
readonly script_dir="$( cd "$( dirname "${BASH_SOURCE[0]}" )" && pwd )"
readonly RUN_ARGS="$@"
readonly target_image=json-path-bash
docker build -t "$target_image" "$script_dir"
docker run --rm -v "$(pwd):/localdir" -i "$target_image" sh -c "cd /localdir && $RUN_ARGS"
and the Dockerfile
FROM ubuntu
RUN apt-get update && apt-get -y install python
Doesn't work on Mac OS
When I try to use JSONPath on Mac OS it outputs the following error:
line 250: readarray: command not found
Output with line numbers would be great
It would be great to have the option to get the line number where the match was found.
Fix error when percent sign is in a key name
JSONPath comparison and standardisation
@cburgmer's JSONPath comparison project is currently discussing some issues relating to a proposed implementation of JSONPath known as "Proposal A". May I encourage you to get involved if you are interested in influencing the direction or, indeed, if you want to make your own proposals.
Similarly, please join us in slack (invitation) for informal discussions on the comparison project and potential JSONPath standardisation.
Failing test case related to case insensitivity
I can reproduce the following two queries failing when running ./all-tests.sh:
$ cat test/valid/goessner.net.expanded.argp20
$.store.book[?(@.title<"j")].title
$ cat test/valid/goessner.net.expanded.argp21
$.store.book[?(@.title>"j")].title
It seems all the entries missing in the latter query (i.e. greater than j) turn up in the previous one instead (i.e. lighter than j). This would be correct if case sensitivity is respected.
Output:
TEST: test/valid-test.sh
1..73
ok 1 - valid/api.github.com.argp1
... snip ...
ok 22 - valid/goessner.net.expanded.argp2
--- - 2019-09-29 20:51:08.000000000 +0200
+++ valid/goessner.net.expanded_argp20.parsed 2019-09-29 14:01:19.000000000 +0200
@@ -1,25 +1,11 @@
-["store","book",0,"title"] "Sayings of the Century"
-["store","book",1,"title"] "Sword of Honour"
-["store","book",2,"title"] "Moby Dick"
-["store","book",3,"title"] "The Lord of the Rings"
-["store","book",4,"title"] "Planet Urth"
["store","book",5,"title"] "Girl From Above: Betrayal (The 1000 Revolution)"
["store","book",6,"title"] "Girl From Above: Escape (The 1000 Revolution Book 2)"
["store","book",7,"title"] "Girl From Above: Trapped (The 1000 Revolution Book 3)"
["store","book",8,"title"] "Girl From Above: Trust (The 1000 Revolution Book 4)"
-["store","book",9,"title"] "Leviathan Wakes: Book 1 of the Expanse"
["store","book",10,"title"] "Caliban's War: Book 2 of the Expanse"
["store","book",11,"title"] "Abaddon's Gate: Book 3 of the Expanse"
-["store","book",12,"title"] "The Reality Dysfunction (Nights Dawn Book 1)"
-["store","book",13,"title"] "The Naked God (Nights Dawn Book 3)"
-["store","book",14,"title"] "The Neutronium Alchemist (Nights Dawn Book 2)"
-["store","book",15,"title"] "The Abyss Beyond Dreams (Chronicle of the Fallers Book 1)"
-["store","book",16,"title"] "Night Without Stars (Chronicle of the Fallers Book 2)"
["store","book",17,"title"] "Dark Space: The Original Trilogy (Books 1-3) (Dark Space Trilogies)"
["store","book",18,"title"] "Dark Space (Book 4): Revenge"
["store","book",19,"title"] "Dark Space (Book 5): Avilon"
["store","book",20,"title"] "Dark Space (Book 6): Armageddon"
-["store","book",21,"title"] "The Honour of the Knights (Battle for the Solar System, #1)"
-["store","book",22,"title"] "The Third Side (Battle for the Solar System, #2)"
-["store","book",23,"title"] "Split Second"
["store","book",24,"title"] "BrainWeb"
not ok 23 - valid/goessner.net.expanded.argp20
--- - 2019-09-29 20:51:09.000000000 +0200
+++ valid/goessner.net.expanded_argp21.parsed 2019-09-29 14:01:19.000000000 +0200
@@ -0,0 +1,14 @@
+["store","book",0,"title"] "Sayings of the Century"
+["store","book",1,"title"] "Sword of Honour"
+["store","book",2,"title"] "Moby Dick"
+["store","book",3,"title"] "The Lord of the Rings"
+["store","book",4,"title"] "Planet Urth"
+["store","book",9,"title"] "Leviathan Wakes: Book 1 of the Expanse"
+["store","book",12,"title"] "The Reality Dysfunction (Nights Dawn Book 1)"
+["store","book",13,"title"] "The Naked God (Nights Dawn Book 3)"
+["store","book",14,"title"] "The Neutronium Alchemist (Nights Dawn Book 2)"
+["store","book",15,"title"] "The Abyss Beyond Dreams (Chronicle of the Fallers Book 1)"
+["store","book",16,"title"] "Night Without Stars (Chronicle of the Fallers Book 2)"
+["store","book",21,"title"] "The Honour of the Knights (Battle for the Solar System, #1)"
+["store","book",22,"title"] "The Third Side (Battle for the Solar System, #2)"
+["store","book",23,"title"] "Split Second"
not ok 24 - valid/goessner.net.expanded.argp21
ok 25 - valid/goessner.net.expanded.argp3
... snip ...
2 test(s) failed
FAIL: test/valid-test.sh 1
I can reproduce this consistently on macOS, Ubuntu (Docker's latest) and Alpine 3.10.
Consider implementing RFC 9535
IETF RFC 9535 defines a proposed standard for JSONPath. Other resources:
- A partial Compliance Test Suite.
- Behaviours of various implementations can be checked in the comparison project. Please note that the JSONPath Reference Implementation referenced in the comparison project is stalled/archived and incomplete.
- At the time of writing, online sandboxes for RFC 9535 are available in C# and in Rust.
- For more background and comments on the status of the CTS and RI, see this this post (also here in the IETF blog).
- Public discussion occurs in the IETF JSONPath Working Group's mailing list (archive).
(See here for updates to this list of resources.)
Valid approaches for this project are to (a) implement the RFC in whole or in part, (b) document its behaviour relative to the RFC, and (c) do nothing.
How do I insert this in to a JSON file?
JSON file:
{
"type": "FeatureCollection",
"name": "Loop_Trail",
"features": [{
"type": "Feature",
"properties": {
"type": "Improved",
"tident": "loop trail",
"ident": "T1",
"Latitude": 29.817366931975631,
"Longitude": -82.398078610345138,
"y_proj": 3458853.014648342,
"x_proj": -9172512.1532477476,
"comment": "",
"new_trk": null,
"new_seg": null,
"display": "",
"color": "",
"altitude": 0.0,
"depth": 0.0,
"temp": 0.0,
"time": "",
"model": "",
"filename": "",
"ltime": "",
"desc": "",
"link": ""
},
"geometry": {
"type": "MultiLineString",
"coordinates": [
[
[-9172512.153247747570276, 3458853.014648342039436],
[-9172516.276892628520727, 3458853.380998854525387],
[-9172527.76284097880125, 3458852.102625732310116],
[-9172553.879514154046774, 3458848.130447461735457],
[-9172580.67746327072382, 3458859.028638950083405],
[-9172616.376520942896605, 3458853.466818191576749],
[-9172650.237693645060062, 3458859.092683436814696],
[-9172656.787734545767307, 3458863.902595946099609],
[-9172668.544189484789968, 3458867.509711052756757],
[-9172677.165886519476771, 3458867.102373105473816],
[-9172685.451398713514209, 3458870.011378903407604],
[-9172702.079195905476809, 3458871.042532142717391],
[-9172716.728845156729221, 3458871.858488189987838],
[-9172737.918516477569938, 3458874.896869246847928],
[-9172750.40300078690052, 3458868.487062633037567],
[-9172799.155385768041015, 3458864.794122570194304],
[-9172811.462871868163347, 3458853.273390656337142],
[-9172822.929895531386137, 3458841.194182069040835],
[-9172829.060260597616434, 3458808.147421804722399],
[-9172826.886188978329301, 3458767.283252045512199],
[-9172822.733970277011395, 3458751.994127082638443],
[-9172828.482510069385171, 3458740.506800198927522],
[-9172829.03242801502347, 3458724.745046642143279],
[-9172817.09897468239069, 3458710.562687636353076],
[-9172809.718490494415164, 3458716.252537921071053],
[-9172793.622800694778562, 3458717.852408944163471],
[-9172788.276123931631446, 3458715.458367603830993],
[-9172773.860245944932103, 3458722.963286225218326],
[-9172770.604149956256151, 3458724.691249841824174],
[-9172766.386253135278821, 3458721.900121493730694],
[-9172759.146031018346548, 3458711.63610094320029],
[-9172755.264319203794003, 3458709.972185332328081],
[-9172749.777380080893636, 3458715.081778103020042],
[-9172745.345749905332923, 3458716.112919636070728],
[-9172731.087945682927966, 3458725.293284447863698],
[-9172725.359443029388785, 3458724.862894857302308],
[-9172715.403024978935719, 3458728.052391129080206],
[-9172699.496902368962765, 3458730.442288412712514]
]
]
}
},
{
"type": "Feature",
"properties": {
"type": "Improved",
"tident": "loop trail",
"ident": "T39",
"Latitude": 29.816406762108624,
"Longitude": -82.399761547028433,
"y_proj": 3458730.4422884127,
"x_proj": -9172699.496902369,
"comment": "",
"new_trk": null,
"new_seg": null,
"display": "",
"color": "",
"altitude": 0.0,
"depth": 0.0,
"temp": 0.0,
"time": "",
"model": "",
"filename": "",
"ltime": "",
"desc": "",
"link": ""
},
"geometry": {
"type": "MultiLineString",
"coordinates": [
[
[-9172699.496902368962765, 3458730.442288412712514],
[-9172693.455310769379139, 3458732.608350321650505]
]
]
}
},
{
"type": "Feature",
"properties": {
"type": "Improved",
"tident": "loop trail",
"ident": "T41",
"Latitude": 29.816423730022812,
"Longitude": -82.399707274487682,
"y_proj": 3458732.6083503217,
"x_proj": -9172693.4553107694,
"comment": "",
"new_trk": null,
"new_seg": null,
"display": "",
"color": "",
"altitude": 0.0,
"depth": 0.0,
"temp": 0.0,
"time": "",
"model": "",
"filename": "",
"ltime": "",
"desc": "",
"link": ""
},
"geometry": {
"type": "MultiLineString",
"coordinates": [
[
[-9172693.455310769379139, 3458732.608350321650505],
[-9172689.781723748892546, 3458733.925421689171344],
[-9172673.330925136804581, 3458749.428440262097865],
[-9172668.105586804449558, 3458751.844263973645866],
[-9172659.90467806905508, 3458768.217052695341408],
[-9172658.355110624805093, 3458777.654920535627753],
[-9172655.481954025104642, 3458786.909622615668923],
[-9172655.052261060103774, 3458798.398271592799574],
[-9172648.380882352590561, 3458808.329319821670651],
[-9172633.862591058015823, 3458835.805302318651229],
[-9172626.714764591306448, 3458839.30481178779155],
[-9172616.376520942896605, 3458853.466818191576749]
]
]
}
}
]
}
I want to add the following segment in to the above JSON file (in to ["features",0,"properties"]
"properties": {
"name": "Some Other Trailname",
"filters": [
"Cycling",
"dogAccess",
"kidFriendly"
],
"price": "Free",
"hours": {
"mon": "Sunrise - Sundown",
"tue": "Sunrise - Sundown",
"wed": "Sunrise - Sundown",
"thu": "Sunrise - Sundown",
"fri": "Sunrise - Sundown",
"sat": "Sunrise - Sundown",
"sun": "Sunrise - Sundown"
}
},
How do I do that using JSONPath.sh ?
Results do not match other implementations
The following queries provide results that do not match those of other implementations of JSONPath
(compare https://cburgmer.github.io/json-path-comparison/):
-
$[1:10]
Input:["first", "second", "third"]Expected output:
["second", "third"]Error:
sed: bad regex '\[(([0-9]+|"[^"]+")[],]){9999}(.*)': Invalid contents of {} No JSON object could be decoded -
$[-1:]
Input:["first", "second", "third"]Expected output:
["third"]Actual output:
["first", "second", "third"] -
$[1:]
Input:["first", "second", "third", "forth", "fifth"]Expected output:
["second", "third", "forth", "fifth"]Error:
sed: bad regex '\[(([0-9]+|"[^"]+")[],]){9999}(.*)': Invalid contents of {} No JSON object could be decoded -
$[0:3:2]
Input:["first", "second", "third", "forth", "fifth"]Expected output:
["first", "third"]Actual output:
["first", "second", "third", "forth", "fifth"] -
$[0:3:1]
Input:["first", "second", "third", "forth", "fifth"]Expected output:
["first", "second", "third"]Actual output:
["first", "second", "third", "forth", "fifth"] -
$['key']
Input:{"key": "value"}Expected output:
["value"]Actual output:
{"key": "value"} -
$['key','another']
Input:{"key": "value", "another": "entry"}Expected output:
["value", "entry"]Actual output:
{"another": "entry", "key": "value"} -
$['0']
Input:{"0": "value"}Expected output:
["value"]Error:
sed: bad regex '\[(([0-9]+|"[^"]+")[],]){9999}(.*)': Invalid contents of {} No JSON object could be decoded -
$['special:"chars']
Input:{"special:\"chars": "value"}Expected output:
["value"]Error:
sed: bad regex '\[(([0-9]+|"[^"]+")[],]){9999}(.*)': Invalid contents of {} No JSON object could be decoded -
$['*']
Input:{"*": "value"}Expected output:
["value"]Actual output:
{"*": "value"} -
$.key
Input:{"key": "value"}Expected output:
["value"]Actual output:
{"key": "value"} -
$.key
Input:{"key": ["first", "second"]}Expected output:
[["first", "second"]]Actual output:
["first", "second"] -
$.key
Input:{"key": null}Expected output:
[null]Actual output:
{"key": null} -
$..key
Input:{"object": {"key": "value", "array": [{"key": "something"}, {"key": {"key": "russian dolls"}}]}, "key": "top"}Expected output:
["top", "value", "something", {"key": "russian dolls"}, "russian dolls"]Actual output:
{"key": "top", "object": {"array": [{"key": "something"}, {"key": {"key": "russian dolls"}}], "key": "value"}} -
$.store..price
Input:{"store": {"book": [{"category": "reference", "author": "Nigel Rees", "title": "Sayings of the Century", "price": 8.95}, {"category": "fiction", "author": "Evelyn Waugh", "title": "Sword of Honour", "price": 12.99}, {"category": "fiction", "author": "Herman Melville", "title": "Moby Dick", "isbn": "0-553-21311-3", "price": 8.99}, {"category": "fiction", "author": "J. R. R. Tolkien", "title": "The Lord of the Rings", "isbn": "0-395-19395-8", "price": 22.99}], "bicycle": {"color": "red", "price": 19.95}}}Expected output:
[8.95, 12.99, 8.99, 22.99, 19.95]Actual output:
{"bicycle": {"price": 19.95}, "book": [{"price": 8.95}, {"price": 12.99}, {"price": 8.99}, {"price": 22.99}]} -
$
Input:{"key": "value", "another key": {"complex": ["a", 1]}}Expected output:
[{"another key": {"complex": ["a", 1]}, "key": "value"}]Error:
sed: bad regex '\[(([0-9]+|"[^"]+")[],]){9999}(.*)': Invalid contents of {} No JSON object could be decoded -
$[*]
Input:["string", 42, {"key": "value"}, [0, 1]]Expected output:
["string", 42, {"key": "value"}, [0, 1]]Error:
Extra data: line 4 column 6 - line 12 column 1 (char 28 - 94) -
$[*]
Input:{"some": "string", "int": 42, "object": {"key": "value"}, "array": [0, 1]}Expected output:
["string", 42, {"key": "value"}, [0, 1]]Actual output:
{"array": [0, 1], "int": 42, "object": {"key": "value"}, "some": "string"} -
$.*
Input:["string", 42, {"key": "value"}, [0, 1]]Expected output:
["string", 42, {"key": "value"}, [0, 1]]Error:
Extra data: line 4 column 6 - line 12 column 1 (char 28 - 94) -
$.*
Input:{"some": "string", "int": 42, "object": {"key": "value"}, "array": [0, 1]}Expected output:
["string", 42, {"key": "value"}, [0, 1]]Actual output:
{"array": [0, 1], "int": 42, "object": {"key": "value"}, "some": "string"}
For reference, the output was generated by the program in https://github.com/cburgmer/json-path-comparison/tree/Bash_JSONPath.sh/implementations/Bash_JSONPath.sh.
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.