Incorporate tests from "Parsing JSON as a Minefield"

[vielmetti]

The article http://seriot.ch/parsing_json.html from @nst has an exhaustive comparison of various JSON parsers, which does not include jq. It also has a whole set of test cases, which are available at https://github.com/nst/JSONTestSuite

At least one of the test cases that's supposed to fail, at "n_structure_object_with_trailing_garbage.json", is parsed by jq as correct.

The PR that would close this is a script that would run each of these tests in turn and validate that they produce the expected results.

[pkoppstein]

@vielmetti - Thanks for providing the information about the JSONTestSuite.

Please note that one of jq's great strengths is that it can read a stream of JSON texts (i.e. JSON entities), not just one. Thus, the test case you mention (n_structure_object_with_trailing_garbage.json) is, from the point of view of any stream-oriented processor that does not require any boundary between valid "texts", misnamed:

• in the .json file, the text after the JSON object is a JSON string, and thus not "garbage";
• the "n_" prefix in this case is inapplicable -- jq should and does parse the file as two JSON entities.

One other very important point is that the JSON specification (https://tools.ietf.org/html/rfc7159) states:

"A JSON parser MAY accept non-JSON forms or extensions."

Thus although 012 is invalid JSON, jq will parse it in the sensible way, and also emits valid JSON (12). This is a strength of jq that is entirely compatible with the JSON specification.

With these points in mind, using the "master" version of jq (jq-1.5rc2-150-g1740fd0) it can be said:

1. In all 317 test cases, jq either parses the input or issues a parsing error message.
2. jq passes all the y_ tests.
3. jq passes all the n_ tests except (perhaps) for the 8 identified below.

n_number_then_00.json (the result: 1, not followed by an error)
n_string_UTF8_surrogate_U+D800.json
n_string_invalid_utf-8.json
n_string_iso_latin_1.json
n_string_lone_utf8_continuation_byte.json
n_string_overlong_sequence_2_bytes.json
n_string_overlong_sequence_6_bytes.json
n_string_overlong_sequence_6_bytes_null.json

The first of these exceptional cases involves NUL after a valid JSON text. In this particular case, jq ignores the NUL, so one might say that this is a permitted extension, but jq does NOT simply treat NUL as "whitespace", so the exact nature of the "extension" might be a bit tricky to characterize.

The other putative failures involve encoding issues, and it may be difficult to justify all of them as permissible extensions, but it is reassuring to know that in all cases, the jq output is valid JSON.

Footnote regarding jq 1.5

There is one test case for which jq 1.5 deviates from the master version:
$ jq1.5 . n_structure_open_array_object.json
$
That is, jq 1.5 does not produce an error message.

[nicowilliams]

We should do this, yes.

[wtlangford]

I ran this against the jq-1.5 release (as installed by brew) and got some very exciting results.

For the implementation defined (i_) tests, dark blue represents a parsing failure, and light blue a parsing success.
The rest of the tests shown are yellow because failure was expected (n_), but parsing was successful.
The output doesn't show tests that went as expected.

The only tests we "fail" are ones related to invalid UTF-8 sequences, which isn't actually a problem, because they're test cases that are "implementation defined", which means we have the option to pass or fail these JSON texts.

We're particularly permissive re: number parsing. A lot of tests that were expected to fail actually succeed, which is fine, since any JSON parser is allowed to support non-strict-JSON extensions.

Overall, I like that this didn't find parsing issues. I would still like to see this test suite integrated somehow, though. It would make for a good regression tester.

[wtlangford]

And now I see that @pkoppstein had also run it.
This might just be a change in the tests, but I don't see the test failure for n_structure_open_array_object.json on mine.

[pkoppstein]

@wtlangford - jq1.5 and jq master still give different results for n_structure_open_array_object.json

For the record:

~/github/JSONTestSuite/test_parsing$ git log -1
commit 299189e5a233a00531550211a4bd6d185b2b36a7
Merge: 9f93b50 9d3da69
Author: Nicolas Seriot <nicolas@seriot.ch>
Date:   Thu Oct 27 00:39:30 2016 +0200

    Merge pull request #14 from krono/master
    
    Add Squeak w/ JSON-tonyg

~/github/JSONTestSuite/test_parsing$ jq --version
jq-1.5rc2-174-g597c1f6

~/github/JSONTestSuite/test_parsing$ jq . n_structure_open_array_object.json
parse error: Exceeds depth limit for parsing at line 1, column 427

~/github/JSONTestSuite/test_parsing$ git pull
Updating 299189e..bf70853

~/github/JSONTestSuite/test_parsing$ jq . n_structure_open_array_object.json
parse error: Exceeds depth limit for parsing at line 1, column 427

~/github/JSONTestSuite/test_parsing$ jq1.5 --version
jq-1.5

~/github/JSONTestSuite/test_parsing$ jq1.5 . n_structure_open_array_object.json
~/github/JSONTestSuite/test_parsing$ 

[wtlangford]

Oops! My brew-installed jq was `jq-1.5rc2-150-g1740fd0`. Those results I posted apply to `master`, not 1.5.

…

On Thu, Feb 9, 2017 at 5:42 AM pkoppstein ***@***.***> wrote: @wtlangford <https://github.com/wtlangford> - jq1.5 and jq master still give different results for n_structure_open_array_object.json For the record: ~/github/JSONTestSuite/test_parsing$ git log -1 commit 299189e5a233a00531550211a4bd6d185b2b36a7 Merge: 9f93b50 9d3da69 Author: Nicolas Seriot ***@***.***> Date: Thu Oct 27 00:39:30 2016 +0200 Merge pull request #14 from krono/master Add Squeak w/ JSON-tonyg ~/github/JSONTestSuite/test_parsing$ jq --version jq-1.5rc2-174-g597c1f6 ~/github/JSONTestSuite/test_parsing$ jq . n_structure_open_array_object.json parse error: Exceeds depth limit for parsing at line 1, column 427 ~/github/JSONTestSuite/test_parsing$ git pull Updating 299189e..bf70853 ~/github/JSONTestSuite/test_parsing$ jq . n_structure_open_array_object.json parse error: Exceeds depth limit for parsing at line 1, column 427 ~/github/JSONTestSuite/test_parsing$ jq1.5 --version jq-1.5 ~/github/JSONTestSuite/test_parsing$ jq1.5 . n_structure_open_array_object.json ~/github/JSONTestSuite/test_parsing$ — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1264 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/ADQ4V00yk2wgeJ5DWax4O3bhARFXFzdvks5rau2rgaJpZM4Khjul> .