ebdrup / json-schema-benchmark Goto Github PK

To have request-validator run the actual validation, you need to call validate(value) on the object returned after creating the validator with a particular schema.

Link to source

This line should really be instance.validate(json).

Your criteria for including a validator into the performance tests shouldn't really take optional tests into account. All the tested validators are run either on node or in the browser. The JavaScript runtimes there cannot handle JSON parsing in a way that is compatible with the bignum and zeroTerminatedFloats test cases. As for the format tests, the JSON-schema spec clearly states that conforming validators MAY choose to support the format keyword, but are not required to.

I believe this one should be benchmarked as well.

schema-inspector

Very informative benchmark, however it will be helpful if comparison of lib sizes are taken into account.

I just realized there were updates to the test suite and released v1.2.4 of skeemas which fixes about 11 of the test failures currently reported.

I'm happy to submit PRs to help update benchmark results but I understand the benchmark owners would prefer to run them independently. What is the correct mechanism for library owners to request an update of these results?

Hi @ebdrup @epoberezkin
I've just released the new version with a bunch tests fixes. Could you please regenerate the overview!

korzio/djv#35 (comment)
https://gitter.im/korzio/djv?at=5970b79889aea4761db944ee

Thanks!

It would be interesting to see a comparative with Joi.

Hi, thanks very much for putting this benchmark together.

I found this repo very helpful in evaluating which JSON schema validator to use in my latest project. However, I found the benchmarks for @exodus/schemasafe misleading.

schemasafe disables error reporting by default, which seems like significantly different (and unexpected) behavior compared to the other validators in this benchmark.

I did see this note in CONTRIBUTING.md:

Also, do not tweak any of the settings or options of a validator. Each validator should be run in it's standard setting. This is to make sure that nothing breaks if the benchmarks changes. But also to reflect the kind of performance a user of the validator would get, if using it without tweaking.

However, in my opinion, to make an "apples to apples" comparison between schemasafe and the other validators, we should apply the includeErrors option to schemasafe.

zod is a json-schema validator designed for TypeScript that is quickly gaining popularity, recently crossing 3m downloads/wk.

I'm curious how it compares to ajv and others

Could it be added to the benchmark?

I notice contributing docs; it looks like all you have to do is add it to this file? https://github.com/ebdrup/json-schema-benchmark/blob/master/validators.js

If so, someone on my team can probably submit a PR.

git submodule update --init && npm install && coffee benchmarks/

this doc shows JSCK failing some official JSON Schema tests.

this code explicitly articulates that we don't support those features, as do the JSCK README and this comment.

it would be nice to put this info in the doc, since "failing a test" doesn't usually imply "deliberately elected not to support a feature."

Some validators are not supporting all drafts.
It would be interesting to see rankings per draft-version and also which validator is supporing which draft-version.

Vine.js claims to be one of the fastest, would be interesting if you could add it.

Hi ebdrup.

I am wondering why the latest report for json-schema-library failed. The report shows version 4.0.0 being used on node version 9. Cloning this repository, updating the submodule and switching to node 9 does pass all tests for json-schema-library.

Do you have any idea why this failed in your run?

Thank you for your help!
sagold

In "Number of test runs per second", does "test runs" mean running the whole JSON schema test suite (about 200+ tests), or just one test in JSON schema test suite?

Thank you.

discussion started in JSCK repo (pandastrike/jsck#78) but belongs here.

JSCK passes all the JSON Schema Test Suite tests, except for a few features we've elected not to support. but with more insight into these tests, we might want to support them also.

Also some of the numbers are smaller now it seems. These are the numbers I am getting: https://github.com/epoberezkin/json-schema-consolidate#validators-compatibility

Moved here from #5
It might be nice to differentiate between required test failures and the optional feature tests as well.

Honestly, the optional/zeroTerminatedFloats tests are not relevant to javascript implementations of json-schema at all, they are intended for languages that differentiate between 1.0 and 1. Given the wording of the spec you could even argue that the tests are not valid in any implementation as 1.0 is a "mathematical integer".

...for interoperability reasons, JSON values used in the context of JSON Schema, whether that JSON be a JSON Schema or an instance, SHOULD ensure that mathematical integers be represented as integers as defined by this specification.

The current benchmark is run on a subset of the tests in the official JSON-schema test suite.
Although the results seem very much in line with what other stable benchmarks produce (like themis), and exercising as much of the standard as possible is good to catch any slow corners in a validator. It might be a good idea to do a benchmark on data that is more like what you see in production.

Perhaps an performance benchmark on one simple and one advanced schema.

To keep things simple for anyone wanting to chose a validator, the benchmark should still produce one single number for each validator like it does now. Nitty gritty details about performance are for schema validator authors to do in their efforts.

The whole test suite fails (throws exception) if I just move to the latest commit