Do the benchmark on a more realistic schema about json-schema-benchmark HOT 4 CLOSED

I was thinking of machine-generating a bigger schema from the schemas in the official tests, to make sure that the test touches as many parts of the spec as possible. Just to make sure to hit any troublesome parts of a validator.

from json-schema-benchmark.

ok, so the bad news is I don't have as much time as I'd like, but the JSCK benchmarks use five schemas, and you're more than welcome to use them.

here's a categorization of those 5 schemas:

• draft 3, trivial
• draft 3, medium
• draft 4, trivial
• draft 4, medium
• draft 4, complex

so you can use them to benchmark a range of complexities and draft support.

from json-schema-benchmark.

Realistic benchmark idea: Validating the draft 4 schema with the draft 4 schema.

from json-schema-benchmark.

@ebdrup The more I think about it, the more I believe that using test-suite for performance benchmarking is completely wrong.

With the exception of some really old and slow libraries, where slow validation may affect UX in the browser, validation performance only really matters for server-side validation. This is where the performance advantage achieved by compiling validators is becoming important... Server-side validation passes 99+% of the time (excluding the cases of DOS attacks and or broken API consumer - but performance is not going to help too much in these cases).

So I think to be relevant, benchmark should only test passing cases on some data samples of small/average/large size.

Tests can still be used to benchmark compliance, separately from performance.

What do you think?

from json-schema-benchmark.