Comments (6)
I attach the problematic schema here too, for convenience.
schema.json
from json-schema-validator.
Maybe it'll be of help, bisect shows this commit as the one after which the code hangs: 7f1ec11 (#931).
from json-schema-validator.
Unfortunately this typically means that there's not enough memory to load your schema and you would need to allocate more heap for your JVM.
Previously the default was to lazily load the schema. Now the default is to preload the schema.
You can still configure it to lazily load the schema with the setPreloadJsonSchema
option.
@Test
void testNoErrorsForEmptyObject() throws IOException {
SchemaValidatorsConfig config = new SchemaValidatorsConfig();
config.setPreloadJsonSchema(false);
getJsonSchemaFromClasspath(resource("schema.json"), SpecVersion.VersionFlag.V7, config);
}
However, you should note that if you have inputs that exercise all the possible evaluation paths of your schema as defined, then it's just going to OOM later when if validates such an input.
from json-schema-validator.
That seems... improbable, @justin-tay ? I mean - this schema is not that large (or complex). Something seems wrong if it needs in excess of gigabytes of memory to load validators for a mere 300kb?
from json-schema-validator.
The state of the validators / schemas are different along the evaluation path. Therefore the code will need to create validator / schema instances at each evaluation path to store this state. This is cached since throwing it away impacts performance as this state is fixed for a particular schema. Throwing it away means that the same state needs to be regenerated between each run and also increases the objects that need to be garbage collected.
The following is a small sample of the evaluation paths that get loaded for your schema.
/properties/components/additionalProperties/$ref/oneOf/5/$ref/properties/query/allOf/0/$ref/oneOf/8/$ref/properties/documents/allOf/0/$ref/oneOf/7/$ref/properties/clusters/allOf/0/$ref/oneOf/2/$ref/properties/matrixRows/allOf/0/$ref/oneOf/3/$ref/properties/matrix/allOf/0/$ref/oneOf/6/$ref/properties/vectors/allOf/0/$ref/oneOf/1/$ref/properties/labels/allOf/0/$ref/oneOf/1/$ref/properties/labelFilter/allOf/0/$ref/oneOf/7/$ref/properties/exclude/items/$ref/oneOf/2/$ref/properties/entries/oneOf/1/$ref
/properties/components/additionalProperties/$ref/oneOf/5/$ref/properties/query/allOf/0/$ref/oneOf/8/$ref/properties/documents/allOf/0/$ref/oneOf/7/$ref/properties/clusters/allOf/0/$ref/oneOf/2/$ref/properties/matrixRows/allOf/0/$ref/oneOf/3/$ref/properties/matrix/allOf/0/$ref/oneOf/6/$ref/properties/vectors/allOf/0/$ref/oneOf/1/$ref/properties/labels/allOf/0/$ref/oneOf/1/$ref/properties/labelFilter/allOf/0/$ref/oneOf/7/$ref/properties/exclude/items/$ref/oneOf/2/$ref/properties/comment/$ref
/properties/components/additionalProperties/$ref/oneOf/5/$ref/properties/query/allOf/0/$ref/oneOf/8/$ref/properties/documents/allOf/0/$ref/oneOf/7/$ref/properties/clusters/allOf/0/$ref/oneOf/2/$ref/properties/matrixRows/allOf/0/$ref/oneOf/3/$ref/properties/matrix/allOf/0/$ref/oneOf/6/$ref/properties/vectors/allOf/0/$ref/oneOf/1/$ref/properties/labels/allOf/0/$ref/oneOf/1/$ref/properties/labelFilter/allOf/0/$ref/oneOf/7/$ref/properties/exclude/items/$ref/oneOf/3/$ref/properties/dictionary/oneOf/1/$ref
/properties/components/additionalProperties/$ref/oneOf/5/$ref/properties/query/allOf/0/$ref/oneOf/8/$ref/properties/documents/allOf/0/$ref/oneOf/7/$ref/properties/clusters/allOf/0/$ref/oneOf/2/$ref/properties/matrixRows/allOf/0/$ref/oneOf/3/$ref/properties/matrix/allOf/0/$ref/oneOf/6/$ref/properties/vectors/allOf/0/$ref/oneOf/1/$ref/properties/labels/allOf/0/$ref/oneOf/1/$ref/properties/labelFilter/allOf/0/$ref/oneOf/7/$ref/properties/exclude/items/$ref/oneOf/3/$ref/properties/comment/$ref
/properties/components/additionalProperties/$ref/oneOf/5/$ref/properties/query/allOf/0/$ref/oneOf/8/$ref/properties/documents/allOf/0/$ref/oneOf/7/$ref/properties/clusters/allOf/0/$ref/oneOf/2/$ref/properties/matrixRows/allOf/0/$ref/oneOf/3/$ref/properties/matrix/allOf/0/$ref/oneOf/6/$ref/properties/vectors/allOf/0/$ref/oneOf/1/$ref/properties/labels/allOf/0/$ref/oneOf/1/$ref/properties/labelFilter/allOf/0/$ref/oneOf/7/$ref/properties/exclude/items/$ref/oneOf/4/$ref/properties/comment/$ref
/properties/components/additionalProperties/$ref/oneOf/5/$ref/properties/query/allOf/0/$ref/oneOf/8/$ref/properties/documents/allOf/0/$ref/oneOf/7/$ref/properties/clusters/allOf/0/$ref/oneOf/2/$ref/properties/matrixRows/allOf/0/$ref/oneOf/3/$ref/properties/matrix/allOf/0/$ref/oneOf/6/$ref/properties/vectors/allOf/0/$ref/oneOf/1/$ref/properties/labels/allOf/0/$ref/oneOf/1/$ref/properties/labelFilter/allOf/0/$ref/oneOf/7/$ref/properties/exclude/items/$ref/oneOf/5/$ref/properties/comment/$ref
/properties/components/additionalProperties/$ref/oneOf/5/$ref/properties/query/allOf/0/$ref/oneOf/8/$ref/properties/documents/allOf/0/$ref/oneOf/7/$ref/properties/clusters/allOf/0/$ref/oneOf/2/$ref/properties/matrixRows/allOf/0/$ref/oneOf/3/$ref/properties/matrix/allOf/0/$ref/oneOf/6/$ref/properties/vectors/allOf/0/$ref/oneOf/1/$ref/properties/labels/allOf/0/$ref/oneOf/3/$ref/properties/labelAggregator/allOf/0/$ref/oneOf/0/$ref/properties/labelCollector/allOf/0/$ref/oneOf/1/$ref/properties/labelFilter/allOf/0/$ref
/properties/components/additionalProperties/$ref/oneOf/5/$ref/properties/query/allOf/0/$ref/oneOf/8/$ref/properties/documents/allOf/0/$ref/oneOf/7/$ref/properties/clusters/allOf/0/$ref/oneOf/2/$ref/properties/matrixRows/allOf/0/$ref/oneOf/3/$ref/properties/matrix/allOf/0/$ref/oneOf/6/$ref/properties/vectors/allOf/0/$ref/oneOf/1/$ref/properties/labels/allOf/0/$ref/oneOf/3/$ref/properties/labelAggregator/allOf/0/$ref/oneOf/0/$ref/properties/labelCollector/allOf/0/$ref/oneOf/1/$ref/properties/labelFilter/allOf/0/$ref/oneOf/0/$ref
/properties/components/additionalProperties/$ref/oneOf/5/$ref/properties/query/allOf/0/$ref/oneOf/8/$ref/properties/documents/allOf/0/$ref/oneOf/7/$ref/properties/clusters/allOf/0/$ref/oneOf/2/$ref/properties/matrixRows/allOf/0/$ref/oneOf/3/$ref/properties/matrix/allOf/0/$ref/oneOf/6/$ref/properties/vectors/allOf/0/$ref/oneOf/1/$ref/properties/labels/allOf/0/$ref/oneOf/3/$ref/properties/labelAggregator/allOf/0/$ref/oneOf/0/$ref/properties/labelCollector/allOf/0/$ref/oneOf/1/$ref/properties/labelFilter/allOf/0/$ref/oneOf/1/$ref
/properties/components/additionalProperties/$ref/oneOf/5/$ref/properties/query/allOf/0/$ref/oneOf/8/$ref/properties/documents/allOf/0/$ref/oneOf/7/$ref/properties/clusters/allOf/0/$ref/oneOf/2/$ref/properties/matrixRows/allOf/0/$ref/oneOf/3/$ref/properties/matrix/allOf/0/$ref/oneOf/6/$ref/properties/vectors/allOf/0/$ref/oneOf/1/$ref/properties/labels/allOf/0/$ref/oneOf/3/$ref/properties/labelAggregator/allOf/0/$ref/oneOf/0/$ref/properties/labelCollector/allOf/0/$ref/oneOf/1/$ref/properties/labelFilter/allOf/0/$ref/oneOf/1/$ref/properties/operator/oneOf/1/$ref
/properties/stages/additionalProperties/$ref/oneOf/5/$ref/properties/clusters/allOf/0/$ref/oneOf/1/$ref/properties/matrix/allOf/0/$ref/oneOf/2/$ref/properties/matrixRows/allOf/0/$ref/oneOf/2/$ref/properties/vectors/properties/columns/allOf/0/$ref/oneOf/1/$ref/properties/labels/allOf/0/$ref/oneOf/1/$ref/properties/labelFilter/allOf/0/$ref/oneOf/7/$ref/properties/exclude/items/$ref/oneOf/5/$ref/properties/query/allOf/0/$ref/oneOf/8/$ref/properties/documents/allOf/0/$ref/oneOf/11/$ref/properties/documentPairs/allOf/0/$ref/oneOf/1/$ref/properties/validation/allOf/0/$ref/properties/pairwiseSimilarity/allOf/0/$ref/oneOf/1/$ref/properties/features/allOf/0/$ref/oneOf/9/$ref/properties/minFeatureCount/oneOf/1/$ref
/properties/stages/additionalProperties/$ref/oneOf/5/$ref/properties/clusters/allOf/0/$ref/oneOf/1/$ref/properties/matrix/allOf/0/$ref/oneOf/2/$ref/properties/matrixRows/allOf/0/$ref/oneOf/2/$ref/properties/vectors/properties/columns/allOf/0/$ref/oneOf/1/$ref/properties/labels/allOf/0/$ref/oneOf/1/$ref/properties/labelFilter/allOf/0/$ref/oneOf/7/$ref/properties/exclude/items/$ref/oneOf/5/$ref/properties/query/allOf/0/$ref/oneOf/8/$ref/properties/documents/allOf/0/$ref/oneOf/11/$ref/properties/documentPairs/allOf/0/$ref/oneOf/1/$ref/properties/validation/allOf/0/$ref/properties/pairwiseSimilarity/allOf/0/$ref/oneOf/1/$ref/properties/features/allOf/0/$ref/oneOf/9/$ref/properties/maxFeatureCount/oneOf/0/allOf/0/$ref
/properties/stages/additionalProperties/$ref/oneOf/5/$ref/properties/clusters/allOf/0/$ref/oneOf/1/$ref/properties/matrix/allOf/0/$ref/oneOf/2/$ref/properties/matrixRows/allOf/0/$ref/oneOf/2/$ref/properties/vectors/properties/columns/allOf/0/$ref/oneOf/1/$ref/properties/labels/allOf/0/$ref/oneOf/1/$ref/properties/labelFilter/allOf/0/$ref/oneOf/7/$ref/properties/exclude/items/$ref/oneOf/5/$ref/properties/query/allOf/0/$ref/oneOf/8/$ref/properties/documents/allOf/0/$ref/oneOf/11/$ref/properties/documentPairs/allOf/0/$ref/oneOf/1/$ref/properties/validation/allOf/0/$ref/properties/pairwiseSimilarity/allOf/0/$ref/oneOf/1/$ref/properties/features/allOf/0/$ref/oneOf/9/$ref/properties/maxFeatureCount/oneOf/1/$ref
/properties/stages/additionalProperties/$ref/oneOf/5/$ref/properties/clusters/allOf/0/$ref/oneOf/1/$ref/properties/matrix/allOf/0/$ref/oneOf/2/$ref/properties/matrixRows/allOf/0/$ref/oneOf/2/$ref/properties/vectors/properties/columns/allOf/0/$ref/oneOf/1/$ref/properties/labels/allOf/0/$ref/oneOf/1/$ref/properties/labelFilter/allOf/0/$ref/oneOf/7/$ref/properties/exclude/items/$ref/oneOf/5/$ref/properties/query/allOf/0/$ref/oneOf/8/$ref/properties/documents/allOf/0/$ref/oneOf/11/$ref/properties/documentPairs/allOf/0/$ref/oneOf/1/$ref/properties/validation/allOf/0/$ref/properties/pairwiseSimilarity/allOf/0/$ref/oneOf/1/$ref/properties/features/allOf/0/$ref/oneOf/9/$ref/properties/comment/$ref
Looking at this it looks like even if you don't preload, you could still potentially get an OOM during execution if you have inputs that exercise the evaluation paths and if you don't want to allocate memory for all the evaluation paths as there is currently no option not to cache the loaded schema during the run itself.
This probably needs more memory than the consolidated FHIR schema, which like a 4 MB schema so it's not really the size in bytes of the schema itself.
from json-schema-validator.
Right, I see it now. It's the fanout on evaluation of refs then - you can easily make this exponential and crash schema parsing even with a very small input. It's basically what's happening with the example I attached. Thank you for looking for a workaround - I don't think multiple refs are going to be that uncommon in the wild, so it's probably good to have an option to turn off caching.
from json-schema-validator.
Related Issues (20)
- "http://" should be considered as a valid iri-reference HOT 1
- Downgrade maven-source-plugin to 3.2.1
- Resolve javadoc warning messages
- `JsonSchema.getSubSchema()` fails with JsonPointer containing a number HOT 2
- Possible performance improvement: JsonNodePath.hashCode() HOT 3
- How to generate the json structure acording to the schema? HOT 2
- oneOf with discriminator does not verify json correctly HOT 4
- The location of the `module-info.class` in the jar file is wrong HOT 1
- anyOf and OneOf unhelpful error messages HOT 2
- Exponential algorithm in BaseJsonValidator.hasAdjacentKeywordInEvaluationPath HOT 13
- Validation error for "id" field in referenced schema HOT 1
- Missing error text details for values of enumeration when enum type is JSON object HOT 3
- is date-time supported in V201909 and latest version of jar HOT 1
- `oneOf` for empty array. HOT 1
- SLF4J 2 dependency, despite being fulfilled by older version HOT 9
- Unevaluated properties and subschemas HOT 4
- Unable to validate all elements of the array using 2020-12 Json schema HOT 4
- Schema validate always print "Failed to apply pattern" when validate fail HOT 1
- Invalid Schema Structure Not Being Enforced HOT 4
- Undertow version containing CVEs HOT 3
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from json-schema-validator.