$data about json-schema-spec HOT 39 CLOSED

Just a note from perspective of me as an AJV user. I've been using AJV for 3 years and only today I discovered $data keyword in AJV; and that it can also reference array lengths. It felt like epiphany. Before, JSON Schema was just a way for some very basic and dumb validation, that still required some supporting scripting for more complex validation scenarios. With $data, it suddenly became so much more "intelligent" and useful.

So as a user of JSON Schema I'm totally up for this (or an at least equally powerful alternative) being considered as a part or at least an extension of some sort.

from json-schema-spec.

My 2¢: people seem to use it a lot with Ajv, judging by the questions. So it must be useful.

I think relative JSON pointer should be extended to allow navigating array items (see #115)

from json-schema-spec.

@askirmas $ref connects schemas to schemas. Meaning you need to know what other schema you are loading at the time that you write the schema containing the $ref. $data loads information from the instance into the schema. So it is (potentially) different for every instance and cannot be determined before the instance is provided. These are fundamentally different behaviors.

from json-schema-spec.

@exist3nz this issue is still open- several linked issues are closed, which can look confusing the way GitHub displays it, but this is still very much open.

It's in the "draft-future" milestone which means that we do intend to consider it, but not in the next two drafts (draft-07, which will likely go out next week, or draft-08 which will focus on issues around re-usability). At some point I will make a draft-09 milestone, and assuming we can resolve the target set of issues for draft-08 in that one draft, $data will likely be the focus for draft-09. Although that's not set in stone by any means.

This would be an expansion of the scope of JSON Schema, which is why we are trying to nail down some key things within the current scope first.

from json-schema-spec.

The sentiment is shared @laurisvan. However it's disagreement about how it should work that has kept it out of the spec. AJV's implementation is how that author thought it should be done.

I also have implemented my own idea (and put it in a vocabulary) for my library JsonSchema.Net.

While AJV may be in wider use, my implementation is arguably more in line with the spec's current state since it defines a vocabulary.

Until alignment on how it should work can be achieved, we can't put it in the spec. If you would like to open this conversation again, I would suggest opening a new issue that:

• summarizes the discussion here
• describes how known approaches (notably mine and AJV's) work along with their advantages and deficiencies

from json-schema-spec.

I'm solidly of the opinion that checking data consistency is solidly out of the scope of JSON Schema. Although it's certainly an option for validators that do want to offer the feature.

And if it's a popular feature then... maybe it's something we have to look into, perhaps as a separate document though.

from json-schema-spec.

Those languages would just be more reliant upon validating via schema than via the framework

I have to make sure in my schema that a user-inputted max is larger than a certain minimum;
Though I can gladly use $data with AJV package/framework, I want to see this feature in the JSON-Schema;

This enhances my code maintainability:
If I ever stop using AJV or move to another package/framework for whichever reason, I don't want to lose such an ability, or have to go over the burden of re-writing the code to check for data consistency.

from json-schema-spec.

See @awwright 's Scope of JSON Schema Validation document for why this is unlikely to be taken up.

from json-schema-spec.

See @awwright 's Scope of JSON Schema Validation document for why this is unlikely to be taken up.

I can't understand why the scope document is against this proposal.
in the scope it says: Validating data consistency may involve: Scanning the rest of the document for a referenced value
Isn't that all this proposal about?

from json-schema-spec.

Hey there @flq. So that's the vocab spec itself. There's a subfolder in that "Prebuilt Vocabularies" folder called "Examples". In there you'll find examples for instance and external data. I suspect you want instance data.

The link to that doc page is https://docs.json-everything.net/schema/examples/data-ref.

If you have any other issues you find with my site or libraries, please feel free to open an issue in my repo.

from json-schema-spec.

I feel that there are several use cases here, and it might be best to split them up.

URI Template resolution

For hypermedia interactions, where the instance data must be referenced in order for hyperlinking to work at all, the extended templating syntax from issue #52 covers the necessary use cases. It narrowly targets hyperlinking and does not involve instance data in any other aspect of JSON Schema, as it applies only in situation where we were already referencing instance data.

Link title

I can see how a link title might reference data in the same way as the URI Template. The template may include the id of a related thing, while the title may include the related thing's name. Either way, the URI and the title are both things that are presented back to the user, which should be affected by the instance data as they are describing a relation involving that data.

Therefore I would prefer to see the URI Template extended syntax (with "vars") be used here rather than a more generic approach that applies to more than just hypermedia values.

rel and mediaType

The use cases for mediaType and rel are not immediately obvious to me. The relation type should not, in my mind, change based on the instance data. Only the specific instance to which the relation points should change. A mediaType specified at runtime from instance data would not be of use in planning what a program can do with different representations. It's not clear to me why you wouldn't just list out the possibilities. If some media type links may only be present some of the time, there are other ways to express that using "oneOf" (or possibly "switch") to associate links with only certain variations of the content.

Interactions during validation

I feel like this should somehow express the constraints in terms of the relations among the fields rather than pulling in data that will produce the desired result. So somehow explicitly saying that "larger" should be strictly greater than "smaller" without loading the data into the schema before validation.

I'm not 100% sure what that would look like. In this case, possibly very much like the $data example, as either way you need to reference the related field. But I would be more comfortable with something that clearly states "this is describing relationships among data" rather than "this loads a value from instance data, and treats it as part of the schema, whatever that happens to mean."

I don't feel like I'm articulating this well, but I'm going to go ahead and post this comment in the interest of provoking discussion :-)

from json-schema-spec.

@handrews
So, for me, my use cases sit entirely in the interactions during validation category, so that's what I'm going to speak to.

I think I have a basic understanding of what you're trying to say - you'd prefer a solution where JSON Schema validators use $data (or some other similar feature) as a pointer whose value is checked during validation, instead of recompiling the schema before validation even begins, where it inserts the value of those pointers directly into the schema.

Would it be possible to modify the proposal to remove the portions about modifying the schema directly, and include language elsewhere in v6+ that validation shouldn't modify schemas for any reason? Then, $data would be functionally identical to the way it works now, but doesn't encourage "bad behavior" among the various JSON Schema validators.

from json-schema-spec.

@HotelDon this is what I mean by "I'm not articulating this well"

It's not so much the reading/loading of the data (which likely has to be done lazily because of $refs), it's what you can do with it after it is loaded. Although thinking about this more I may be OK with it.

The way the proposal is written, with a list of allowed properties that trails off with "more..." left me very concerned about the scope. However what's not explicitly called out, but I think would be better than listing the fields, is that all of the fields proposed for $data take a literal value, and not a schema. We really need to make sure that data is never interpreted as a schema- that's a security nightmare- just use it to shove in links to all of your favorite malware sites!

But I think the intent here is that $data is only used to load data in place of a literal value. I can get behind that.

I still think it is valuable to separate the hypermedia template resolution cases out and use vars as specified in issue #52 for that. Since the values in vars are already assumed to be pointers into the instances, so requiring them to be little{"$data": "/pointer"} objects instead of just pointers is overkill.

from json-schema-spec.

I had never considered the possibility of someone trying to load schema's with $data, so I guess that is what got me confused.

So maybe fix it to sound more like this:

This keyword would be available:

• inside any schema
• contained in an object ({"$data": ...}) for most schema properties that accept literal values. For example:
  • minimum/maximum
  • minItems/maxItems
  • pattern
  • enum
  • etc...

I'm still having a hard time wrapping my head around the hypermedia/LDO portion of this proposal, so I don't have much of an opinion on it. It might be helpful if @geraintluff chimed in to defend his original proposal a bit, assuming he still has any interest in doing so.

from json-schema-spec.

I've become more receptive of this proposal while working with some of the more difficult hyper-schema problems such as discussed in #108

from json-schema-spec.

@awwright $data has important uses in hyper-schema whether it is available in general validation or not.

from json-schema-spec.

I can see this could be useful. A few clear usecases might be helpful if anyone has the time or inclination.

from json-schema-spec.

@Relequestual I'd like to see whether PR #179 is accepted or not before digging into use cases here. If it is accepted, that will clarify how to present the future use cases. If it is not, I'll need to come up with a different approach anyway.

from json-schema-spec.

I'm moving this out of draft-07/wright-*-02. It is a huge topic that has seen no progress and almost no real discussion in the past year. And there is no clear advocate with time available to move it forward.

from json-schema-spec.

Random thought: Would it make sense to define $data as part of a separate vocabulary for data interaction? (for lack of a better term)

If we went this route, it would also add to the use cases for #314 for understanding multiple vocabularies in use simultaneously.

from json-schema-spec.

Separate vocabulary seems overkill...

from json-schema-spec.

Hi everyone, I can see this issue is closed but it's the most relevant for what I'm looking for.
We use json schemas to defined configurations and some of our fields have a min and max field. Now, it's quite easy to see that in this situation we would want somehow to restrict via the schema that the max cannot be less than the min and it seemed this $data solution would have been perfect.

After researching around the topic I am very confused if this is still in the road map or not, or if there are alternative solutions to point to other values for validation. Please let me know what's the status as keeping the standard and not finding workaround hacks would be my first choice.

from json-schema-spec.

Please note that this remark comes from the point of view of a hardcore .Net developer.

I'm not sure about other frameworks/languages, but in the web side of .Net , we have data contracts. These are implemented via attributes (annotations) that we can place on properties of our DTO models. This mechanism has no way to reference other properties in the way that we're thinking of here. (To be honest I think it's rather limited in other ways, too.)

Given that it's built into the framework, though, it is very widely used, and it raises a concern that some languages would not be able to model this kind of feature.

Now, that's not to say that it shouldn't be done. Those languages would just be more reliant upon validating via schema than via the framework, which is counter to having these annotations included in the first place.

from json-schema-spec.

Bringing over some commentary from #541 and #549 since this is the oldest issue in this area of functionality:

I'm proposing that all of the possible $data-tagged features (including but not limited to loading instance data into the schema, loading external data into the schema, and asserting relationships among instance locations) be worked on as a new vocabulary. All of the proposals in this area add substantial complexity, and also do not need to change the existing core and validation specification concepts. With vocabulary support being added in draft-08, having one or more vocabularies for this area would allow it to develop independent from core and validation, which we hope are approaching a final draft. I expect this area would be pretty active with new ideas and feedback, which would delay finalization significantly if added to core or validation (and it's unrelated to hyper-schema, which has its own mechanisms for working with instance data in URI Templates).

from json-schema-spec.

@handrews Nitpicking: In your example

        "minimum": {"$data": "1/smaller"},
        "exclusiveMinimum": true

should be replaced by

        "exclusiveMinimum": {"$data": "1/smaller"},

lest people get confused.

from json-schema-spec.

@equaeghe that example was copied from the original proposal, and both when written and at the time I filed it here was correct as that is the draft-04 and earlier syntax.

from json-schema-spec.

Why does this and many similar 'features' couldn't be done with $ref?

from json-schema-spec.

@handrews

$ref connects schemas to schemas
$data loads information from the instance

For validator both are load some data on demand when pointer see this key. Difference is only in context

{
  "$schema": "http://json-schema.org/draft-07/schema",
  "$ref": "#/definitions/type-object",
  "properties": {
    "type": {
      "$ref": "#/definitions/type-object",
      "$data": "Deprecated? It is description layer",
      "default": {
        "$data": "#/definitions/type-object",
        "$ref": "Deprecated? It is data layer. The only thing - if I need exactly $ref property, not as an instruction to validator",
        "\\$data": "But same question for $data and any other",
        "$$ref": "Some escaping should be used in any case, but it is not $ref vs $data question" 
      }
    }
  },
  "definitions": {
    "type-object": {
      "type": "object"
    }
  }
}

from json-schema-spec.

The context is of huge importance. Most validators split validation into two parts, parsing the schema and validating a document with this parsed schema. With $data you are missing some values when parsing the schema as their content is part of the document. It's not extremely difficult to implement in a validator, as it indeed uses the same reference mechanics as $ref, but it does require a fundamental change in the way schemas are processed.

from json-schema-spec.

Most validators

Does another way exists? I expect all validators

The context is of huge importance

And it's obvious for validator. That's why I see bingo

from json-schema-spec.

Also I see huge question for $data and not so critical for $ref - merge strategy. For $ref it is replace as Object.assign({}, $ref, holder) but for data I'd prefer to use all the variety (and 1 option per project depend on it's specification).
Ajv propose $merge and $patch but my opinion same like for $data - it is just

entities should not be multiplied unnecessarily

{
  "$schema": "http://json-schema.org/draft-07/schema",
  "properties": {
    "$data": {
      "oneOf": [
        {"type": "string", "format": "uri-reference"},
        {
          "type": "object",
          "required": ["uri"],
          "properties": {
            "uri": {
              "type": "string", "format": "uri-reference"
            },
            "strategy": {
              "type": "string",
              "default": "replace",
              "enum": ["replace", "merge", "replace_recursive", "merge_recursive"]
            },
            "mergeRecursiveTypeConflicts": {
              "description": "For next step of processing $data will be instruction. Let's say like $ref/type. So merge ['boolean', 'integer'] with 'string' can produce ['boolean', 'integer', 'string'] that is OOP extending"
            }
          }
        }
      ]     
    }
  }
}

from json-schema-spec.

Can we close this out since the general consensus is against this?

from json-schema-spec.

@awwright yes, I think it's time. The most relevant current discussions are in #855 and #549.

from json-schema-spec.

@yww325 Find this paragraph:

Many applications simply wish to test that an ID is defined in another part of the same document. Even though this case would be compute-bound, it is still outside the scope of JSON Schema validation for several reasons...

from json-schema-spec.

While commenting for the already closed issues might not be. a good practice, I must also say that $data in ajv has been extremely beneficial for us. I don't know how I could otherwise have done cross-references between the values in validated data.

Leaving it out makes JSON Schema much less expressive to us, and we need to do more validation in application code that we could have left for schema.

from json-schema-spec.

For those in the future who will read @gregsdennis's last comment, here's the updated link to the vocabulary he mentioned.

from json-schema-spec.

🤦 It's changed again. I've updated the link above. Hopefully this one will stick.

from json-schema-spec.

Sorry @gregsdennis but the documentation doesn't bring me to the point of understanding how to use the "data" keyword in the context of Schema.NET.

I'm looking at https://docs.json-everything.net/schema/vocabs/data-2022/ and there's only an example for minValue and that data MUST be an object and may use a JSON pointer. I'm trying to express the (apparently often requested) constraint that an item within an array must be one of the names laid out in a different array in the instance.

from json-schema-spec.

Thank you for the response - I've joined the slack channel to elaborate.

from json-schema-spec.