We have run into an issue with URI Templates and the use of the single quote character and we are looking for some clarification

RFC 6570 says:

2.1. Literals

The characters outside of expressions in a URI Template string are
intended to be copied literally to the URI reference if the character
is allowed in a URI (reserved / unreserved / pct-encoded) or, if not
allowed, copied to the URI reference as the sequence of pct-encoded
triplets corresponding to that character's encoding in UTF-8
[RFC3629].

 literals      =  %x21 / %x23-24 / %x26 / %x28-3B / %x3D / %x3F-5B
               /  %x5D / %x5F / %x61-7A / %x7E / ucschar / iprivate
               /  pct-encoded
                    ; any Unicode character except: CTL, SP,
                    ;  DQUOTE, "'", "%" (aside from pct-encoded),
                    ;  "<", ">", "\", "^", "`", "{", "|", "}"

https://datatracker.ietf.org/doc/html/rfc6570#section-2.1

The literals syntax intentionally excludes the single quote. However, single quote is listed as a reserved character in RFC 3986 https://datatracker.ietf.org/doc/html/rfc3986/#section-2.2

Looking further into RFC6570 in section 3.1 it says

If the literal character is allowed anywhere in the URI syntax
(unreserved / reserved / pct-encoded ), then it is copied directly to
the result string.

This seems to be a contradiction. According to the literal ABNF, single quote is not allowed in a literal. According to the text, it should be.

Can we get a clarification on what is the expectation? If indeed this is an error in the specification we can add the test case to cover it.

The readme says that all the cases from spec-examples-by-section.json are included in the spec-examples.json file, but this is not true. There are various missing examples.

Hello,

As to errors:

• percent encoding is case insensitive, as such %3A and %3a are the same; tests do not account for it;
• in extended tests, empty_assoc is defined as [], which make it undistinguishable from empty_list, which is also defined as [].

Now, as to extended tests and empty map/list expansion: what is the reasoning behind the fact that {?empty_list*} should expand to the empty string? Same goes for empty_assoc.

Section 3 states that a URI Template process should stop processing a template when it encounters an expression that does not match URI template grammar. I have a fews of tests in mind but they don't really fit nicely into the current framework.

Given the test expression

 [ "{/keys", "???" ]

Obviously we're looking for a template processor to fail when there is no closing } but I'm not sure how this should be expressed in a test case. Given that these tests are optional according to the spec, I'm not a 100% sure that they'd be applicable to the standard test suite. At the very least, I'd envision them being optional.

Structures are processed as if they are an associative array with names corresponding to the fields in the structure definition and "." separators used to indicate name hierarchy in substructures. https://tools.ietf.org/html/rfc6570#section-2.4.2

Given a composite value, { "foo": { "bar": "baz" } }, and a URI Template, {foo.bar}, I expect this to expand to baz.

This is the only place in the spec where hierarchical names are discussed. There are no examples in the spec, there are no tests in this suite, and none of the implementations I tried support . as anything but a literal character in a variable name. So, I was hoping someone here could tell me if I've interpreted this part of the spec correctly. If not, what is it intended to mean?

extended-tests contains:

        ["/search.{format}{?q,geocode,lang,locale,page,result_type}",
            [ "/search.json?q=URI%20Templates&lang=en&geocode=37.76,-122.427&page=5",
              "/search.json?q=URI%20Templates&lang=en&geocode=-122.427,37.76&page=5"]
            ],

Note the ordering of geocode and lang.

In spec-examples.json, the{/keys} test contains a duplicate result combination:

     ["{/keys}", [
        "/comma,%2C,dot,.,semi,%3B",
        "/comma,%2C,semi,%3B,dot,.",
        "/dot,.,comma,%2C,semi,%3B",
        "/dot,.,semi,%3B,comma,%2C",
        "/comma,%2C,dot,.,semi,%3B",
        "/comma,%2C,semi,%3B,dot,."
      ]]

It should be

     ["{/keys}", [
        "/comma,%2C,dot,.,semi,%3B",
        "/comma,%2C,semi,%3B,dot,.",
        "/dot,.,comma,%2C,semi,%3B",
        "/dot,.,semi,%3B,comma,%2C",
        "/semi,%3B,comma,%2C,dot,.",
        "/semi,%3B,dot,.,comma,%2C"
      ]]

In spec-examples-by-section.json (and the spec), are some test pairs:

["{/var,empty}", "/value/"],
["{/var,undef}", "/value"]

However, I can't find these in spec-examples.json.

I'm contributing to a URI template lib (JS), which uses PEG - grncdr/uri-template#6

and the grammar actually parses some of the templates in negative-tests.json, instead of failing:

• {hello:2*}
• {keys:1}, {+keys:1}, {;keys:1*}

Why should the first category fail, rather than expand into "He" ?
As for the second category, I'm assuming it should fail because prefixed values are incompatible with lists/maps - can someone confirm ?

Thanks

I can see from the closed issues and the RFC errata history that the "'" character is allowed in literals now. However, I didn't realise that initially, and was quite confused as to why the second test case in spec-examples.json included what I thought to be an invalid literal. Obviously I've tracked that down, but would it be worth making this more obvious?

One way to do so might be to remove that test from the spec-examples.json file (it doesn't actually appear in the RFC itself) and creating a small new test case file (something like spec-errata-examples.json perhaps) which would include any cases which reflect changes to the spec? That would potentially make it clearer for example that the spec has changed, and also point people in the direction of checking errata during implementation (yes, I know I should anyway, but it's not always glaringly obvious).

Hi! I'd love to use this in my project. Would you mind granting me (or ideally, everyone) an MIT or ISC license for this package?

Looking again, I found a libre enough license in the readme. Would you mind declaring it in a way that Github understands?

Also, would you like to add npm metadata and publish it on npm, or should I do that myself?

Consider expanding {+abcdef:5} with abcdef := "%61%62%63%64%65%66".
(And in similar way, {+invalid_utf8:2} with invalid_utf8 := %FF%FF%FF%FF.)
The spec (RFC 6570) is not so clear in what the result should be, and various implementations would produce different results, so this kind of inputs should be included to the test suite.

Although this is not explicitly called out by the specification, it seems to me that

{ "foo": ["a", "b"] }

https://localhost?foo={foo}
https://localhost?foo={foo*}

will both result in 
https://localhost?foo=a,b

Should we add those cases to the test suite?

I don't really understand Git well enough to know how this is possible, but this changeset a3ef0fe#diff-0
does not seem to have been applied. The ordering of the {keys} test cases do not match the keys variable.

I wrote a JavaScript library (GitHub repo) that as well as expanding templates, can also guess values from URIs:

var template = uriTemplate("{/id*}{?fields,token}");
var guessedValues = template.fromUri("/person/albums?fields=id,name,picture&token=12345");
/* {
       id: ["person", 'albums"],
       fields: ["id", "name", "picture"],
       token: "12345"
   } */
var reconstructed = template.fillFromObject(guessedValues);

I'm currently testing it using this test suite, by checking that the guessed values will reconstruct the original URI.

However, I can't actually check that the calculated values are "correct". Most of them I could probably handle with a fuzzy comparison against the entries in variables, but that doesn't work for some cases, e.g.:

• "?{x,undef,y}" => "?1024,768"
• "{var1}{var2}" => "string1string2"

(Note that the second case is OK when var2 has a prefix, like "{var1}{?var2}")

Would it be possible to assemble a sub-set of the tests where the original data should be recoverable? If I assembled such a subset, would you be open to including it in this repo?

https://tools.ietf.org/html/rfc6570#section-3.2.3 -

Reserved expansion, as indicated by the plus ("+") operator for Level 2 and above templates, is identical to simple string expansion except that the substituted values may also contain pct-encoded triplets and characters in the reserved set.

I just reported this here: bricas/uri-template#7 - and then realised the test suite doesn't cover them either (no explicit examples in the RFC). I will create some.

Hi.

In negative-tests.json all result values (second member) are false and the readme says:

if the second member is boolean false, expansion is expected to fail (i.e., the template was invalid).

I don't understand why the following templates are considered invalid:

[ "{keys:1}",  false  ],
[ "{+keys:1}",  false  ],

As far as I can tell those templates are valid according to the spec:

expression      =  "{" [ operator ] variable-list "}"
variable-list   =  varspec *( "," varspec )
varspec         =  varname [ modifier-level4 ]
modifier-level4 =  prefix / explode
prefix          =  ":" max-length
max-length      =  %x31-39 0*3DIGIT   ; positive integer < 10000

I see that these two test cases have been around for a long time (since 2012), so there is obviously something going on that I don't fully understand. Could you please enlighten me?

The extended tests contain a test for pct encoded varname for form style query expansion with string value:

["/lookup{?Stra%C3%9Fe}", "/lookup?Stra%C3%9Fe=Gr%C3%BCner%20Weg"],

I believe that this test should also exist for lists, i.e have in variables:

"Stra%C3%9Fe"  : ["Grüner",  "Weg"]

and

["/lookup{?Stra%C3%9Fe_list}", "/lookup?Stra%C3%9Fe_list=Gr%C3%BCner&Stra%C3%9Fe_list=Weg"],

I propose that these tests are removed from
"negative-tests.json":

[ "{keys:1}",  false  ],
[ "{+keys:1}",  false  ],

I guess that the author of these tests thought of this
extract from the RFC:

Prefix modifiers are not applicable to variables that
have composite values.

The normative part of the RCF does not seem to explicitly
say what "not applicable" means. The algorithm in
"Appendix A" of the RFC
(https://www.rfc-editor.org/rfc/rfc6570#appendix-A) seems
to ignore the prefix modifier for complex values (and would
therefore not pass the above tests).

Also the algorithm in "Appendix A" explicitly ignores
variables with an absent or undefined value. Keeping the
above tests would mean that evaluating a
"prefix-modifier-varspec" with absent/undefined value
should be tolerated while the evaluation must fail for
complex values. This is not the behavior I would have
expected after reading the specification.

In section 3.2.8 of the specification stated that.

if the variable has a simple string value or no explode modifier is given, append the variable name (encoded as if it were a literal string) and an equals character ("=") to the result string

But in the test file called extended-tests.json. The test entry with value /lookup{?Stra%C3%9Fe} doesn't get encoded as stated by the spec. It just copy varname to the result string without being encoded. (i.e. /lookup?Stra%C3%9Fe=Gr%C3%BCner%20Weg")

I think the correct value should be /lookup?Stra%25C3%259Fe=Gr%C3%BCner%20Weg (encoed varname too).

Or maybe I misunderstood the spec?

Hello,
I am creating a regex for uri-templates and I need help with the following points:

1- In paragraphe 2.1. Literals, it is mentioned that literals can any Unicode character except: CTL, SP, DQUOTE, "'", "%" (aside from pct-encoded), "<", ">", "", "^", "`", "{", "|", "}"
I can’t find the name or the code for CTL, in any of the following resources:
https://en.wikipedia.org/wiki/C0_and_C1_control_codes
https://en.wikipedia.org/wiki/Unicode_control_characters
http://www.unicode.org/Public/UNIDATA/Index.txt
http://www.unicode.org/Public/UNIDATA/NamesList.txt
http://www.unicode.org/charts/PDF/U0000.pdf

Could you please tell me the name and the unicode code point for CTL?

2- I am testing my regex using the test cases provided in https://github.com/uri-templates/uritemplate-test.
My regex does match the following cases from negative-tests.json
[ "{!hello}", false ] ,
[ "{=path}", false ] ,
[ "{|var*}", false ],
[ "{keys:1}", false ],

2.1- The first 3 are matched because I have included the op-reserve ("=" / "," / "!" / "@" / "|") as legal operators.
Should I remove them for now, and update my regex once they become used by the specification ?

2.2- I don’t know why is the last case wrong expression. Could you please explain ?

Thank you very much in advance

Hassan

The spec states

The expression syntax specifically excludes use of the dollar ("$")
and parentheses ["(" and ")"] characters so that they remain
available for use outside the scope of this specification.

I've come across an implementation at https://github.com/tavis-software/Tavis.UriTemplates that is using this test suite and does throw when it encounter a parameter with a $ sign, like this one that could be an OData endpoint:

https://example.com/{?$filter,$orderby,$top,$skip}

The extended tests suggest that lists are unordered, however I cannot find any basis for this.

The spec doesn't mention ordered/unordered, however the word "list" usually implies an ordering (in contrast to "set"), and in this particular example, it would be very odd if the the order of a list in the hierarchical part didn't matter (notice how "albums" and "person" are listed in both orders):

Hello everyone,

Is this test [ "/vars/:var" , false ] from negative tests correct? Could you please explain the reason? I'm doing some work related to URI templates and this one confused me.

Thanks in advance!

ioseb

Could you please include test cases for
Variables: strange: “:/?#[]@!$&'()*+,;=”;

Templates:
{strange}
{+strange}

(from Franz Antesberger [email protected])

Currently all variables are alpha-only. We could use some numbers, underscores, dots and pct-encoded characters ( see: http://tools.ietf.org/html/rfc6570#section-2.3 ).