elm / url Goto Github PK

We're running an Elm app as an embedded element, so we don't actually control the url where the elm app runs. We are interested in variables stored in query parameters though, and as far as I know there's no way to construct a parser which allows us to just parse queries.

Our current workaround is to alter the Url object and always set path to "/", but this feels a bit hackish.

In the documentation for Url.Parser.parse, the code uses </> without importing it at the top of the example. This caused a little confusion when I was talking with someone about how to parse URLs today!

Url.Builder.absolute ["file", "/path/bla"] []

should result to

/file/%2Fpath%2Fbla

but currently results to

/file//path/bla

This is especially surprising due to documentation

Use Url.Builder instead! Functions like absolute, relative, and crossOrigin already do this automatically! percentEncode is only available so that extremely custom cases are possible, if needed.

Example:

> Url.Builder.absolute [ "foo", "/", "bar" ] []
"/foo///bar" : String

Should be /foo%2Fbar, right?

I've seen #21, didn't read the spec, but here's what the Wikipedia article says about this concrete case:

The reserved character /, for example, if used in the "path" component of a URI, has the special meaning of being a delimiter between path segments. If, according to a given URI scheme, / needs to be in a path segment, then the three characters %2F or %2f must be used in the segment instead of a raw /.

I've run across this while debugging an issue in our application with Mapbox search API returning errors on punctuation/special symbols (the URL looks like https://api.mapbox.com/geocoding/v5/mapbox.places/<query>.json).

The documentation for Url.percentEncode claims: "Use Url.Builder instead! Functions like absolute, relative, and crossOrigin already do this automatically! percentEncode is only available so that extremely custom cases are possible, if needed."
https://package.elm-lang.org/packages/elm/url/latest/Url#percentEncode

But as you can see at https://github.com/elm/url/blob/master/src/Url/Builder.elm#L52-L54 Url.absolute doesn't use Url.percentEncode at all.

absolute : List String -> List QueryParameter -> String
absolute pathSegments parameters =
   "/" ++ String.join "/" pathSegments ++ toQuery parameters

What

Parse a complete pathname with all segments at once.
The requested feature is implemented in this fork here : http://package.elm-lang.org/packages/terezka/url-parser/1.0.0/UrlParser#any

Why

Our clients are able to define custom paths for the pages they create. Which means they could write something like /this/random/path/1/. Having the ability to accept that complete path and still allow for parsing queryParams will prevent workarounds that don't make full use of this package.

Note

We're having this issue in evancz/url-parser but as this looks to become the next version of that I'm posting this here.

https://package.elm-lang.org/packages/elm/url/latest/Url-Parser-Query#custom
This is the example in the documentation:

posts : Parser (Maybe (List Int))
posts =
  custom "post" (List.maybeMap String.toInt)

-- ?post=2        == [2]
-- ?post=2&post=7 == [2, 7]
-- ?post=2&post=x == [2]
-- ?hats=2        == []

List.maybeMap doesn't exist.

Meta issue to remind, that the example has to be updated. There will be Elm 0.19 specific changes. But the example is also "evancz/url-parser": "2.0.0 <= v < 3.0.0".

Hello, I need to parse username and password from url, can we added that to elm/url?

Documentation of Url.Parser.query says

This may be handy if you need query parameters but are not parsing any path segments.

but it is not possible to parse query parameters without path segments - see #17.

Url.Parser.s doesn't percentDecode path segments for matching so it returns Nothing instead of match if percent-encoding is used.

Hey!

I was informed that the Url Parser is being moved to this repository and to check and see if this issue still exists: evancz/url-parser#51

It does! Some users in the #beginner thread mentioned that had no idea the purpose of the first argument to the custom function. I took a look and it used to be used for error messages, but those have since been removed and the argument tipe is not used in the function. Here is an example of what the current use cases are in the documentation and what it would look like without the unused argument.

-- The int example in the docs. The confusion lay with the "NUMBER" argument.

int : Parser (Int -> a) a
int =
    custom "NUMBER" String.toInt

-- The int example without the argument

int : Parser (Int -> a) a
int =
    custom String.toInt

-- The CSS example in the docs. It is "CSS_FILE" here

css : Parser (String -> a) a
    css =
      custom "CSS_FILE" <| \segment ->
        if String.endsWith ".css" segment then
          Just segment
        else
          Nothing

-- The CSS example without the argument

css : Parser (String -> a) a
css =
    custom 
        (\segment ->
            if String.endsWith ".css" segment then
              Just segment
            else
              Nothing
        )

Problem

Some issues on this repo are saying that more decoding should happen on path segments by default:

• #16
• #20
• #25
• #41

Issue #20 also says that this decoding should not occur in all cases, referencing this issue that points to implementations in many languages that do not call percentDecode in a naive way.

What should be done?

Current Status

As mentioned in #16, it is currently possible to use Url.Parser.custom to create custom path parsers that have whatever behavior you personally think is best. For example, custom "STRING" Url.percentDecode would do very aggressive percent decoding.

So nobody is blocked on this topic. It is a question of defaults, and any change should be considered a breaking change that triggers a MAJOR version bump.

Goal

Make a table of scenarios to try to find the ideal defaults for a broad range of people. The default options for path parsing and building are:

1. no percent decoding
2. percent decoding for specific characters
3. percent decoding for all characters

We currently do (1) but maybe it'd be good to make a table to show the various options. Right now I think it would be ideal for someone interested in this topic to:

1. Build a table of interesting paths and see how they all work under different defaults.
2. Check the defaults of implementations in other languages.

Hopefully this will reveal more information / some sort of consensus. I think being thorough about this is very important, since any change here could break a lot of peoples' code in ways that might be very hard to detect. Please share your efforts here or on the Elm Discourse.

I expected the Url.Builder to create Urls, not Strings. It loses all guarantees that I expected from it.

If it would create a Url, I could simply run Url.toString on it. The reverse is much more hassle, since Url.fromString may produce Nothing, even though I just used the Url.Builder that should prevent me from generating invalid Urls.

Maybe I missed something and I see that changing it means breaking the interface. But I think this is an important consideration.

Issue

evancz/url-parser had parseHash. It was removed.

In my case "Url-Routing" is not an option, because the http server is not under my control and will issue 404.

Parsing a url like http://localhost:3000/#/mydata/abe8e05f-e58a-4bfc-81e8-2081c7638407 is hacky. There is fragment. It returns Maybe String. There is no way to parse String with the library. Therefore the workaround shown is working, but not what anybody would want.

Are there alternatives without adding the functionality? If not, can we support hash routing again?

Workaround

parseUrl : Url -> Maybe Route
parseUrl url =
    UrlParser.parse parseRoute (urlFragmentToPath url)

urlFragmentToPath : Url -> Url
urlFragmentToPath url =
    { url | path = Maybe.withDefault "" url.fragment, fragment = Nothing }

My application requires me to handle query parameters somewhat generically (in particular, to handle query parameters with variable names). The exposed interface of Url.Parser doesn't support that.

Url.Parser internally uses:

prepareQuery : Maybe String -> Dict String (List String)

for the low-level query parameter parsing.

I'd be happy to reimplement this myself, but for the fact that it calls out to kernel code for percent-decoding, which I'm not permitted to do myself, and I'd be loathe to implement differently from core Elm.

Thus, please consider exposing a way to get at a full query parameter dictionary without jumping through hoops.

The parsing of urls misses an edge case where there is additional invalid info at the end of a url.

in elm repl:

import Url
Url.fromString "https://www.test.org (will be online next week)"

Result: Just { fragment = Nothing, host = "www.test.org (will be online next week)", path = "/", port_ = Nothing, protocol = Https, query = Nothing }

expected Result: Nothing

Also:

import Url
Url.fromString "https://In development"

Result: Just { fragment = Nothing, host = "In development", path = "/", port_ = Nothing, protocol = Https, query = Nothing }
: Maybe Url.Url

Expected Result: Nothing

Assuming that my app runs at http://localhost/project/?foo=bar.
And I want to have link to homepage as ./.
I found no way how create that via Url.Builder.custom.

https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/encodeURIComponent#description

Note that a URIError will be thrown if one attempts to encode a surrogate which is not part of a high-low pair

> import Url
> Url.percentEncode "\u{d800}"
URIError: URI malformed

Url.percentDecode is already wrapped in try-catch but Url.percentEncode isn’t:

Note: I have never encountered this for real. I just noticed this because I went to the MDN page for encodeURIComponent to better learn how it works, and realized this is another way an exception can be thrown in Elm. So I’m reporting this more for documentation and collecting all the weird edge cases.

https://package.elm-lang.org/packages/elm/url/latest/Url-Parser-Query#map

Example is:

page : Parser Int
page =
  map (Result.withDefault 1) (int "page")

According to signature for int function it should be

page : Parser Int
page =
  map (Maybe.withDefault 1) (int "page")

Our front page has a standard html search form. If the user types a search string, eg. "hello world" and submit the form, the user is taken to /search?query=hello+world where the elm application is located. When parsing the query parameter I expect the output to be "hello world" - however the actual output is "hello+world". Is this intentional or is this something that should be fixed? I haven't found specific documentation on this, but it seems to be common practice to handle + in query paramaters as space.

Example (21 Mar 2021):

let
    queryParser = 
        Url.Parser.s "search" <?> Url.Parser.Query.string "q"
in
Url.fromString "https://www.example.com/search?q=how+much+is+1%2B1%3F"
    |> Maybe.andThen (Url.Parser.parse queryParser)

-- Result:    Just (Just "how+much+is+1+1?")
-- Expected:  Just (Just "how much is 1+1?")

For my current app I need a parser which only checks that some initial segments of path match the parser, and then ignore the rest of the segments, if any, and regardless of how many there are.

This doesn't seem to be possible currently. Custom parser doesn't help as it would require me to know exactly how many segments I want to ignore, which I don't know.

Consider an application that is not in the root of a site e.g.

example.com/some/path/

In this case we would pass the basepath to the app via flags. e.g

flags =
    {
    basePath = "some/path/"
    }

I attempted to parse this like:

matchers : Parser (Route -> a) a
matchers =
    Parser.s flags.basePath  </> Parser.oneOf
        [ Parser.map Home Parser.top
        , Parser.map About (Parser.s "about")
        ]

But Parser.s won't parse the /.

Example here https://ellie-app.com/3grXTSD4Bkxa1

It would be great to have something to parse a basepath e.g.

matchers : Parser (Route -> a) a
matchers =
    Parser.base flags.basePath  </> Parser.oneOf
        [ Parser.map Home Parser.top
        , Parser.map About (Parser.s "about")
        ]

Thanks

Hello everyone,
I'm reading Url.Parser source code to understand Elm's url parsing mechanism. I noticed that the type signature of parse function to be:

parse : Parser (a -> a) a -> Url -> Maybe a

Is that signature correct? But I think the correct signature should be:

parse : Parser (a -> b) b -> Url -> Maybe b

Did I misunderstand something here?
Would anyone please give me a hint, thank you very much!

I can't seem to install the package, getting the following error:

Error: Could not find any packages named elm-lang/url.

Here are some packages that have similar names:

    elm-lang/core
    elm-lang/dom
    elm-lang/html
    elm-lang/svg

Is this normal? If not, is it possible to publish the package?

I tried to use URLs with IPv6 host for a Browser.application and the navigation didn't work. The log message in the console directed me to this page: https://github.com/elm/core/blob/1.0.0/hints/1.md . When tried the same with IPv4 or a domain name, then it worked.

The URLs that I tried:

• http://[::]:7272/ui/
• http://[::1]:7272/ui/

Is this a bug or just the feature is not supported currently?

From the second paragraph of the docs (https://package.elm-lang.org/packages/elm/url/latest/Url):

When you are creating a single-page app with Browser.fullscreen, you use the Url.Parser module to turn a Url into even nicer data.

However, I cannot find a "fullscreen" function anywhere inside the Browser module. (I'm a newbie so correct me if I'm wrong.)

Context

What I mean by a "dirty" path is simply a path that has trailing and leading slashes, without accounting for what other URL parts are following or leading.

Current behavior

If we try to merge parts of a URL that contains leading and trailing slashes, building a relative path out of these data would lead to an incorrect path.

import Url.Builder

Url.Builder.relative ["http://localhost/", "/ping/"] [] -- "http://localhost///ping/"

Expected behavior

It would be nice if the function somehow cleaned the URLs. The end result (with the above data) could look like that.

import Url.Builder

Url.Builder.relative ["http://localhost/", "/ping/"]  [] -- "http://localhost/ping/"
Url.Builder.relative ["http://localhost/", "ping/"]   [] -- "http://localhost/ping/"
Url.Builder.relative ["http://localhost/", "/ping"]   [] -- "http://localhost/ping"
Url.Builder.relative ["http://localhost/", "ping"]    [] -- "http://localhost/ping"
Url.Builder.relative ["http://localhost", "/ping/"]   [] -- "http://localhost/ping/"
Url.Builder.relative ["http://localhost", "ping/"]    [] -- "http://localhost/ping/"
Url.Builder.relative ["http://localhost", "/ping"]    [] -- "http://localhost/ping"
Url.Builder.relative ["http://localhost", "ping"]     [] -- "http://localhost/ping"

Additional Context

In JavaScript, there is a module path that has a join function that act in a similar fashion, but will merge the parts of a path and make a valid url at the end (it also uses the operating system's default path separator but this is irrelevant here).

There could be a new Url.Builder.join function to prevent any breaking change in the current API instead.

import {join} from "path";

console.log(join("http://localhost/", "/ping/")); // "http://localhost/ping/"
console.log(join("http://localhost/", "ping/"));  // "http://localhost/ping/"
console.log(join("http://localhost/", "/ping"));  // "http://localhost/ping"
console.log(join("http://localhost/", "ping"));   // "http://localhost/ping"
console.log(join("http://localhost", "/ping/"));  // "http://localhost/ping/"
console.log(join("http://localhost", "ping/"));   // "http://localhost/ping/"
console.log(join("http://localhost", "/ping"));   // "http://localhost/ping"
console.log(join("http://localhost", "ping"));    // "http://localhost/ping"

Like

import Url.Builder

Url.Builder.join ["http://localhost/", "/ping/"]  [] -- "http://localhost/ping/"
Url.Builder.join ["http://localhost/", "ping/"]   [] -- "http://localhost/ping/"
Url.Builder.join ["http://localhost/", "/ping"]   [] -- "http://localhost/ping"
Url.Builder.join ["http://localhost/", "ping"]    [] -- "http://localhost/ping"
Url.Builder.join ["http://localhost", "/ping/"]   [] -- "http://localhost/ping/"
Url.Builder.join ["http://localhost", "ping/"]    [] -- "http://localhost/ping/"
Url.Builder.join ["http://localhost", "/ping"]    [] -- "http://localhost/ping"
Url.Builder.join ["http://localhost", "ping"]     [] -- "http://localhost/ping"

It's often useful to have an optional path at the end.

This can be currently done with two parsers.

Example from elm repl

> import Url

> Url.fromString "gio"
Nothing : Maybe Url.Url

> Url.fromString "https://"
Nothing : Maybe Url.Url

> Url.fromString "https://g"
Just { fragment = Nothing, host = "g", path = "/", port_ = Nothing, protocol = Https, query = Nothing }

The first 2 calls to fromString are correct. However, the 3rd call (Url.fromString "https://g") generates a Just Url when it should be returning a Nothing.

I am no expert, but I do believe a valid URL requires a TLD.

I also validated this against npm's validator library. Code here

I wanted to take an existing url and modify the query parameters. I noticed Url is just a record type so I figured the correct way to do this is

{ currentUrl
    | query =
        Url.Builder.toQuery [ Url.Builder.string "data" "test" ] |> Just
}
    |> Url.toString

but when I did that the result was http://localhost:3000/??data=test instead of http://localhost:3000/?data=test.

This seems to be a bug or at least unintuitive behavior and what I ended up doing to make it work was add String.dropLeft 1 to the toQuery output.

In light of the question of how URLs should be percent-encoded (#42), I think there is a more pressing underlying issue.

There is confusion about how Url.Builder and Url.Parser methods work, since the documentation on percentEncode makes it seem that functions like absolute, relative, and crossOrigin will also do percent-encoding on the paths, as evidenced by the top comment in #25.

To address the misunderstanding of what gets percent-encoded by those functions and how to achieve the desired behavior, we can change the documentation for percentEncode from

Use Url.Builder instead! Functions like absolute, relative, and crossOrigin already do this automatically! percentEncode is only available so that extremely custom cases are possible, if needed.

to something like

Use Url.Builder instead! Functions like absolute, relative, and crossOrigin already encode query parameters automatically! percentEncode is only available so that extremely custom cases are possible, if needed.

Note that these functions do not percent-encode path components, because not everyone needs the same encoding. If you need to do that, you need to call this function on each path, for example Url.Builder.absolute (List.map percentEncode ["encode me"]) [] to get /encode%20me.

Similarly, the documentation for percentDecode can be changed from

Use Url.Parser instead! It will decode query parameters appropriately already! percentDecode is only available so that extremely custom cases are possible, if needed.

to

Use Url.Parser instead! It will decode query parameters appropriately already! percentDecode is only available so that extremely custom cases are possible, if needed.

Note that Url.Parser does not percent-decode path components, because because not everyone needs the same encoding. If you need to do that, you can create a custom Url.Parser with custom "STRING" Url.percentDecode.

I have originally reported elm/browser#20 but as far as I can tell reason is not in Browser.application but rather in the fact that following code explicitly deals with just http / https protocols

The string combinator from Url.Parser does not decode the value using Url.percentDecode.
I think the implementation should be something like this:

string : Parser (String -> a) a
string =
    custom "STRING" Url.percentDecode

Since the library uses a protocol type, defined like this:

{-| Is the URL served over a secure connection or not?
-}
type Protocol = Http | Https

it is possible to use the library for anything other than http and https.
But what about if somebody needs a websocket url? The schemes ... | Ws | Wss would have to be added to support that (which I would argue is the most common scheme in the web after http). But maybe there are also other cases not covered by this, so having ... | Other String might be a good idea too. It would be a shame if people coulnd't use this nice library because their scheme is not supported.

Also toUrl has a check for http or https hardcoded which makes it impossible for other schemes to be a URL. My personal opinion is that this library should be not make any assumptions on the scheme which is used and just treat it as the part before the : and //.