ponylang / http Goto Github PK

These should benchmark the current HTTP parser against a set of typical HTTP requests, ranging from very small and simple GET requests to a chunked huge POST request encoded as mutipart/form-data etc.

Use ponybench

Copied rom ponylang/ponyc#2614 :

Reported by @BlackYoup

I have a problem with the HTTP server where a long request URI would make the server close the connection even if the request is entirely valid. This is because the state machine doesn't seem to implement the "we need more information to decide" case, or at least I couldn't find it / may have not seen it.

Reproduce

I can reproduce on both 0.21.3 and current master fb10149e116460f908f949310ff2876fe1a38688

I used the httpserver example with default options. I then made a request like this:

curl -v http://127.0.0.1:50000/testtesttesttesttesttesttesttesttesttesttesttestt
*   Trying 127.0.0.1...
* TCP_NODELAY set
* Connected to 127.0.0.1 (127.0.0.1) port 50000 (#0)
> GET /testtesttesttesttesttesttesttesttesttesttesttestt HTTP/1.1
> Host: 127.0.0.1:50000
> User-Agent: curl/7.59.0
> Accept: */*
>
* Empty reply from server
* Connection #0 to host 127.0.0.1 left intact
curl: (52) Empty reply from server

If you remove the last t from the URI, it won't close the connection and proceed the request.

Debug

I traced the problem up until here, into the _ServerConnHandler.received() method:
https://github.com/ponylang/ponyc/blob/fb10149e116460f908f949310ff2876fe1a38688/packages/net/http/_server_conn_handler.pony#L50...L74

If I print the output of the data variable when the first chunk comes in, it shows like this:

0x00: 4745 5420 2F74 6573 7474 6573 7474 6573  GET /testtesttes
0x10: 7474 6573 7474 6573 7474 6573 7474 6573  ttesttesttesttes
0x20: 7474 6573 7474 6573 7474 6573 7474 6573  ttesttesttesttes
0x30: 7474 6573 7474 2048 5454 502F 312E 310D  ttestt HTTP/1.1.
0x40:

As you can see, there is no line return (\n, 0xA) at the end of the chunk, which comes in the next chunk we receive:

0x00: 0A48 6F73 743A 2031 3237 2E30 2E30 2E31  .Host: 127.0.0.1
0x10: 3A35 3030 3030 0D0A 5573 6572 2D41 6765  :50000..User-Age
0x20: 6E74 3A20 6375 726C 2F37 2E35 392E 300D  nt: curl/7.59.0.
0x30: 0A41 6363 6570 743A 202A 2F2A 0D0A 0D0A  .Accept: */*....
0x40:

But just that missing line return makes the parse() call fail and closes the connection.

I'm not sure how to fix this yet but given the amount of time I've put into that debug, I'd appreciate if you could give me a few insights as to how it could be fixed, if it's not too much work :)

It is obsolete and defect by now. For HTTP servers https://github.com/ponylang/http_server should be used.

Headers, on look up from a payload, are case-sensitive. They should be stored in a case insensitive fashion and lookup should also be case insensitive.

Probably by lowercasing all the headers.

Description

Steps to reproduce

I wrote a program to demonstrate this issue. It uses the GitHub API.

NOTE:

• You must replace the string YOUR_GITHUB_PERSONAL_AUTH_TOKEN_HERE with your own GitHub personal access token.
• You must include the http library using stable: stable add github ponylang/http
• You must build with stable: stable env ponyc
• The first (and only) argument should be the name of a repo, like ponylang/ponyc.

use "http"
use "net/ssl"

actor Main
  new create(env: Env) =>
    let auth_token = "YOUR_GITHUB_PERSONAL_AUTH_TOKEN_HERE"

    let sslctx = try
      recover val
        SSLContext
          .>set_client_verify(true)
          .>set_authority(None)?
        end
      end

    let repo = try
      env.args(1)?
    else
      env.err.print("first argument must be the name of a repo")
      env.exitcode(-1)
      return
    end

    try
      let auth = env.root as AmbientAuth
      let client = HTTPClient(auth, sslctx)
      let nf = recover val NotifyFactory(env.out) end

      let url_string = "https://api.github.com/repos/" + repo + "/issues"

      env.out.print("getting data at " + url_string)

      let url = URL.valid(url_string)?

      let req = Payload.request("GET", url)
      req("User-Agent") = "Pony httpget"

      req("Authorization") = "token " + auth_token

      let sentreq = client(consume req, nf)?
    end


class NotifyFactory is HandlerFactory
  let _os: OutStream

  new create(os: OutStream) =>
    _os = os

  fun apply(session: HTTPSession): HTTPHandler ref^ =>
    Notify(_os)

class Notify is HTTPHandler
  let _os: OutStream

  new create(os: OutStream) =>
    _os = os

  fun ref apply(response: Payload val) =>
    _os.print("received response")

  fun ref chunk(data: ByteSeq) =>
    _os.print("received chunk")

  fun ref finished() =>
    _os.print("finished chunks")

  fun failed(reason: HTTPFailureReason) =>
    _os.print("failed")

If you pass a valid repo with lots of issues that should cause GitHub to send the response using chunked encoding, which will trigger the issue and the program will not terminate. If you pass an invalid repo name then the response will be short and GitHub will not use chunked encoding, so the program will terminate.

Expected Behavior

When the program is finished receiving chunked data the HTTPClient will close the connection and terminate.

Actual Behavior

The connection is kept open. As a consequence, the program does not terminate.

There appears no way to re-use a HTTPClient despite the documentation suggesting that one should create one HTTPClient and then re-using it.

The HandlerMaker's apply method is only called once when the HttpSession is created and then stored against the (scheme, host, port) tuple, thus when the second request is made, it uses the Handler from the first request.

Code that reproduces the issue & a more in-depth discussion of the problem:
https://gist.github.com/CandleCandle/260a284ece6d2e99520cfbd99798e739

The workflow release.yml is referencing action actions/checkout using references v1. However this reference is missing the commit a6747255bd19d7a757dbdda8c654a9f84db19839 which may contain fix to the some vulnerability.
The vulnerability fix that is missing by actions version could be related to:
(1) CVE fix
(2) upgrade of vulnerable dependency
(3) fix to secret leak and others.
Please consider to update the reference to the action.

I'm removing as part of the "automated release process" work.

I'd like to make this standardized across library projects including, the library-project scaffolding so I pulled out for now to figure out how to address across the board.

フェデラー風

I'm removing as part of the "automated release process" work.

I'd like to make this standardized across library projects including, the library-project scaffolding so I pulled out for now to figure out how to address across the board.

I've just started with Pony and found that the HTTPClient doesn't deliver connection errors back to the caller. I can see the code should be making it to _ClientConnection._connect_failed but things stop propagating pretty soon after that.

I see there's a comment saying TODO send fail response in the _cancel_all method, and was wondering if anybody was looking at/fixing this?

My guess would be to put something into the HandlerFactory but am new so thought I should ask for suggestions first!

Thanks, and hope that all makes sense!

https://github.com/ponylang/main-actor-documentation-action

When calling HTTPClient.apply for the second time using a new HandleFactory, the behaviour isn't quite clear.

I'm trying to work with HTTPClient and promises, but the second promise is not used, only the first one.

use "net"
use "promises"
use "http"


actor Main
  new create(env: Env) =>
    let client = NetworkClient(TCPConnectAuth(env.root), env.out)

    try
      let url = URL.build("https://echo.sacovo.ch")?

      let p1 = Promise[Payload val]
      let p2 = Promise[Payload val]

      p1.next[Array[ByteSeq] val](ToBody)
        .next[None](PrintBody(env.out))

      p2.next[Array[ByteSeq] val](ToBody)
        .next[None](PrintBody(env.out))

      client.get(url, p1)
      client.get(url, p2)
    end

class ToBody
  fun apply(p: Payload val): Array[ByteSeq] val? =>
    recover val p.body()?.clone() end


class PrintBody
  let out: OutStream

  new create(out': OutStream) =>
    out = out'

  fun apply(a: Array[ByteSeq] val) =>
    for line in a.values() do
      out.print(line)
    end


class MyHandlerFactory is HandlerFactory
  let promise: Promise[Payload val]
  let out: OutStream

  new create(promise': Promise[Payload val], out': OutStream) =>
    promise = promise'
    out = out'

  fun box apply(session: HTTPSession tag): HTTPHandler ref^ =>
    out.print("Received session")
    MyHandler(promise, out, session)


class MyHandler is HTTPHandler
  let promise: Promise[Payload val]
  let out: OutStream
  let session: HTTPSession

  new create(promise': Promise[Payload val], out': OutStream, session': HTTPSession) =>
    promise = promise'
    out = out'
    session = session'

  fun ref apply(payload: Payload val): None tag =>
    out.print("Received payload, status: " + payload.status.string())
    promise(payload)
    out.print("Called promise")


actor NetworkClient
  let client: HTTPClient
  let out: OutStream

  new create(auth: TCPConnectAuth, out': OutStream) =>
    client = HTTPClient.create(auth)
    out = out'

  be get(url: URL, promise: Promise[Payload val]) =>
    out.print("Sending get...")
    try
      let payload = Payload.request("GET", url)
      payload.update("My-Header", "This is a test header!")
      client.apply(consume payload, MyHandlerFactory(promise, out))?
      out.print("Sent request")
    else
      promise.reject()
    end

As far as I understand from this conversation, this is because a session already exists and the HandlerFactory that is provided in the second call is not used because of that.

Description

The net/ssl package and regex package were recently removed from the standard library.

Steps to reproduce

1. Install a version of ponyc commit b0b68f7a394ca6cd13198f2fdf7240bf7a116d8e.
2. build the http library using by running make.

Expected behavior

The tests build and pass.

Actual behavior

The tests fail to build with messages like "can't load package 'net/ssl'" and "can't load package 'regex'".

The HTTP Parser is way too slow.

It uses error as control flow, which is not cool in the hot path.
See https://www.ponylang.org/reference/pony-performance-cheatsheet/#avoid-error

Use the benchmarks from #1 to validate performance improvements.

The workflow pr.yml is referencing action actions/checkout using references v1. However this reference is missing the commit a6747255bd19d7a757dbdda8c654a9f84db19839 which may contain fix to the some vulnerability.
The vulnerability fix that is missing by actions version could be related to:
(1) CVE fix
(2) upgrade of vulnerable dependency
(3) fix to secret leak and others.
Please consider to update the reference to the action.

From #3 :

Here's the big question to me. What does the release process look like for this. If we can answer that, I'll take care of setting everything up.

So, what does it mean to release the HTTP library?

Are is it a tag on master? Tag on a release branch? If its tag based, I prefer it being on a release branch. Then from that branch we can trigger Travis doing... something...

That something would be

    generating documentation and putting "somewhere"
    ? is that it? probably yes for right now as folks would use stable to get the release or download from the releases page.

If we can agree on that, I'll work on setting it up.

I think for now, what I would attempt and we can see how it works out would be to store the generated documentation on the release branch. It kind of makes sense as it is an "easy" way to get at the documentation for a given release.

For compat with ponyc 0.47.0

before release:

• update to latest library-documentation-action

after release we need to rev:

• pr.yml back from latest for linux and nightlies for windows

Hi,

I'm not sure if I have an issue for ponylang/http or my network or raspbian configuration.

I compiled the httpserver example an run it on my raspbian stretch:

pi@homeserver:~/src/.deps/ponylang/http/examples/httpserver $ sudo ./httpserver 8080 connected: ::1

It runs fine, a test with wget shows:

pi@homeserver:~/src$ wget localhost:8080/Hello
--2019-03-21 11:40:22--  http://localhost:8080/Hello
Auflösen des Hostnamens »localhost (localhost)« … ::1, 127.0.0.1
Verbindungsaufbau zu localhost (localhost)|::1|:8080 … verbunden.
HTTP-Anforderung gesendet, auf Antwort wird gewartet … 200 200 OK
Länge: 20
Wird in »»Hello«« gespeichert.

Hello               100%[===================>]      20  --.-KB/s    in 0s      

2019-03-21 11:40:22 (645 KB/s) - »»Hello«« gespeichert [20/20]

But when I try to access from another machine (in the same network), it's not possible (the Raspberry is named 'homeserver.local'):

woody@lappi:~$ wget homeserver.local:8080/Hello
--2019-03-21 11:55:43--  http://homeserver.local:8080/Hello
Auflösen des Hostnamens homeserver.local (homeserver.local) … 192.168.0.236
Verbindungsaufbau zu homeserver.local (homeserver.local)|192.168.0.236|:8080 … fehlgeschlagen: Verbindungsaufbau abgelehnt.

I also installed lighttp on the Raspberry, no problem to access it from another machine:

woody@lappi:~$ wget homeserver.local
--2019-03-21 11:57:35--  http://homeserver.local/
Auflösen des Hostnamens homeserver.local (homeserver.local) … 192.168.0.236
Verbindungsaufbau zu homeserver.local (homeserver.local)|192.168.0.236|:80 … verbunden.
HTTP-Anforderung gesendet, auf Antwort wird gewartet … 200 OK
Länge: 3378 (3,3K) [text/html]
Wird in »index.html« gespeichert.

index.html          100%[===================>]   3,30K  --.-KB/s    in 0s      

2019-03-21 11:57:35 (16,4 MB/s) - »index.html« gespeichert [3378/3378]

I can not see any reason why the Raspberry refuses the connection to httpserver.

pi@homeserver:~/src $ sudo iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

pi@homeserver:~/src $ sudo netstat -tlnpu
Aktive Internetverbindungen (Nur Server)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name    
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      1368/lighttpd       
tcp        0      0 0.0.0.0:8083            0.0.0.0:*               LISTEN      463/perl            
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      457/sshd            
tcp6       0      0 ::1:8080                :::*                    LISTEN      4257/./httpserver   
tcp6       0      0 :::80                   :::*                    LISTEN      1368/lighttpd       
tcp6       0      0 :::22                   :::*                    LISTEN      457/sshd            
udp        0      0 0.0.0.0:57862           0.0.0.0:*                           381/avahi-daemon: r 
udp        0      0 0.0.0.0:68              0.0.0.0:*                           353/dhcpcd          
udp        0      0 0.0.0.0:5353            0.0.0.0:*                           381/avahi-daemon: r 
udp6       0      0 :::60888                :::*                                381/avahi-daemon: r 
udp6       0      0 :::546                  :::*                                353/dhcpcd          
udp6       0      0 :::5353                 :::*                                381/avahi-daemon: r 

Just an idea..: The httpserver is only listed as tcp6 listening. Maybe that could be the problem?

Thanks for any help, I'm not so familiar with the network stuff.

In RFC 55, this http server seems to have perfomance problem as a http server

It still has poor performance characteristics

What is the most serious performance problem?
I think this sever creates one actor per one connection. I think that way is good for perfomance in terms of concurrent process of connections.