mccutchen / go-httpbin Goto Github PK

First, would love to say great work on the port! It's been well needed.

Both /status/301 and /status/307 have a location field but 308 does not. The entire point of 308 is to act the same as 301 but without the possible method switching problems.

Hi, started using your httpbin thank you for porting this work to Go!

I have been having a problem when using this solution for headers.

I am wondering if you would be OK with simply returning all headers as a string like httpbin instead of setting as list types. It looks like that is the default behavior for httpbin.org

$ curl -H 'foo: bar' -H 'foo: baz' https://httpbin.org/get
{
  "args": {},
  "headers": {
    "Accept": "*/*",
    "Foo": "bar,baz",        <===========================
    "Host": "httpbin.org",
    "User-Agent": "curl/7.54.0"
  },
  ...
  "url": "https://httpbin.org/get"
}

vs

$ curl -H 'foo: bar' -H 'foo: baz' https://httpbin.mccutchen.awesome.com/get
{
   "url" : "https://httpbin.mccutchen.awesome.com/get",
   "origin" : "100.112.202.128",
   "args" : {},
   "headers" : {
      "Foo" : [ "bar", "baz" ],     <===============================

   }
}

The /headers endpoint seem to sort the request headers alphabetically. I was trying to use httpbin to research on HTTP clients fingerprinting which makes use of headers case and order.

API /IP return server ip, so it shoud be :

{
"origin":"xxx",
"server":"172.0.2.1"
}

Hi, I don't seem to find an option to test the PATCH method as the old httpbin had.
Is it available somehow?

Thanks

We're using go-httpbin as a part of our tests in apache/trafficserver. We noticed a difference in behavior between httpbin.org and go-httpbin with respect to Expect: 100-continue:

https://www.rfc-editor.org/rfc/rfc7231#section-5.1.1

Essentially, httpbin.org replies with a 100 Continue response when the Expect: 100-continue header is received, while go-httpbin seems to ignore it. It could be valuable to add Expect: 100-continue support.

Thanks for go-httpbin. It has been helpful to us.

Rather than using the redirect and having the client make another request to the new url - provide another endpoint which will forward the original request to the provided url.

Correct:
curl -v "http://httpbin.org/range/100?duration=6&chunk_size=5" -r1-67 -N

The content is returned in sections.

Wrong:
curl -v "http://httpbingo.org/range/100?duration=6&chunk_size=5" -r1-67 -N

The content is returned at once

do you think you can start pushing multi-arch images to docker-hub ala'

docker buildx build --platform linux/amd64,linux/arm64,linux/arm/v7

https://www.docker.com/blog/multi-arch-images/

It's up to you to decide of course, but I think this should be considered a bug:

$ curl -s 'https://httpbingo.org/anything?one=two' | jq .data
"data:application/octet-stream;base64,"

The data field for a GET request, as I understand it, should always be empty.

update: now I looked at the tests and I see you're emulating httpbin, which will handle GETs with bodies. Which is weird! But if the request doesn't have a body, it still should have an empty data field.

(Not that you need to match httpbin byte for byte, but it does return the empty string here)

Let's use this as a place to track compatibility with original httpbin:

Should fix

• #141
• #172

Fixed

• #89
• #90
• #123
• #125
• #145
• #163

Won't fix

• #15
• #37
• Expect: 100-continue with empty request body

Hello!

Availability of httpbin was not that great recently, and seems like it not really maintained.
We were looking for alternatives and go-httpbin looks great!

We need to have some service for our demos, for doing various tests, offer it as test url for our trial user first experience, right in UI etc.

Initially we wanted to have it hosted just under our own domain like httpbin.tyk.io, so we can ensure quality of its availability, but I wonder if we can just offer you help of hosting your service https://httpbingo.org, and take care of its availability?

Cheers!

As discovered in #89 (comment), the original httpbin implementation takes an interesting approach to handling binary data that we might want to match. Copying from that comment:

Let's look at what happens when we post binary data (in this case a 1x1 png image).

Original httpbin returns it as a base64-encoded data: URL:

$ curl -sSL --data-binary @tiny.png -H "Content-Type: image/png"  https://httpbin.org/post | jq .data
"data:application/octet-stream;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAAAAAA6fptVAAAACklEQVQIW2O4AwAA3gDddw4iTwAAAABJRU5ErkJggg=="

go-httpbin returns it basically as-is, accounting for encoding the bytes in JSON-compatible format:

$ curl -sSL --data-binary @tiny.png -H "Content-Type: image/png"  https://httpbingo.org/post | jq .data
"�PNG\r\n\u001a\n\u0000\u0000\u0000\rIHDR\u0000\u0000\u0000\u0001\u0000\u0000\u0000\u0001\b\u0000\u0000\u0000\u0000:~�U\u0000\u0000\u0000\nIDAT\b[c�\u0003\u0000\u0000�\u0000�w\u000e\"O\u0000\u0000\u0000\u0000IEND�B`�"

I got a different response.

Original
 curl https://httpbin.org/response-headers\?access_token\=faketoken\&token_type\=Bearer   
{
  "Content-Length": "128", 
  "Content-Type": "application/json", 
  "access_token": "faketoken", 
  "token_type": "Bearer"
}

go-httpbin
 curl http://httpbin:9100/response-headers\?access_token\=faketoken\&token_type\=Bearer  
{
  "access_token": [
    "faketoken"
  ],
  "token_type": [
    "Bearer"
  ]
}

The cookie created using cookies/set?foo=bar isn't deleted by calling cookies/delete?foo

Could you help to add a -v option to make httpbin output detailed request and response information (like curl -v)? That would make the program more useful. I searched around and found that the httputil package has DumpRequest and DumpResponse methods, which could be used for this feature.

Btw, thanks for creating such a useful tool!

This library is a great candidate for integration into custom applications for testing purposes. Unfortunately it hardcodes the global logger which makes using it harder.

I was hoping to use go-httpbin today to figure out what the ip of my load balancer actually comes through as, but it was masked by the cloudflare CDN the request went through

Could you return the .RemoteAddr field somewhere in the response?

Code

It's relatively easy to implement an endpoint that will use Server-Sent Events to stream a response to a client, and it's also relatively easy to write HTTP proxy code that does not correctly support SSE responses.

So, we should expose an endpoint in go-httpbin that makes it easier to test/validate that proxies correctly handle SSE.

When I request /base64/dGVzdC1pbWFnZQ==, I get a 12 byte response back, including two trailing 0x00 bytes. IMHO, dGVzdC1pbWFnZQ== should decode to test-image, no trailing 0-bytes.

curl -i  http://localhost:8080/base64/dGVzdC1pbWFnZQ== | hexdump -C
00000000  48 54 54 50 2f 31 2e 31  20 32 30 30 20 4f 4b 0d  |HTTP/1.1 200 OK.|
00000010  0a 41 63 63 65 73 73 2d  43 6f 6e 74 72 6f 6c 2d  |.Access-Control-|
00000020  41 6c 6c 6f 77 2d 43 72  65 64 65 6e 74 69 61 6c  |Allow-Credential|
00000030  73 3a 20 74 72 75 65 0d  0a 41 63 63 65 73 73 2d  |s: true..Access-|
00000040  43 6f 6e 74 72 6f 6c 2d  41 6c 6c 6f 77 2d 4f 72  |Control-Allow-Or|
00000050  69 67 69 6e 3a 20 2a 0d  0a 43 6f 6e 74 65 6e 74  |igin: *..Content|
00000060  2d 54 79 70 65 3a 20 74  65 78 74 2f 70 6c 61 69  |-Type: text/plai|
00000070  6e 0d 0a 44 61 74 65 3a  20 46 72 69 2c 20 30 35  |n..Date: Fri, 05|
00000080  20 4d 61 79 20 32 30 32  33 20 31 36 3a 35 35 3a  | May 2023 16:55:|
00000090  32 37 20 47 4d 54 0d 0a  43 6f 6e 74 65 6e 74 2d  |27 GMT..Content-|
000000a0  4c 65 6e 67 74 68 3a 20  31 32 0d 0a 0d 0a 74 65  |Length: 12....te|
000000b0  73 74 2d 69 6d 61 67 65  00 00                    |st-image..|
000000ba

I would like to know if it would be possilbe to create a version of the docker image with version 2.4.0 because a git tag already exists.

Can I support in any way to help here?

Can we enable basic auth for all methods (GET, POST, PUT, DELETE, PATCH) by adding all methods on httpbin:
mux.HandleFunc("/basic-auth/", methods(h.BasicAuth, "GET", "POST", "PUT", "DELETE", "PATCH"))

Hi there,

I build a dumb (quick) web server like httpbin that (for the moment) mostly reply with malformed data. It was really useful to me to tests some scenarios.
For the moment, I have:

• send data before headers
• send duplicate headers
• delay answer

I also plan to add some ping/pong GRPC endpoint (It may break your only require http package ).

I was wondering if that could be of interest if I made a PR (after a rewrite to compete with your clean code) or if i'm better fork your code and live on my own ?

Thanks

Original httpbin supports requests like /status/200,201,400 and /status/200:0.6,201:0.3,400:0.1, and will return a weighted random choice from the specified codes.

Some quick notes on the upstream implementation:

• If not specified, weight is 1
• If specified, weights are parsed as floats, but there is no requirement that they sum to 1.0 or are otherwise limited to any particular range (see weighted_choice implementation)

httpbingo.org currently redirects all requests form HTTP to HTTPS, making explicit HTTP tests impossible. For maximum compatibility with httpbin.org it should answer in exactly the protocol that was requested.

It would be beneficial if there would be an available endpoint which implements rate limiting.

If too many requests would be sent to /rate-limit, the endpoint would return HTTP-429 with header Retry-After.

It would be useful for testing adaptive methods of limiting.

Independent of the support for chunked responses with issue#72, it would be nice to have the the Transfer-Encoding header being returned when the client sends it.

Unless I am missing something, testing the client for sending this request header (i.e. chunking the request), is therefore currently not possible. This feature exists on the original httpbin.org instance by Kenneth Reitz, as evident by calling

curl -X POST ... -d '{}'  -H 'Content-Type: application/json' -H 'Transfer-Encoding: chunked'

with httpbin.org/post and httpbingo.org/post.

/status/1xx results in failures when tested. They are unusual codes but should be testable regardless since they are in the spec.

We got a security alert regarding this line of code in the doRedirect() helper function:

Redirect URLs should be checked to ensure that user input cannot cause a site to redirect to arbitrary domains. This is often done with a check that the redirect URL begins with a slash, which most of the time is an absolute redirect on the same host. However, browsers interpret URLs beginning with // or /\ as absolute URLs. For example, a redirect to //example.com will redirect to https://example.com. Thus, redirect checks must also check the second character of redirect URLs.

It can return some env to user, like

set HTTPBIN_VERSION=v1 for go-httpbin adn request /env, it can be return :

{
  "envs":{
   "version":"v1"
  }
}

As proposed by @x70b1 in #84, updating the Dockerfile to set the go-httpbin binary as the ENTRYPOINT rather than the CMD

- CMD ["/bin/go-httpbin"]
+ ENTRYPOINT ["/bin/go-httpbin"]

would be a small, useful UX improvement by allowing folks who run the image using command line arguments to slightly simplify their workflows:

-  nerdctl run mccutchen/go-httpbin go-httpbin -use-real-hostname
+  nerdctl run mccutchen/go-httpbin -use-real-hostname

I'm currently a little reluctant to do this out of fear of unexpectedly breaking existing uses in CI processes if they're not pinning to a specific release, but it's a good candidate to include in any future breaking change we need to make.

Love httpbingo & httpbingo.org, thanks for creating! How about an endpoint that fails intermittently? That would be handy for testing retries and such. Perhaps /intermittent fails 75% of the time, returning HTTP 500?

Hey,

We've found a Cross-site Scripting (XSS) - Reflected at https://httpbingo.org

POC
navigate to URL :

https://httpbingo.org/base64/ZXhhbXBsZS5vcmciPjxzdmcvb25sb2FkPXByb21wdCgneHNzJyk+
xss will pop up

Impact
attacker is able to execute javascript code on users

A /ws endpoint that provides a basic WebSockets-based echo server would be a useful addition, if it's not too difficult to implement without pulling in any external dependencies. MDN's guide to writing a WebSocket server might be a good starting point.

This would be very useful to help test HTTP proxies to ensure that they correctly support WebSockets, similar to the SSE endpoint proposed in #150.

We could potentially leverage something like https://github.com/crossbario/autobahn-testsuite to verify the basic functionality of the implementation, ideally wired up into our CI test suite.

Hello any idea why this string fails to decode?

🙋, 🌍!\n…

It seems to work fine on command line echo "8J+Ziywg8J+MjSEK4oCm"|base64 --decode but https://httpbingo.org/base64/decode/8J+Ziywg8J+MjSEK4oCm returns an error.

I guess it rejects certain encodes?

The following httpbin.org features (besides /brotli) are not present in the library:

• now.httpbin.org The current time, in a variety of formats. (this probably will be more useful as a /now url...)
• /uuid Returns UUID4.
• /anything Returns request data, including method used.
• /anything/:anything Returns request data, including the URL.
• /base64/:value Decodes base64url-encoded string.

None of them are a big deal, just posting this for completion. If I have the time, I may implement some of them, though I doubt that will be very soon.

300 is "Multiple Choice" but no multiple choices are given. Unfortunately how these choices are to be made was never really written into the spec.

Given the prefix "/foo", httpbin will render its default page only at "/foo/" and not just "/foo".

I'm planning on using go-httpbin for integration tests of an HTTP intercepting proxy. Thanks for maintaining it!

I will come up with some more "exotic" feature requests and have to see how easy it is to add those myself.

I'd love to see support for chunked encoding with trailers.

https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Trailer
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Server-Timing

Here's a rough idea:

/chunked/:n Generates a chunked response with n random chunks of variable size, a Transfer-Encoding: chunked header and an optional Server-Timing trailer

Parameters:

• minsize=1: Min number of random bytes
• maxsize=1024: Max number of random bytes
• trailer=1: Include the Trailer: Server-Timing header and Server-Timing trailer

Does that sound like something you would accept? Do you see any problems? I don't really care about Server-Timing directly, but I need any trailer. Foo would do but I thought we might as well use something that's actually used IRL and has some browser support.

To support max-body-size we could just check the worst case maxsize * n before doing anything.

Not sure if this is a deliberate or an accidental change, but starting around 2022-Nov-11, https://httpbingo.org/redirect-to started returning HTTP-403 errors instead of the expected HTTP-302:

% curl -D - 'https://httpbingo.org/redirect-to?url=https://example.com'
HTTP/2 403 
access-control-allow-credentials: true
access-control-allow-origin: *
content-type: text/plain; charset=utf-8
x-content-type-options: nosniff
date: Wed, 16 Nov 2022 15:56:03 GMT
content-length: 51
server: Fly/5f2bf728 (2022-11-11)
via: 2 fly.io
fly-request-id: 01GJ0JF2S5RG7Z8EV44KC0GJQ6-iad

Forbidden redirect URL. Be careful with this link.

Compare to https://httpbin.org:

% curl -D - 'https://httpbin.org/redirect-to?url=https://example.com'
HTTP/2 302 
date: Wed, 16 Nov 2022 15:56:39 GMT
content-type: text/html; charset=utf-8
content-length: 0
location: http://example.com
server: gunicorn/19.9.0
access-control-allow-origin: *
access-control-allow-credentials: true

I've dismissed a few Reflected cross-site scripting code scanning alerts (e.g. 1, 2, 6) as irrelevant because the error responses generated by go-httpbin on bad input are served as Content-Type: text/plain and therefore pose no XSS risk.

We should codify that in the test suite, so if in the future we ever decide to change it we can make sure we're taking appropriate steps to guard against (re)introducing XSS issues.

Summary

Querying against the /bytes/0 endpoint should return a 0 byte body, but instead returns a 1 byte body.

Tested with go-httpbin version 2.5.6:

Reproduction Steps

go-httpbin Server:

$  go-httpbin -host 127.0.0.1 -port 61000
go-httpbin listening on http://127.0.0.1:61000
time="2023-02-28T17:36:51.9741" status=200 method="GET" uri="/bytes/0" size_bytes=1 duration_ms=0.12 user_agent="curl/7.85.0-DEV" client_ip=127.0.0.1:55130                                                                                                                                                       

Curl client:

$  curl -vs  http://127.0.0.1:61000/bytes/0
*   Trying 127.0.0.1:61000...
* Connected to 127.0.0.1 (127.0.0.1) port 61000 (#0)
> GET /bytes/0 HTTP/1.1
> Host: 127.0.0.1:61000
> User-Agent: curl/7.85.0-DEV
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Origin: *
< Content-Type: application/octet-stream
< Date: Tue, 28 Feb 2023 17:36:51 GMT
< Content-Length: 1
<
* Connection #0 to host 127.0.0.1 left intact

Note that both go-httpbin (size_bytes=1) and curl (Content-Length: 1) indicate that go-httpbin returned a 1 byte response. The expected response would have 0 bytes.

I took a packet dump and it looks like go-httpbin is responding with a 1 byte body of value 0x48, ascii H.

go.mod:

...
require (
	github.com/mccutchen/go-httpbin v2.1.0
)

$ go mod verify
go: finding github.com/mccutchen/go-httpbin v2.1.0
go: errors parsing go.mod:
[path redacted]/go.mod:5: invalid module version "v2.1.0": unknown revision v2.1.0

Removing v prefix from version fixes situation just partially. go mod verify Succecedes on finding version but the next call of go mod tidy resets requirement to latest revision on master branch:

$ go mod verify
go: finding github.com/mccutchen/go-httpbin 2.1.0
all modules verified
$ go mod tidy
$ cat go.mod|grep httpbin
	github.com/mccutchen/go-httpbin v0.0.0-20190116014521-c5cb2f4802fa

My guess is go mod expects version tags to have v prefix which is missing here.

In the last 2-3 days we've been seeing quite a few instances of timeouts, 'no reply', incomplete bodies and other stuff when accessing the httpbingo.org API, causing most of our CI runs to fail. Just thought I'd note here in case there's DDOSing or anything like that causing these issues.

Finding this project very useful! Super easy to deploy and use for CI. One endpoint that seems to be broken is the /drip endpoint. When I fetch it with libcurl, the progress callback shows that I only get one response packet. By comparison, when I use https://httpbin.org/drip I get ten packets of a byte each.

Hi @mccutchen, first of all thanks for this great project! I am using it in tests in the https://github.com/adobe/helix-fetch project.

I noticed that https://httpbingo.org is not reachable anymore. Is this temporary?

The answers between httpbin and httpbingo differ. httpbingo will always return an array for each parameter or header, while httpbin will return simple objects and use sub arrays only when needed. I think your behaviour is maybe a more sensible approach, but it's harder to read for humans and makes it hard to use httpbingo as a drop-in replacement for httpbin.

curl -s  'http://httpbin.org/get?foo=bar&foo=baz&what=this' |jq
{
  "args": {
    "foo": [
      "bar",
      "baz"
    ],
    "what": "this"
  },
...

$ curl -s  'http://httpbingo.org/get?foo=bar&foo=baz&what=this' |jq 
{
  "args": {
    "foo": [
      "bar",
      "baz"
    ],
    "what": [
      "this"
    ]
  },
...

While looking into #40 I made the unpleasant (and quite tardy) discover that Google App Engine explicitly disallows streaming responses like those generated by the /drip, /stream, and /stream-bytes endpoints. Per the docs:

App Engine does not support streaming responses where data is sent in incremental chunks to the client while a request is being processed. All data from your code is collected as described above and sent as a single HTTP response.

That's unfortunate, because App Engine is otherwise a fairly nice place to host a project like this: free to run at current (extremely minimal) request volumes, but with autoscaling built in on the off chance it becomes the next Facebook.