mccutchen / go-httpbin Goto Github PK
View Code? Open in Web Editor NEWA reasonably complete and well-tested golang port of httpbin, with zero dependencies outside the go stdlib.
Home Page: https://httpbingo.org
License: MIT License
A reasonably complete and well-tested golang port of httpbin, with zero dependencies outside the go stdlib.
Home Page: https://httpbingo.org
License: MIT License
This library is a great candidate for integration into custom applications for testing purposes. Unfortunately it hardcodes the global logger which makes using it harder.
We're using go-httpbin as a part of our tests in apache/trafficserver. We noticed a difference in behavior between httpbin.org and go-httpbin with respect to Expect: 100-continue
:
https://www.rfc-editor.org/rfc/rfc7231#section-5.1.1
Essentially, httpbin.org replies with a 100 Continue
response when the Expect: 100-continue
header is received, while go-httpbin seems to ignore it. It could be valuable to add Expect: 100-continue
support.
Thanks for go-httpbin. It has been helpful to us.
300 is "Multiple Choice" but no multiple choices are given. Unfortunately how these choices are to be made was never really written into the spec.
I was hoping to use go-httpbin today to figure out what the ip of my load balancer actually comes through as, but it was masked by the cloudflare CDN the request went through
Could you return the .RemoteAddr
field somewhere in the response?
Code
Lines 44 to 66 in 00a70e7
In the last 2-3 days we've been seeing quite a few instances of timeouts, 'no reply', incomplete bodies and other stuff when accessing the httpbingo.org
API, causing most of our CI runs to fail. Just thought I'd note here in case there's DDOSing or anything like that causing these issues.
The answers between httpbin and httpbingo differ. httpbingo will always return an array for each parameter or header, while httpbin will return simple objects and use sub arrays only when needed. I think your behaviour is maybe a more sensible approach, but it's harder to read for humans and makes it hard to use httpbingo as a drop-in replacement for httpbin.
curl -s 'http://httpbin.org/get?foo=bar&foo=baz&what=this' |jq
{
"args": {
"foo": [
"bar",
"baz"
],
"what": "this"
},
...
$ curl -s 'http://httpbingo.org/get?foo=bar&foo=baz&what=this' |jq
{
"args": {
"foo": [
"bar",
"baz"
],
"what": [
"this"
]
},
...
/status/1xx results in failures when tested. They are unusual codes but should be testable regardless since they are in the spec.
I'm planning on using go-httpbin for integration tests of an HTTP intercepting proxy. Thanks for maintaining it!
I will come up with some more "exotic" feature requests and have to see how easy it is to add those myself.
I'd love to see support for chunked encoding with trailers.
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Transfer-Encoding
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Trailer
https://developer.mozilla.org/en-US/docs/Web/HTTP/Headers/Server-Timing
Here's a rough idea:
/chunked/:n
Generates a chunked response with n
random chunks of variable size, a Transfer-Encoding: chunked
header and an optional Server-Timing
trailer
Parameters:
minsize=1
: Min number of random bytesmaxsize=1024
: Max number of random bytestrailer=1
: Include the Trailer: Server-Timing
header and Server-Timing
trailerDoes that sound like something you would accept? Do you see any problems? I don't really care about Server-Timing
directly, but I need any trailer. Foo
would do but I thought we might as well use something that's actually used IRL and has some browser support.
To support max-body-size
we could just check the worst case maxsize * n
before doing anything.
The /headers
endpoint seem to sort the request headers alphabetically. I was trying to use httpbin to research on HTTP clients fingerprinting which makes use of headers case and order.
Could you help to add a -v
option to make httpbin output detailed request and response information (like curl -v
)? That would make the program more useful. I searched around and found that the httputil package has DumpRequest
and DumpResponse
methods, which could be used for this feature.
Btw, thanks for creating such a useful tool!
It's up to you to decide of course, but I think this should be considered a bug:
$ curl -s 'https://httpbingo.org/anything?one=two' | jq .data
"data:application/octet-stream;base64,"
The data
field for a GET
request, as I understand it, should always be empty.
update: now I looked at the tests and I see you're emulating httpbin, which will handle GETs with bodies. Which is weird! But if the request doesn't have a body, it still should have an empty data
field.
(Not that you need to match httpbin byte for byte, but it does return the empty string here)
Finding this project very useful! Super easy to deploy and use for CI. One endpoint that seems to be broken is the /drip
endpoint. When I fetch it with libcurl, the progress callback shows that I only get one response packet. By comparison, when I use https://httpbin.org/drip
I get ten packets of a byte each.
While looking into #40 I made the unpleasant (and quite tardy) discover that Google App Engine explicitly disallows streaming responses like those generated by the /drip
, /stream
, and /stream-bytes
endpoints. Per the docs:
App Engine does not support streaming responses where data is sent in incremental chunks to the client while a request is being processed. All data from your code is collected as described above and sent as a single HTTP response.
That's unfortunate, because App Engine is otherwise a fairly nice place to host a project like this: free to run at current (extremely minimal) request volumes, but with autoscaling built in on the off chance it becomes the next Facebook.
Hi, I don't seem to find an option to test the PATCH method as the old httpbin had.
Is it available somehow?
Thanks
Hello!
Availability of httpbin was not that great recently, and seems like it not really maintained.
We were looking for alternatives and go-httpbin looks great!
We need to have some service for our demos, for doing various tests, offer it as test url for our trial user first experience, right in UI etc.
Initially we wanted to have it hosted just under our own domain like httpbin.tyk.io, so we can ensure quality of its availability, but I wonder if we can just offer you help of hosting your service https://httpbingo.org, and take care of its availability?
Cheers!
It's relatively easy to implement an endpoint that will use Server-Sent Events to stream a response to a client, and it's also relatively easy to write HTTP proxy code that does not correctly support SSE responses.
So, we should expose an endpoint in go-httpbin that makes it easier to test/validate that proxies correctly handle SSE.
As proposed by @x70b1 in #84, updating the Dockerfile to set the go-httpbin
binary as the ENTRYPOINT
rather than the CMD
- CMD ["/bin/go-httpbin"]
+ ENTRYPOINT ["/bin/go-httpbin"]
would be a small, useful UX improvement by allowing folks who run the image using command line arguments to slightly simplify their workflows:
- nerdctl run mccutchen/go-httpbin go-httpbin -use-real-hostname
+ nerdctl run mccutchen/go-httpbin -use-real-hostname
I'm currently a little reluctant to do this out of fear of unexpectedly breaking existing uses in CI processes if they're not pinning to a specific release, but it's a good candidate to include in any future breaking change we need to make.
httpbingo.org currently redirects all requests form HTTP to HTTPS, making explicit HTTP tests impossible. For maximum compatibility with httpbin.org it should answer in exactly the protocol that was requested.
We got a security alert regarding this line of code in the doRedirect()
helper function:
go-httpbin/httpbin/handlers.go
Line 960 in 8f905de
Redirect URLs should be checked to ensure that user input cannot cause a site to redirect to arbitrary domains. This is often done with a check that the redirect URL begins with a slash, which most of the time is an absolute redirect on the same host. However, browsers interpret URLs beginning with // or /\ as absolute URLs. For example, a redirect to //example.com will redirect to https://example.com. Thus, redirect checks must also check the second character of redirect URLs.
Correct:
curl -v "http://httpbin.org/range/100?duration=6&chunk_size=5" -r1-67 -N
The content is returned in sections.
Wrong:
curl -v "http://httpbingo.org/range/100?duration=6&chunk_size=5" -r1-67 -N
The content is returned at once
Original httpbin supports requests like /status/200,201,400
and /status/200:0.6,201:0.3,400:0.1
, and will return a weighted random choice from the specified codes.
Some quick notes on the upstream implementation:
weighted_choice
implementation)Independent of the support for chunked responses with issue#72, it would be nice to have the the Transfer-Encoding
header being returned when the client sends it.
Unless I am missing something, testing the client for sending this request header (i.e. chunking the request), is therefore currently not possible. This feature exists on the original httpbin.org instance by Kenneth Reitz, as evident by calling
curl -X POST ... -d '{}' -H 'Content-Type: application/json' -H 'Transfer-Encoding: chunked'
with httpbin.org/post and httpbingo.org/post.
Hey,
We've found a Cross-site Scripting (XSS) - Reflected at https://httpbingo.org
POC
navigate to URL :
https://httpbingo.org/base64/ZXhhbXBsZS5vcmciPjxzdmcvb25sb2FkPXByb21wdCgneHNzJyk+
xss will pop up
Impact
attacker is able to execute javascript code on users
I would like to know if it would be possilbe to create a version of the docker image with version 2.4.0 because a git tag already exists.
Can I support in any way to help here?
It can return some env to user, like
set HTTPBIN_VERSION=v1 for go-httpbin adn request /env, it can be return :
{
"envs":{
"version":"v1"
}
}
Hi @mccutchen, first of all thanks for this great project! I am using it in tests in the https://github.com/adobe/helix-fetch project.
I noticed that https://httpbingo.org
is not reachable anymore. Is this temporary?
Hi there,
I build a dumb (quick) web server like httpbin that (for the moment) mostly reply with malformed data. It was really useful to me to tests some scenarios.
For the moment, I have:
I also plan to add some ping/pong GRPC endpoint (It may break your only require http package
).
I was wondering if that could be of interest if I made a PR (after a rewrite to compete with your clean code) or if i'm better fork your code and live on my own ?
Thanks
Rather than using the redirect and having the client make another request to the new url - provide another endpoint which will forward the original request to the provided url.
Not sure if this is a deliberate or an accidental change, but starting around 2022-Nov-11, https://httpbingo.org/redirect-to started returning HTTP-403 errors instead of the expected HTTP-302:
% curl -D - 'https://httpbingo.org/redirect-to?url=https://example.com'
HTTP/2 403
access-control-allow-credentials: true
access-control-allow-origin: *
content-type: text/plain; charset=utf-8
x-content-type-options: nosniff
date: Wed, 16 Nov 2022 15:56:03 GMT
content-length: 51
server: Fly/5f2bf728 (2022-11-11)
via: 2 fly.io
fly-request-id: 01GJ0JF2S5RG7Z8EV44KC0GJQ6-iad
Forbidden redirect URL. Be careful with this link.
Compare to https://httpbin.org:
% curl -D - 'https://httpbin.org/redirect-to?url=https://example.com'
HTTP/2 302
date: Wed, 16 Nov 2022 15:56:39 GMT
content-type: text/html; charset=utf-8
content-length: 0
location: http://example.com
server: gunicorn/19.9.0
access-control-allow-origin: *
access-control-allow-credentials: true
The cookie created using cookies/set?foo=bar
isn't deleted by calling cookies/delete?foo
I've dismissed a few Reflected cross-site scripting code scanning alerts (e.g. 1, 2, 6) as irrelevant because the error responses generated by go-httpbin on bad input are served as Content-Type: text/plain
and therefore pose no XSS risk.
We should codify that in the test suite, so if in the future we ever decide to change it we can make sure we're taking appropriate steps to guard against (re)introducing XSS issues.
Hi, started using your httpbin thank you for porting this work to Go!
I have been having a problem when using this solution for headers.
I am wondering if you would be OK with simply returning all headers as a string like httpbin instead of setting as list types. It looks like that is the default behavior for httpbin.org
$ curl -H 'foo: bar' -H 'foo: baz' https://httpbin.org/get
{
"args": {},
"headers": {
"Accept": "*/*",
"Foo": "bar,baz", <===========================
"Host": "httpbin.org",
"User-Agent": "curl/7.54.0"
},
...
"url": "https://httpbin.org/get"
}
vs
$ curl -H 'foo: bar' -H 'foo: baz' https://httpbin.mccutchen.awesome.com/get
{
"url" : "https://httpbin.mccutchen.awesome.com/get",
"origin" : "100.112.202.128",
"args" : {},
"headers" : {
"Foo" : [ "bar", "baz" ], <===============================
}
}
Can we enable basic auth for all methods (GET, POST, PUT, DELETE, PATCH) by adding all methods on httpbin:
mux.HandleFunc("/basic-auth/", methods(h.BasicAuth, "GET", "POST", "PUT", "DELETE", "PATCH"))
As discovered in #89 (comment), the original httpbin implementation takes an interesting approach to handling binary data that we might want to match. Copying from that comment:
Let's look at what happens when we post binary data (in this case a 1x1 png image).
Original httpbin returns it as a base64-encoded data:
URL:
$ curl -sSL --data-binary @tiny.png -H "Content-Type: image/png" https://httpbin.org/post | jq .data
"data:application/octet-stream;base64,iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAAAAAA6fptVAAAACklEQVQIW2O4AwAA3gDddw4iTwAAAABJRU5ErkJggg=="
go-httpbin returns it basically as-is, accounting for encoding the bytes in JSON-compatible format:
$ curl -sSL --data-binary @tiny.png -H "Content-Type: image/png" https://httpbingo.org/post | jq .data
"�PNG\r\n\u001a\n\u0000\u0000\u0000\rIHDR\u0000\u0000\u0000\u0001\u0000\u0000\u0000\u0001\b\u0000\u0000\u0000\u0000:~�U\u0000\u0000\u0000\nIDAT\b[c�\u0003\u0000\u0000�\u0000�w\u000e\"O\u0000\u0000\u0000\u0000IEND�B`�"
Given the prefix "/foo", httpbin will render its default page only at "/foo/" and not just "/foo".
A /ws
endpoint that provides a basic WebSockets-based echo server would be a useful addition, if it's not too difficult to implement without pulling in any external dependencies. MDN's guide to writing a WebSocket server might be a good starting point.
This would be very useful to help test HTTP proxies to ensure that they correctly support WebSockets, similar to the SSE endpoint proposed in #150.
We could potentially leverage something like https://github.com/crossbario/autobahn-testsuite to verify the basic functionality of the implementation, ideally wired up into our CI test suite.
Love httpbingo & httpbingo.org, thanks for creating! How about an endpoint that fails intermittently? That would be handy for testing retries and such. Perhaps /intermittent
fails 75% of the time, returning HTTP 500?
go.mod:
...
require (
github.com/mccutchen/go-httpbin v2.1.0
)
$ go mod verify
go: finding github.com/mccutchen/go-httpbin v2.1.0
go: errors parsing go.mod:
[path redacted]/go.mod:5: invalid module version "v2.1.0": unknown revision v2.1.0
Removing v
prefix from version fixes situation just partially. go mod verify
Succecedes on finding version but the next call of go mod tidy
resets requirement to latest revision on master branch:
$ go mod verify
go: finding github.com/mccutchen/go-httpbin 2.1.0
all modules verified
$ go mod tidy
$ cat go.mod|grep httpbin
github.com/mccutchen/go-httpbin v0.0.0-20190116014521-c5cb2f4802fa
My guess is go mod
expects version tags to have v
prefix which is missing here.
First, would love to say great work on the port! It's been well needed.
Both /status/301 and /status/307 have a location field but 308 does not. The entire point of 308 is to act the same as 301 but without the possible method switching problems.
Hello any idea why this string fails to decode?
🙋, 🌍!\n…
It seems to work fine on command line echo "8J+Ziywg8J+MjSEK4oCm"|base64 --decode
but https://httpbingo.org/base64/decode/8J+Ziywg8J+MjSEK4oCm returns an error.
I guess it rejects certain encodes?
The following httpbin.org features (besides /brotli
) are not present in the library:
now.httpbin.org The current time, in a variety of formats.
(this probably will be more useful as a /now
url...)/uuid Returns UUID4.
/anything Returns request data, including method used.
/anything/:anything Returns request data, including the URL.
/base64/:value Decodes base64url-encoded string.
None of them are a big deal, just posting this for completion. If I have the time, I may implement some of them, though I doubt that will be very soon.
I got a different response.
Original
curl https://httpbin.org/response-headers\?access_token\=faketoken\&token_type\=Bearer
{
"Content-Length": "128",
"Content-Type": "application/json",
"access_token": "faketoken",
"token_type": "Bearer"
}
go-httpbin
curl http://httpbin:9100/response-headers\?access_token\=faketoken\&token_type\=Bearer
{
"access_token": [
"faketoken"
],
"token_type": [
"Bearer"
]
}
API /IP
return server ip, so it shoud be :
{
"origin":"xxx",
"server":"172.0.2.1"
}
Querying against the /bytes/0
endpoint should return a 0 byte body, but instead returns a 1 byte body.
Tested with go-httpbin version 2.5.6:
go-httpbin Server:
$ go-httpbin -host 127.0.0.1 -port 61000
go-httpbin listening on http://127.0.0.1:61000
time="2023-02-28T17:36:51.9741" status=200 method="GET" uri="/bytes/0" size_bytes=1 duration_ms=0.12 user_agent="curl/7.85.0-DEV" client_ip=127.0.0.1:55130
Curl client:
$ curl -vs http://127.0.0.1:61000/bytes/0
* Trying 127.0.0.1:61000...
* Connected to 127.0.0.1 (127.0.0.1) port 61000 (#0)
> GET /bytes/0 HTTP/1.1
> Host: 127.0.0.1:61000
> User-Agent: curl/7.85.0-DEV
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 200 OK
< Access-Control-Allow-Credentials: true
< Access-Control-Allow-Origin: *
< Content-Type: application/octet-stream
< Date: Tue, 28 Feb 2023 17:36:51 GMT
< Content-Length: 1
<
* Connection #0 to host 127.0.0.1 left intact
Note that both go-httpbin (size_bytes=1
) and curl (Content-Length: 1
) indicate that go-httpbin returned a 1 byte response. The expected response would have 0 bytes.
I took a packet dump and it looks like go-httpbin is responding with a 1 byte body of value 0x48
, ascii H
.
do you think you can start pushing multi-arch images to docker-hub ala'
docker buildx build --platform linux/amd64,linux/arm64,linux/arm/v7
When I request /base64/dGVzdC1pbWFnZQ==
, I get a 12 byte response back, including two trailing 0x00
bytes. IMHO, dGVzdC1pbWFnZQ==
should decode to test-image
, no trailing 0-bytes.
curl -i http://localhost:8080/base64/dGVzdC1pbWFnZQ== | hexdump -C
00000000 48 54 54 50 2f 31 2e 31 20 32 30 30 20 4f 4b 0d |HTTP/1.1 200 OK.|
00000010 0a 41 63 63 65 73 73 2d 43 6f 6e 74 72 6f 6c 2d |.Access-Control-|
00000020 41 6c 6c 6f 77 2d 43 72 65 64 65 6e 74 69 61 6c |Allow-Credential|
00000030 73 3a 20 74 72 75 65 0d 0a 41 63 63 65 73 73 2d |s: true..Access-|
00000040 43 6f 6e 74 72 6f 6c 2d 41 6c 6c 6f 77 2d 4f 72 |Control-Allow-Or|
00000050 69 67 69 6e 3a 20 2a 0d 0a 43 6f 6e 74 65 6e 74 |igin: *..Content|
00000060 2d 54 79 70 65 3a 20 74 65 78 74 2f 70 6c 61 69 |-Type: text/plai|
00000070 6e 0d 0a 44 61 74 65 3a 20 46 72 69 2c 20 30 35 |n..Date: Fri, 05|
00000080 20 4d 61 79 20 32 30 32 33 20 31 36 3a 35 35 3a | May 2023 16:55:|
00000090 32 37 20 47 4d 54 0d 0a 43 6f 6e 74 65 6e 74 2d |27 GMT..Content-|
000000a0 4c 65 6e 67 74 68 3a 20 31 32 0d 0a 0d 0a 74 65 |Length: 12....te|
000000b0 73 74 2d 69 6d 61 67 65 00 00 |st-image..|
000000ba
It would be beneficial if there would be an available endpoint which implements rate limiting.
If too many requests would be sent to /rate-limit
, the endpoint would return HTTP-429
with header Retry-After
.
It would be useful for testing adaptive methods of limiting.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.