Giter VIP home page Giter VIP logo

oxy's Introduction

Vulcand

Vulcand is a programmatic extendable proxy for microservices and API management. It is inspired by Hystrix and powers Mailgun microservices infrastructure.

Focus and priorities

Vulcand is focused on microservices and API use-cases.

Features

  • Uses etcd as a configuration backend.
  • API and command line tool.
  • Pluggable middlewares.
  • Support for canary deployments, realtime metrics and resilience.

Vulcan diagram

Project info

documentation https://vulcand.github.io/
status Used in production@Mailgun on moderate workloads. Under active development.
discussions https://groups.google.com/d/forum/vulcan-proxy
roadmap roadmap.md
build status Build Status

Opentracing Support

Vulcand has support for open tracing via the Jaeger client libraries. Users who wish to use tracing support should use the --enableJaegerTracing flag and must either run the Jaeger client listening on localhost:6831/udp or set the environment variables JAEGER_AGENT_HOST and JAEGER_AGENT_POST. (See the Jaeger client libraries for all available configuration environment variables.)

When enabled vulcand will create 2 spans: one span called vulcand which covers the entire downstream request and another span called middleware which only spans the processing of the middleware before the request is routed downstream.

Aliased Expressions

When running vulcand in a kubernetes DaemonSet vulcand needs to know requests from the local node can match Host("localhost") rules. This --aliases flag allows an author of a vulcand DaemonSet to tell vulcand the name of the node it's currently running on, such that vulcand correctly routes requests for Host("localhost"). The --aliases flag allows the user to pass in multiple aliases separated by commas.

Example

$ vulcand --aliases 'Host("localhost")=Host("192.168.1.1")'

oxy's People

Contributors

a-palchikov avatar archis-polyverse avatar colynn avatar crholm avatar emilevauge avatar faascape avatar fotinakis avatar fracklen avatar gheibia avatar herver avatar horkhe avatar jbdoumenjou avatar jd3nn1s avatar jeremyschlatter avatar juliens avatar klizhentas avatar ldez avatar magicshui avatar matthewedge avatar mgjv avatar mmatur avatar mouminoux avatar nexomichael avatar nvartolomei avatar pquerna avatar rtribotte avatar somethingnew2-0 avatar sylr avatar westurner avatar wrotki avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

oxy's Issues

Trace plugin improvements

  • Compute and record request body hash
  • Add ability to capture all headers e.g. via special '*' parameter

buffer middleware still attempt the request even though client cancel it

httputil.ReverseProxy will return 502 BadGateway error if client cancel the request.

buffer middleware will retry if we configured retry with IsNetworkError() which capture 502 code.

If this 502 code is caused by client canceled request, it's not necessary to retry.

BTW, Nginx will return it's self-defined code 499 in this case.

No keepalives for streaming connections

Hey,

Because you "NewSingleHostReverseProxy(urlcpy)" in each connection in forwarder, the connections are never kept alive for streaming connections. This is an issue when backend server can't handle many open connections. For example when using docker for Mac: docker/for-mac#815

Ideally you'd just create one reverse proxy per ip address, or create custom ReverseProxy with director that changes target url dynamically

Change min go version to 1.8

Hi!

I'm trying to backport our fork of oxy.

And when I want to backport the websocket part, I'm facing an issue. I use the Clone method of tls.Config which appear in go 1.8.

WDYT about change the minimum go version to 1.8 (in travis) ?

forwarder as a fallback in circuit breaker

We've been looking at vulcand for our environment, but we're missing some functionality that we'd like to have.

We have a pattern of falling back to a second copy of a service (in another region) before giving up, i.e. the pattern is:

  1. try the local (read) service
  2. if fail (using circuit breaker) try the remote (read) service
  3. if fail -> error

Ideally, we'd handle that in the internals of vulcand, with a second backend or so, but that seems like a lot more change than necessary. We think it should be possible to make a new circuit breaker fallback using the forwarder in oxy that could do this job.

If we were to implement this, is there an appetite to accept a pull request of that nature?

fwd_websocket_test.go failed randomly

Travis CI

----------------------------------------------------------------------
FAIL: fwd_websocket_test.go:20: FwdSuite.TestWebSocketEcho
127.0.0.1:35895
[79 75 0 0]
1 [79 75 0 0] <nil>
fwd_websocket_test.go:61:
    c.Assert(runtime.NumGoroutine(), Equals, num)
... obtained int = 14
... expected int = 15
time="2018-03-13T14:51:37Z" level=error msg="vulcand/oxy/forward/websocket: Error dialing \"127.0.0.1:36593\": websocket: bad handshake with resp: 403 403 Forbidden"
2018/03/13 14:51:38 http: TLS handshake error from 127.0.0.1:46260: remote error: tls: bad certificate
time="2018-03-13T14:51:38Z" level=error msg="vulcand/oxy/forward/websocket: Error dialing \"127.0.0.1\": websocket: bad handshake with resp: 400 400 Bad Request"
OOPS: 19 passed, 1 FAILED
--- FAIL: TestFwd (1.76s)

Local

time="2018-03-22T11:47:40+08:00" level=error msg="vulcand/oxy/forward/websocket: Error dialing \"127.0.0.1:51863\": websocket: bad handshake with resp: 403 403 Forbidden"
2018/03/22 11:47:40 http: TLS handshake error from 127.0.0.1:51879: remote error: tls: bad certificate
time="2018-03-22T11:47:40+08:00" level=error msg="vulcand/oxy/forward/websocket: Error dialing \"127.0.0.1\": websocket: bad handshake with resp: 400 400 Bad Request"

----------------------------------------------------------------------
FAIL: fwd_websocket_test.go:257: FwdSuite.TestWebSocketUpgradeFailed

fwd_websocket_test.go:315:
    c.Assert(err, Equals, io.ErrUnexpectedEOF)
... obtained *net.OpError = &net.OpError{Op:"read", Net:"tcp", Source:(*net.TCPAddr)(0xc4204cc0c0), Addr:(*net.TCPAddr)(0xc4204cc0f0), Err:(*os.SyscallError)(0xc420376300)} ("read tcp 127.0.0.1:51888->127.0.0.1:51887: read: connection reset by peer")
... expected *errors.errorString = &errors.errorString{s:"unexpected EOF"} ("unexpected EOF")

OOPS: 19 passed, 1 FAILED
--- FAIL: TestFwd (1.36s)
FAIL

Websocket does not forward Ping

Hi, it seams that oxy does not forward some control messages but rather use gorilla/websocket defaults in handling message types such as Ping (code 9), Pong (code 10)

msgType, msg, err := src.ReadMessage()

Only really reads Text (code 1) or Binary (code 2)

https://github.com/gorilla/websocket/blob/master/conn.go#L944
Pings and Pongs are discarded, since gorilla uses a specific callback for this type of control messages.

Is there any particular reason for letting the proxy answer Ping/Pong messages? instead of the destination.

Forward to insecure https

When I forward to a self signed internal https server (dev environment) I get:

ERRO[0002] Error forwarding to https://localhost:3001, err: x509: certificate signed by unknown authority

I've gone through the docs and I can't see where it's possible to configure a tls.Config with InsecureSkipVerify set to true.

Is this possible?

Small bug with forwarder when path changes

Wanted to check the best way to fix this bug before starting a PR.

I'm using Forwarder as a hijacking reverse proxy which rewrites the destination URL (so, it doesn't just reverse proxy like normal to a new destination host, it changes the host and path).

I thought I could just do something like replace req.URL:

import (
	"github.com/vulcand/oxy/forward"
	"net/http"
	"net/url"
)

func proxyHandler(w http.ResponseWriter, req *http.Request) {
        // Replace every request with example.com.
	newUrl, _ := url.ParseRequestURI("http://example.com")
	req.URL = newUrl

	fwd, _ := forward.New()
	fwd.ServeHTTP(w, req)
}

func main() {
	http.HandleFunc("/", proxyHandler)
	http.ListenAndServe("127.0.0.1:8080", nil)
}

...except, that attempting to use this always returns a 400 Bad Request:

$ curl -I -x localhost:8080 http://foo/
HTTP/1.1 400 Bad Request

I was able to trace the bug to this line:

outReq.URL.Opaque = req.RequestURI

which does:

outReq.URL.Opaque = req.RequestURI

The bug: If you call outReq.URL.String() after that line you get: http:http://foo/, which obviously will die.

I've read the docs on how Opaque is used but don't quite understand what the best global fix is for this. I was able to work around it by just removing that line entirely — and now everything works, both normal reverse-proxying and hijacking the request URL. :)

$ curl -I -x localhost:8080 http://foo/
HTTP/1.1 200 OK

But some path tests break (but they also set some Opaque things manually), so I'm not sure what the right fix is here.

RedirectFallback does not preserve path

To be able to use the RedirectFallback in fallback.go for slightly more complex situations, it would be nice if it could be used to preserve the path of the current request on the redirect. For example, if the current request is

http://something/basepath/rest/of/path

and if redirect is set to

http://somethingelse/otherbasepath

then I'd expect the redirect to go to

http://somethingelse/otherbasepath/rest/of/path

I'd like to add some code to either redirectfallback, or create a new fallback method, that can support this. Would a change like this be acceptable?

Malicious user can access one server forever by forging cookie

Malicious user can access the specific server forever by setting backend server in cookie, it's very dangerous if the project is used in API gateway.

I will show how to reproduce the issue:

gateway code

package main

import (
	"net/http"
	"net/url"

	"github.com/vulcand/oxy/forward"
	"github.com/vulcand/oxy/roundrobin"
)

func main() {
	fwd, _ := forward.New()

	ss := roundrobin.NewStickySession("zzz")
	lb, err := roundrobin.New(fwd, roundrobin.EnableStickySession(ss))
	if err != nil {
		panic(err)
	}

	url1, _ := url.Parse("http://127.0.0.1:8001")
	url2, _ := url.Parse("http://127.0.0.1:8002")

	err = lb.UpsertServer(url1, roundrobin.Weight(1))
	err = lb.UpsertServer(url2, roundrobin.Weight(2))

	s := &http.Server{
		Addr:    ":8000",
		Handler: lb,
	}
	panic(s.ListenAndServe())
}

exploit

  • get one cookie

curl -i http://127.0.0.1:8000

    Set-Cookie: zzz=http://127.0.0.1:8001; Path=/
  • forge an request that access 8001 port forever

curl -H "Cookie: zzz=http://127.0.0.1:8001; Path=/" http://127.0.0.1:8000

even without Path

curl -H "Cookie: zzz=http://127.0.0.1:8001" http://127.0.0.1:8000

Do you think we should enhance it?

Measuring request latency when using forward package

This is related to the closed PR #31.

If you are working on providing additional information (backend URL) from requests made by forward, please consider adding information about request latency, meaning the time that it takes for the upstream server to return a response -- this is already printed in logs as "duration", but there seems to be no way to use this value outside forward.

Package "stream" is misleadingly named

github.com/mailgun/oxy/stream seems to do the opposite of streaming. "Reads the entire request and response into buffer, [...]"

Maybe it should really be named buffer?

TTL of ratelimit

bucketSetI, exists := tl.bucketSets.Get(source)
var bucketSet *TokenBucketSet
if exists {
bucketSet = bucketSetI.(*TokenBucketSet)
bucketSet.Update(effectiveRates)
} else {
bucketSet = NewTokenBucketSet(effectiveRates, tl.clock)
// We set ttl as 10 times rate period. E.g. if rate is 100 requests/second per client ip
// the counters for this ip will expire after 10 seconds of inactivity
tl.bucketSets.Set(source, bucketSet, int(bucketSet.maxPeriod/time.Second)*10+1)
}

As the comment above says:

E.g. if rate is 100 requests/second per client ip the counters for this ip will expire after 10 seconds of inactivity

However, I could not find where the TTL is reset, which means each bucketSet would been expired at its creation time + TTL even if some "source" is always active.

Logging URLs

Hi,
Simple project to print all the HTTP requests from my machine. I redirect to localhost:8080.
It logs the request like: www.reddit.com:443

But my browser doesn't return the requested page.

This site can’t be reached
The webpage at https://www.reddit.com might be temporarily down or it may have moved permanently to a new web address.
ERR_TUNNEL_CONNECTION_FAILED

What am I doing wrong? Thanks!

package main

import (
	"net/http"
	"github.com/vulcand/oxy/forward"
  )
  
 
func main() { 
	
	fwd, _ := forward.New()
	
	handler := http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
		println(r.Host, req.URL.Path)
		fwd.ServeHTTP(w, req)
	})
	
	s := &http.Server{
		Addr:           ":8080",
		Handler:        handler,
	}
	
	s.ListenAndServe()

}

Proper rate limiting headers

I was told to x-post this on here since it's more of an issue/feature with/for the oxy middleware.
traefik/traefik#3440

Do you want to request a feature or report a bug?

Feature

What did you expect to see?

The way the current rate limits are implemented are a bit of lackluster. I hope this can be improved upon.
Currently when you define your rate limits as suggested in the .yml or .toml file, you can specify a duration, an average and a burst. Additionally to an expression it should trigger on.

Now iIf you hit one of those rate limits in the loadbalancer it will return a string which says the following:

max rate reached: retry-in 30s

func (m *MaxRateError) Error() string {
return fmt.Sprintf("max rate reached: retry-in %v", m.delay)
}

While this may be sort of okay, the headers that get set are:

X-Retry-In: 30s

func (e *RateErrHandler) ServeHTTP(w http.ResponseWriter, req *http.Request, err error) {
if rerr, ok := err.(*MaxRateError); ok {
w.Header().Set("X-Retry-In", rerr.delay.String())
w.WriteHeader(429)
w.Write([]byte(err.Error()))
return
}
utils.DefaultHandler.ServeHTTP(w, req, err)
}

For users that want to properly build around those rate limits and throttle depending on how close you are to hitting them, it would be kind of nice to have a way of always sending rate limit headers?

Maybe something along the lines of on every request:

X-Ratelimit-Limit: 50
X-Ratelimit-Remaining: 20 // or 0 if you hit the ratelimit
X-Ratelimit-Reset: Unix Timestamps of reset

This way the users that interface with any service could properly integrate their application into those ratelimits.

Additionally, it would be nice to have a configurable response. Currently as mentioned above it returns a string, which can certainly throw off any implementation on a JSON based REST API since the response body would not match the expected JSON response.

Issue with the forward package

Hi,
I'm trying to implement a reverse proxy on top of the forward and roundrobin packages.
It works, but not as expected. When I bench the service with wrk, it starts to slow down the requests' processing after a few seconds, and starts to display error messages:

ERR: 2015/09/01 17:26:24.843441 Error forwarding to http://127.0.0.1:8082, err: dial tcp 127.0.0.1:8082: can't assign requested address

I created a full example here which reproduces the issue: https://gist.github.com/Arkan/3f344293fd79560a96c4

Do you have any idea?

Thanks

can't redirect to correct url

I started a program using forward in localhost, and it will be redirected to another url, just like:
localhost:9090---> localhost:8081/receive
but there is only a request to localhost:8081/. It lost receive.

then I modified the code in forward/fwd.go#copyRequest:

outReq.URL.Opaque = req.URL.Path + req.RequestURI // it's `/receive` + some params

and it worked.

But I think it may not be the best way to use this package.So could I get some suggestions?

Unable to proxy requests

Hi,

I am trying to proxy a request from serviceA/login to serviceB/token,

But I'm having 404 from the proxied service which is serviceB/token,

I noticed that serviceA/login successful proxies to serviceB/login, note the path login which is from serviceA/login, what I need is serviceA/token?q=foo&z=bar but I dunno how to exactly copy the request like I do with httputil.ReverseProxy{Director: director}

Here is the not working code using oxy/forward

func (h *AuthProxyHandler) Proxy(w http.ResponseWriter, r *http.Request) {
  target, _ := url.Parse(h.tokenURL)
  targetQuery := target.RawQuery
  r.URL.Scheme = target.Scheme
  r.URL.Host = target.Host
  r.URL.Path = target.Path
  if targetQuery == "" || r.URL.RawQuery == "" {
    r.URL.RawQuery = targetQuery + r.URL.RawQuery
  } else {
    r.URL.RawQuery = targetQuery + "&" + r.URL.RawQuery
  }
  fwd, _ := forward.New()
  fwd.ServeHTTP(w, r)
}

here is the working version using httputil.ReverseProxy

func (h *AuthProxyHandler) Proxy(w http.ResponseWriter, r *http.Request) {
	target, _ := url.Parse(h.tokenURL)
	targetQuery := target.RawQuery
	director := func(req *http.Request) {
		r.URL.Scheme = target.Scheme
		r.URL.Host = target.Host
		r.URL.Path = target.Path
		if targetQuery == "" || r.URL.RawQuery == "" {
			r.URL.RawQuery = targetQuery + r.URL.RawQuery
		} else {
			r.URL.RawQuery = targetQuery + "&" + r.URL.RawQuery
		}
	}
	p := &httputil.ReverseProxy{Director: director}
	p.ServeHTTP(w, r)
}

I've debugged the response, err := f.roundTripper.RoundTrip(f.copyRequest(req, req.URL)) code noticed it does it in a different manner,

What am I missing?

Thanks

Buffer tests not working on master

Since there is no CI on this project, I figured this out only when I cloned the project (to hack on it) and ran the tests.

EDIT: I noticed there is Travis CI but only for build, not for the tests.
EDIT2: It seems go1.5 works OK, but 1.6 and 1.7 do not - see #58

$ go test
INFO[0000] Round trip: http://127.0.0.1:56109, code: 200, duration: 628.619µs

----------------------------------------------------------------------
FAIL: buffer_test.go:94: BFSuite.TestChunkedEncodingLimitReached

buffer_test.go:122:
    c.Assert(status, Equals, "HTTP/1.0 413 Request Entity Too Large\r\n")
... obtained string = "HTTP/1.0 200 OK\r\n"
... expected string = "HTTP/1.0 413 Request Entity Too Large\r\n"

INFO[0000] Round trip: http://127.0.0.1:56113, code: 200, duration: 445.303µs

----------------------------------------------------------------------
FAIL: buffer_test.go:55: BFSuite.TestChunkedEncodingSuccess

buffer_test.go:89:
    c.Assert(reqBody, Equals, "testtest1test2")
... obtained string = ""
... expected string = "testtest1test2"

INFO[0000] Round trip: http://127.0.0.1:56117, code: 200, duration: 316.075µs
ERRO[0000] Error copying upstream response Body: total size of 33 exceeded allowed 4
ERRO[0000] failed to read response, err no data ready
INFO[0000] Round trip: http://127.0.0.1:56121, code: 200, duration: 394.326µs
INFO[0000] Round trip: http://127.0.0.1:56125, code: 200, duration: 404.963µs
INFO[0000] Round trip: http://127.0.0.1:56129, code: 304, duration: 302.626µs
INFO[0000] Round trip: http://127.0.0.1:56133, code: 200, duration: 447.161µs tls:version: 303, tls:resume:false, tls:csuite:c02f, tls:server:
INFO[0000] request body over limit: Maximum size &{4} was reached
INFO[0000] Round trip: http://127.0.0.1:56140, code: 200, duration: 392.388µs
ERRO[0000] Error copying upstream response Body: total size of 33 exceeded allowed 4
ERRO[0000] failed to read response, err no data ready
INFO[0000] Round trip: http://127.0.0.1:56144, code: 200, duration: 406.956µs
ERRO[0000] Error forwarding to http://localhost:64321, err: dial tcp [::1]:64321: getsockopt: connection refused
INFO[0000] retry Request(GET /) attempt 2
ERRO[0000] Error forwarding to http://localhost:64322, err: dial tcp [::1]:64322: getsockopt: connection refused
INFO[0000] retry Request(GET /) attempt 3
ERRO[0000] Error forwarding to http://localhost:64323, err: dial tcp [::1]:64323: getsockopt: connection refused
ERRO[0000] Error forwarding to http://localhost:64321, err: dial tcp [::1]:64321: getsockopt: connection refused
INFO[0000] retry Request(GET /) attempt 2
INFO[0000] Round trip: http://127.0.0.1:56157, code: 200, duration: 1.180046ms
INFO[0000] Round trip: http://127.0.0.1:56163, code: 200, duration: 853.58µs
OOPS: 11 passed, 2 FAILED
--- FAIL: TestBuffer (0.05s)
FAIL
exit status 1

forward to localhost returns 404

Do you know why my localhost is returning 404?

Here's the code:

// broken.go
package main

import (
  "fmt"
  "html"
  "net/http"
  "github.com/vulcand/oxy/forward"
  "github.com/vulcand/oxy/testutils"
  "github.com/urfave/negroni"
)

func main() {

  // Forwards incoming requests to whatever location URL points to,
  // adds proper forwarding headers
  fwd, _ := forward.New()

  redirect := func(w http.ResponseWriter, r *http.Request) {
    url := testutils.ParseURI("http://localhost:3838/sample-apps/hello")
    r.URL = url
    r.Host = url.Host
    fwd.ServeHTTP(w, r)
  }

  mux := http.NewServeMux()

  mux.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) {
    fmt.Fprintf(w, "Hello, %q", html.EscapeString(r.URL.Path))
  })

  mux.HandleFunc("/myapp", redirect)

  n := negroni.Classic() // Includes some default middlewares
  n.UseHandler(mux)

  http.ListenAndServe(":8011", n)
}
go run broken.go

[negroni] 2018-01-31T01:52:23Z | 200 |   97.89µs | localhost:8011 | GET /shiny
INFO[0006] vulcand/oxy/forward/http: Round trip: http://localhost:3838/sample-apps/hello, code: 404, Length: 849, duration: 6.862264ms
[negroni] 2018-01-31T01:52:27Z | 404 |   6.946216ms | localhost:3838 | GET /sample-apps/hello
^Csignal: interrupt

In contrast, this works perfectly well and the localhost app loads as it is supposed to:

// works.go
package main

import (
  "net/http"
  "github.com/vulcand/oxy/forward"
  "github.com/vulcand/oxy/testutils"
)

func main() {
  fwd, _ := forward.New()

  redirect := http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
    req.URL = testutils.ParseURI("http://localhost:3838/sample-apps/hello")
    fwd.ServeHTTP(w, req)
  })

  s := &http.Server{
      Addr:           ":8011",
      Handler:        redirect,
  }
  s.ListenAndServe()
}

Maintain Oxy

Hello oxy maintainers!
We use Oxy quite intensively on Traefik.
We had to make few fixes and enhancements in the past and we decided to fork because Oxy was inactive at that time.
It seems the project is more active today and it may be a good idea to merge back from our fork.
But we think it would be even better to help you maintain Oxy :) WDYT?

"Te" removed as hop-by-hop header in HTTP/2 - kills GRPC forwarding

Was trying to make GRPC work with Traefik, and after a deep rabbit-hole stumbled upon https://nghttp2.org/blog/2015/03/24/proxying-grpc-with-nghttpx/

This is the usual HTTP/2 request, but one thing to note is te: trailers. According to RFC 7230, section 4.3.:

The “TE” header field in a request indicates what transfer codings, besides chunked, the client is willing to accept in response, and whether or not the client is willing to accept trailer fields in a chunked transfer coding.

HTTP/2 disallows transfer-encoding stuff, but it only allows “trailers” in “TE” header field:

The presence of the keyword “trailers” indicates that the client is willing to accept trailer fields in a chunked transfer coding, as defined in Section 4.1.2, on behalf of itself and any downstream clients.

So in HTTP/1.1, trailer fields are only allowed in chunked transfer encoding. In HTTP/2, chunked transfer encoding is deprecated because it has more elegant framing and thanks to it, trailer fields are always allowed.

gRPC does not work without “TE” header field.

Is it possible to remove the "te" header from the HopHeader list?

dialer.DialContext undefined (type *websocket.Dialer has no field or method DialContext)

I get the following error whenever I'm trying to use the forward package: vulcand/oxy/forward/fwd.go:347:33: dialer.DialContext undefined (type *websocket.Dialer has no field or method DialContext). I understand that the error occurs because the function websocket.DefaultDialer.DialContext() does not exist (anymore?) in gorilla/websocket but I don't quite understand the code to fix it. If someone is willing to point out where to go from here, I'd really appreciate it.

Allow disabling logrus

First up, thanks for writing oxy! It's neat and helped us quite a bit so far.

However, we need the ability to disable logging from oxy, because we want to decide what to log ourselves.

To that end, we have a tiny fork that has a new github.com/vulcand/oxy/log package, which exposes only the parts of logrus' API that oxy uses, and delegates it to a package-internal *logrus.Logger. All imports of logrus have been changed to import the new package. In addition, it's now possible to .Disable() the logger, which sets the output to io.Discard.

Would you be interested in merging these changes? If not, would you be interested in merging changes that allow swapping out the logger?

And to the wider community: Do you also need to disable (or even change) the logger that oxy uses? If so, say so by :+1_tone3:ing this issue.

Feature request: rate limit on HTTP query parameter

Hello,

At Planet Labs we're considering replacing an internally developed edge router with Traefik. Our edge router is very similar to Traefik, including its use of oxy. We have one rate limiting use case that's not supported by oxy out of the box; we'd like to rate limit on HTTP URL query parameter. We need to rate limit on query parameters because some of our legacy clients provide their API key as a query param.

We'd effectively want to add the following:

// NewExtractor creates a new SourceExtractor
func NewExtractor(variable string) (SourceExtractor, error) {
	if variable == "client.ip" {
		return ExtractorFunc(extractClientIP), nil
	}
	if variable == "request.host" {
		return ExtractorFunc(extractHost), nil
	}
	if strings.HasPrefix(variable, "request.header.") {
		header := strings.TrimPrefix(variable, "request.header.")
		if len(header) == 0 {
			return nil, fmt.Errorf("wrong header: %s", header)
		}
		return makeHeaderExtractor(header), nil
	}
	if strings.HasPrefix(variable, "request.queryparam.") {
		param := strings.TrimPrefix(variable, "request.queryparam.")
		if len(param) == 0 {
			return nil, fmt.Errorf("wrong param: %s", header)
		}
		return makeQueryParamExtractor(param), nil
	}
	return nil, fmt.Errorf("unsupported limiting variable: '%s'", variable)
}

func makeQueryParamExtractor(param string) SourceExtractor {
	return ExtractorFunc(func(req *http.Request) (string, int64, error) {
		return req.URL.Query().Get(param), 1, nil
	})
}

Would you accept a PR with such functionality?

github.com/Sirupsen/logrus package name changed

Hi,

Several days ago package github.com/Sirupsen/logrus changed it's name to github.com/sirupsen/logrus.
Because of that using of vulcand/oxy as library package is breaking dependency tree.

It seems like simply imports update will fix all integration and futher issues

Client Host header not sent

Hi @klizhentas !

First thanks for this great library and open sourcing it. I am currently using it in a different project (https://github.com/martensson/moxy) and it's working great so far. But I encountered a deal breaking bug that would be nice to fix, which is that oxy does not respect the clients Host header being sent.
This causes problems when you want to route traffic to nodes that respond differently depending on vhost settings.

Being a transparent reverse proxy the library should maintain all header as they are unless its a way to override them at least. I checked the code and saw a quick fix to the problem, and there is a PR that you can check out.

If you believe there is a better way of doing it please tell me, if not I will have to fork oxy just because of this small fix, and I hope not :/

Thanks!

Best regards,
Benjamin

how to capture/ignore client-cancel-request error in buffer middleware

When I use buffer and reverseproxy to handle failover, I found that buffer can not handle the case that reverseproxy.go return 502 if client canceled the request (maybe the connection has gone).

Currently, as I have remove the strict check for expectBody, buffer return 500 at here

If I ignore this specific case, buffer received 502 from reverseproxy, it will attempt to resend the request.

My question is that is there any method to identify client-cancel-request error and gracefully ignored it (not logged 502/500 error)?

buffer: mailgun panic

panic: runtime error: slice bounds out of range

goroutine 277423 [running]:
io.(*multiReader).Read(0xc420a12320, 0xc421408000, 0x2000, 0x2000, 0x46bcb2, 0x1, 0x5313eb)
	/opt/go/src/io/multi.go:31 +0x2da
xxxxxxxx/vendor/github.com/mailgun/multibuf.(*multiReaderSeek).Read(0xc4203d7ec0, 0xc421408000, 0x2000, 0x2000, 0xc4299fe0a0, 0xc420c9b810, 0xc420612b58)
	/opt/build/src/xxxxxxxx/vendor/github.com/mailgun/multibuf/buffer.go:229 +0x52
io/ioutil.(*nopCloser).Read(0xc420f370a0, 0xc421408000, 0x2000, 0x2000, 0x0, 0xc420612ba0, 0x514107)
	<autogenerated>:1 +0x6b
io/ioutil.devNull.ReadFrom(0x0, 0x7fa9d1f15c68, 0xc420f370a0, 0x8d02c0, 0xc76201, 0x7fa9d1f15d08)
	/opt/go/src/io/ioutil/ioutil.go:144 +0x85
io/ioutil.(*devNull).ReadFrom(0xccf440, 0x7fa9d1f15c68, 0xc420f370a0, 0x7fa9d1f15d08, 0xccf440, 0x1)
	<autogenerated>:6 +0x61
io.copyBuffer(0xc782e0, 0xccf440, 0x7fa9d1f15c68, 0xc420f370a0, 0x0, 0x0, 0x0, 0x1d8, 0x0, 0x0)
	/opt/go/src/io/io.go:384 +0x2cb
io.Copy(0xc782e0, 0xccf440, 0x7fa9d1f15c68, 0xc420f370a0, 0x1d8, 0x0, 0x0)
	/opt/go/src/io/io.go:360 +0x68
net/http.(*transferWriter).WriteBody(0xc4202e5f00, 0xc75e20, 0xc420381900, 0x2, 0x2)
	/opt/go/src/net/http/transfer.go:324 +0x6c4
net/http.(*Request).write(0xc4211ccd00, 0xc75e20, 0xc420381900, 0x0, 0x0, 0x0, 0x0, 0x0)
	/opt/go/src/net/http/request.go:622 +0x6e9
net/http.(*persistConn).writeLoop(0xc420788fc0)
	/opt/go/src/net/http/transport.go:1707 +0x1ad
created by net/http.(*Transport).dialConn
	/opt/go/src/net/http/transport.go:1118 +0xa5a

Header set after sent in fwd.go

At this line.

The headers have already been set and sent over the wire here, so, as far as I can tell, updating the content-length has no effect. Am I missing something?

Does Buffer not support Transfer-Encoding ?

Try adding the following case to buffer_test.go to reproduce the issue

func (s *BFSuite) TestChunkedResponseConversion(c *C) {
       srv := testutils.NewHandler(func(w http.ResponseWriter, req *http.Request) {
               h := w.(http.Hijacker)
               conn, _, _ := h.Hijack()
               fmt.Fprintf(conn, "HTTP/1.1 200 OK\r\nTransfer-Encoding: chunked\r\n\r\n4\r\ntest\r\n5\r\ntest1\r\n5\r\ntest2\r\n0\r\n\r\n")
               conn.Close()
       })
       defer srv.Close()

       fwd, err := forward.New()
       c.Assert(err, IsNil)

       rdr := http.HandlerFunc(func(w http.ResponseWriter, req *http.Request) {
               req.URL = testutils.ParseURI(srv.URL)
               fwd.ServeHTTP(w, req)
       })
       st, err := New(rdr)
       c.Assert(err, IsNil)
       proxy := httptest.NewServer(st)

       defer proxy.Close()

       re, body, err := testutils.Get(proxy.URL)
       c.Assert(err, IsNil)
       c.Assert(string(body), Equals, "testtest1test2")
       c.Assert(re.StatusCode, Equals, http.StatusOK)
       c.Assert(re.Header.Get("Content-Length"), Equals, fmt.Sprintf("%d", len("testtest1test2")))
}

X-Forwarded-For wrong format for IPv6 address

In case if req.RemoteAddr contain IPv6 address in format with zone information, like:
"RemoteAddr":"[fe80::d806:a55d:eb1b:49cc%vEthernet (vmxnet3 Ethernet Adapter - Virtual Switch)]:64692"

net.SplitHostPort will do it best and split this address on host and port part, but according to RFC that defines IPv6 format for this header - https://tools.ietf.org/html/rfc3986#section-3.2.2 - it tells -

This syntax does not support IPv6 scoped addressing zone identifiers.

so I suggest to filter out clientIP in case if it contains "%" in it's value.

Pull Request - #72

Roundrobin overwrites request URL

It seems that using roundrobin in a way that is similar to the example in README does not work correctly -- or at least it does not work the way I expect it to.

We have the following code:

fwd, _ := forward.New()
lb, _ := roundrobin.New(fwd)

lb.UpsertServer(url1)
lb.UpsertServer(url2)

s := &http.Server{
    Addr:           ":8080",
    Handler:        lb,
}
s.ListenAndServe()

Where:

Let's consider the following scenario:

  1. Server receives request: http://server/path/to/a/file.txt
  2. Roundrobin gets a suitable server from the list using code and sets the URL for forwarder in: https://github.com/vulcand/oxy/blob/master/roundrobin/rr.go#L72
  3. The URL for forwarder is either: "http://backend1:8080/" or "http://backend2:8080/", while I would have expected it to be: "http://backendN:8080/path/to/a/file.txt"

I cannot see how the code is supposed to be used to achieve a trivial reverse proxy setup.

Is there something I've missed? :)

Concurrency issue in memmetrics

Hi!

It seems there is a concurrency issue in the memmetrics used by circuit breakers.
RTMetrics.statusCodes is accessed in rw without mutex:
https://github.com/vulcand/oxy/blob/master/memmetrics/roundtrip.go#L216

Here is a stack we got benchmarking traefik using cbreaker:

error: concurrent map writes

goroutine 911054 [running]:
runtime.throw(0xca0dc0, 0x15)
        /usr/local/go/src/runtime/panic.go:530 +0x90 fp=0xc8206493a8 sp=0xc820649390
runtime.mapassign1(0x9e7220, 0xc820dac870, 0xc820649478, 0xc820649488)
        /usr/local/go/src/runtime/hashmap.go:445 +0xb1 fp=0xc820649450 sp=0xc8206493a8
github.com/containous/traefik/vendor/github.com/mailgun/oxy/memmetrics.(*RTMetrics).recordStatusCode(0xc82041e940, 0xc8, 0x0, 0x0)
        /go/src/github.com/containous/traefik/vendor/github.com/mailgun/oxy/memmetrics/roundtrip.go:216 +0x118 fp=0xc8206494a8 sp=0xc820649450
github.com/containous/traefik/vendor/github.com/mailgun/oxy/memmetrics.(*RTMetrics).Record(0xc82041e940, 0xc8, 0x365199)
        /go/src/github.com/containous/traefik/vendor/github.com/mailgun/oxy/memmetrics/roundtrip.go:160 +0x59 fp=0xc8206494d0 sp=0xc8206494a8
github.com/containous/traefik/vendor/github.com/mailgun/oxy/cbreaker.(*CircuitBreaker).serve(0xc8202517a0, 0x7fb8a18dc0f0, 0xc8207f28c0, 0xc82049ec40)
        /go/src/github.com/containous/traefik/vendor/github.com/mailgun/oxy/cbreaker/cbreaker.go:160 +0x1f0 fp=0xc820649580 sp=0xc8206494d0
github.com/containous/traefik/vendor/github.com/mailgun/oxy/cbreaker.(*CircuitBreaker).ServeHTTP(0xc8202517a0, 0x7fb8a18dc0f0, 0xc8207f28c0, 0xc82049ec40)
        /go/src/github.com/containous/traefik/vendor/github.com/mailgun/oxy/cbreaker/cbreaker.go:108 +0xbc fp=0xc8206495c0 sp=0xc820649580
github.com/containous/traefik/middlewares.(*CircuitBreaker).ServeHTTP(0xc820030b00, 0x7fb8a18dc0f0, 0xc8207f28c0, 0xc82049ec40, 0xc82058cc00)
        /go/src/github.com/containous/traefik/middlewares/cbreaker.go:21 +0x42 fp=0xc8206495e8 sp=0xc8206495c0
github.com/containous/traefik/vendor/github.com/codegangsta/negroni.middleware.ServeHTTP(0x7fb8a195c3c8, 0xc820030b00, 0xc82061a5c0, 0x7fb8a18dc0f0, 0xc8207f28c0, 0xc82049ec40)
        /go/src/github.com/containous/traefik/vendor/github.com/codegangsta/negroni/negroni.go:33 +0xaa fp=0xc820649630 sp=0xc8206495e8
github.com/containous/traefik/vendor/github.com/codegangsta/negroni.(*Negroni).ServeHTTP(0xc820dac750, 0x7fb8a18dc168, 0xc82058cba0, 0xc82049ec40)

/cc @gbjk

Responses larger than 1MB have incorrect content length header

I just found a problem with vulcand that uses the stream package in this library. When the responses is more than 1MB, it attempts to chunk the response into 1MB chunks and send them back to the client. However the content-length header does not match the actual content size.

Curl returns the following error:

curl: (18) transfer closed with 16384 bytes remaining to read

When I look at the headers, the Content-Length is exactly 1MB + 16384 bytes and there is only 1MB of content in the response. I have not been able to pinpoint exactly how the chunking logic works, but I will try to create a unit test showing the problem as soon as I can. If you have any ideas where the problem may be let me know.

Web Socket forwarder forwards close codes that are not valid on the wire

When faced with a Web Socket client that closes its TCP connection gracefully, but does not send a Web Socket close frame, gorilla/websocket correctly reports this as websocket close code 1006.

Oxy forwards code 1006 to the target Web Socket server. This is in breach of the Web Socket RFC6455 that states:

      1006 is a reserved value and MUST NOT be set as a status code in a
      Close control frame by an endpoint.  It is designated for use in
      applications expecting a status code to indicate that the
      connection was closed abnormally, e.g., without sending or
      receiving a Close control frame.

This can (and in the case of Jetty/Atmosphere does) lead to unexpected behavior on the server.

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.