google / martian Goto Github PK
View Code? Open in Web Editor NEWMartian is a library for building custom HTTP/S proxies
License: Apache License 2.0
Martian is a library for building custom HTTP/S proxies
License: Apache License 2.0
While the tag created in #261 is very helpful for module enabled users, the version shows up a v2.1.0+incompatible
due to the lack of semantic import versioning. Can we get this repo into compatibility?
Based on the suggestions made in the wiki on Modules, it sounds like one option is to bump to v3.0.0
. I am aware that this may not be a popular decision, since there would not be any API breakage, but it could be an opportunity to do so, if that was needed.
The wiki also suggests solutions for source management for packages like yours.
These appear to be the options available:
v3.0.0
v2.2.0
go build main.go
to compile it to binary../main -v 10
{
"url.Filter": {
"scope": ["response"],
"host": "hiweeds.net",
"modifier": {
"header.Modifier": {
"name": "Martian-Test",
"value": "true"
}
}
}
}
404 page not found
I once using this program. and it can work fine. But now it fail, can anybody check that ?
All the core modifiers, filters, and verifiers should have increased debug log output (log.Debugf
) so it's easier to trace which modifiers were called and which values were modified.
I am newbie to golang, and found https://github.com/netroby/gohttpproxy.git this httpproxy using martian, so I run it and with description in martian readme.md page, try to set the modifier, but it is not work out, main reason is as below, there is no martian.proxy
in host,should I add it in hostname ?or what should I do the make the set of modifiers work.
martian logs as below.
--------------------------------------------------------------------------------
Request to http://martian.proxy/configure
--------------------------------------------------------------------------------
POST http://martian.proxy/configure HTTP/1.1
Host: martian.proxy
Content-Length: 98
Accept: */*
Content-Type: application/json
User-Agent: curl/7.49.1
Via: 1.1 martian
X-Forwarded-For: ::1
X-Forwarded-Host: martian.proxy
X-Forwarded-Proto: http
{ "header.Modifier": { "scope": ["response"], "name": "Test-Header", "value": "true" }}
--------------------------------------------------------------------------------
2016/10/21 09:03:57 ERROR: martian: failed to round trip: dial tcp: lookup martian.proxy: no such host
2016/10/21 09:03:57 INFO:
--------------------------------------------------------------------------------
Response from http://martian.proxy/configure
--------------------------------------------------------------------------------
HTTP/1.1 502 Bad Gateway
Content-Length: 0
Warning: 199 "martian" "dial tcp: lookup martian.proxy: no such host" "Fri, 21 Oct 2016 09:03:57 GMT"
There is an unnecessary amount of duplication between verifications and filters. Additionally, the verifiers have inconsistent behavior as to whether the verification requirement is 0, 1, or n times. It would be nice to create an API that handles the verification requirement in one place while allowing filters to be used or verification requirements.
For example, using the basic verification framework together with a url.Filter to create a verification for "all requests must be HTTPS":
v := verify.NewVerification("HTTPS enforcement")
c := v.VerifyAll()
// c := v.VerifyAny()
// c := v.VerifyCount(10)
// c := v.VerifyNone()
f := martianurl.NewFilter(&url.URL{Scheme: "https"})
f.SetRequestModifier(c)
v.SetRequestModifier(f)
Or in the JSON API:
{
"verify.Verification": {
"name": "HTTPS enforcement",
"modifier": {
"url.Filter": {
"scheme": "https",
"modifier": {
"verify.All": { }
}
}
}
}
}
Conditions need to be tied to their parent verify.Verification
.
In the Go API this is simple enough since we can just have the Verification
keep track of it's child conditions using the constructors available on Verification
(VerifyAll
, VerifyNone
, etc.).
This will allow us to ask the verification container to generate errors without requiring complete traversal of the modifier tree. Additionally, this will allow us to remove all the existing methods on filters and groups that currently deal with passing along verification information through the stack (VerifyRequests
, VerifyResponses
, ResetRequestVerifications
, and ResetResponseVerifications
).
The parse
API is trickier. The JSON API currently does not provide any context during the parse and thus has no way to tie a modifier with the key verify.All
to the outer verify.Verification
.
The only solution I currently have to this is to provide a *parse.Context that can carry this information through the parse phase. It should itself be similar to net/context's Context
such that we can push and pop the current verify.Verification
off as needed since verify.Verifications
may be nested underneath of each other, though I currently don't have a good explanation for when that would even make sense. Maybe as a general context API for others to use in case they need to store information that is nestable.
The har
package concatenates all values with the same header together, separated by commas. This makes it impossible to round-trip. E.g. the headers
A: 1
A: 2
are represented the same as
A: 1, 2
Namely, as Header{Name: "A", Value: "1, 2"}
.
The fix is for cases like the first to be represented by two distinct Header
values.
I saw the sample of add key-value pair in example, but not find how to modify the content of response in the README.
could any body show me how to do that?
For example, when the http response body have the data "hello", replace it with a word "world".
BTW, will martian decompress the http response data that compressed?
A notifier is a modifier that executes a WebHook (HTTP callback) when the modifier is evaluated in the modifier tree.
A notifier would take a target URL and a message string. When MoidfyRequest or ModifyResponse is executed, a POST request is sent to the the target URL with the message string as the request body.
Might take separate target URLs and messages for request or response to provide control over whether to execute the callback on request or response.
The body of requests and responses are not logged for performance reasons (chunked encoding).
https://github.com/google/martian/wiki/Modifier-Reference#logging
It'd be nice if one could optionally log request & response bodies. I'm looking for an easy way to dump requests/responses as JSON (ideally HAR).
In addition to posting the config.json to http://martian.proxy/configure via POST,
it would nice to be able to view what is on the server via GET.
Example:
curl -X GET http://martian.proxy/configure => returns current json configuration
Martian is, first and foremost, a set of tools to build custom HTTP/S proxies.
We've always had an example proxy implementation that we've maintained to show others how one could be built which has been confusing to users. We should change the README to present Martian as a toolkit for custom proxies rather than the documentation for the example implementation.
At some point we may want to move the testing proxy implementation to a separate repo and name it something else.
The popular online HAR viewer tool at http://www.softwareishard.com/har/viewer has the ability of loading (via an ajax request so even local URLs work) any HAR from its URL if this HAR is formatted as
onInputData(harJson);
If Martian allowed for this format change via, let's say, a query param (?jsonp=1
) we could go to http://www.softwareishard.com/har/viewer?inputUrl=http://martian.proxy/logs?jsonp=1 to get instant rendering of the HAR timeline.
Currently, a request to http://martian.proxy/some/path will not 404, and will unsuccessfully attempt to establish a TCP connection. The error returned is difficult to diagnose. Martian should return a 404.
har.PostData.Text
, the body of a POST, has type string
. This is a mistake. It should be []byte
.
The HAR spec does say that text
is a string, but it means a JSON string, not a Go string. POST data can contain arbitrary bytes (e.g. if the MIME type is application/octet-stream).
The reason this matters is that encoding/json.Marshal
expects strings to be valid UTF-8; if they are not, it loses information by replacing invalid code points with the replacement character. So when a har.Log
is marshaled to JSON, the result is wrong.
Example:
pd := &har.PostData{Text: string([]byte{150, 151, 152})}
bytes, err := json.Marshal(pd)
if err != nil {
log.Fatal(err)
}
fmt.Printf("%v\n", string(bytes))
Output:
{"mimeType":"","params":null,"text":"\ufffd\ufffd\ufffd"}
Now that github.com/google/martian/cmd/proxy is going to be our de facto proxy implementation we should write some integration tests for it to ensure we don't break it.
In the martian.Context
for each http.Request
is a short lived unique(ish) ID. It would be helpful for debugging to be able to include this "trace ID" in every log line that occurs to be able to follow the execution/modification of a request/response.
My initial idea is that a package creates a Logger instance that it holds and passes the martian.Context
to that will obtain the information it needs and reformat as appropriate.
package mypkg
var log = martian.NewLogger("mypkg.Modifier")
type Modifier struct {
...
}
func (m *Modifier) ModifyRequest(req *http.Request) error {
ctx := martian.NewContext(req)
// Logs with ctx.ID(), the package and the formatted log line.
log.Debugf(ctx, "modifying request: %s", req.URL)
...
return nil
}
The entries in the generated har files have all 3 members of the timings struct set to 0. After the struct is initialized, it looks like it is never filled out, so the 3 fields always stay at 0.
messageview.MessageView.BodyReader
tries to uncompress the body by looking only at the Content-Encoding. But it neglects to check for a Range header (or for a 206 Partial Content status). This results in an error when attempting to unzip a partial gzip file.
We are having a problem in Debian when building the martian where TestStreamsInSentOrder occasionally deadlocks. There's a little more info at https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=923819 but tl;dr in a small but consistent number of builds TestStreamsInSentOrder runs for 10 minutes until it is killed. It looks this test was previously flaky and fixed in #265. Is it possible there are still issues? I'm also doing some investigation on my end to see what I can find. A log of the failure is below.
=== RUN TestStreamsInSentOrder
SIGQUIT: quit
PC=0x45e770 m=0 sigcode=0
goroutine 0 [idle]:
runtime.epollwait(0x7fff00000004, 0x7fff30199818, 0xffffffff00000080, 0x7fff00000001, 0xffffffff00000001, 0x1, 0x0, 0xda5f5e3000000005, 0x7f17, 0x0, ...)
/usr/lib/go-1.11/src/runtime/sys_linux_amd64.s:671 +0x20
runtime.netpoll(0xc000028501, 0xc000028501)
/usr/lib/go-1.11/src/runtime/netpoll_epoll.go:71 +0x132
runtime.findrunnable(0xc000026000, 0x0)
/usr/lib/go-1.11/src/runtime/proc.go:2469 +0x51c
runtime.schedule()
/usr/lib/go-1.11/src/runtime/proc.go:2613 +0x13a
runtime.park_m(0xc000001800)
/usr/lib/go-1.11/src/runtime/proc.go:2676 +0xae
runtime.mcall(0x0)
/usr/lib/go-1.11/src/runtime/asm_amd64.s:299 +0x5b
goroutine 1 [chan receive, 9 minutes]:
testing.(*T).Run(0xc00010c100, 0x72cf41, 0x16, 0x73d830, 0x478386)
/usr/lib/go-1.11/src/testing/testing.go:879 +0x383
testing.runTests.func1(0xc00010c000)
/usr/lib/go-1.11/src/testing/testing.go:1119 +0x78
testing.tRunner(0xc00010c000, 0xc0000a5e08)
/usr/lib/go-1.11/src/testing/testing.go:827 +0xbf
testing.runTests(0xc00008e320, 0x967200, 0x8, 0x8, 0x40d3ff)
/usr/lib/go-1.11/src/testing/testing.go:1117 +0x2aa
testing.(*M).Run(0xc00010a000, 0x0)
/usr/lib/go-1.11/src/testing/testing.go:1034 +0x165
main.main()
_testmain.go:56 +0x13d
goroutine 19 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x7f17da5f5e30, 0x72, 0xc000046b80)
/usr/lib/go-1.11/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc00010a318, 0x72, 0xffffffffffffff00, 0x776bc0, 0x934560)
/usr/lib/go-1.11/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc00010a318, 0xc00013c000, 0x1000, 0x1000)
/usr/lib/go-1.11/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Read(0xc00010a300, 0xc00013c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/lib/go-1.11/src/internal/poll/fd_unix.go:169 +0x179
net.(*netFD).Read(0xc00010a300, 0xc00013c000, 0x1000, 0x1000, 0xc0004d9630, 0x40, 0x50)
/usr/lib/go-1.11/src/net/fd_unix.go:202 +0x4f
net.(*conn).Read(0xc00008a058, 0xc00013c000, 0x1000, 0x1000, 0x0, 0x0, 0x0)
/usr/lib/go-1.11/src/net/net.go:177 +0x68
bufio.(*Reader).fill(0xc00006a4e0)
/usr/lib/go-1.11/src/bufio/bufio.go:100 +0x10f
bufio.(*Reader).ReadByte(0xc00006a4e0, 0xc0002cd630, 0x0, 0xf)
/usr/lib/go-1.11/src/bufio/bufio.go:242 +0x39
golang.org/x/net/websocket.hybiFrameReaderFactory.NewFrameReader(0xc00006a4e0, 0x73e108, 0xc0001ae028, 0xc000464000, 0xc000046e90)
/<<BUILDDIR>>/golang-github-google-martian-2.1.0+git20181219.d0b5ad3/obj-x86_64-linux-gnu/src/golang.org/x/net/websocket/hybi.go:123 +0x60
golang.org/x/net/websocket.Codec.Receive(0x73d868, 0x73d870, 0xc0001ae000, 0x6a3c40, 0xc0004ed520, 0x0, 0x0)
/<<BUILDDIR>>/golang-github-google-martian-2.1.0+git20181219.d0b5ad3/obj-x86_64-linux-gnu/src/golang.org/x/net/websocket/websocket.go:341 +0xaf
github.com/google/martian/marbl.TestStreamsInSentOrder(0xc00010c100)
/<<BUILDDIR>>/golang-github-google-martian-2.1.0+git20181219.d0b5ad3/obj-x86_64-linux-gnu/src/github.com/google/martian/marbl/handler_test.go:55 +0x32a
testing.tRunner(0xc00010c100, 0x73d830)
/usr/lib/go-1.11/src/testing/testing.go:827 +0xbf
created by testing.(*T).Run
/usr/lib/go-1.11/src/testing/testing.go:878 +0x35c
goroutine 21 [IO wait, 9 minutes]:
internal/poll.runtime_pollWait(0x7f17da5f5f00, 0x72, 0x0)
/usr/lib/go-1.11/src/runtime/netpoll.go:173 +0x66
internal/poll.(*pollDesc).wait(0xc00010a118, 0x72, 0xc00005e000, 0x0, 0x0)
/usr/lib/go-1.11/src/internal/poll/fd_poll_runtime.go:85 +0x9a
internal/poll.(*pollDesc).waitRead(0xc00010a118, 0xffffffffffffff00, 0x0, 0x0)
/usr/lib/go-1.11/src/internal/poll/fd_poll_runtime.go:90 +0x3d
internal/poll.(*FD).Accept(0xc00010a100, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0)
/usr/lib/go-1.11/src/internal/poll/fd_unix.go:384 +0x1a0
net.(*netFD).accept(0xc00010a100, 0xc00013a140, 0x3f26326a14b63c0c, 0x44c488)
/usr/lib/go-1.11/src/net/fd_unix.go:238 +0x42
net.(*TCPListener).accept(0xc00008a050, 0x5c7e7212, 0xc00004be78, 0x478386)
/usr/lib/go-1.11/src/net/tcpsock_posix.go:139 +0x2e
net.(*TCPListener).Accept(0xc00008a050, 0xc00004bec8, 0x18, 0xc000070600, 0x645b85)
/usr/lib/go-1.11/src/net/tcpsock.go:260 +0x47
net/http.(*Server).Serve(0xc000136000, 0x778ca0, 0xc00008a050, 0x0, 0x0)
/usr/lib/go-1.11/src/net/http/server.go:2826 +0x22f
net/http.Serve(0x778ca0, 0xc00008a050, 0x775d00, 0xc00008e460, 0x0, 0x0)
/usr/lib/go-1.11/src/net/http/server.go:2423 +0x6e
created by github.com/google/martian/marbl.TestStreamsInSentOrder
/<<BUILDDIR>>/golang-github-google-martian-2.1.0+git20181219.d0b5ad3/obj-x86_64-linux-gnu/src/github.com/google/martian/marbl/handler_test.go:38 +0x163
goroutine 4 [chan receive, 9 minutes]:
github.com/google/martian/marbl.(*Handler).streamLogs(0xc00008e460, 0xc00010e240)
/<<BUILDDIR>>/golang-github-google-martian-2.1.0+git20181219.d0b5ad3/obj-x86_64-linux-gnu/src/github.com/google/martian/marbl/handler.go:77 +0x19d
github.com/google/martian/marbl.(*Handler).streamLogs-fm(0xc00010e240)
/<<BUILDDIR>>/golang-github-google-martian-2.1.0+git20181219.d0b5ad3/obj-x86_64-linux-gnu/src/github.com/google/martian/marbl/handler.go:61 +0x34
golang.org/x/net/websocket.Server.serveWebSocket(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/<<BUILDDIR>>/golang-github-google-martian-2.1.0+git20181219.d0b5ad3/obj-x86_64-linux-gnu/src/golang.org/x/net/websocket/server.go:89 +0x17e
golang.org/x/net/websocket.Server.ServeHTTP(0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, ...)
/<<BUILDDIR>>/golang-github-google-martian-2.1.0+git20181219.d0b5ad3/obj-x86_64-linux-gnu/src/golang.org/x/net/websocket/server.go:70 +0x6a
github.com/google/martian/marbl.(*Handler).ServeHTTP(0xc00008e460, 0x778ee0, 0xc000146000, 0xc00010c800)
/<<BUILDDIR>>/golang-github-google-martian-2.1.0+git20181219.d0b5ad3/obj-x86_64-linux-gnu/src/github.com/google/martian/marbl/handler.go:61 +0xd1
net/http.serverHandler.ServeHTTP(0xc000136000, 0x778ee0, 0xc000146000, 0xc00010c800)
/usr/lib/go-1.11/src/net/http/server.go:2741 +0xab
net/http.(*conn).serve(0xc000140000, 0x779120, 0xc00008ca40)
/usr/lib/go-1.11/src/net/http/server.go:1847 +0x646
created by net/http.(*Server).Serve
/usr/lib/go-1.11/src/net/http/server.go:2851 +0x2f5
rax 0xfffffffffffffffc
rbx 0xffffffff
rcx 0x45e770
rdx 0x80
rdi 0x4
rsi 0x7fff30199818
rbp 0x7fff30199e18
rsp 0x7fff301997d8
r8 0x0
r9 0x3
r10 0xffffffff
r11 0x246
r12 0xffffffffffffffff
r13 0x6a
r14 0x69
r15 0x200
rip 0x45e770
rflags 0x246
cs 0x33
fs 0x0
gs 0x0
*** Test killed with quit: ran too long (10m0s).
FAIL github.com/google/martian/marbl 600.017s
我需要代理访问https请求的网站,并且修改请求头,请求如何做?
Would it be possible to have timings recorded as well, at least optionally?
Main use case for this is analysis of web site performance. It would be handy to be able to view the HAR as it would have been seen at the user's end (sans the browser level metrics of course).
I am trying to setup http(s) proxy server using martian
.
I generated key/certificate files and tried to run a proxy in cmd using the following command.
./proxy -addr=:8080 -tls-addr=:8081 -api-addr=:9090 -api=localhost -cert=ca.crt -key=ca.key -har -v=2
I will show a history I tried. Please tell me how to configure or what was wrong.
curl
.It worked fine.
$ http_proxy=localhost:8080 https_proxy=localhost:8081 curl -p http://google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>301 Moved</TITLE></HEAD><BODY>
<H1>301 Moved</H1>
The document has moved
<A HREF="http://www.google.com/">here</A>.
</BODY></HTML>
2019/04/08 15:55:53 INFO:
--------------------------------------------------------------------------------
Request to http://google.com:80
--------------------------------------------------------------------------------
CONNECT http://google.com:80 HTTP/1.1
Host: google.com:80
Content-Length: 0
User-Agent: curl/7.54.0
Via: 1.1 martian-006e5d3bdb04c9a1b2f8
X-Forwarded-For: ::1
X-Forwarded-Host: google.com:80
X-Forwarded-Proto: http
X-Forwarded-Url: http://google.com:80
curl
It didn't work fine.
$ http_proxy=localhost:8080 https_proxy=localhost:8081 curl -p https://google.com
curl: (56) Proxy CONNECT aborted
2019/04/08 15:58:30 ERROR: martian: failed to read request: tls: oversized record received with length 20037
Chrome
It didn't work fine.
ERR_PROXY_CONNECTION_FAILED
2019/04/08 15:59:10 ERROR: martian: failed to read request: mitm: SNI not provided, failed to build certificate
migrate to "github.com/gorilla/websocket" from "golang.org/x/net/websocket"
Recommended by golang.org/x/net/websocket
This package currently lacks some features found in an alternative and more actively maintained WebSocket package:
Would allow customising the har logger via the REST API and leverage the existing bodyLogging
and postDataLogging
options currently only accessible when martian is used as a library.
{
"har.Logger": {
"scope": ["request", "response"],
"bodyLogging": false,
"postDataLogging": false
}
}
Use case: avoid transmitting heavy request/response bodies when they are not necessary for what's being tested. For e.g. when testing the HAR logs in a separate process that communicates with martian over the REST api.
Filters often end up in a large amount of duplication and boilerplate code to work with all of the APIs in Martian (Verification, JSON, whatever else). They also have the problem that it's not possible to "invert" a filters condition or provide an "else" when the condition fails.
I think it would be more maintainable going forward to create a standard filter that takes care of acting as a modifier and can accept conditions, essentially creating a rather basic strategy pattern.
The API I have in my head looks something like:
package filter
type RequestCondition interface {
MatchRequest(req *http.Request) (bool, error)
}
type ResponseCondition interface {
MatchResponse(res *http.Response) (bool, error)
}
type Filter struct {
reqcond filter.RequestCondition
rescond filter.ResponseCondition
reqmod martian.RequestModifier
resmod martian.ResponseModifier
}
func (f *Filter) ModifyRequest(req *http.Request) error {
match, err := f.reqcond.MatchRequest(req)
if err != nil {
return err
}
if !match {
return nil
}
return f.reqmod.ModifyRequest(req)
}
This would allow us to trivially add a filter.NotCondition
wrapper around any filter.RequestCondition
or filter.ResponseCondition
that could be used to invert the condition and create an "else".
My remote proxy which is password protected and supports https, when applies to martian as downstream proxy, 407 is got.
martian -addr=:8000 -api martian -organization Martian -downstream-proxy-url http://xxxx:password@someproxy:60000
curl -x http://localhost:8000 https://ip.cn -v
then comes the following result:
CONNECT ip.cn:443 HTTP/1.1
Host: ip.cn:443
User-Agent: curl/7.59.0
Proxy-Connection: Keep-Alive
< HTTP/1.1 407 Proxy Authentication Required
< Connection: close
< Content-Type: text/html
< Server: someproxy
<
While the tag created in #261 is very helpful for module enabled users, the version shows up a v2.1.0+incompatible
due to the lack of semantic import versioning. Can we get this repo into compatibility?
Based on the suggestions made in the wiki on Modules, it sounds like one option is to bump to v3.0.0
. I am aware that this may not be a popular decision, since there would not be any API breakage, but it could be an opportunity to do so, if that was needed.
The wiki also suggests solutions for source management for packages like yours.
These appear to be the options available:
v3.0.0
v2.2.0
A test specifically fails on ARM arch:
ok github.com/google/martian 42.997s
Testing: "/builddir/build/BUILD/martian-2.1.0/_build/src/github.com/google/martian/api"
+ GOPATH=/builddir/build/BUILD/martian-2.1.0/_build:/usr/share/gocode
+ go test -buildmode pie -compiler gc -ldflags '-extldflags '\''-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld '\'''
PASS
ok github.com/google/martian/api 0.019s
Testing: "/builddir/build/BUILD/martian-2.1.0/_build/src/github.com/google/martian/auth"
+ GOPATH=/builddir/build/BUILD/martian-2.1.0/_build:/usr/share/gocode
+ go test -buildmode pie -compiler gc -ldflags '-extldflags '\''-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld '\'''
PASS
ok github.com/google/martian/auth 0.020s
Testing: "/builddir/build/BUILD/martian-2.1.0/_build/src/github.com/google/martian/body"
+ GOPATH=/builddir/build/BUILD/martian-2.1.0/_build:/usr/share/gocode
+ go test -buildmode pie -compiler gc -ldflags '-extldflags '\''-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld '\'''
PASS
ok github.com/google/martian/body 0.020s
Testing: "/builddir/build/BUILD/martian-2.1.0/_build/src/github.com/google/martian/cmd/proxy"
+ GOPATH=/builddir/build/BUILD/martian-2.1.0/_build:/usr/share/gocode
+ go test -buildmode pie -compiler gc -ldflags '-extldflags '\''-Wl,-z,relro -Wl,--as-needed -Wl,-z,now -specs=/usr/lib/rpm/redhat/redhat-hardened-ld '\'''
2018/10/22 16:58:41 martian: starting proxy on [::]:44869 and api on [::]:46069
2018/10/22 16:58:41 martian: shutting down
2018/10/22 16:58:50 martian: starting proxy on [::]:35927 and api on [::]:33689
2018/10/22 16:58:50 martian: starting proxy on [::]:36391 and api on [::]:46289
2018/10/22 16:58:50 martian: shutting down
2018/10/22 16:58:50 martian: shutting down
--- FAIL: TestProxyMain (13.92s)
--- FAIL: TestProxyMain/HttpsGenerateCert (8.71s)
main_test.go:51: waitForProxy: did not start up within 5.0 seconds
FAIL
exit status 1
FAIL github.com/google/martian/cmd/proxy 13.942s
error: Bad exit status from /var/tmp/rpm-tmp.AEwrYc (%check)
It works fine on any other arches.
You can have access to the full log here: https://koji.fedoraproject.org/koji/taskinfo?taskID=30397630
The last release tag is from three years ago. People depending on martian through go modules will get that by default, so I think making a new tag for something more recent might be warranted.
Users have asked for the capability to record responses from external services, and then to use those responses as a mechanism to stub out the external service during the execution of a test run.
Modfifier is "martianurl.Modifier", Filter is "url.Filter" and Verifier is "martianurl.verifier"
The "martianurl" bit is an artifact from the Go package naming, they should all be "url.Something"
Some prelimary investigation reveals several problems with supporting WebSockets, both secure and insecure. When using a proxy, browsers will send a CONNECT request to the destination when attempting to open a WebSocket, regardless of whether it is secure (ws:// or wss://).
Martian assumes CONNECT requests are requests to upgrade to TLS for HTTPS. In the non MITM case, Martian will blindly make a TCP connection by default for CONNECT requests which means that both secure and insecure WebSocket requests work.
Everything starts to break when MITM is enabled. Here's how:
To start, we'll look at an example request that is made to the proxy when attempting to open a WebSocket to echo.websocket.org.
CONNECT echo.websocket.org:80 HTTP/1.1
Host: echo.websocket.org:80
Proxy-Connection: keep-alive
Content-Length: 0
User-Agent: ...
Martian will wrap the connection with a TLS connection after sending a 200 OK
response to the client. In the case of an insecure WebSocket request, the connection will then have a normal HTTP request sent to it for the WebSocket handshake. This fails with a TLS handshake error: tls: first record does not look like a TLS handshake
.
This is the tricky case. We actually need to inspect the first couple bytes of traffic from the connection to be able to make a guess whether the connection is TLS or something else. The current idea is to inspect the bytes for:
GET https://echo.websocket.org/?encoding=text HTTP/1.1
Host: echo.websocket.org
Upgrade: websocket
Connection: Upgrade
Content-Length: 0
Origin: http://www.websocket.org
Sec-Websocket-Extensions: permessage-deflate
Sec-Websocket-Key: 5/02dRzDuskoI7ZWhnvL1w==
Sec-Websocket-Version: 13
The TLS handshake succeeds in the secure WebSockets case, but fails with the following response to the prior request. This is because Martian will remove the Upgrade
header (because it is listed in the Connection header) and thus prevent the downstream server from recognizing it as a WebSocket request.
HTTP/1.1 400 WebSocket Upgrade Failure
Content-Length: 77
Access-Control-Allow-Credentials: true
Access-Control-Allow-Headers: content-type
Access-Control-Allow-Headers: authorization
Access-Control-Allow-Headers: x-websocket-extensions
Access-Control-Allow-Headers: x-websocket-version
Access-Control-Allow-Headers: x-websocket-protocol
Access-Control-Allow-Origin: http://www.websocket.org
Content-Type: text/html
Date: Tue, 24 Nov 2015 21:59:21 GMT
I expect this case will be slightly easier to handle with a custom modifier that checks for the Upgrade
header, and if it finds websocket
specified, will issue context.Hijack()
and stitch the connections together.
The idea is that session.Context
provides a Hijack()
method similar to ResponseWriter.Hijack()
that will return net.Conn, bufio.ReadWriter, error
.
This will allow a modifier aware of WebSockets to take control of the connection while allowing modifiers before it to run.
A sub modification system (such as modifiers specifically designed to manipulate Websocket messages) could be created to further enhance the system.
To deal with the default case that is currently broken, we can create a modifier that will by default take any request with WebSocket headers and hijack the connection and stitch the two connections together reusing behavior similar to the CONNECT tunnel, send a 101 Switching Protocols response, and then waiting until the connection is closed.
The package verify
needs to be reworked to be more useful for end users.
The current JSON API looks like:
{
"errors": [
{ "message": "request(url): message: got %v, want %v" }
]
}
Having the HTTP message type and verification type mixed in with the verification failure makes it hard to distinguish what type of failure happened and forces the consumer to blindly accept the message string as is.
The message
key should continue to exist for those who want to blindly accept the full message formatting from the verifier, but should be able to use the metadata about the failure to construct their own message.
The proposed new format:
{
"errors": [
{
"header.Verifier": {
"scope": "request",
"url": "http://example.com/path",
"message": "[ existing message format ]",
"actual": "Content-Type: text/plain",
"expected": "Content-Type: image/png"
}
}
]
}
This format closely resembles the existing modifier JSON API with standard keys for the verification error.
Would you please support socks5 as downstream proxy server?
To enhance Martian’s trafficshape package so that it can simulate desired network conditions more precisely. These features are needed in order to allow developers to write tests that exercise application features that are designed to react to suboptimal network conditions.
Key features of this effort include:
These features are useful to anyone who is interested in writing tests that simulate poor network conditions: especially any tests that are serving video. More specifically, the test authors will be able to specify the “shape” of the video traffic as a configuration in the test and write assertions based on the intended behavior of client in dealing with the traffic.
Currently, the trafficshape package only allows setting the bandwidth and latency for all the requests and responses; the user cannot pick a few resources for which to shape traffic. This is insufficient for the testing situation where one wants to throttle a particular resource while letting the rest of the test infrastructure use the network normally.
The trafficshape package also does not have a way to keep track of what byte offsets are currently being written back to the client for a specific resource. This can be useful when the user wants to perform different actions or have different bandwidths depending on what byte ranges are being written back to the client.
{
"trafficshape": {
"default": {
"bandwidth": {
"up": 10000000,
"down": 1000000
},
"latency": {
"up": 1000,
"down": 100000
}
},
"shapes": [{
"url_regex": "example.com/video/stream",
"throttles": [{
"throttle": {
"bytes": "100-150000",
"bandwidth": 100000
}
}],
"halts": [{
"halt": {
"byte": 200000,
"duration": 5000,
"count": 1
}
}],
"close_connections":[{
"close_connection": {
"byte": 250000,
"count": 1
}
}]
}]
}
}
These are the values that will be applied for the byte ranges that do not have throttles specified for them.
Every throttle action takes in a byte range and a bandwidth. When the proxy is writing back bytes that fall in the range of a throttle’s byte range for the current url, it will be subject to the corresponding bandwidth.
Every halt action takes in a byte, duration and count (Explained below). The byte specifies at which byte to halt writing data back to the client and duration specifies how long to remain in this state before starting to write data back to the client.
Every close_connection action takes in a byte and a count (Explained below). The byte specifies what byte the proxy should be about to write back to the client before it should close the current connection with the client.
Note the “count” property in the “halt” and “close_connection” actions . It is possible that in the course of writing the video, the client asks for byte range multiple times (this might happen when the existing connection is interrupted, and the client didn’t read part of the data written back by the proxy already, so it asks again for a range of the data that we have already written). So it is possible that they would want the action to be performed only the first time a specific byte is written back, or possibly more times. That is what the count specifies. A count of -1 (default) will specify that the action will happen everytime the specific byte is written back.
So the example json says:
For the byte_ranges, we support ranges to be expresses as “-5000”, which means “up till but not including byte 5000”, or “5000-”, which would mean from “5000 onwards”.
RFC 7231 4.3.6:
A server MUST NOT send any Transfer-Encoding or Content-Length header
fields in a 2xx (Successful) response to CONNECT. A client MUST
ignore any Content-Length or Transfer-Encoding header fields received
in a successful response to CONNECT.
Full bug report see curl/curl#1317
The current martian proxy handle CONNECT, will response
CONNECT www.google.com:443 HTTP/1.1
Host: www.google.com:443
User-Agent: curl/7.29.0
Proxy-Connection: Keep-Alive
< HTTP/1.1 200 OK
< Content-Length: 0
<
The Content-Length must not response
This is useful if a modifier needs to release resources before being replaced and garbage collected.
I tried installing the martian package using the following command
go get github.com/google/martian
This was the error I received.
$GOPATH......./src/github.com/google/martian/proxy.go:142: tr.TLSNextProto undefined (type *http.Transport has no field or method TLSNextProto)
When using the url.Filter, when the path is malformed (does not have a leading slash), Martian silently ignores it.
I saw this on one of our Travis runs:
WARNING: DATA RACE
Read at 0x00c000097a08 by goroutine 30:
internal/race.Read()
/Users/travis/.gimme/versions/go1.11.6.darwin.amd64/src/internal/race/race.go:37 +0x38
sync.(*WaitGroup).Add()
/Users/travis/.gimme/versions/go1.11.6.darwin.amd64/src/sync/waitgroup.go:71 +0x16b
github.com/google/martian.(*Proxy).handleLoop()
/Users/travis/gopath/pkg/mod/github.com/google/[email protected]+incompatible/proxy.go:221 +0x7f
Previous write at 0x00c000097a08 by goroutine 77:
internal/race.Write()
/Users/travis/.gimme/versions/go1.11.6.darwin.amd64/src/internal/race/race.go:41 +0x38
sync.(*WaitGroup).Wait()
/Users/travis/.gimme/versions/go1.11.6.darwin.amd64/src/sync/waitgroup.go:128 +0xef
github.com/google/martian.(*Proxy).Close()
/Users/travis/gopath/pkg/mod/github.com/google/[email protected]+incompatible/proxy.go:145 +0xcc
cloud.google.com/go/httpreplay/internal/proxy.(*Proxy).Close()
/Users/travis/gopath/pkg/mod/cloud.google.com/[email protected]/httpreplay/internal/proxy/record.go:200 +0x54
cloud.google.com/go/httpreplay.(*Replayer).Close()
/Users/travis/gopath/pkg/mod/cloud.google.com/[email protected]/httpreplay/httpreplay.go:152 +0x4b
gocloud.dev/internal/testing/setup.NewAWSSession2.func2()
/Users/travis/gopath/src/gocloud.dev/internal/testing/setup/setup.go:139 +0x41
gocloud.dev/pubsub/awssnssqs.(*harness).Close()
/Users/travis/gopath/src/gocloud.dev/pubsub/awssnssqs/awssnssqs_test.go:195 +0x4c
gocloud.dev/pubsub/drivertest.testMetadata()
/Users/travis/gopath/src/gocloud.dev/pubsub/drivertest/drivertest.go:714 +0x901
gocloud.dev/pubsub/drivertest.RunConformanceTests.func1()
/Users/travis/gopath/src/gocloud.dev/pubsub/drivertest/drivertest.go:167 +0x6f
testing.tRunner()
/Users/travis/.gimme/versions/go1.11.6.darwin.amd64/src/testing/testing.go:827 +0x162
Goroutine 30 (running) created at:
github.com/google/martian.(*Proxy).Serve()
/Users/travis/gopath/pkg/mod/github.com/google/[email protected]+incompatible/proxy.go:216 +0x392
Goroutine 77 (finished) created at:
testing.(*T).Run()
/Users/travis/.gimme/versions/go1.11.6.darwin.amd64/src/testing/testing.go:878 +0x659
gocloud.dev/pubsub/drivertest.RunConformanceTests()
/Users/travis/gopath/src/gocloud.dev/pubsub/drivertest/drivertest.go:167 +0x5c3
gocloud.dev/pubsub/awssnssqs.TestConformance()
/Users/travis/gopath/src/gocloud.dev/pubsub/awssnssqs/awssnssqs_test.go:212 +0xa5
testing.tRunner()
/Users/travis/.gimme/versions/go1.11.6.darwin.amd64/src/testing/testing.go:827 +0x162
==================
It would be great to verify that a request has gone through the proxy. This can be very handy when testing to verify that a specific callback has been called for example. In this case there would be an error if the verifier has still not matched any request/response when GET http://martian.proxy/verify
is called.
Verifier:
{
"atLeastOnce.Verifier": {
"scope": ["request"],
"host": "example.com",
"path": "/testing"
}
}
Error:
{
"errors" : [
{
"message": "request(http://example.com/testing) call verify failure"
}
]
}
Hi,
command: go test -buildmode pie -compiler gc -ldflags "-X github.com/google/martian/version=2.1.0 -extldflags '-Wl,-z,relro '"
testing: github.com/google/martian/static
--- FAIL: TestFileExistsInBothExplictlyMappedPathAndInferredPath (0.00s)
static_file_modifier_test.go:115: res.Header.Get('Content-Type'): got , want text/plain; charset=utf-8
--- FAIL: TestStaticModifierExplicitPathMapping (0.01s)
static_file_modifier_test.go:171: res.Header.Get('Content-Type'): got , want text/plain; charset=utf-8
--- FAIL: TestStaticModifierOnRequest (0.00s)
static_file_modifier_test.go:222: res.Header.Get('Content-Type'): got , want text/plain; charset=utf-8
--- FAIL: TestRequestOverHTTPS (0.00s)
static_file_modifier_test.go:275: res.Header.Get('Content-Type'): got , want text/plain; charset=utf-8
--- FAIL: TestModifierFromJSON (0.00s)
static_file_modifier_test.go:359: res.Header.Get('Content-Type'): got , want text/plain; charset=utf-8
static_file_modifier_test.go:398: res.Header.Get('Content-Type'): got , want text/plain; charset=utf-8
--- FAIL: TestStaticModifierSingleRangeRequest (0.00s)
static_file_modifier_test.go:473: res.Header.Get('Content-Type'): got , want text/plain; charset=utf-8
FAIL
exit status 1
FAIL github.com/google/martian/static 0.020s
Hi team,
I am trying to use martian proxy on windows 10 host to intercept traffic from my Android mobile.
I started the proxy using the command proxy -v=2 - har=true.
Proxy is started with default settings [traffic on 8080 and api on 8181]
I am getting following errorss when I try to get the har logs
GET http://localhost:8181/logs- 400 Bad Request
GET http://localhost:8080/logs- 404 Not Found
I am using Advanced Rest Client to call Martian rest APIs.
Could you please help?
When martian gets a request from another proxy, it may already have x-forwarded-... headers that represent the original client request. The current implementation (link below) overwrites those values. I would expect the values to be preserved if already present, as I expect these headers to tell me about the original client request, not the request as received from the second-to-last proxy in the chain.
martian/header/forwarded_modifier.go
Line 32 in 195b986
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.