aws-samples / http-requests-mirroring Goto Github PK

License: BSD 3-Clause "New" or "Revised" License

Go 100.00%

http-requests-mirroring's Introduction

Mirror production traffic to test environment with VPC Traffic Mirroring

This repository contains the artifacts for the AWS blog post Mirror production traffic to test environment with VPC Traffic Mirroring.

Additional Considerations

Parameters

When creating the stack, you can optionally specify additional parameters. For example, you can use the parameter “ForwardPercentage” to define the percentage of requests that are replicated (by default, this is 100%). You can even choose to only replicate requests coming from a percentage of header values or remote addresses - for example, to mirror all requests that come from only a percentage of users (rather than a percentage of requests from all users). To do that, set the parameter “PercentageBy” to “header” or “remoteaddr”. When “PercentageBy” is set to “header”, you need to provide the header name in the parameter “PercentageByHeader”.

X-Forwarded headers

When the replay handler generates new requests, it manupulates the following headers:

X-Forwarded-For: appends the IP of the client or the IP of the latest proxy.
X-Forwarded-Port: sets it to the outermost port from the chain of client and proxies.
X-Forwarded-Proto: sets it to the outermost protocol from the chain of client and proxies.
X-Forwarded-Host: sets it to the outermost host from the chain of client and proxies.

Protocols support

The only protocol supported is HTTP. HTTPS is not supported. Therefore, SSL offloading should happen before the traffic reaches the EC2 instances in the production environment.

Scaling up the EC2 instances in the replay handler

If you increase the number of instances in the autoscaling group, traffic may get unbalanced in some cases due to how Network Load Balancer flow hash algorithm works. This may happen during scale out operations in the replay handler. To prevent this from happening, when a scale out action is needed from n to m instances (e.g. from 3 to 4), you can scale out to n+m first (e.g. to 3+4=7) and then scale in to m (e.g. 4). You can do this operation with two subsequent updates of the "InstanceNumber" parameter of the CloudFormation Stack. The CloudFormation template provided is already configured to remove the oldest instances first, so that traffic is re-distributed equally to the newer instances.

Security

See CONTRIBUTING for more information.

License

This library is licensed under the BSD-3-Clause License. See the LICENSE file.

http-requests-mirroring's People

Contributors

Stargazers

Watchers

http-requests-mirroring's Issues

Unable to install: non-boolean condition in if statement

It looks as if the changes in #4 may have had unintended consequences -- when attempting to boot this up, the cloud init script fails to install the package:

$ go install github.com/aws-samples/http-requests-mirroring@latest
go: finding module for package github.com/google/gopacket/examples/util
go: finding module for package github.com/google/gopacket
go: finding module for package github.com/google/gopacket/layers
go: finding module for package github.com/google/gopacket/pcap
go: finding module for package github.com/google/gopacket/tcpassembly
go: finding module for package github.com/google/gopacket/tcpassembly/tcpreader
go: found github.com/google/gopacket in github.com/google/gopacket v1.1.19
go: found github.com/google/gopacket/examples/util in github.com/google/gopacket v1.1.19
go: found github.com/google/gopacket/layers in github.com/google/gopacket v1.1.19
go: found github.com/google/gopacket/pcap in github.com/google/gopacket v1.1.19
go: found github.com/google/gopacket/tcpassembly in github.com/google/gopacket v1.1.19
go: found github.com/google/gopacket/tcpassembly/tcpreader in github.com/google/gopacket v1.1.19
# github.com/aws-samples/http-requests-mirroring
go/pkg/mod/github.com/aws-samples/[email protected]/main.go:158:5: non-boolean condition in if statement

Getting Invalid Method & malformed HTTP request error messages and replay handler instances can't forward requests because of this

2022/01/25 08:36:16 Error reading stream 100.65.2.144->100.69.242.111 52492->443 : invalid method "\xfb\x1f)\x95]\xab\xfbt\x87\xe6:\xd0\x10\xec\x00\xdf\x04<\x19\x04?X\xae_\xf7s\xad\xc1^\x16\x01\x92\xc0\xa8\xa9\x93\xd7\xdbt\t\xc22\x84\xe2"
2022/01/25 08:36:16 Error reading stream 100.65.2.144->100.69.242.111 52492->443 : malformed HTTP request ""
2022/01/25 08:36:16 Error reading stream 100.65.2.144->100.69.242.111 52492->443 : malformed HTTP request "\xde\x0f\xba\xed\xad\xf8\x87-b_A\x80Ϝ(ɞ\xb3\xa1\x06\x97\x159\xaa~"a\x8a\x97\xdaO\xf7\xba\xf2\xaa\xaa\xe1\xf5p\x83\xd9\xf5zc\u0380\xa8k\xderD\xfdu\xa4\xc5\xce\xe5\x86\xfe\x98\xfd{\x1dܩ\xba!\xb1\xc8\x12m\x88\x86\x98\x80\x05\x94\x82:\xbc\xdb\xf3\xe7\xf7\x90\xd6\a/^\x96?\f(\x94\xb2\xb3K\x1c\xd9z}\xbdS\xbfb\xa0\xa26\xc3n\xe0j\xd6^\xa3w\x01>5>\xb7\xeeJ\x9e\t\x12\xb4ز\xf1\xb5(\xf5\xf5\xf8k\r|\x1a\xab\xff\x93\xbd8\x06\xbd\xa7\xb8\xc3m\xaeI\xa3\xf1\xc2\xf4H\xc8\xfc\x9e\x9d\xb0\xee\x01a\xecO\xa56\x0e\xe93g\xf7\xd8\xf5L\x88,\xc6P\xa57OF\xec\xfb}o\xeb>rxVZ\xf8\x12\xdar*,\xf8m\x10\f*\xf8\xb4"
2022/01/25 08:36:16 Error reading stream 100.65.2.144->100.69.240.237 48034->443 : malformed HTTP request "\x0f\x8b\xae\xbc\x845\xfd\x032s/t\xc5C\xd7\xef=^Qۋ\x1b\x89\xa6\x87xm|5\x89\x9dC\x89\x96\xf4\xd4\xd6\xd6R\x90\xc2.{F\x1dd\x89R\xb1\xa4\v\f\xac\xea8\xbc9\xc8]J\r\x8e\x9c\xfek6\x1e+\xb8\x17M\x14,\x1b&\x02\x96^\xd2\xf8\xb9\x821\b\xa9\x1f\xd6aGƤrj\xf2\xda0\x81\xfe&f\x9f\x87\xde\x13\xe4\x14\x89t9\xe70\x1a\x8d\xc9\xf6\x90\r\x13\x0f\xe5\b\x8a\x8a\x05\xee\x90\xd93[b\x89\xee\x89\xfbn\xb8\xb1\xbcA\xa4\x1e\xf9\xec\x18\xb17\xdaI?\xb10;i4C\x91Ů\xc2M\xfb\xf1ۯe\adu\x88[t\xb3\xe1P\xb0\xf8\x97\x0f\x98\xd1\x1b\x8e\x80\x98\xec\x01\xe0\xf5'3\x90\v\x80\x9b'\x97Z\xd5O\x06\xefV\x93)鲛Ns]ߪu\xbb3H\xd0f}\xb9yܝ}\b\x90\x8b\x10lT\x8f\a\xca\ue04d̏Q=\xbb\x00d\a \xf6쀷\xbe\x92\xe0֯\xa8\xeb\xe6\xf0\xe2\x1e<r[];%\xda\x11\x12\xee\x8d\x03\xbf"
2022/01/25 08:36:16 Error reading stream 100.65.2.144->100.69.242.111 52492->443 : malformed HTTP request "\xee\x14%\xb2\x1d\xa8\x90\xb1U1\xed\aU\xe3\x1f\x16\x86\xed\xcdZ\xc98\xee\xb6\xc7 =o\x15V\x88\xb8\x96*\x99\\\x15\x19\xd9\xcf\xfe\xec\x8e\u0093\xa5\xe3\xf4\xde,\xc9

cloud formation is no longer working: go get is no longer supported

from /var/log/cloud-init-output.log

++ GOPATH=/root/go
+ go get github.com/google/gopacket
go: go.mod file not found in current directory or any parent directory.
        'go get' is no longer supported outside a module.
        To build and install a command, use 'go install' with a version,
        like 'go install example.com/cmd@latest'
        For more information, see https://golang.org/doc/go-get-install-deprecation
        or run 'go help get' or 'go help install'.
Oct 17 09:00:29 cloud-init[1237]: util.py[WARNING]: Failed running /var/lib/cloud/instance/scripts/part-001 [1]
Oct 17 09:00:29 cloud-init[1237]: cc_scripts_user.py[WARNING]: Failed to run module scripts-user (scripts in /var/lib/cloud/instance/scripts)
Oct 17 09:00:29 cloud-init[1237]: util.py[WARNING]: Running module scripts-user (<module 'cloudinit.config.cc_scripts_user' from '/usr/lib/python2.7/site-packages/cloudinit/config/cc_scripts_user.pyc'>) failed

Unable to install "vxlan-to-http-request"

Hello,
the script suddenly fails to install the go module:

[root@ip-172-30-3-44 ~]# mkdir $GOPATH"/src/vxlan-to-http-request"
mkdir: cannot create directory ‘/root/go/src/vxlan-to-http-request’: No such file or directory

we fixed it with mkdir -p $GOPATH"/src/vxlan-to-http-request"

but then

[root@ip-172-30-3-44 ~]# wget https://github.com/aws-samples/http-requests-mirroring/raw/main/main.go -P $GOPATH"/src/vxlan-to-http-request"
--2022-05-05 11:14:27--  https://github.com/aws-samples/http-requests-mirroring/raw/main/main.go
Resolving github.com (github.com)... 140.82.113.4
Connecting to github.com (github.com)|140.82.113.4|:443... connected.
HTTP request sent, awaiting response... 302 Found
Location: https://raw.githubusercontent.com/aws-samples/http-requests-mirroring/main/main.go [following]
--2022-05-05 11:14:27--  https://raw.githubusercontent.com/aws-samples/http-requests-mirroring/main/main.go
Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 185.199.109.133, 185.199.110.133, 185.199.111.133, ...
Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|185.199.109.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 7806 (7.6K) [text/plain]
Saving to: ‘/root/go/src/vxlan-to-http-request/main.go’

100%[====================================================================================================================================================================================================================================================================================>] 7,806       --.-K/s   in 0s

2022-05-05 11:14:27 (108 MB/s) - ‘/root/go/src/vxlan-to-http-request/main.go’ saved [7806/7806]

[root@ip-172-30-3-44 ~]# go install "vxlan-to-http-request"
go install: version is required when current directory is not in a module
        Try 'go install vxlan-to-http-request@latest' to install the latest version```

Replay handler fails health check from Target group

I tried running the stack to spin up the replay handler but the Target group fails health checks for the instances. The health check is configured for TCP: 4789 (Target port). I am not quite sure what I am missing here.