Giter VIP home page Giter VIP logo

privacy-gateway-server-go's Introduction

Privacy Gateway Application Server

This project contains a gateway implementation for an Oblivious HTTP (OHTTP) gateway in Go.

Overview

This gateway implements a simple version of the Oblivious Gateway Resource as described in the specification. In particular, it accepts encapsulated Binary HTTP requests and then uses the corresponding HTTP requests to fetch a Target resource. The response from this Target is encapsulated back to the original client of the encapsulated request.

By default, the gateway exposes the following API endpoints:

  • "/gateway": An endpoint that will accept OHTTP requests, fetch the corresponding target resource, and return an OHTTP response.
  • "/gateway-echo": An endpoint that will echo the contents of the encapsulated OHTTP request back in an OHTTP response.
  • "/ohttp-configs": An endpoint that will provide an encoded KeyConfig.
  • "/health": An endpoint for inspecting the health of the gateway (returns 200 in normal conditions).

The gateway only supports the HPKE ciphersuite based on DHKEM(X25519, HKDF-SHA256), HKDF-SHA256, and AES-128-GCM.

The gateway does not currently support key rotation. This issue tracks adding this feature.

Deployment

This section describes deployment instructions for the gateway.

Configuration Variables

The behavior of the gateway is configurable via a number of environment variables. These are explained below.

  • SEED_SECRET_KEY: This environment variable is a hex-encoded byte array representing a secret seed used to derive the gateway private and public key pair. It MUST be 32 randomly generated bytes produced from a cryptographically secure random number generator, such as /dev/urandom. See this guidance for additional information.
  • ALLOWED_TARGET_ORIGINS: This environment variable contains a comma-separated list of target origin names that the gateway is allowed to access. When configured, the gateway will only attempt to resolve requests to target origins in this list. Any other request will yield a HTTP 403 Forbidden return code.
  • CERT: This environment variable is the name of a file containing the certificate (chain) used to serve TLS connections.
  • KEY: This environment variable is the name of a file containing the private key used to serve TLS connections.
  • TARGET_REWRITES: This environment variable contains a JSON document instructing the gateway to rewrite the target URL found in an encapsulated request to some specified scheme and host.

Target URL rewrites

The TARGET_REWRITES configuration option is useful to set up forwarding between the gateway and target when both are on a private network or if they share a loopback interface. For example, suppose that the target is exposed to the internet at https://example.org, but also reachable by the gateway at http://localhost:8080 (note http and not https). It's more efficient to redirect traffic over localhost than back out over the internet, so you could set TARGET_REWRITES to:

{
	"example.org": { "Scheme": "http", "Host": "localhost:8080" }
}

Then the encapsulated HTTP requests

POST /some-cool-api HTTP/1.1
Host: example.org

some content

or

POST https://example.org/some-cool-api HTTP/1.1

some content

...would both be rewritten to:

POST http://localhost:8080/some-cool-api HTTP/1.1

some content

Custom Application Payloads {#custom-config}

The gateway can be configured to service Binary HTTP (BHTTP) messages or custom application payloads. To use custom applciation payloads, you must specify the type of application request and response encodings using the CUSTOM_REQUEST_TYPE and CUSTOM_RESPONSE_TYPE environment variables. For example, if you were using protobuf as the application data encoding, you might set CUSTOM_REQUEST_TYPE="message/protohttp request" and CUSTOM_RESPONSE_TYPE="message/protohttp response". See the OHTTP library and OHTTP standard for additional information about choosing custom content types. This example protobuf file contains an example protobuf encoding of HTTP messages as an alternate to BHTTP.

When specifying a custom application format, it is also required to implement a new handler for the format. This can be done by adding a new ContentType handler that implements the logic for producing an application response for your application request. As an example, if the custom content type corresponded to DNS messages, the handler might resolve the DNS query and produce an encoded DNS response. Alternatively, if using the example protobuf-based HTTP encoding, the ContentType handler might be implemented as follows:

func protobufHandler(binaryRequest []byte) ([]byte, error) {
	request := &Request{}
	if err := proto.Unmarshal(binaryRequest, request); err != nil {
		return nil, err
	}

        // Convert the protohttp Request to a http.Request equivalent value
	targetRequest, err := protoHTTPToRequest(request)
	if err != nil {
		return nil, err
	}

	client := &http.Client{}
	targetResponse, err := client.Do(targetRequest)
	if err != nil {
		return nil, err
	}

	response, err := responseToProtoHTTP(targetResponse)
	if err != nil {
		return nil, err
	}

	return proto.Marshal(response)
}

That's it!

Local development

To deploy the server locally, first acquire a TLS certificate using mkcert as follows:

$ mkcert -key-file key.pem -cert-file cert.pem 127.0.0.1 localhost

Then build and run the server as follows:

$ make all
$ CERT=cert.pem KEY=key.pem PORT=4567 ./gateway

Preconfigured deployments

Deploy deploy to Scalingo

Manual deployment

This server can also be manually deployed on any bare metal machine, or in cloud providers such as GCP. Instructions for both follow.

Bare metal

Deployment on bare metal servers, such as Equinix, can be done following the instructions below. These steps assume that git and go are both installed on the metal.

  1. Configure a certificate on the metal using certbot. Once complete, the output should be something like the following, assuming the server domain name is "example.com":
Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/example.com/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/example.com/privkey.pem

You must configure certbot to renew this certificate periodically. The simplest way to do this is via a cron job:

$ 00 00 1 * 1 certbot renew
  1. Configure two environment variables to reference these files:
$ export CERT=/etc/letsencrypt/live/example.com/fullchain.pem
$ export KEY=/etc/letsencrypt/live/example.com/privkey.pem
  1. Clone and build the server:
$ git clone [email protected]:cloudflare/app-relay-gateway-go.git
$ cd app-relay-gateway-go
$ go build ./...
  1. Run the server:
$ PORT=443 ./gateway &

This will run the server until completion. You must configure the server to restart should it terminate prematurely.

GCP

To deploy, run:

$ gcloud app deploy

To check on its status, run:

$ gcloud app browse

To stream logs when deployed, run

$ gcloud app logs tail -s default

privacy-gateway-server-go's People

Contributors

alexander-chervony avatar bkochendorfer avatar bwesterb avatar chris-wood avatar dependabot[bot] avatar make4me-flo avatar tgeoghegan avatar thibmeu avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

privacy-gateway-server-go's Issues

Add key rotation support

The key is currently fixed per gateway instance. The gateway should support key rotation for applications and environments that require this feature.

Gateway<>Target ACL configuration

Right now the gateway will act on any valid HTTP request it receives from a client. That means if a client sends an encoded HTTP request for, say, GET github.com/index.html, it will honor it. We should probably allow the gateway to restrict the set of accessible targets. We could configure this type of ACL based on another environment variable.

Encapsulate all errors after request decapsulation

If there's something wrong with the decapsulated request body, that error needs to be generated and returned to the client in an encapsulated response. It should not yield a 4xx response to the client.

Add end-to-end latencies monitoring

It is beneficial to measure OHTTP-added latency before applying the protocol on a big scale.

To reliably trace the time on each step of the OHTTP pipeline in ideal world it would be needed to use same Precision Time Protocol service on all the actors: client, relay, gateway, target server. But it's quite hard to achieve since all of this is on the different networks and require substantial efforts.

But similar metrics can be implemented WITHOUT PTP servers if to be done on RELAY side with just the execution time provided by the gateway. The idea is simple: on the relay in runtime measure total execution time and request start time. Gateway should add 3 extra custom headers into the response to the gateway (along with encapsulated response): 1 gateway code execution duration; 2. Roundtrip to Target server duration; 3. Target server execution time (returned by target server and not from all endpoints but the completeness here is not needed as these network latencies to one Target endpoint or another are pretty similar to the same server/AWS env).
Having all these values from Relay, Gateway and Target the RELAY can summarise them into quite a comprehensive metrics for added latencies on every step and the network latencies in-between.

Redirecting to a target on loopback

I'm interested in using privacy-gateway-server-go in a setup where it and the OHTTP target are on the same loopback (localhost) interface (concretely, they're both in the same Kubernetes pod, but we don't need to get into Kubernetes details to discuss this issue). I want the gateway to be able to forward decapsulated requests to the target without going back out to the internet. The problem is that privacy-gateway-server-go uses the encapsulated request's control data to decide what scheme and authority to forward the request to, but that will be the publicly routable DNS name for the target, not its port on the loopback interface.

To make this concrete: let's imagine that I have the OHTTP gateway exposed to the internet at https://example.com/gateway and the target exposed at https://example.com/target. Further, let's suppose that the target is visible to the gateway at localhost:80.

If a client wants to hit some API on the target, it would construct and encapsulate an HTTP request:

POST /some-cool-api HTTP/1.1

some content

...and then POST that to https://example.com/gateway. This won't work, because privacy-gateway-server-go will try to forward the request to https:///some-cool-api. OK, let's put the entire target URL in the encapsulated request:

POST https://example.org/target/some-cool-api HTTP/1.1

some content

This does work, but not the way I want: privacy-gateway-server-go will go do a DNS lookup for example.org and connect to the target over the internet, but I want to do this over loopback and without TLS. This encapsulated request works (note http and not https):

POST http://localhost:80/target/some-cool-api HTTP/1.1

some content

...but this solution isn't great, because now the client needs to know details of the private network namespace that the target and gateway are on. Similar things can be achieved with a Host header instead of putting the authority in the first line of the HTTP request, but spelling out the entire URL makes it easier to control the scheme.

I think what I would like is a configuration option on privacy-gateway-server-go that allows operators to provide a map of origins to origins used to rewrite requests forwarded to targets. So in the scenario above, I could run the gateway with:

{ "https://example.org": "http://localhost:80" }

And then when the gateway encounters the key in an encapsulated request, it forwards the request to the value (preserving the request path). If this solution sounds good to the maintainers, then I'd be happy to work on a PR for this.

BHTTP: support truncation for received known-length requests

BHTTP requests with truncated empty content and trailer field sections are not processed as expected according to BHTTP specs.

Example:

Request with truncated empty content and trailer field section (doesn't work):
0004504f5354056874747073186170692e73746167696e672e6f776865616c74682e636f6d1c2f63616c6c2f76312f75736572732f6d652f616e6f6e796d697a6564409c06416363657074106170706c69636174696f6e2f6a736f6e09582d4170692d4b65792466336161646639652d616562632d343061322d383265322d38616362643439346665326611582d4170706c69636174696f6e2d4b65792435646133323532612d323931392d343033622d393035342d3538643965653430656435620c436f6e74656e742d54797065106170706c69636174696f6e2f6a736f6e

Same request with zero-length empty content and trailer field section (works):
0004504f5354056874747073186170692e73746167696e672e6f776865616c74682e636f6d1c2f63616c6c2f76312f75736572732f6d652f616e6f6e796d697a6564409c06416363657074106170706c69636174696f6e2f6a736f6e09582d4170692d4b65792466336161646639652d616562632d343061322d383265322d38616362643439346665326611582d4170706c69636174696f6e2d4b65792435646133323532612d323931392d343033622d393035342d3538643965653430656435620c436f6e74656e742d54797065106170706c69636174696f6e2f6a736f6e0000

Refactor metrics reporting

should be:

  • v2 prefix;
  • metric labels/tags inferred from error names;
  • custom metric calls removed afterwards

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.