initc3 / sting-flashbots Goto Github PK

License: MIT License

Python 61.95% Dockerfile 2.25% Makefile 3.59% C++ 1.58% Shell 0.66% Go 25.01% Solidity 4.96%

sting-flashbots's Introduction

Sting-Flashbots

Prerequisites

Set Intel IAS SPID

export RA_CLIENT_SPID=<spid>
export RA_TYPE=<dcap or epid>
export RA_CLIENT_LINKABLE=<0 or 1>
export RA_TLS_EPID_API_KEY=<api key>

or add it in a .env file, at the root of this repository, e.g.:

# .env file
RA_CLIENT_SPID=0123456789abcdefghijklmnopqrstuv
RA_TYPE=dcap
RA_CLIENT_LINKABLE=0
RA_TLS_EPID_API_KEY=vutsrqponmlkjihgfedcba9876543210

Set SGX driver environment variables

Set SGX_DRIVER and GRAMINE_IMG_TAG in your .env file or via export statements.

For the out-of-tree (legacy) driver set SGX_DRIVER to oot, and for the in-kernel (dcap) driver set SGX_DRIVER to inkernel.

The GRAMINE_IMG_TAG refers to the tag of an image hosted at https://github.com/initc3/docker-gramine/pkgs/container/gramine. For instance, to use ghcr.io/initc3/gramine:dcap-f160357 set GRAMINE_IMG_TAG to dcap-f160357.

Example of a .env file for the out-of-tree driver:

# .env file
SGX_DRIVER=oot
GRAMINE_IMG_TAG=legacy-f160357

Example of a .env file for the in-kernel driver:

# .env file
SGX_DRIVER=inkernel
GRAMINE_IMG_TAG=dcap-f160357

Running in SGX Simulation mode or without SGX

To run in simulation mode, set in .env or environment

# .env file
SGX=0

To run without SGX use docker-compose-nosgx.yml file

Add --file docker-compose-nosgx.yml to the docker compose commands., e.g.:

docker compose --file docker-compose-nosgx.yml build

Build docker image

docker compose build

Run demo

Create docker containers for blockchain network, builder, and relayer

docker compose up -d

Look at Searcher container logs

docker compose logs -f searcher

Stop containers and delete volumes

docker compose down -v

Running on Sepolia

Setup

Add Sepolia private keys and address to .env file

# .env file
...
SEARCHER_ADDRESS=<address for $SEARCHER_KEY>
SEARCHER_KEY=<Sepolia account private key for searcher (with balance)>
BOUNTY_CONTRACT_ADMIN_PK=<Sepolia account for admin who approves sting enclaves (can be the same as $SEARCHER_KEY)>
STINGER_KEY=<Another Sepolia account private key for sending the stinger (with balance)>
POF_KEYS=[<list of Sepolia account private keys to use for private order flow transaction simulations (with balances)>]
BUILDER_KEY=<builder bls secret key to set BUILDER_TX_SIGNING_KEY and BUILDER_SECRET_KEY>

Generate jwt secret

mkdir -p sepolia
openssl rand -hex 32 | tee ./sepolia/jwtsecret

Running on Sepolia without SGX

build images for Sepolia

docker compose -f docker-compose-sepolia-nosgx.yml build

Start containers

docker compose -f docker-compose-sepolia-nosgx.yml up -d

Look at searcher logs

docker compose -f docker-compose-sepolia-nosgx.yml logs -f searcher

Delete containers and volume

docker compose -f docker-compose-sepolia-nosgx.yml down -v

Running on Sepolia with SGX

Download network snapshot

Build non-sgx docker containers for downloading snapshot

docker compose -f docker-compose-nosgx.yml build builder beacon-chain

create docker network

docker network create sting-sync-net

Set environment variables for builder key

export BUILDER_KEY=<builder bls secret key>

start builder

docker run --publish 8551:8551 --publish 8545:8545 --net sting-sync-net --name builder \
  -e BUILDER_SECRET_KEY=$BUILDER_KEY \
  -e BUILDER_TX_SIGNING_KEY=$BUILDER_KEY \
  -v $PWD/sepolia:/root/sepolia  \
  --rm flashbots-builder:local --sepolia \
  --http --http.api=engine,eth,web3,net,debug,flashbots \
  --http.corsdomain=* \
  --http.addr=0.0.0.0 \
  --ws --ws.api=engine,eth,web3,net,debug \
  --authrpc.jwtsecret=/root/sepolia/jwtsecret \
  --authrpc.vhosts=* --authrpc.addr=0.0.0.0 \
  --datadir=/root/sepolia/synced

start beacon-chain (in seperate terminal)

docker run --publish 4000:4000 --publish 3500:3500 --publish 8080:8080 --net sting-sync-net --name beacon-chain \
  -v $PWD/sepolia:/root/sepolia \
  --rm ghcr.io/initc3/flashbots-prysm:cecd2d9cb \
  --datadir=/root/sepolia/beacondata --sepolia \
  --checkpoint-sync-url=https://sepolia.beaconstate.info \
  --genesis-beacon-api-url=https://sepolia.beaconstate.info \
  --grpc-gateway-host=0.0.0.0 \
  --execution-endpoint=http://builder:8551 \
  --accept-terms-of-use \
  --jwt-secret=/root/sepolia/jwtsecret

wait for sync to complete

docker logs builder # | grep "Snap sync complete"
...
...
INFO [06-02|11:18:23.743] Syncing: chain download in progress      synced=100.00% chain=12.23GiB   headers=3,609,[email protected]    bodies=3,609,[email protected]    receipts=3,609,[email protected]    eta=0s
INFO [06-02|11:18:23.743] Snap sync complete, auto disabling 
INFO [06-02|11:18:23.747] Upgrading chain index                    type=bloombits               percentage=0
INFO [06-02|11:18:24.390] New local node record                    seq=1,685,715,593,349 id=9b35988b6158e5af ip=38.65.223.112 udp=30303 tcp=30303
INFO [06-02|11:18:24.396] Resuming state snapshot generation       root=aa662a..06f1ed in=087068..71eb36 at=a1228e..84f3fc accounts=168,867              slots=654,628              storage=59.48MiB dangling=0 elapsed=6.167s        eta=3m0.938s
INFO [06-02|11:18:24.398] Imported new potential chain segment     number=3,609,949 hash=5428a4..d8898d blocks=1   txs=113  mgas=11.586  elapsed=63.766ms      mgasps=181.691 dirty=34.11MiB
INFO [06-02|11:18:24.411] Chain head was updated                   number=3,609,949 hash=5428a4..d8898d root=2d6bc9..202ecf elapsed=2.04271ms
INFO [06-02|11:18:24.411] Entered PoS stage 
...
...

check state root against other sources e.g. https://checkpoint-sync.sepolia.ethpandaops.io/

curl -s http://localhost:3500/eth/v1/beacon/headers/finalized | jq .'data.header.message'

stop containers and delete network

docker stop builder beacon-chain
docker network rm sting-sync-net

Run demo

add Fake propose to environment (you can create a new one using: geth-sgx-gramine)

export FAKE_PROPOSER=$(cat sepolia/validator_data.json)

build images for Sepolia

docker compose -f docker-compose-sepolia.yml build

if you have less than 64G memory on the machine increase the swap file size to 64G
Start containers

docker compose -f docker-compose-sepolia.yml up -d

Look at searcher logs

docker compose -f docker-compose-sepolia.yml logs -f searcher

Delete containers and volume

docker compose -f docker-compose-sepolia.yml down -v

Sepolia Demo

We demonstrated this on Sepolia Test Net using docker-compose-sepolia-combined.yml running the searcher in SGX and the builder not in SGX

Instance of the contract on Sepolia Reward Contract
Block containing the sting bundle Block
Sting transaction signed in SGX with sig_nonce 64393256116112811812186352716830363224904947493293752501720077651081401306790 sting_tx
Evidence transaction with nonce 30136919085670681544726071097920847459004406643340331248827579930501700917868 = Commitment(sting_tx.sig_nonce) evidence_tx

sting-flashbots's People

Contributors

Stargazers

Watchers

Forkers

luluzhou1

sting-flashbots's Issues

Check builder Ra-tls certificate in gramine python

error with attestation phase on remote server with in-kernel driver

See gramineproject/gramine#1217

SGX mode on sepolia

mrenclave does not show up in logs on remote server

Figure out how to get the MRENCLAVE with a gramine command.

questions

Does changing environment variables change mrenclave in gramine?

If not we should double check which variables we should lock down at build time

Document both SGX and non-SGX modes in README

To prevent misunderstandings, organize the README into two main sections, one for each mode.

use condition/healthcheck for searcher startup

healthcheck in builder can be something like:

 curl -k -X POST https://localhost:8545 -H "Content-Type: application/json" -d "{\"method\":\"eth_blockNumber\",\"params\":[],\"id\":1,\"jsonrpc\":\"2.0\"}" | jq .result | xargs printf "%d\n" | > some_block_height

last pipe needs some work

geth-sgx-gramine remote attestation flow

Te goal of this issue is to properly understand the remote attestation flow in geth-sgx-gramine.

See https://github.com/flashbots/geth-sgx-gramine#how-does-the-attestation-work for an introduction.

geth-sgx-gramine leverages gramine's ra-tls interface, documented in https://gramine.readthedocs.io/en/stable/attestation.html#mid-level-ra-tls-interface. A minimal example is provided by gramine at https://github.com/gramineproject/gramine/tree/master/CI-Examples/ra-tls-mbedtls.

It's important to note that gramine's ra-tls flow supports both EPID and ECDSA-based attestations. However, as per their documentation, geth-sgx-gramine only supports DCAP (ECDSA). It seems that it's possible to support EPID as well though by making small modifications to geth-sgx-gramine/attest.c (@njeans did this work).

For both EPID and DCAP, communication with a trusted entity is necessary to verify the SGX quote. In the case of EPID it's IAS (Intel Attestation Service), and in the case of DCAP, it's some attestation infrastructure, like PCCS.

If we are using EPID, we can just keep on using IAS, which we know how.

If we are to use DCAP, then we could use PCCS. We currently know how to run a PCCS server, in a docker container, such that it could be integrated with our docker based setup. A more thorough understanding of the overall flow is necessary.

Links

remote attestation: is the report sent to IAS?

It looks like the remote atesttation report is generated for the given SPID (RA_CLIENT_SPID env var), but it is not sent to Intel for verification. Sending a the report to Intel is done via an HTTP POST request and requires an API key is paired with the SPID.

Build docker images for beacon chain and geth

These images don't change, so we can build them and just use the pre-built ones. This will save time for the initial setup when invoking docker compose build.

docker image for kernel driver

To build for the in-kernel driver, access to in-kernel driver is needed, with the --privileged option. See, for instance:
https://github.com/falcosecurity/falco/blob/cfc96e899b352f68992b1214fec23d7584d1114b/docker/falco/Dockerfile#L5

The idea is to have an image with all deps installed, and then to run a container, and mount the include/ dir where sgx.h is (/usr/src/linux-headers-5.15.0-69-generic/arch/x86/include/uapi/asm/sgx.h), and perhaps more, like /usr/src/linux-headers-5.15.0-69-generic/arch/x86/include/, and to build gramine then.

Alternatively, just use the current gramine release (deb package), in a docker image. Perhaps there will be issues with Python dependencies, but perhaps is simpler than the above.

relayer setup.sh problem

root@8e93ba3683fb:/Sting-Flashbots/relayer/src# ./setup.sh
+ ./build.sh
rm -f -r /shared-data/relayer_key.pem
rm -f -r /Sting-Flashbots/relayer/enclave_data/* /Sting-Flashbots/relayer/output_data/* /Sting-Flashbots/relayer/input_data/*
rm -f *.manifest *.manifest.sgx *.token *.sig OUTPUT* *.PID
rm -f -r /Sting-Flashbots/relayer/src/enclave/__pycache__
gramine-manifest \
	-Dlog_level=error \
	-Darch_libdir=/lib/x86_64-linux-gnu \
	-Dentrypoint=/usr/local/bin/python3.10 \
	-Dra_type=none \
	-Dra_client_spid= \
	-Dra_client_linkable=0 \
	-Dpython=python3.10 \
	-Ddebug= \
	-Dvenvpath=/root/.venvs/sting \
	-Dsealing_key=default \
	-Dsealing_filetype=chroot \
	python.manifest.template >python.manifest
+ [[ 0 == 1 ]]
+ GRAMINE=gramine-direct
+ cd /Sting-Flashbots/relayer/src/
+ make reset
rm -f -r /shared-data/relayer_key.pem
rm -f -r /Sting-Flashbots/relayer/enclave_data/* /Sting-Flashbots/relayer/output_data/* /Sting-Flashbots/relayer/input_data/*
+ cp /Sting-Flashbots/chain/build/contracts/Honeypot.json /Sting-Flashbots/relayer/input_data/Honeypot.json
+ cp -r /Sting-Flashbots/chain/keystores/relayer/ /Sting-Flashbots/relayer/input_data/relayer/
+ gramine-direct ./python ./enclave/gen_key.py
error: PAL failed get_topology_info() failed: No such file or directory (ENOENT)

Simulation mode problem: Permission denied: '/data/...'

To reproduce the first error that pops up when running the relayer setup.sh script, just run:

make reset
cp /Sting-Flashbots/chain/build/contracts/Honeypot.json /Sting-Flashbots/relayer/input_data/Honeypot.json
cp -r /Sting-Flashbots/chain/keystores/relayer/ /Sting-Flashbots/relayer/input_data/relayer/

gramine-direct python ./enclave/gen_key.py 
Traceback (most recent call last):
  File "//./enclave/gen_key.py", line 18, in <module>
    ecall_gen_private_key()
  File "//./enclave/gen_key.py", line 10, in ecall_gen_private_key
    with open(relayer_private_key_path, 'wb') as f:
PermissionError: [Errno 13] Permission denied: '/data/private_key.pem'

Note: /data is an encrypted (aka sealed) mounted directory, in the generated python.manifest:

[[fs.mounts]]
type = "encrypted"
path = "/data/"
uri = "file:/Sting-Flashbots/relayer/enclave_data/"
key_name = "_sgx_mrenclave"

It is defined in python.manifest.template:

fs.mounts = [
  # ...
  { type = "encrypted", path = "/data/", uri = "file:/Sting-Flashbots/relayer/enclave_data/", key_name = "_sgx_mrenclave" },
]

root@8602ac5bbf63:/Sting-Flashbots/relayer/src# ./setup.sh
+ ./build.sh
rm -f -r /shared-data/relayer_key.pem
rm -f -r /Sting-Flashbots/relayer/enclave_data/* /Sting-Flashbots/relayer/output_data/* /Sting-Flashbots/relayer/input_data/*
rm -f *.manifest *.manifest.sgx *.token *.sig OUTPUT* *.PID
rm -f -r /Sting-Flashbots/relayer/src/enclave/__pycache__
gramine-manifest \
        -Dlog_level=error \
        -Darch_libdir=/lib/x86_64-linux-gnu \
        -Dentrypoint=/usr/local/bin/python3.10 \
        -Dra_type=epid \
        -Dra_client_spid= \
        -Dra_client_linkable=0 \
        -Dpython=python3.10 \
        -Ddebug= \
        -Dvenvpath=/root/.venvs/sting \
        python.manifest.template >python.manifest
+ [[ 0 == 1 ]]
+ GRAMINE=gramine-direct
+ cd /Sting-Flashbots/relayer/src/
+ make reset
rm -f -r /shared-data/relayer_key.pem
rm -f -r /Sting-Flashbots/relayer/enclave_data/* /Sting-Flashbots/relayer/output_data/* /Sting-Flashbots/relayer/input_data/*
+ cp /Sting-Flashbots/chain/build/contracts/Honeypot.json /Sting-Flashbots/relayer/input_data/Honeypot.json
+ cp -r /Sting-Flashbots/chain/keystores/relayer/ /Sting-Flashbots/relayer/input_data/relayer/
+ gramine-direct ./python ./enclave/gen_key.py
Traceback (most recent call last):
  File "//./enclave/gen_key.py", line 18, in <module>
    ecall_gen_private_key()
  File "//./enclave/gen_key.py", line 10, in ecall_gen_private_key
    with open(relayer_private_key_path, 'wb') as f:
PermissionError: [Errno 13] Permission denied: '/data/private_key.pem'

initc3 / sting-flashbots Goto Github PK

sting-flashbots's Introduction

Sting-Flashbots

Prerequisites

Set Intel IAS SPID

Set SGX driver environment variables

Running in SGX Simulation mode or without SGX

Build docker image

Run demo

Create docker containers for blockchain network, builder, and relayer

Look at Searcher container logs

Stop containers and delete volumes

Running on Sepolia

Setup

Running on Sepolia without SGX

Running on Sepolia with SGX

Download network snapshot

Run demo

Sepolia Demo

sting-flashbots's People

Contributors

Stargazers

Watchers

Forkers

sting-flashbots's Issues

Links

Recommend Projects

Recommend Topics

Recommend Org