xlabs / wormhole Goto Github PK

User Story
As a developer, I want to be able to effectively test load balancing for the generic relayers by changing the number of wallets used from 1 to multiple wallets, so that I can increase the throughput of messages hitting the generic relayer

Acceptance Criteria

• Precondition: Wallets already all exist
• Will be an array of wallets that the service will be able to use for load testing
• This will need to be able to support the main load test script and the matrix load test script

• On chain deployments:
  • WH-CCTP Integration contract deployment
  • CCTP Relayer Contract Deployment & registrations
• Off chain configuration
  • USDC cannonical address
  • CCTP Contract address
  • CCTP Relayer Contract Address
  • Wallet Balance for Base
  • Wallet monitoring for Base
  • Wallet alerts for Base

Overview

This issue is for grouping tasks that relate to extending the core relayer functionality with wallet monitoring capabilities for metrics and alerting.

Related Tasks:

Not to be mistaken with relay-status-api.

We need a way to expose the health and status of our relayers to our integrators.

Health probably needs to be exposed at a chain level (emitter chain? target chain? both? each?)

We can probably expose a list of compromised/delayed/missing(needing re-observation) vaas.

Would it make sense to consider streaming events instead of an api? (maybe VAAs for applications for which the time a vaa is redeemed is critical?)

Chains oku will be deployed on:

• Ethereum
• Optimism
• Arbitrum
• Base
• Polygon

Notes:

• Reward address is configured by the gfx team. Our relayer might want to duble check that the configured reward address matches what we expect so that the rewards are not paid to someone else.
• Wont have a testnet deployment
• We can assume the tx gas_units spent is fixed (probably hardcoded/configured on our side)
• The first front end to use this will be connect to we should reach out to @kevin peters
• Cost model:
  • Will always be paid on eth or weth (wweth? stETH and wstETH) on the target(?) chain
  • Won’t perform swaps (the relayer, uniswap will instead)
  • The gas spent by the relayer is:
    • Uniswap contract execution
    • VAA verification
  • Gotchas:
    • we can’t simulate the transaction so we don’t know the gas cost of a relay.

TODOs:

• document the interface we’ll use on our API and communicate it to gfx
• review Oku contracts:
  • are we paid in the case of reverts?
  • how is reward address handled/configured?
  • do they expose a way for us to check what the reward address is?
  • are contracts upgradable?
  • what would be a good way for us to manage the reward address registration on their contracts?
• Implementation of API
  • api endpoint implementation
  • kubernetes manifests
  • integration tests
  • endpoint that responds OpenAPI spec

Open questions/tech debt:
- Relayer validation of
- the quote matches the VAA (target chain, )
- the quote is not stale
- the reward address is the expected one

The API should contain these items as part of its high level documentation (i.e. its OpenAPI/swagger description).

• Relayer gets paid in destination chain with tokens denominated in the result of the swap.
  • If the swap fails, the tokens paid to the relayer are the original tokens instead.
• We expect to support only one pair for now but the API should handle quoting different pairs.
• The API should require specifying the destination chain to ensure we quote based on its gas price.

Things not supported:

• Routing a swap through several token hops to maximize swap returns.

The reasons to do it off-chain:

• We don't need to pay for transactions to keep fees updated on the contracts (cheaper)

Stakholders:

• Oku
• W7

• unhandled rejections
• relayer already redeemed error metrics
• spy connection dropped?
• redis connection dropped?

Working document: https://www.notion.so/xlabsxyz/Relayers-PnL-Model-0410be2eb96b4843b8dd0f4133c17378

Questions to answer:

• Where will this code live?
• How can we feed the alert data from a "foreign" json file
• We want to focus on a simple solution that can be implemented on ~3 weeks

Notes: We don't need dashboards in the first iteration.

When the relayer, through missed-vaa worker, knows that a VAA has been missed by the guardian, we should try to issue a re-observation request.
Re-observation is performed by issuing a command that requires access to the guardian private socket, which poses a challenge in terms of security.

The following diagram is the result of the preliminary talk about this subject:

The goal of this spike is to:

• Formalize a design we would use to automate re-observation
• Get the solution design approved by the wormhole-prod-ops team (Jeff?)
• Create the necessary tasks to perform such automation

• Deploy new core implementation on Testnet.
• Write Testnet self sign and upgrade script.
• Execute Testnet upgrade.
• Deploy new core implementation on Mainnet.
• Generate Mainnet message and request Guardians to sign it.
• Execute Mainnet upgrade.

Notion Doc With Diagramas and Code:
Liquidity Layer Off-Chain

- Pays for CCTP transfers on slow finality chains upfront and earns a fee against assuming the risk of loosing the “slow transfer” due to a chain re-org. (Needs to have a higher usdc liquidity on the chains it pays for relays)
- Uses a fast finality chain as a transfer router called “hub” (avalanche). All transfers are routed through this Hub, which in turn distributes the different payments
- Competes on an auction against other providers for the relay (Needs to be able to react to the auction result)
- When a fast transfer is requested, there are x possible outcomes:
    - No relayer bids on the opportunity to do a fast transfer and a normal “slow” transfer is executed instead
    - Our relayer bids on the opportunity to do a fast transfer but doesn’t win the auction. (nothing else to do? what happens if the relayer that won the bid never finishes the relay? how do we get the slow transfer to still?)
        - might be worth to close the auction if the winning relayer failed to close the auction
    - Our relayer bids on the opportunity to do a fast transfer and wins the auction:
        - it completes the relay in time (grace period?) and gets the full reward
        - it completes the relay later than expected but before the slow transfer and gets (its compensation?) slashed
        - the slow transfer kicks in before the relayer finished the transfer. The slow transfer is executed and the fast transfer cannot be executed any longer

Setup the xlabs NPM registry on Github

• CCTP monitoring
  • update relayer engine version to include last monitoring updates
  • dashboard revamp

• On chain deployments:
  • WH-CCTP Integration contract deployment
  • CCTP Relayer Contract Deployment & registrations
• Off chain configuration
  • USDC cannonical address
  • CCTP Contract address
  • CCTP Relayer Contract Address
  • Wallet Balance for Base
  • Wallet monitoring for Base
  • Wallet alerts for Base

The current RE supports setting the number of workflow retries, from zero to positive infinity through the retries configuration field. This is useful but special applications could require special handling about how often you trigger the same operation.

In particular, at C3 we call special endpoints when relaying deposit operations. If we do not space the frequency when hitting the remote server, Cloudflare will reject us with timeouts , 502 or similar errors. In this scenario, something like retrying for 2, 4, 8, 16 /.... seconds can work.

I dont know about the precise underlying BullMQ support for this, but probably the backoff should be increasing up to a limit defined by 'plugin' configuration.

Thanks in advance.

On chain deployments:

• WH-CCTP Integration contract deployment
• CCTP Relayer Contract Deployment & registrations

Off chain configuration

• USDC cannonical address
• CCTP Contract address
• CCTP Relayer Contract Address
• Wallet Balance for Polygon
• Wallet monitoring for Polygon
• Wallet alerts for Polygon

Testnet Information

TokenMessenger: Testnet

MessageTransmitter: Testnet

TokenMinter: Testnet

USDC Contract: Testnet
0x9999f7fea5938fd3b1e26a12c3f2fb024e194f97
https://mumbai.polygonscan.com/token/0x9999f7fea5938fd3b1e26a12c3f2fb024e194f97

Goals:

• Optimize VAA parse and verify in a way that is backwards compatible. I.e. so that current contracts see a benefit without changing any code.
• Write tests that are necessary to ensure any new low level implementation details are correct.

Approach:

• Store guardian set in the bytecode of another account created with CREATE2.
• Minimize memory pressure by avoiding repeated use of abi.* family of functions.
• Cherry pick packed guardian set representation that Drew wrote.

There's two small problems with the Fetch VAA implmenetation on the relayer engine:

• if the guardian is down, the request will never timeout effectively blocking all other processing on the relayer engine (more info on thread
• If there's an error fetching the VAA for the missed-vaa worker, the error message will not be logged (instead only the error stack is logged). More info on thread
• When hitting wormscan to get the VAA we often see a rate-limitting error (status-code = 429). Fix this issue by setting a backoff for retries and/or speaking to wormscan team to review the rate-limitting configuration

On chain deployments:

• WH-CCTP Integration contract deployment
• CCTP Relayer Contract Deployment & registrations

Off chain configuration

• USDC cannonical address
• CCTP Contract address
• CCTP Relayer Contract Address
• Wallet Balance for Polygon
• Wallet monitoring for Polygon
• Wallet alerts for Polygon