Giter VIP home page Giter VIP logo

sygma-fee-oracle's Introduction

Sygma Fee Oracle

Sygma fee oracle is a go implemented, centralized service that provides the endpoints to Sygma UI for all necessary data related to bridging fee.

Architecture Overview

There are three main parts in fee oracle codebase in an abstract way: App base, Data fetcher, Data provider.

  1. App base: This is the combination of base, app, config packages. App base loads the config, preforms health check and starts the server when fee oracle launches. It maintains the cleanup process when the app gets the termination signal.
  2. Data fetcher: This is the combination of oracle, store, cronjob packages. Data fetcher follows the scheduled jobs to fetch data from registered external oracles and store data into store module.
  3. Data provider: This is the combination of store, api, consensus, identity packages. Data provider queries the store based on the request from the endpoint, refines the data based on the configured strategy, then returns the data along with the signature of the fee oracle identity key.

Installation & Build

Make sure Go1.17 has been installed.

This will clone the main branch of fee oracle codebase to your workplace dir and compile the binary into your $GOPATH/bin

$ mkdir workplace && cd workplace
$ git clone https://github.com/ChainSafe/Sygma-fee-oracle.git
$ make install

Configuration

Fee oracle needs three config files in the ./ dir of the codebase:

  1. config.yaml: this is the fee oracle application config file.
  2. domain.json: this is the domain config file.

Application config file config.yaml

Template of the config.yaml can be found in ./config/config.template.yaml.

Domain config

For domain configuration, it is possible to use local file, or the shared configuration file that is uploaded to some remote service (eg. ipfs). Depending on if the URL or file path is provided to the flag domain_config_path, the application will use the domain configuration from local file or from the remote service. This file indicates all the domains and resources the fee oracle needs to fetch data for. Details need to be matched with Sygma core configuration, such as id.

Sygma currently does not support bridging the native currency, such as Ether on Ethereum, Matic on Polygon, however, the id is constructed with zero address and its native domainId and is used in baseRate calculation internally.

Example:

{
  "domains": [
    {
      "id": 0,
      "name": "ethereum",
      "nativeTokenDecimals": 18,
      "nativeTokenSymbol": "eth",
      "resources": [
        {
          "resourceId": "0x0000000000000000000000000000000000000000000000000000000000000001",
          "decimals": 18,
          "symbol": "usdt"
        }
      ]
    },
    {
      "id": 1,
      "name": "polygon",
      "nativeTokenDecimals": 18,
      "nativeTokenSymbol": "matic",
      "resources": [
        {
          "resourceId": "0x0000000000000000000000000000000000000000000000000000000000000001",
          "decimals": 18,
          "symbol": "usdt"
        }
      ]
    },
    {
      "id": 2,
      "name": "moonbeam",
      "nativeTokenDecimals": 18,
      "nativeTokenSymbol": "glmr",
      "resources": [
        {
          "resourceId": "0x0000000000000000000000000000000000000000000000000000000000000001",
          "decimals": 18,
          "symbol": "usdt"
        }
      ]
    }
  ]
}

Fee Oracle Identity

Each fee oracle server associates with a private key, which is used to sign the endpoint response data. There should be a keyfile.priv keyfile in the root dir of the fee oracle codebase, or you can specify which keyfile to use in CLI.

Fee oracle provides key generation CLI, keyfile needs to be generated separately.

Quick Start

To quickly start from makefile, make sure config.yaml, domain.json, resource.json and keyfile.priv are ready in the root dir of the codebase, then execute:

$ make start

Command Line

Fee oracle provides CLI.

For general help:$ sygma-fee-oracle -h

$ sygma-fee-oracle server

Start Sygma fee oracle main service

Usage:
  sygma-fee-oracle server [flags]

Flags:
  -c, --config_path string
  -d, --domain_config_path string
  -h, --help                          help for server
  -t, --key_type string               Support: secp256k1
  -k, --keyfile_path string
  -r, --resource_config_path string

$ sygma-fee-oracle key-generate

Start Sygma fee oracle identity key generation

Usage:
  sygma-fee-oracle key-generate [flags]

Flags:
  -h, --help                  help for key-generate
  -t, --key_type string       Support: secp256k1 (default "secp256k1")
  -k, --keyfile_path string   Output dir for generated key file, filename is required with .priv as file extension (default "keyfile.priv")

Unit Test

$ make test

Gosec Checking

$ make check

Lint Checking

$ make lint

Using Docker

Fee oracle provides a Dockerfile to containerize the codebase. To build docker image:

$ docker build -t fee_oracle .

To run docker container:

$ docker run -p 8091:8091 -it fee_oracle

8091 will be the exposed part for the endpoint access.

Note: fee oracle requires a private key file when starting, this key file must be a secp256k1 type and named as keyfile.priv and put in the ./ dir of the codebase when building docker image, if no keyfile exists in ./ dir, fee oracle will auto generate a secp256k1 keyfile to use.

End to End Test

This will start fee oracle service, ganache-cli locally, install solcjs, abigen and generate contract go binding code, deploy fee handler contracts to local ganache. $ make e2e-test

EVN Params

APP_MODE=release                                             // app mode: debug or release. app mode is used for internal testing only.
IDENTITY_KEY=                                                // fee oracle prikey in hex, without 0x prefix
IDENTITY_KEY_TYPE=secp256k1                                  // fee oracle prikey type, only support secp256k1 for now
WORKING_ENV=production                                       // fee oracle app running mode: dev or production
LOG_LEVEL=4                                                  // log level, 4 is info, 5 is debug, using 4 on production
HTTP_SERVER_MODE=release                                     // fee oracle http server running mode: debug or release
HTTP_SERVER_PORT=8091                                        // fee oracle http server exposed port
CONVERSION_RATE_JOB_FREQUENCY="* * * * *"                    // conversion rate job frequency, using cron schedule expressions(https://crontab.guru)
GAS_PRICE_JOB_FREQUENCY="* * * * *"                          // gas price job frequency, using cron schedule expressions(https://crontab.guru)
ETHERSCAN_API_KEY=                                           // api key of etherscan
POLYGONSCAN_API_KEY=                                         // api key of polygonscan
COINMARKETCAP_API_KEY=                                       // api key of coinmarketcap
MOONSCAN_API_KEY=                                            // api key of moonscan
DATA_VALID_INTERVAL=3600                                     // Time of valid fee oracle response data in seconds
CONVERSION_RATE_PAIRS=eth,usdt,matic,usdt                    // conversion rate pairs that enabled for price fetching. Must be paired

API Documentation

Swagger API Doc

License

Business Source License 1.1

sygma-fee-oracle's People

Contributors

eedygreen avatar freddyli7 avatar makmuftic avatar mpetrun5 avatar mschirbel avatar nmlinaric avatar p1sar avatar rbadillap avatar tcar121293 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

sygma-fee-oracle's Issues

Filter gas price by domainID

Seems to be more correct to filter gas price by domainID instead of domain name which seems arbitrary and is not used anywhere else.

Will be a lot more crucial when we implement proper database support and currently we have to edit shared config names to match gas price domains instead of matching domain IDs which are always correct.

Implementation details

Testing details

Acceptance Criteria

Move local config.yaml into env var

Move local config.yaml into env var

Implementation details

Read the data that is currently in config.yaml from env var
Make necessary changes in devops repo also

Testing details

Acceptance Criteria

config.yaml file is removed
configuration data is read from env var

Add support for shared configuration [fee oracle]

Fee oracle needs to be refactored so it supports the new shared configuration.

Implementation details

  • Load shared configuration from IPNS URL
  • Restructure fee oracle codebase so it only uses one configuration (currently, it loads domains and resources separately)
  • Extract address prefix information to in-code constant

Testing details

  • Update all unit and e2e tests regarding made changes
  • Add additional tests for loading shared configuration

Acceptance Criteria

  • Passing unit and e2e tests
  • Successful devnet shared configuration loading

Add custom tags to external integration logs

Add specific tags for each service (Etherscan, Coinmarketcap...) to logs when the query is successfull and it fails (not 200).

Implementation details

Add custom Str() field to the logs with the tag for service.

Testing details

Acceptance Criteria

Add substrate chain RPC fetching support

The fee oracle must make an RPC call to the substrate to fetch inclusion_fee when providing a fee estimate for a bridging request to the substrate-based chain. Support should be added in a general way so that fee oracle can call arbitrary substrate RPC calls.

Implementation details

Testing details

  • Add unit tests

Acceptance Criteria

  • Fee oracle is able to make substrate RPC calls
  • Fee oracle can call Phala<>Sygma pallet to fetch inclusion_fee

Implement environment config with viper

Instead of fetching each param separate from environment use something like viper (relayer has custom logic because of chain configs) to do that to remove ifology when fetching params.

Implementation details

Testing details

Acceptance Criteria

Automated swagger documentation

The Swagger API documentation is currently not automatically updated upon new releases. This results in the link in the README file pointing to v1.0.0 of the API, which is hardcoded. Even if we change the URL to point to the latest version, the Swagger docs will still show v1.0.0 as it is the last released version of swagger docs.

Implementation details

To resolve this issue, we should automate the release of new versions of the Swagger docs using a GitHub action. This will ensure that the latest version of the documentation is always available and up to date.

Acceptance Criteria

  • The link inside the README file should point to the latest version of the Swagger API documentation.
  • A GitHub action should be created to release new Swagger docs versions automatically.

Add PHA market price rate support

Add PHA asset support in fee oracle.

Implementation details

As PHA is supported on the marketplace, this change should be just a matter of expanding configuration, so information on PHA token price is periodically fetched and persisted.

Testing details

  • Manually test

Acceptance Criteria

  • Fee oracle should be able to return the PHA price rate

Refactor conversion rate pairs property

Currently, fee oracle requires us to define all possible conversion pairs manually. This, paired with the format of this configuration property, makes it really hard to maintain this in any real production environment.

It would be ideal for removing this as a configuration property fully.

Implementation details

I see two approaches here:

To further facilitate these approaches, we can expand the shared configuration for each individual resource.
Add property feeType - "basic" / "oracle". This would mark all tokens that are supported by fee oracle

  1. Currently, we are calculating rates for each defined pair and persisting it. We could parse shared config, extract tokens that are marked to use oracle, and then only fetch the value of each supported token (in $) and persist these values. Then, when someone requires specific pair, we just divide $ values to get the rate. This would significantly reduce complexity, as adding a new token to fee oracle requires us just to add one more query toward a service like coinmarketcap. The biggest con to this approach is that we are using $ price as a medium for calculating rate (but probably services like coinmarketcap are also using this under the hood).

  2. Second approach would be to parse shared config, extract tokens that are marked to use oracle, and then query rates for all possible pairs between them. This approach removes the necessity for us to define all the pairs for fee oracle manually but still has this problem of rapidly increasing the number of queries as we start supporting new tokens.

cc: @freddyli7 @P1sar @mpetrun5, what do you think about this? I am leaning more toward the 1) option.

Testing details

TBD

Acceptance Criteria

TBD

Periodically fetch shared configuration

Having fee oracle periodically fetch shared configuration would enable us not to have downtime on fee oracle service.

Implementation details

Testing details

Acceptance Criteria

Update config.yaml

Update config.yaml

Implementation details

Update config.yaml so that is up to date with shared config(add additional config if necessary)
Update ecs in devops repo if necessary

Testing details

Acceptance Criteria

devnet is working without errors

Fix SSM params fetching

Currently, SSM params are being fetched by hardcoded value

remoteParamDomainData = "/chainbridge/fee-oracle/domainData"

Unfortunately among different instances, this value should be different
TestNet https://github.com/sygmaprotocol/sygma-fee-oracle/pull/11/files#diff-276fe3cf568afda7033fcc8889aff02c5d11d9977ae4c261572d3d491d2bee99R98
DevNet https://github.com/sygmaprotocol/sygma-fee-oracle/pull/11/files#diff-8ea3a6a5a58f6a3b68fd479c8a2fc6972adb12b777348a44d28b18521424baf1R110
So we need to make these params configurable

Implementation details

Testing details

  • check thet local setup works
  • Unit tests
  • e2e tests

Acceptance Criteria

  • Code changed
  • Ronny's PR with new task definition updaated with new params

Assign gas price APIs per domain

Related to #24. Avoid using domain names to match domains and assign gas price APIs per domain.
Also, remove custom polygonscan implementation as it is the same as Etherscan.

Currently, it limits future possibilites to resuse Etherscan implementations on networks outside of ethereum (there are many which have Etherscan supported) and it cannot match gas prices per domainID which will be important for proper database support.

Implementation details

Testing details

Acceptance Criteria

Fix large floating point calculation error

Expected Behavior

Current Behavior

Possible Solution

Steps to Reproduce (for bugs)

Versions

ChainBridge commit (or docker tag):
chainbridge-solidity version:
chainbridge-substrate version:
Go version:

FeeOracle changes to include gas limit in feeData

Implementation details

Expand endpoint from: /{version}/rate/from/{fromDomainID}/to/{toDomainID}/resourceid/{resourceID}
to something like this:
/{version}/rate/from/{fromDomainID}/to/{toDomainID}/resourceid/{resourceID}/gasLimit/{msgGasLimit}

  • Add this gas limit information to the signed fee data.
  • Do some basic value checks on this parameter

Testing details

  • Expand unit and e2e tests based on changes

Acceptance Criteria

  • Passing unit/e2e tests
  • Working endpoint

Better support for testnets

Currently, we need to have a hardcoded mapping for resources on testnets for which we want to query values from APIs like coinmarketcap. The reason behind this is that price APIs are being queried with the token symbols, so we need to remap testnet token symbols to something that actually has a price.

Suggested solution

The problem with this is that currently, we have a hardcoded remap inside code. My idea would be to add an additional work mode for fee oracle. Currently, fee oracle works in two modes:

  • debug => local setup/e2e tests
    Always returns hardcoded response
  • release => devnet/testnet/mainet

After refactoring, it would have three different modes:

  • debug => local setup/e2e tests
    Always returns hardcoded response
  • test => devnet/testnet
    Same as release, just on startup fee oracle can accept additional mapping of resourceID<>symbol that would allow us to remap test tokens to some actual mainet tokens for which we can fetch price data.
  • release => mainet

Testing details

  • Manually test different modes localy
  • Add unit test for any new logic

Acceptance Criteria

  • Removed hardcoded mapping for testnet tokens
  • Working e2e tests
  • Working devnet fee oracle
  • Working testnet fee oracle

Add MongoDB support

Currently, fee oracle is using levelDB, but we should migrate to something that will scale better with the growing data set.

Implementation details

Ensure that communication toward DB is abstracted so we can always exchange it with another implementation.

Testing details

  • Add units tests

Acceptance Criteria

  • Fee oracle supports persisting data to MongoDB
  • Passing unit and e2e tests

Add subtrate support to API

Expand fee oracle API to support fee data requests where the destination chain is a substrate-based chain with deployed Sygma pallet.

Implementation details

  • Details of fee data message format can be found in technical documentation (will be available after this PR is merged)
  • API endpoint handler function should be expanded so it takes into consideration the destination domain (network) type when constructing a response
  • Additional query property should be supported (length_fee) when the user requests a generic call fee estimate with the substrate-based destination chain.

Testing details

  • Add unit test

Acceptance Criteria

  • Fee Oracle should return appropriate fee data when requested for the destination chain being substrate-based

Make sure that feeOracle is testnet ready

  • test feeOracle on testnet environment and make sure that all configuration is correctly configured
  • make all necessary changes and updates (check contract configuration, env variables...)

Implementation details

Testing details

Acceptance Criteria

feeOracle can be used on testnet for GMP

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.