there is plenty of config options scattered across the codebase, i think it's time we move them to a dedicated config.

use pydantic for loading the config file and schema validation

• https://pydantic-docs.helpmanual.io/usage/settings/

In this transaction, we can see that 1 curve LP for the rocketpool wsteth pool is valued near 1 wsteth.
https://etherscan.io/tx/0x7684daf36307027d2c724b1349d62a36428be75cafa8d9badc8085cb3338b768

In yearn exporter, we have an inaccurate price returned when querying for the price of this same LP. We get $0.066 when it should be closer to $2,900.

with v3 release around the corner (next week-ish), it would be helpful to have it supported in the exporter

check if it's feasible to bump the python requirement to 3.10, which gives up pattern matching and some performance improvements.

there were some incompatible libraries which have upgraded since

• https://github.com/ponyorm/pony/releases/tag/v0.7.15rc1
• https://github.com/psf/black/releases/tag/21.12b0

need to check if there are other problems.

0xJiji, [15 Nov 2021, 05:22:59 (15 Nov 2021, 05:17:40)]:
yvUSDC Theta Vault address: 0x8FE74471F198E426e96bE65f40EeD1F8BA96e54f

Lookup from Opyn Controller: https://etherscan.io/address/0x4ccc2339F87F6c59c6893E1A678c2266cA58dC72#readProxyContract

1. First, call the accountVaultCounter function accountVaultCounter(0x8FE74471F198E426e96bE65f40EeD1F8BA96e54f)
2. Then call the getVault function getVault(0x8FE74471F198E426e96bE65f40EeD1F8BA96e54f, 20)
3. The deposited yvUSDC amount is the last number in the tuple

Relevant telegram thread: https://t.me/c/1410227556/4218

Multiple networks using ID 'mainnet-custom'

Aggregate the calls needed to describe each vault/strategy and request all the needed data at once.

This should help with Infura limits.

Background

APY data for all data is is generated and published to S3. The data is consumed and displayed in various places, including https://yearn.finance .

It is critical this data be reliably generated and published to S3. Currently, the data is generated a published using systemd just running on a random machine that is not actively managed or monitored.

Goal

We want to publish publish this data in a more reliable way. More specifically we want:

1. Run the script in a docker container using a scheduled task on AWS Fargate. The infrastructure should be defined using AWS CDK .
2. Capture any errors when generating this data with Sentry.
3. Export useful metrics to Graphite. Create a dashboard in Grafana with alerting.

Each step can be implemented in parallel.

[2022-02-27 15:45:14.379505]
🔥 API (vaults) update for Fantom failed!
Traceback (most recent call last):
File "/yearn-exporter-fantom/scripts/s3.py", line 237, in with_monitoring
main()
File "/yearn-exporter-fantom/scripts/s3.py", line 180, in main
raise EmptyS3Export(f"No data for vaults was found in generated data, aborting upload")
yearn.exceptions.EmptyS3Export: No data for vaults was found in generated data, aborting upload

since ironbank has spun out into its own product, tvl exporter on all networks should stop counting it as yearn tvl after 2022-01-22T00:00:00Z.

https://twitter.com/ibdotxyz/status/1484526289167867909

https://defillama.com/protocol/iron-bank

it should accept an array of tokens, and return token prices for each, ideally in the format [{"address": "0x...", "priceUsdc: ..."}]

0x0100546F2cD4C9D97f798fFC9755E47865FF7Ee6
alETH token

Sometimes S3 script generates empty files which overwrites the data in the S3 bucket and effectively brings down the API used to populate the APY on the website.
Add a simple check before S3 upload is done to prevent above situation.

The following contract crv3crypto-gauge fails because it does not have a controller function and its curve registry also doesn't have a gauge_controller function:
https://ftmscan.com/address/0x00702BbDEaD24C40647f235F15971dB0867F6bdB

https://github.com/yearn/yearn-exporter/blob/master/yearn/apy/curve/simple.py#L52
https://github.com/yearn/yearn-exporter/blob/master/yearn/prices/curve.py#L64

sentry issue:
https://sentry.yearn.vision/organizations/yearn/issues/917/?environment=production&project=2&query=is%3Aunresolved

A high-level overview can be found here https://docs.yearn.finance/partners/integration_guide#delegated-deposit

Blocks #193

add minimal support for gnosis chain since we are contemplating deploying there.

• add to networks
• update oracles
• vault registry

Previously the vault (0xA696a63cc78DfFa1a63E9E50587C197387FF6C7E) fee split for badger the recipient was 0xB65cef03b9B89f99517643226d76e286ee999e77, moving forward it will require to be 0xD0A7A8B98957b9CD3cFB9c0425AbE44551158e9e

add basic support for avalanche since yearn is contemplating deploying there

• add to networks
• add price oracles

as per @mariuspod idea, we could avoid duplicating the data and store everything in postgres, getting the best of both worlds

we could even keep the grafana dashboard code intact by using promscale, which would translate the promql queries

it's currently cash + borrowed - reserves

so we can onboard partners on other networks like fantom

token https://etherscan.io/address/0x29059568bb40344487d62f7450e78b8e6c74e0e5
pool https://etherscan.io/address/0xc26b89a667578ec7b3f11b2f98d6fd15c07c54ba

1. git clone [email protected]:yearn/yearn-exporter.git
   2.brownie run treasury throws an error:

milkyklim@laptop ~/p/y/yearn-exporter (master)> brownie run treasury
Brownie v1.15.2 - Python development framework for Ethereum
Generating interface ABIs...
YearnExporterProject is the active project.
  File "brownie/_cli/run.py", line 49, in main
    return_value, frame = run(
  File "brownie/project/scripts.py", line 53, in run
    module = _import_from_path(script)
  File "brownie/project/scripts.py", line 149, in _import_from_path
    _import_cache[import_str] = importlib.import_module(import_str)
  File "/Users/milkyklim/.pyenv/versions/3.8.6/lib/python3.8/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen, line line, in in
  File "<frozen, line line, in in
  File "<frozen, line line, in in
  File "<frozen, line line, in in
  File "<frozen, line line, in in
  File "<frozen, line line, in in
  File "./scripts/treasury.py", line 8, in <module>
    from ypricemagic import magic
ModuleNotFoundError: No module named 'ypricemagic'

Introduced in #111. @BobTheBuidler can you check if all files are in place?

token https://etherscan.io/address/0x29059568bb40344487d62f7450e78b8e6c74e0e5
pool https://etherscan.io/address/0xc26b89a667578ec7b3f11b2f98d6fd15c07c54ba

creation tx https://ethtx.info/mainnet/0x9718eb7885e31ac87e72ecd57cb826438574295f25c19db17988687fab1445fd/
factory https://etherscan.io/address/0xf18056bbd320e96a48e3fbf8bc061322531aac99#code

To reproduce

In a file e.g. scripts/incorrect_prices.py

from yearn.prices.magic import get_price

def eursusdc():
    price = get_price("0x3D229E1B4faab62F621eF2F6A610961f7BD7b23B")
    print(price)

def eurtusd():
    price = get_price("0x3b6831c0077a1e44ed0a21841c3bc4dc11bce833")
    print(price)

brownie run incorrect_prices eursusdc
brownie run incorrect_prices eurtusd

For eursusdc the outputted price is ~$10415.46
This should be (1 eurs + 1 usdc) i.e. ~$2.16

The cause

The registry is providing incorrect information about the number of decimals

1. Go to https://etherscan.io/address/0x90E00ACe148ca3b23Ac1bC8C240C2a7Dd9c2d7f5#readContract
2. Call get_decimals with the eursusdc pool (0x9838eccc42659fa8aa7daf2ad134b53984c9427b)
3. The output is (2,6)
4. This is in the incorrect order, and should be (6,2)

The fix

Use the new V2 registry at 0x4AacF35761d06Aa7142B9326612A42A2b9170E33 which provides the correct values. Fall back to the current registry if get_pool_name is blank.

Likewise the decimal data provided by the registry for eurtusd is incorrect

this is a second test from gh

Makes no sense to show the chart (only) with dollar TVL. ETH as base currency should at least be an option. Otherwise it is difficult to say whether there is really any growth at all or whether the chart just shows the price growth of ETH and the corresponding tokens.

Currently we have our code for the various api endpoints in multiple repos:

• /v1 routes: https://github.com/yearn/yearn-exporter/blob/master/yearn/api.py
• apybot(kx9x): https://yearnapybot.azurewebsites.net/api/chains/250/vaults/0x0DEC85e74A92c52b7F708c4B10207D9560CEFaf0/pps/1d
• yearn-exporter-api (not published yet): https://github.com/mariuspod/yearn-exporter-api

This has the following drawbacks:

• request sending/parsing is duplicated, possibly in different programming languages
• portability is weak

Therefore we should merge this into a single repo, possibly yearn-exporter as it's the closest to the data and we have already the /v1 routes in it.

We can infer which numeric parameters an unknown strategy has to offer by introspecting the contract's ABI.

It would still require to manually specify which ones are to be scaled.

A simple example:

>>> {x['name']: getattr(s, x['name'])() for x in s.abi if x['type'] == 'function' and x['stateMutability'] == 'view' and
 not x['inputs'] and x['outputs'][0]['type'] == 'uint256'}
{
    'blocksToLiquidationDangerZone': 46500,
    'collateralTarget': 730000000000000000,
    'debtThreshold': 0,
    'dyDxMarketId': 3,
    'estimatedTotalAssets': 0,
    'expectedReturn': 0,
    'getCurrentPosition': (0, 0),
    'getblocksUntilLiquidation': 115792089237316195423570985008687907853269984665640564039457584007913129639935,
    'minCompToSell': 100000000000000000,
    'minReportDelay': 86400,
    'minWant': 0,
    'netBalanceLent': 0,
    'predictCompAccrued': 0,
    'profitFactor': 100,
    'storedCollateralisation': 0
}

We have prod vaults on arbitrum this would add them to the exporter.

A high-level overview can be found here https://docs.yearn.finance/partners/integration_guide#delegated-deposit

Blocks #193

This can be seen if you try to view yvsUSD 3.3 vault's strategies, it only has one SingleSidedCrvsUSD when in fact there are two.

using Contract.from_explorer fails

>>> c1 = Contract.from_explorer('0x2f2a2543B76A4166549F7aaB2e75Bef0aefC5B0f')
>>> c1.name()
  File "<console>", line 1, in <module>
  File "brownie/network/contract.py", line 765, in __getattribute__
    raise AttributeError(f"Contract '{self._name}' object has no attribute '{name}'")
AttributeError: Contract 'ClonableBeaconProxy' object has no attribute 'name'
>>> c1.abi
[
    {
        'inputs': [],
        'stateMutability': "nonpayable",
        'type': "constructor"
    },
    {
        'stateMutability': "payable",
        'type': "fallback"
    },
    {
        'stateMutability': "payable",
        'type': "receive"
    }
]

using interface.ERC20 works

>>> c2 = interface.ERC20('0x2f2a2543B76A4166549F7aaB2e75Bef0aefC5B0f')
>>> c2.name()
'Wrapped BTC'

https://sentry.yearn.vision/organizations/yearn/issues/1050/?environment=aws-production&project=2&query=is%3Aunresolved

check if it's feasible to bump the python requirement to 3.10, which gives up pattern matching and some performance improvements.

there were some incompatible libraries which have upgraded since

• https://github.com/ponyorm/pony/releases/tag/v0.7.15rc1
• https://github.com/psf/black/releases/tag/21.12b0

need to check if there are other problems.

There's an unverified contract for the PHP token 0xFa247d0D55a324ca19985577a2cDcFC383D87953 of a test vault on arbitrum which breaks the s3 apy calculations for arb:

https://sentry.yearn.vision/organizations/yearn/issues/1001/?environment=aws-production&project=2

Running it at its current speed will use up a free 100k requests infura account in a few hours.

APY is erroring for the EURT vault 0x0d4EA8536F9A13e4FBa16042a46c30f092b06aA5

Add this to the bottom of scripts/apy.py

def eurt():
    calculate_apy("0x0d4EA8536F9A13e4FBa16042a46c30f092b06aA5")

then run brownie run apy eurt

The following error should be outputted:

Running 'scripts/apy.py::eurt'...\
File "brownie/_cli/run.py", line 51, in main\
return_value, frame = run(\
File "brownie/project/scripts.py", line 103, in run\
return_value = f_locals\[method_name\](\*args, \*\*kwargs)\
File "./scripts/apy.py", line 135, in eurt\
calculate_apy("0x0d4EA8536F9A13e4FBa16042a46c30f092b06aA5")\
File "./scripts/apy.py", line 55, in calculate_apy\
print(json.dumps(dataclasses.asdict(vault.apy(samples)), indent=2))\
File "./yearn/v2/vaults.py", line 201, in apy\
return apy.curve.simple(self, samples)\
File "./yearn/apy/curve/simple.py", line 40, in simple\
raise ApyError("crv", "no gauge")\
ApyError: ('crv', 'no gauge')

there is no feed registry on fantom, so we need to hardcode the list of feeds for now
https://docs.chain.link/docs/fantom-price-feeds/

brownie run apy eurs_usdc

fails with this error:

Running 'scripts/apy.py::eurs_usdc'...
  File "brownie/_cli/run.py", line 51, in main
    return_value, frame = run(
  File "brownie/project/scripts.py", line 103, in run
    return_value = f_locals[method_name](*args, **kwargs)
  File "./scripts/apy.py", line 132, in eurs_usdc
    calculate_apy("0x801Ab06154Bf539dea4385a39f5fa8534fB53073")
  File "./scripts/apy.py", line 55, in calculate_apy
    print(json.dumps(dataclasses.asdict(vault.apy(samples)), indent=2))
  File "./yearn/v2/vaults.py", line 201, in apy
    return apy.curve.simple(self, samples)
  File "./yearn/apy/curve/simple.py", line 43, in simple
    raise ApyError("crv", "no gauge")
ApyError: ('crv', 'no gauge')

Hi
I was trying to run the yearn-exporter on my Ubuntu but I am getting the error related to nginx client starting.
Here is my error message.

2021/07/13 19:02:23 Could not create Nginx Client: expected 200 response, got 404

Is there any additional settings to be customized in prometheus?
Thanks.

Would be effective if we allocated funds so users who access v3 Yearn when it is passed beta phase can earn a reward. I believe Yearn has done this in the past. This would be a good way to reach out to the community and grow our brand

Add Earn v1 and v2 (and v3?) for better tvl stats.

Implement a uniswap v3 price oracle because some tokens don't have enough liquidity and return incorrect values.

rETH: https://etherscan.io/token/0xae78736Cd615f374D3085123A210448E74Fc6393

>>> magic.get_price('0x5c0A86A32c129538D62C106Eb8115a8b02358d57')
DEBUG yearn.prices.magic:101 getting price for token 0xae78736Cd615f374D3085123A210448E74Fc6393 with market UniswapV2Multiplexer
DEBUG yearn.prices.magic:106 UniswapV2Multiplexer -> 0.06600946269098168
DEBUG yearn.prices.magic:106 CurveRegistry -> 0.06606106891317451

https://docs.uniswap.org/protocol/reference/smart-contracts

FTM

could not fetch price for malformed token 0x58e57cA18B7A47112b877E31929798Cd3D703b0f on Fantom at 27501504
https://sentry.yearn.vision/organizations/yearn/issues/534/?environment=staging&project=2

reproduce with brownie:

from yearn.prices import magic
magic.get_price('0x58e57cA18B7A47112b877E31929798Cd3D703b0f', block=27501504)

ETH

failed to get price for malformed token 0xEd4064f376cB8d68F770FB1Ff088a3d0F3FF5c4d on Mainnet at 13967922
https://sentry.yearn.vision/organizations/yearn/issues/651/?environment=staging&project=2

reproduce with brownie:

from yearn.prices import magic
magic.get_price('0xEd4064f376cB8d68F770FB1Ff088a3d0F3FF5c4d', block=13967922)

"Hey guys, we use yearn.science for pulling supported tokens and https://github.com/yearn/yearn-assets for images. I see that you've refactored the GitHub paths to be /multichain-tokens but yearn.science doesn't reflect that just yet"

When BlockNotFound comes up it crashes the exporter, and it has to be manually restarted.