The Bedrock deployment configuration doesn't make sense for cross-team use cases, and should therefore be pulled out into a Bedrock specific repository for managing the Bedrock development instance.

• Autoretrieve container building to ECR on push to master will stay in the autoretrieve repository. For now we'll leave the ECR in the STI account. We might want to move this later to something ARG owns?
• Kubernetes deployment manifests and CD will be moved to a new Bedrock specific repository. It will then just make PRs to that repo's cd/dev branch and deploy to the same sti-dev cluster it's being deployed to now.

Goals

Autoretrieve can produce useful statistics about the performance of retrieval on the network

Requested Metrics

1. Deal Success Rate
2. Deal Acceptance Rate
3. Deal Volume
4. Time to First Byte (TTFB)
5. Avg transfer speed
6. (unique) number of SPs
7. (unique) number of IPFS clients
8. total data volume transferred

What

Add a unit test for the indexer endpoint.

Suggest using a mock like: https://github.com/ipfs-shipyard/w3rc/blob/main/contentrouting/delegated/mock/http.go and modelling after https://github.com/ipfs-shipyard/w3rc/blob/main/contentrouting/delegated/http_test.go

The only other thing I would do here is check to make sure valid records of a type OTHER than FilecoinGraphsyncV1 are dropped.

Currently, setting up the autoretrieve service on EC2 requires a lot of manual steps. Automate the steps to allow for CI/CD build systems and quick deploys.

Add a metric to the dashboard to understand at which retrieval stage did a retrieval fail

An integration test of the flow of autoretrieve:

• Bitswap node makes request to auto retrieve
• Auto retrieve bitswap provider converts to filecoin retrieval
• Pings an indexer node to find content
• Fetches data via filecoin retrieval from a miner

The trick here will be properly setting up all the players, ideally using simple versions or mocked values for the services.

• can use itests for Lotus miner
• probably can manually construct libp2p + bitswap node and just pair it
• not sure how to get the indexer but it's just an http endpoint so we could mock it

Add some system operational metrics to monitor the status of the Autoretrieve service, which can help identify scaling issues and help with capacity planning:

• Uptime graph (how often is the service restarting)
• Storage usage (ensure pruning is working)
• Bandwidth usage (check if we're maxing out on connections or bandwidth, etc.)
• Memory and Process Usage (can the machine handle the request load)

cc @hannahhoward @kylehuntsman

In order to launch autoretrieve at scale, the service must periodically clear out retrieval downloads to not run out of space on the machine.

we should be able to have a knob where we can limit how many concurrent / rate of data we pull from a given provider.

What

https://github.com/application-research/autoretrieve/blob/master/blocks/randompruner.go looks like a decent sized file to test

We need a config property for limiting the number of active retrievals for an autoretrieve instance.

What

The main logic should not live directly inside the file that parse CLI args and reads config. The CLI logic should move to a cmd directory

After discussing w/ @kylehuntsman, some additional metrics to monitor on grafana include measuring the overall autoretrieve funnel (feel free to break out each metric into its own issue as needed):

• Bitswap requests
• Cache Hits (i.e. previously retrieved by Autoretrieve, available to serve immediately)
• Cache Misses (this should result in an Filecoin retrieval request)
• Indexer Requests (should map 1:1 to a cache miss)
• Indexer Response Codes (200, 404, errors)
• Index Misses (either the content is not indexed, or there's an error)
• Filecoin retrieval attempt (should map 1:1 to a 200 indexer response)
• Filecoin retrieval stages (connect, query ask, deal accept, transfer, complete, etc.)
• Filecoin retrieval successes and failures

cc @hannahhoward

from hannah - "I think you’ve got a deadlock in your code after this commit to autoretrieve — be0a8b3
-- Put takes the lock and doesn’t release till the end, but then calls notifyWaitCallback which attempts to take the lock from within the lock Put has taken. Fix incoming I think though I’m not the expert here."

If we're going to be letting users manually create configs, then we might need to start verifying the structure of the config file to ensure we're not assuming bad state as valid. Verification of the config file would need to happen upon start of the instance after accepting any cli option overrides. If verification of the config structure fails, we should report an error to the user and stop processing. Only start the execution of the instance of verification succeeds.

What

Add unit tests to the Bitswap provider logic

How

Given it's based on the original Bitswap provider implementation, seems like we ought to be able to crib https://github.com/ipfs/go-bitswap/blob/master/internal/decision/engine_test.go or other tests in that directory.

Depending on retrieval volume in production, the autoretrieve service may need to horizontally scale to handle the traffic.

No need for cache
don't want restarted data transfers to record as failures, for now.

Create a default config file using the given AUTORETRIEVE_PATH if autoretrieve is started and a config file is not found. Possibly add a warning that a config file could not be a found and a default will be generated. I could also see the case for making this an error, and guiding the user to create a config file and restart the instance.

In order to launch autoretrieve at scale, CID exclusion lists need to be supported.

Once CID exclusion lists, datastore pruning, and real-time metrics are implemented, relaunch the autoretrieve service in production.

Goals

If I run autoretrieve, especially with FullRT on, I'm going to make a lot of requests. It would be great to be able to set a maximum size for the local blockstore cache, and start to prune it (perhaps the least recently requested block first?) once it reaches that size and I need to make room for more data.

If deemed useful, peer the autoretrieve service to major IPFS gateways in order to help serve more retrievals content, i.e. beyond Estuary.

cc @hannahhoward

This is a placeholder ticket to track the work of @gmelodie.

It will enable faster content discovery via Indexer lookups and a faster retrieval experience as the content will already be in the autoretrieve cache for faster client fetching.

Follow a similar pattern to STI in migrating production metrics to PL grafana.

we have no need for this at the moment and it's causing unwanted retrieval restarts

Currently autoretrieve is being run out of the same AWS account as STI, let's tag appropriately to better understand the cost breakdown.

add github workflow.

Once #80 and #18 are done, we need to review #47 & #46 to verify what we have and don't have, and what is correct.

Add a command for generating a default config file using the given AUTORETRIEVE_PATH. This would allow users to easily create an autoretrieve instance config file, and then configure it, before running the instance.

There is one miner that autoretrieve is connecting to that is constantly failing retrievals. Let's look into why this miner is doing this, and possibly reach out to gain a better understanding.

TODO: Add miner PeerID

Make a grafana dashboard that displays the real-time metrics listed in #15

As an improvement to serving online retrieval requests via autoretrieve (for the bitswap nodes that happen to be peered to the service), this placeholder ticket will track the work to proactively index Estuary content so all content queries (beyond the peered bitswap nodes) can served by autoretrieve.

cc @hannahhoward

Once #85 is done, we need to review the grafana dashboards for correctness, and add additional dashboards as needed.

cc @kylehuntsman can you fill this in with whatever work you are currently doing for @dkkapur?

I think the plot we end up wanting is that at a given time, how many providers are we doing 1 retrieval from? how many are we doing 2 from?, etc. We’re worried that there are defaults at a couple places, and we may see that even as we do a lot of transfers, we may have a bunch of providers who rate limit us to not allow more than e.g. 20 simultaneous transfers.

• Underlying data can be short lived for now
• Grafana metrics don't need the provider ID, just the distribution of number of requests per provider

Send the network level DSR metrics, as measured by autoretrieve, to the Pando service.

• For each deal, record:
  • success or fail (possibly why?)
  • miner who made deal
• periodically, package up recent DSR records into an IPLD data structure written to a store, with a CID
• publish the record to Pando using similar code as used in Dealbot

Pando Spec: https://kencloud.com/pando.html
Dealbot Pando integration: https://github.com/filecoin-project/dealbot/blob/main/controller/publisher/pando_publisher.go

In our testing, we observe the following:

1. When we do a single retrieval with a miner, it finishes
2. When we do multiple retrievals with the same miner at once, one retrieval fails, but then all other retrievals fail shortly afterwards. Automatic data transfer restarts do not seem to solve the issue.

Inspecting the logs, we see the GraphSync stream was reset:

2022-01-27T01:52:23.174Z	[34mINFO[0m	graphsync	impl/graphsync.go:468	Graphsync ReceiveError from 12D3KooWNTiHg8eQsTRx8XV7TiJbq3379EgwG6Mo3V3MdwAfThsx: stream reset

and also on autoretrieve:

2022-01-27T01:52:23.104Z	�[35mDEBUG�[0m	filclient	[email protected]/filclient.go:1256	retrieval event	{"dealID": "134345", "rootCid": "bafybeihpbyg4xizni7voackqxomab6444fhwl2gkk7a2tmuphyqg5ip6mq", "miner": "f08403", "name": "ReceiveDataError", "code": 27, "message": "stream reset", "blocksIndex": 10994, "totalReceived": 2291825565}

Miner logs show the miner attempts several times to re-establish the stream unsuccessfully (this is just reattempting to connect, not yet restarting the data transfer itself):

2022-01-27T01:52:28.179Z	�[34mINFO�[0m	graphsync	messagequeue/messagequeue.go:219	cant open message sender to peer 12D3KooWNTiHg8eQsTRx8XV7TiJbq3379EgwG6Mo3V3MdwAfThsx: failed to dial 12D3KooWNTiHg8eQsTRx8XV7TiJbq3379EgwG6Mo3V3MdwAfThsx:
2022-01-27T01:52:28.180Z	�[34mINFO�[0m	graphsync	messagequeue/messagequeue.go:219	cant open message sender to peer 12D3KooWNTiHg8eQsTRx8XV7TiJbq3379EgwG6Mo3V3MdwAfThsx: failed to dial 12D3KooWNTiHg8eQsTRx8XV7TiJbq3379EgwG6Mo3V3MdwAfThsx:
2022-01-27T01:52:28.180Z	�[34mINFO�[0m	graphsync	messagequeue/messagequeue.go:219	cant open message sender to peer 12D3KooWNTiHg8eQsTRx8XV7TiJbq3379EgwG6Mo3V3MdwAfThsx: failed to dial 12D3KooWNTiHg8eQsTRx8XV7TiJbq3379EgwG6Mo3V3MdwAfThsx:
2022-01-27T01:52:28.180Z	�[34mINFO�[0m	graphsync	messagequeue/messagequeue.go:219	cant open message sender to peer 12D3KooWNTiHg8eQsTRx8XV7TiJbq3379EgwG6Mo3V3MdwAfThsx: failed to dial 12D3KooWNTiHg8eQsTRx8XV7TiJbq3379EgwG6Mo3V3MdwAfThsx:
2022-01-27T01:52:28.180Z	�[34mINFO�[0m	graphsync	messagequeue/messagequeue.go:219	cant open message sender to peer 12D3KooWNTiHg8eQsTRx8XV7TiJbq3379EgwG6Mo3V3MdwAfThsx: failed to dial 12D3KooWNTiHg8eQsTRx8XV7TiJbq3379EgwG6Mo3V3MdwAfThsx:

Miner logs show at the same time, multiple other streams between the miner and autoretrieve:

2022-01-27T01:52:29.306Z	�[34mINFO�[0m	net/identify	identify/id.go:372	failed negotiate identify protocol with peer	{"peer": "12D3KooWNTiHg8eQsTRx8XV7TiJbq3379EgwG6Mo3V3MdwAfThsx", "error": "stream reset"}
2022-01-27T01:52:31.175Z	�[34mINFO�[0m	net/identify	identify/id.go:372	failed negotiate identify protocol with peer	{"peer": "12D3KooWNTiHg8eQsTRx8XV7TiJbq3379EgwG6Mo3V3MdwAfThsx", "error": "stream reset"}
2022-01-27T01:52:33.785Z	�[34mINFO�[0m	net/identify	identify/id.go:372	failed negotiate identify protocol with peer	{"peer": "12D3KooWNTiHg8eQsTRx8XV7TiJbq3379EgwG6Mo3V3MdwAfThsx", "error": "stream reset"}
2022-01-27T01:52:35.349Z	�[34mINFO�[0m	net/identify	identify/id.go:372	failed negotiate identify protocol with peer	{"peer": "12D3KooWNTiHg8eQsTRx8XV7TiJbq3379EgwG6Mo3V3MdwAfThsx", "error": "stream reset"}
2022-01-27T01:52:37.225Z	�[34mINFO�[0m	net/identify	identify/id.go:372	failed negotiate identify protocol with peer	{"peer": "12D3KooWNTiHg8eQsTRx8XV7TiJbq3379EgwG6Mo3V3MdwAfThsx", "error": "stream reset"}
2022-01-27T01:52:37.935Z	�[34mINFO�[0m	net/identify	identify/id.go:372	failed negotiate identify protocol with peer	{"peer": "12D3KooWNTiHg8eQsTRx8XV7TiJbq3379EgwG6Mo3V3MdwAfThsx", "error": "stream reset"}
2022-01-27T01:52:39.002Z	�[34mINFO�[0m	net/identify	identify/id.go:372	failed negotiate identify protocol with peer	{"peer": "12D3KooWNTiHg8eQsTRx8XV7TiJbq3379EgwG6Mo3V3MdwAfThsx", "error": "stream reset"}
2022-01-27T01:52:39.866Z	�[34mINFO�[0m	net/identify	identify/id.go:372	failed negotiate identify protocol with peer	{"peer": "12D3KooWNTiHg8eQsTRx8XV7TiJbq3379EgwG6Mo3V3MdwAfThsx", "error": "stream reset"}
2022-01-27T01:52:40.507Z	�[34mINFO�[0m	net/identify	identify/id.go:372	failed negotiate identify protocol with peer	{"peer": "12D3KooWNTiHg8eQsTRx8XV7TiJbq3379EgwG6Mo3V3MdwAfThsx", "error": "stream reset"}
2022-01-27T01:52:41.069Z	�[34mINFO�[0m	net/identify	identify/id.go:372	failed negotiate identify protocol with peer	{"peer": "12D3KooWNTiHg8eQsTRx8XV7TiJbq3379EgwG6Mo3V3MdwAfThsx", "error": "stream reset"}

Additional notes:

• Autoretrieve also shows other stream errors that appear to reference the resource manager (autoretrieve is running libp2p 1.18 -- the miner is not) -- it shows these messages long before the failure -- at several times:

2022-01-27T01:52:29.236Z	�[31mERROR�[0m	autoretrieve	metrics/basic.go:52	Failed to query miner f08403 for Qme5eFv4f1SY7WyuQ3gG6CPDR2cY68JVZjjdweU4Bgi1Mu (root QmezTdYeKyjPFoREStJQQbvATUP8yRJdHMMZx2rZ86p9g9): failed to open stream to peer: peer:12D3KooWBwUERBhJPtZ7hg5N3q1DesvJ67xx9RLdSaStBz9Y6Ny8: resource limit exceeded

• The connection between the miner and autoretrieve should have Protect called on it, once per transfer, with a different tag for each transfer. When the first transfer finishes, one of these tags is removed, but the other two should still be there.

• One additional temporary stream is established at the end of the transfer in order for the miner to ack to autoretrieve that it's finished the transfer. The message is sent from the miner successfully, but does not appear to be received on the client. It appears sending this message happens right before the failure.

• Autoretrieve is running the FullRT DHT Client ((https://github.com/libp2p/go-libp2p-kad-dht/tree/master/fullrt) which crawls the network to establish a DHT index. As a result, autoretrieve is connected to a LOT of peers. We actually did not expect to get any bitswap requests on autoretrieve till we started publishing records in the DHT, but actually, we get lots presumably cause we're in the swarms of every peer we connect to when building a DHT index. This presents a potential issue for bitswap chatter should FullRT be deployed widely.

Add more unit or integration tests, and hook it up to the ci/cd build pipeline.

Sub tickets:

• #84
• #39
• #41
• #42
• #60
• #88

Goals

Make autoretrieve easier to deploy by dockerizing it. Recommend basing the build off the Lotus Dockerfile, since that already handles the considerable complexity in compiling the FFI: https://github.com/filecoin-project/lotus/blob/master/Dockerfile.lotus

We discussed assuming a default location in ~/.autoretrieve for the data dir. This can be controlled via a AUTORETRIEVE_PATH environment variable on run. File I/O will use this path to assume locations of files and directories.

Once in production, monitor the grafana dashboard to better understand autoretrieve patterns, and tune the dashboard as necessary. Other metrics to consider: number of cache hits vs. making a Filecoin retrieval. Performance comparison between cache and non-cache responses. etc.

Add real-time metrics support to help measure the goals listed in #15