filecoin-project / boost Goto Github PK

View Code? Open in Web Editor NEW

109.0 22.0 65.0 22.32 MB

Boost is a tool for Filecoin storage providers to manage data storage and retrievals on Filecoin.

License: Other

Makefile 0.40% Go 88.15% CSS 1.54% HTML 0.06% JavaScript 9.04% Shell 0.73% Dockerfile 0.07%

filecoin storage

boost's People

Contributors

Stargazers

Watchers

boost's Issues

Transfer should be in megabits/s

Notes on the current state of the UI

I went through the UI today and took some notes as I was going through, dumping them here for visibility.

Storage Deals

Would be great to be able to filter storage deals by client id (helpful for quickly debugging with particular clients)

Deal Detail Page

Hide retry button when the deal is not in a bad state. (If I'm in "publishing" a retry is a no-op correct?)

Storage Space

We have the amounts listed in the graph and they're human readable, do we need it in the description area too?
- Perhaps we can use this area to provide more helpful information related to deals? (Ex: "23 deals remaining @ 32GB" "92 deals remaining at 8GB") Is it possible to see what sizes of deals we support and display deals remaining for those numbers?

Funds

Source wallet tooltip is a bit self descriptive. Maybe we should change to something along the lines of "Move funds from the Source Wallet to Escrow to earmark them for deals". It's not immediately clear why I wouldn't just move all my funds from source to Escrow. Can I also move funds from Source to the publish wallet? I don't see UI for that, is that done via lotus directly?
Escrow tooltip is pretty clear
Publish tooltip is also clear
Perhaps we can add some UX here as a call to action for SPs. Can we get a range of how many deals they have escrow for and maybe add a warning label to recommend they move collateral to escrow ("Escrow only has capacity for ~1 more deal, add funds to escrow before making further deals.")? Effectively I'm thinking about triggers for them too:
1. Move funs to source wallet.
2. Move funds to Escrow.
3. Move funds to Publish storage deals.
  I think leveraging more calls to action would be helpful. We're providing better visibility into the system, and I think adding those nudges to help them keep deals moving will be helpful in several of the views.

Deal Publish

Looks like there's a syncing bug here. I had 8 deals auto publish but that didn't refresh on this page, but did update on the Storage Deals page. Refreshing fixed it.

General

Is there a way to see when the windowpost is? Might be nice to have that as an ever present element.

Implement and test transfer resumption across restarts and connection resumption

Rename DB column DealID to DealUUID

To avoid confusion with the deal ID that is stored on chain

Deal Detail Page: Show sector ID

UI: Add anchors to URLs

Ideally when we are on the Storage Space screen (or any other screen), and we CTRL+R the browser, we should end up on the same screen.

Currently it's difficult to understand how the different bars in the fund manager bar chart relate to each other.
Instead, use a similar design to the storage manager, with multiple bars that sum to a whole.

Bug: Storage Deals Page - Pagination shows "2 deals" when there's only one in the list

Connect Storage Manager to UX

Write a resolver that connects the storage manager UX to the implementation

Add support for SPs to rate limit incoming deals (ASKv2 support)

Currently there is no good backoff mechanism to allow Storage Providers to rate limit incoming storage deal requests. The key problem is around the associated data transfer, as accepting deals immediately triggers data transfer, and too many in progress deals can starve needed resources.

Current Proposal

ASKv2 Protocol

The ARG team has currently proposed an initial iteration of the ASKv2 protocol to enable Storage Providers to respond to ASK requests with a backoff timing, ie: "Hey, come back in 3 hours and let's do this". This should provide an immediate improvement to the existing deal negotiation that Estuary and other clients are making.

Long term solutions may include order books or deal markets to enable easier discovery of SPs who have bandwidth to accept deals immediately.

Dependencies

Requires a "finalized" design of the ASKv2 protocol. (We may wish to test early iterations of this with Estuary to find appropriate flows for backoff)

Have an option for SP to reject deals from a bls account

Checklist

This is not a new feature or an enhancement to the Filecoin protocol. If it is, please open an FIP issue.
This is not a new feature request. If it is, please file a feature request instead.
This is not brainstorming ideas. If you have an idea you'd like to discuss, please open a new discussion on the lotus forum and select the category as Ideas.
I have a specific, actionable, and well motivated improvement to propose.

Lotus component

Improvement Suggestion

while randomly going through some gas trace os PSD message, we realized that for some PSD message that including deals made with clients that's using bls wallet causes up to ~40% gas usage of PSDs to be verifyingsiguature, which can be significantly saved if the client uses secp256k1.

verifySignature: map[crypto.SigType]int64{
			crypto.SigTypeBLS:       16598605,
			crypto.SigTypeSecp256k1: 1637292,
		},

lotus sp should have an option to say they want to reject such deals so that they can decrease the cost of deal onboarding for clients.

Structured logging and comprehensive deal journal

Improve logging in boost state manager and transport layers
Make sure all logging is structured (eg log.Infow instead of log.Info)

Refactor cancellation of deals

At the moment cancellation of deals is racy - refactor the deal state manager to remove the races

Cleanup of all deal state when deal fails or completes.

Ensure that all resources for a deal are cleaned up when the deal fails or completes

Implement deal protocol version 2

Retry deals with start epoch in the future, if the sectors they've been added to initially fail to seal

Checklist

This is not a new feature or an enhancement to the Filecoin protocol. If it is, please open an FIP issue.
This is not a new feature request. If it is, please file a feature request instead.
This is not brainstorming ideas. If you have an idea you'd like to discuss, please open a new discussion on the lotus forum and select the category as Ideas.
I have a specific, actionable, and well motivated improvement to propose.

Lotus component

Improvement Suggestion

Currently in Lotus, after we add a deal to a sector, we remove it from the markets staging area, and pass it to the sealing subsystem which adds a piece to a sector, and then attempts to seal a sector.

However if a sector fails to seal for some reason, even if some deals are still good in it, there is currently no retry mechanism to try to add them to another sector and seal a new sector with the good deals.

We should figure out how to improve this, some ideas:

there should be a mechanism for sector FSM to tell the deal FSM that a sector got removed
each deal should check itself and if it is still good, it should be added to a new sector
we should figure out how to extract a deal from a failed sector, and re-add it to a new sector, given that it is no longer available in the markets staging area.

Transfer percent denominator should be file size not piece size

Storage space page should auto update as space allocation changes

Deal state retry / resumption

Update `devnet` to not use default LOTUS environment variables and paths

At the moment if you use the devnet binary, or if you run integration tests, they will create artefacts at various places under the $HOME directory of the user.

Ideally we should be creating artefacts under a temporary directory that devnet creates and then do a cleanup after we shut the devnet down.

Change boost init params

We need 2 addresses for making deals:

A StorageMinerActor address that
- Must be used in the deal.Proposal.Provider field
- Must be the target address when adding funds to deal escrow
Worker / Controller address of StorageMinerActor address that
- Sends PublishStorageDeals message (ie it's the address in the From field of the message)

So in boost init I suggest we:

Remove the "wallet-collateral-pledge" param
- Get the storage miner actor's address using lotus StorageMiner API's GetActor()
- Rename the config variable to "StorageMinerActor"
Check that the publish storage deals wallet is a worker / controller address of the StorageMinerActor

Auto-repair fund manager and storage manager state on Boost startup

Given that we persist the fund manager and storage manager state to the SQL DB, it is possible to end up in consistent state if the Boost process crashes between doing the DB updates. The process should auto-repair this state on startup before resuming all the deals.

Refactor: static file server

Currently the GQL server uses the default file http.FileServer to serve static files from a directory.
However this file server doesn't sandbox the directory, so a user can request /static/../../some/other/file

Deal acceptance filter

Integrate the deal acceptance filter
Make sure that the deal acceptance filter can access these parameters:
- Available storage space for incoming deals
  Check that there is enough space to store the deal as it is transferred across the wire
- Available storage space for unsealed / sealed deal
  Check that there is enough space to store the deal in sealed format (and unsealed for fast-retrieval deals)
- Data transfer url
  Check that provider supports this type of data transfer
- Sealing pipeline capacity
  Check that pipeline has capacity to seal deal by requested start time
- Control wallet amount
  Check that provider has enough funds for deal collateral
- PublishStorageDeals wallet amount
  Check that provider has enough funds to call PublishStorageDeals
- Data transfer volume
  Check the current data transfer rate and make sure we have bandwidth to accept more
- Data transfer statistics
  eg Provider has 5 transfers in progress with Estuary, wants to limit max transfers
  eg Maximum number of in-progress deal transfers: global, per client, per specific client

Deal updates pubsub mechanism

Currently we are using github.com/libp2p/go-libp2p-core/event as our pubsub mechanism to publish changes to deal state.
This events system

blocks each new event until the previous one is processed (ie back pressure)
requires all events to be drained on unsubscribe (otherwise publish will get blocked)

For our use case each deal state change replaces the last, so in reality we need a pubsub mechanism that

does not block on publish (no back pressure)
only keeps track of the latest event (all previous events can be replaced with the latest)
automatically drains all events on unsubscribe

Add support to retry failed deal states that are not terminal

Restarting markets actor turns deals in progress into StorageDealError state

Checklist

This is not a security-related bug/issue. If it is, please follow please follow the security policy.
This is not a question or a support request. If you have any lotus related questions, please ask in the lotus forum.
This is not a new feature request. If it is, please file a feature request instead.
This is not an enhancement request. If it is, please file a improvement suggestion instead.
I have searched on the issue tracker and the lotus forum, and there is no existing related issue or discussion.
I am running the Latest release, or the most recent RC(release canadiate) for the upcoming release or the dev branch(master), or have an issue updating to any of these.
I did not make any code changes to lotus.

Lotus component

Lotus Version

1.13.1-rc1

Describe the Bug

Storage deals which are in StorageDealSealing or StorageDealAwaitingPreCommit or StorageDealStaged or StorageDealPublish turn into StorageDealError when market service is restarted.

See the following deals;

Nov  6 00:45:58  true      bafyreidmksbovp3cbi2a2ttcw6x3p3zo55yt6bf5eec5dpwkcxjn6hn7ti  2717173  StorageDealActive                        f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  
Nov  6 00:46:19  true      bafyreibcw3fcsnn5cpmzdqwtdts5olvgw66nanjlijsljthnmqr4xh5t2q  2717174  StorageDealActive                        f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  
Nov  6 00:50:19  true      bafyreicj7yrokjukncujkjwjkdqgf26ozl6qv26tynvi6fi7aghjauietu  2717175  StorageDealActive                        f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  
Nov  6 01:05:24  true      bafyreibaata2yd7fhghopiyyvgcys3veyejl35wf6iayuf7kii76nfk4pa  2717238  StorageDealActive                        f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  
Nov  6 02:50:35  true      bafyreic7cwb2r4dvgdg4mevai4mnnp6pqw4ls6d56fkdqmp4p43jgsh4ni  2717483  StorageDealActive                        f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  
Nov  6 02:56:39  true      bafyreicintk4hbdzaxxgrq74f6mvq3xedquy6tljt6ux475xp5gpmzzhre  2717484  StorageDealActive                        f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  
Nov  6 03:50:47  true      bafyreiaxo3ba6jfy6iti4oegh2qpn6rerz2jrr7pcmfde3zcnainawzmci  2717676  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262178): failed to look up deal on chain: deal 2717676 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 03:55:45  true      bafyreiexg7yuqhks6lktlrx7wji7znqya4tv274yf4po63slasf2xtk43m  2717677  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262178): failed to look up deal on chain: deal 2717677 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 04:06:30  true      bafyreiaotmfj5v6ej5zs5ecfkffciu5e7oodktovzwjp6pezikozp4e564  2717729  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262208): failed to look up deal on chain: deal 2717729 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 04:37:16  true      bafyreihc2sripy6xz3pf2c6432cxgt4gechmzxv6pq32rtpksk2loyzjry  2717859  StorageDealActive                        f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  
Nov  6 04:50:59  true      bafyreignyfnbq27sscqzk4yinjxvc27bc66ytib27uw4dkechmvqixmkra  2717931  StorageDealActive                        f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  
Nov  6 05:06:48  true      bafyreibejxbpd4anrd3hadvoxm2b3jiwf2xcmyoenmlsvurxcmiwdfgmfq  2718015  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262318): failed to look up deal on chain: deal 2718015 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 05:35:55  true      bafyreihtg3cpklevcebr72pxoke5vhlnqsc6t67uxcb5twnpcxqvflotbe  2718088  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262378): failed to look up deal on chain: deal 2718088 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 05:50:56  true      bafyreigynvrpqgpahjjsywzu4iri3ygksuur2tzbplbm4ui2oyy7rs4wta  2718124  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262398): failed to look up deal on chain: deal 2718124 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 06:06:42  true      bafyreieca7gnlrj4fq55pimnrbk2jc4cxcykxif7yds4b7r7tvvqcpf324  2718164  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262438): failed to look up deal on chain: deal 2718164 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 06:07:13  true      bafyreidepo53als3yofanwxeqmwv3hc5xaxrwtrh637rtsm73zkuy7cp5e  2718163  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262438): failed to look up deal on chain: deal 2718163 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 06:11:48  true      bafyreie5snk7yw3ptvpl45qup5mvte6tjqmnvbttzezuzoi4otxgtj67m4  2718165  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262438): failed to look up deal on chain: deal 2718165 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 06:21:03  true      bafyreibdt7265sryyfdxhsv4j2eqdhwsrfbqndqhovx67juvcoyovvo6qa  2718196  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262468): failed to look up deal on chain: deal 2718196 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 06:23:40  true      bafyreifhvy4yk2bsf6e2g5o7mxfw5dk3i6nxf2edkdiagi5gua3nqjc4me  2718195  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262468): failed to look up deal on chain: deal 2718195 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 06:26:31  true      bafyreidhnjt57fl2lmbcckfs2clzkdxq5zh5el6dzszh22pkom3kftqo5e  2718197  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262468): failed to look up deal on chain: deal 2718197 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 06:36:06  true      bafyreifrivayunk6sroujq5lr6zadb7tvgddf2bqc75325bbjhpapk7kx4  2718226  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262488): failed to look up deal on chain: deal 2718226 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 06:59:30  true      bafyreifreqmqxebcpb43vhcxssrzz4prvgt2iqyu2vbs3er3rkiozh3pei  2718300  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262520): failed to look up deal on chain: deal 2718300 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 07:06:13  true      bafyreie7nna5jz6kudgig7evt7k67lmyd7nf6iynnxd3ire5ytns2cbjv4  2718301  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262520): failed to look up deal on chain: deal 2718301 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 07:21:11  true      bafyreidixezqfciceongk24addgggjvnliotf3hhbmby2bvuxi6zyjhnca  2718344  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262548): failed to look up deal on chain: deal 2718344 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 07:24:21  true      bafyreiacqqgacsrto4oeumxcpnnlwaydfg63qq7m7gpgkkdfuukvctrgti  2718345  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262548): failed to look up deal on chain: deal 2718345 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 07:36:13  true      bafyreia5jwkeufwqdcse6cz7za3h4u5sbbydcaiv4btokwywvjfwnmvlza  2718405  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262588): failed to look up deal on chain: deal 2718405 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 07:52:07  true      bafyreibrvdwolz7gkea6pjj2gbc3wj7nas26todupzf4in2xv6cdydgsoa  2718489  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262618): failed to look up deal on chain: deal 2718489 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 07:58:12  true      bafyreifwyu4vmydhvlvdsmsrafa2r7n5sgenbigh27kgnq7fgsp5ermqla  2718488  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262618): failed to look up deal on chain: deal 2718488 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 08:06:15  true      bafyreia3rog4gpisuriuxrheeqy6yweoerc2vuvzjgtp2ma4tw2437icnu  2718525  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262628): failed to look up deal on chain: deal 2718525 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 08:21:18  true      bafyreifuptvoaw3vcaxsqgu53emahilja5eq4h3zuj3tknu6uimlx4mre4  2718591  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262649): failed to look up deal on chain: deal 2718591 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 08:52:04  true      bafyreia73niu2qltjrttzsmeuqratimcu24xpew4eiyzilbp7w3hd44uhm  2718658  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262698): failed to look up deal on chain: deal 2718658 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 09:06:26  true      bafyreicxkqy7cqhb6kgt456jtxont3ht5zdqmvxmo2otndslf6d26woz3u  2718707  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262728): failed to look up deal on chain: deal 2718707 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 09:59:48  true      bafyreicpe2m6m46dqwbjzvcce2763m67y3766q7i5xj3r7shi2jfgmvlbm  2718816  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262818): failed to look up deal on chain: deal 2718816 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 10:21:31  true      bafyreiavhubln2lsvezrwkhnoorgz5ntdl3juokxt4wlvl6t3vcp42qmaq  2718855  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262848): failed to look up deal on chain: deal 2718855 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 10:51:41  true      bafyreibencvi42czj5uiacrzu7cgj6ynhnm35rhupyamclkshueqh4gary  2718932  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262889): failed to look up deal on chain: deal 2718932 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 10:54:54  true      bafyreiblhse4maio4pl5i4ooagsdzblxlrty22p2csvuw7mxbjz4g2buye  2718933  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262889): failed to look up deal on chain: deal 2718933 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 10:57:07  true      bafyreiejqxqkk2ehltyrckhhtbodvdciue7g47t6u7bqgly5qyggycdqjm  2718962  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262909): failed to look up deal on chain: deal 2718962 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 10:57:38  true      bafyreigdog3ln2nwzugej3e2t3l2fgfnovztd5zcherty4xsgmqblmg4wq  2718963  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262909): failed to look up deal on chain: deal 2718963 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 11:06:42  true      bafyreihu2s7gtnvvepi7jlgphacr3jjlpn3dzch5mgnc3ckxrjhmkcmyku  2718964  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262909): failed to look up deal on chain: deal 2718964 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 11:21:43  true      bafyreidmih4mw4plrd3o2pgtivjvbxfvl7u7k2ys7old7ar4wito34fstm  2719005  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262928): failed to look up deal on chain: deal 2719005 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 11:36:03  true      bafyreigttdunmsxrmayhishsn2qm6aii2fe3rurmup46px3agfisgzksya  2719041  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262948): failed to look up deal on chain: deal 2719041 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 11:36:36  true      bafyreihpt5attymcqqp35xcodisfqkmignhn36eikt52ade74nf7k7wmum  2719040  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262948): failed to look up deal on chain: deal 2719040 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 11:51:45  true      bafyreieyy5e7a5z3xbalera4xztvdl5ujaw4s37ophitwkoaqjlelbmfpa  2719081  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262978): failed to look up deal on chain: deal 2719081 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 12:06:44  true      bafyreiepsu6ujowyzlj5xw3iw24gv2vk4u6jt3iujbwbqlvcz5z3aoe524  2719125  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1262998): failed to look up deal on chain: deal 2719125 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 12:21:46  true      bafyreibgyyt7yzdiuvlqcvurywc3t7hzkdvlgyzgadji7y6qyvrtigziai  2719167  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1263018): failed to look up deal on chain: deal 2719167 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 12:36:50  true      bafyreidhgj4dfyt2zfqc2lyubprsv2ez6emldrt5p4yle7feste57gtene  2719189  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1263048): failed to look up deal on chain: deal 2719189 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 12:39:41  true      bafyreiatzghj7jdzmmyjj4nbcf4a4ddkilj67l3cws6ymenfzqibkukpsy  2719190  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1263048): failed to look up deal on chain: deal 2719190 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 12:53:38  true      bafyreia5d34tqqacxlzeddy4r3o6fpedsybizsl62ti35vnpmxdiiknkgm  2719202  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1263059): failed to look up deal on chain: deal 2719202 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 12:57:26  true      bafyreifw5pqz3bcau7j35zh33ftplxcew6tlrndunzalcndantopmb37vm  2719203  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 1263059): failed to look up deal on chain: deal 2719203 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Nov  6 13:07:38  true      bafyreieugjvtjggby67kzro7exyys4x4z3qzpfzpap4wsbqyfvtk7jesje  2719266  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  handing off deal to node: packing piece at path fstmp3156441631: AddPiece failed: piece for deal bafy2bzacebdzdazm3qknj6alu7xd6dqjdwcvny2h2dxnbeusl45d46bp7b2eq already pending
Nov  6 13:16:31  true      bafyreihzihk3jj3zf4lnnjmh7wfrdqswuccysnrlreqlhbmbmf2xklyo44  2719267  StorageDealError                         f144zep4gitj73rrujd3jw6iprljicx6vl4wbeavi                                               32GiB   0 FIL              1468800                                                                                                                                  handing off deal to node: packing piece at path fstmp2267266342: AddPiece failed: piece for deal bafy2bzacebwyrtc5cc6yeetz7yysnv6cpxrftcxnm2s3tnw57wcpgawqxd35k already pending

I restarted the service at 13:18 and 04:40. The weird thing is that the sectors have sealed correctly, the data is in there, and the deals themselve are perfectly fine on the chain.
The provider of the data actually sees the deals as succesful as well (@jsign).

There seems to be no way for me to reset / recheck the state, and the error state is clearly wrong;

$ lotus state get-deal 2717173
{
  "Proposal": {
    "PieceCID": {
      "/": "baga6ea4seaqcnqpj3mfmyyo7iu465csahbniknmscj75sx324ctgwumz4uua6ia"
    },
    "PieceSize": 34359738368,
    "VerifiedDeal": true,
    "Client": "f0743060",
    "Provider": "f01392893",
    "Label": "mAXASIMNcA3y8yQJ84uqgojU8or16Uyz+bL+0k7zRN/esjhWB",
    "StartEpoch": 1263370,
    "EndEpoch": 2732170,
    "StoragePricePerEpoch": "0",
    "ProviderCollateral": "5286449256929280",
    "ClientCollateral": "0"
  },
  "State": {
    "SectorStartEpoch": 1262622,
    "LastUpdatedEpoch": -1,
    "SlashEpoch": -1
  }
}

Logging Information

Repo Steps

Have deals sealing (these were manually imported through bidbot)
Restart miner (markets) process
See failed deals

[Deal Making Issue] Client gets stuck in StorageDealAwaitingPreCommit even after deal has failed on miner side

Basic Information

When a miner fails 3 times to seal, it will give up and purge the sector. The problem this issue describes is that on the client side there is never any notification that the deal has failed. The client just hangs in a StorageDealAwaitingPreCommit state forever.

I propose that the miner reports back on a final failure state like this so the client status can change to StorageDealError or some other error state.

Describe the problem

On the client side (1.7.0), I'm seeing repeated instances of deals getting stuck in StorageDealAwaitingPreCommit even days after the miner has failed to seal and given up on the sector.

Version

Client:

$ lotus version
Daemon:  1.7.0-dev+mainnet+git.a40fb1e1d+api1.1.0
Local: lotus version 1.7.0-dev+mainnet+git.a40fb1e1d

Miner

$ lotus version
Daemon:  1.6.0-rc2+mainnet+git.80224bcb8+api1.1.0
Local: lotus version 1.6.0-rc2+mainnet+git.80224bcb8

# Note: this is after a recent downgrade to 1.6.0; one of the failed deals shown below was made last week when miner was on 1.7.0.
# But this bug report is not about the miner-side issue! It is about the client UX, ie, not telling the user that the deal has failed.

To Reproduce

Make a deal from client:

$ lotus client deal --fast-retrieval --verified-deal bafykbzacebzqkg44jer45btajfibw4hnzjpfoxh6f63eg4ridgfcget2wxbyw f0127896  0.0000002 518400
bafyreiffnyigpe3l32gmnkjbv7px6rb3a5xqxgqshd5uq5nup6qsi4gdei

Wait ~5-10 minutes for client-side deal to advance to StorageDealAwaitingPreCommit .
Now wait 3-5 days or longer. The StorageDealAwaitingPreCommit status never changes to a final error status.

Deal status

The output of lotus client list-deals -v:

ubuntu@dealbot-mainnet:~$ lotus client list-deals -v
Created          DealCid                                                      DealId   Provider  State                         On Chain?         Slashed?     PieceCID                                                          Size       Price          Duration  TransferChannelID                                                                                                              TransferStatus  Verified  Message
Apr 16 14:39:48  bafyreicqyarmouxojweqio7fmd55d6qe6qphraq2trkn7cvjjgxg4s3jhy  1742150  f0127896  StorageDealAwaitingPreCommit  Y (epoch 0)       Y (epoch 0)  baga6ea4seaqn2csbrgsdpavbcek5rvj7z52gahph65pta53oanbmqnmw6tgnkoq  1.984 GiB  0.1039372 FIL  519686    12D3KooWLXgNd8PfyRf7eC7xKPVnuZCb65SA7ZcqEFp4zUeC2yah-12D3KooWQrZnhRaBecSZYKR815zH65xkg1TvfyPWUGYwehHvVEKB-1617824480496011385  Completed       true
Apr 20 14:33:21  bafyreiffnyigpe3l32gmnkjbv7px6rb3a5xqxgqshd5uq5nup6qsi4gdei  1770094  f0127896  StorageDealAwaitingPreCommit  N                 N            baga6ea4seaqn2csbrgsdpavbcek5rvj7z52gahph65pta53oanbmqnmw6tgnkoq  1.984 GiB  0.1039398 FIL  519699    12D3KooWLXgNd8PfyRf7eC7xKPVnuZCb65SA7ZcqEFp4zUeC2yah-12D3KooWQrZnhRaBecSZYKR815zH65xkg1TvfyPWUGYwehHvVEKB-1617824480496011386  Completed       true

lotus-miner storage-deals list -v (relevant lines only):

$ lotus-miner storage-deals list -v
Apr 16 17:39:48  true   bafyreicqyarmouxojweqio7fmd55d6qe6qphraq2trkn7cvjjgxg4s3jhy  1742150  StorageDealError                         f1wdxdpqh3hirrhp353i4o6ld7bsw6evh3v7i5jtq                                               2GiB    0.1039372 FIL             519686   12D3KooWLXgNd8PfyRf7eC7xKPVnuZCb65SA7ZcqEFp4zUeC2yah-12D3KooWQrZnhRaBecSZYKR815zH65xkg1TvfyPWUGYwehHvVEKB-1617824480496011385  error awaiting deal pre-commit: failed to set up called handler: called check error (h: 686834): failed to look up deal on chain: deal 1742150 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
Apr 20 17:33:21  true   bafyreiffnyigpe3l32gmnkjbv7px6rb3a5xqxgqshd5uq5nup6qsi4gdei  1770094  StorageDealAwaitingPreCommit             f1wdxdpqh3hirrhp353i4o6ld7bsw6evh3v7i5jtq                                               2GiB    0.1039398 FIL             519699   12D3KooWLXgNd8PfyRf7eC7xKPVnuZCb65SA7ZcqEFp4zUeC2yah-12D3KooWQrZnhRaBecSZYKR815zH65xkg1TvfyPWUGYwehHvVEKB-1617824480496011386

Lotus daemon and miner logs

This issue is about client UX, not the underlying miner bug(s) that caused these deals to fail miner-side. Nevertheless, here are the relevant miner logs.

This is for the 'jhy deal started last Friday and where client side is still stuck in StorageDealAwaitingPreCommit (miner was running 1.7.0):

$ cat lotus-miner-logs* | grep bafyreicqyarmouxojweqio7fmd55d6qe6qphraq2trkn7cvjjgxg4s3jhy
2021-04-17T03:21:24.726+0300	INFO	markets	loggers/loggers.go:20	storage provider event	{"name": "ProviderEventRestart", "proposal CID": "bafyreicqyarmouxojweqio7fmd55d6qe6qphraq2trkn7cvjjgxg4s3jhy", "state": "StorageDealAwaitingPreCommit", "message": ""}
2021-04-20T12:36:49.398+0300	INFO	markets	loggers/loggers.go:20	storage provider event	{"name": "ProviderEventRestart", "proposal CID": "bafyreicqyarmouxojweqio7fmd55d6qe6qphraq2trkn7cvjjgxg4s3jhy", "state": "StorageDealAwaitingPreCommit", "message": ""}
2021-04-20T12:37:14.888+0300	INFO	markets	loggers/loggers.go:20	storage provider event	{"name": "", "proposal CID": "bafyreicqyarmouxojweqio7fmd55d6qe6qphraq2trkn7cvjjgxg4s3jhy", "state": "StorageDealFailing", "message": "error awaiting deal pre-commit: failed to set up called handler: called check error (h: 686834): failed to look up deal on chain: deal 1742150 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed"}
2021-04-20T12:37:14.890+0300	WARN	providerstates	providerstates/provider_states.go:536	deal bafyreicqyarmouxojweqio7fmd55d6qe6qphraq2trkn7cvjjgxg4s3jhy failed: error awaiting deal pre-commit: failed to set up called handler: called check error (h: 686834): failed to look up deal on chain: deal 1742150 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed
2021-04-20T12:37:14.900+0300	INFO	markets	loggers/loggers.go:20	storage provider event	{"name": "ProviderEventFailed", "proposal CID": "bafyreicqyarmouxojweqio7fmd55d6qe6qphraq2trkn7cvjjgxg4s3jhy", "state": "StorageDealError", "message": "error awaiting deal pre-commit: failed to set up called handler: called check error (h: 686834): failed to look up deal on chain: deal 1742150 not found - deal may not have completed sealing before deal proposal start epoch, or deal may have been slashed"}

This is for the 'dei deal that was started 4 hours ago and is still in StorageDealAwaitingPreCommit (miner running 1.6.0`):

$ cat lotus-miner-logs* | grep bafyreiffnyigpe3l32gmnkjbv7px6rb3a5xqxgqshd5uq5nup6qsi4gdei
2021-04-20T17:33:21.924+0300	INFO	markets	loggers/loggers.go:20	storage provider event	{"name": "ProviderEventOpen", "proposal CID": "bafyreiffnyigpe3l32gmnkjbv7px6rb3a5xqxgqshd5uq5nup6qsi4gdei", "state": "StorageDealValidating", "message": ""}
2021-04-20T17:33:21.932+0300	INFO	markets	loggers/loggers.go:20	storage provider event	{"name": "ProviderEventDealDeciding", "proposal CID": "bafyreiffnyigpe3l32gmnkjbv7px6rb3a5xqxgqshd5uq5nup6qsi4gdei", "state": "StorageDealAcceptWait", "message": ""}
2021-04-20T17:33:21.942+0300	INFO	markets	loggers/loggers.go:20	storage provider event	{"name": "ProviderEventDataRequested", "proposal CID": "bafyreiffnyigpe3l32gmnkjbv7px6rb3a5xqxgqshd5uq5nup6qsi4gdei", "state": "StorageDealWaitingForData", "message": ""}
2021-04-20T17:33:22.070+0300	INFO	markets	loggers/loggers.go:20	storage provider event	{"name": "ProviderEventDataTransferInitiated", "proposal CID": "bafyreiffnyigpe3l32gmnkjbv7px6rb3a5xqxgqshd5uq5nup6qsi4gdei", "state": "StorageDealTransferring", "message": ""}
2021-04-20T17:37:55.880+0300	INFO	markets	loggers/loggers.go:20	storage provider event	{"name": "ProviderEventDataTransferCompleted", "proposal CID": "bafyreiffnyigpe3l32gmnkjbv7px6rb3a5xqxgqshd5uq5nup6qsi4gdei", "state": "StorageDealVerifyData", "message": ""}
2021-04-20T17:38:14.816+0300	INFO	markets	loggers/loggers.go:20	storage provider event	{"name": "ProviderEventVerifiedData", "proposal CID": "bafyreiffnyigpe3l32gmnkjbv7px6rb3a5xqxgqshd5uq5nup6qsi4gdei", "state": "StorageDealReserveProviderFunds", "message": ""}
2021-04-20T17:38:14.822+0300	INFO	markets	loggers/loggers.go:20	storage provider event	{"name": "ProviderEventFundsReserved", "proposal CID": "bafyreiffnyigpe3l32gmnkjbv7px6rb3a5xqxgqshd5uq5nup6qsi4gdei", "state": "StorageDealReserveProviderFunds", "message": ""}
2021-04-20T17:38:14.827+0300	INFO	markets	loggers/loggers.go:20	storage provider event	{"name": "ProviderEventFunded", "proposal CID": "bafyreiffnyigpe3l32gmnkjbv7px6rb3a5xqxgqshd5uq5nup6qsi4gdei", "state": "StorageDealPublish", "message": ""}
2021-04-20T18:38:14.993+0300	INFO	markets	loggers/loggers.go:20	storage provider event	{"name": "ProviderEventDealPublishInitiated", "proposal CID": "bafyreiffnyigpe3l32gmnkjbv7px6rb3a5xqxgqshd5uq5nup6qsi4gdei", "state": "StorageDealPublishing", "message": ""}
2021-04-20T18:44:00.484+0300	INFO	markets	loggers/loggers.go:20	storage provider event	{"name": "ProviderEventFundsReleased", "proposal CID": "bafyreiffnyigpe3l32gmnkjbv7px6rb3a5xqxgqshd5uq5nup6qsi4gdei", "state": "StorageDealPublishing", "message": ""}
2021-04-20T18:44:00.489+0300	INFO	markets	loggers/loggers.go:20	storage provider event	{"name": "ProviderEventDealPublished", "proposal CID": "bafyreiffnyigpe3l32gmnkjbv7px6rb3a5xqxgqshd5uq5nup6qsi4gdei", "state": "StorageDealStaged", "message": ""}
2021-04-20T18:44:09.211+0300	INFO	markets	loggers/loggers.go:20	storage provider event	{"name": "ProviderEventDealHandedOff", "proposal CID": "bafyreiffnyigpe3l32gmnkjbv7px6rb3a5xqxgqshd5uq5nup6qsi4gdei", "state": "StorageDealAwaitingPreCommit", "message": ""}

Right now, on the miner, lotus client list-deals is is still showing StorageDealAwaitingPreCommit even though both deals have already failed permanently on the miner:

$ lotus client list-deals
DealCid      DealId   Provider  State                         On Chain?         Slashed?     PieceCID     Size       Price          Duration  Verified
...xg4s3jhy  1742150  f0127896  StorageDealAwaitingPreCommit  Y (epoch 0)       Y (epoch 0)  ...w6tgnkoq  1.984 GiB  0.1039372 FIL  519686    true
...qsi4gdei  1770094  f0127896  StorageDealAwaitingPreCommit  N                 N            ...w6tgnkoq  1.984 GiB  0.1039398 FIL  519699    true

Code modifications
None

Refactor Deal Handler state lifecycle

cleanup: remove blockstore-related components no longer needed with the dagstore

Consider removing the StagingBlockstore, potentially ClientDatastore, etc. These components are a bit tangled, currently inoffensive, and potentially hard to remove because of the blockstore muxing in Graphsync, but we should try to get rid of this as it's functional cruft.

Pass wallet addresses to `boost init`

To run boost the user needs to have two wallets:

One for publish storage deals messages
One for pledging collateral

These wallets should be passed as required parameters to boost init.
We should check that the wallets are distinct.
We should store the wallet addresses in config.toml (rather than in the datastore) so that it's easier for the user to see and update their configuration.

End-to-end Integration Test

An integration test that

Creates a new storage deal
Sends the deal to the storage provider
Provider fetches the deal data (can just use a public URL for now)
Provider publishes the deal and hands off to sealer

boost won't start when there is an active deal

Error:

2021-11-24T11:53:25.381+0200    FATAL   boost   boost/main.go:47        creating node: starting node: getting active deals: scanning deal row: sql: Scan error on column index 1, name "TransferSize": converting NULL to uint64 is unsupported

If there is an active deal waiting to be published and we stop boost, we can't start it back up:

2021-11-24T11:45:20.428+0200    INFO    provider        storagemarket/deal_publisher.go:241     waiting publish deals queue period of 1h0m0s before publishing
^C%                                                                                                                                                                                                                  


                         
➜  boost git:(nonsense/improve-readme) ✗ boost run                                                                                                                                                                                        <<<
Worker: 205; Base: 0; Target: 0 (diff: 0)
State: idle; Current Epoch: 0; Todo: 0

Done!
2021-11-24T11:53:25.106+0200    ERROR   alerting        alerting/alerts.go:105  alert raised    {"type": {"System":"process","Subsystem":"fd-limit"}, "message": {"message":"soft FD limit is low","recommended_min":100000,"soft_limit":256}}
2021-11-24T11:53:25.381+0200    FATAL   boost   boost/main.go:47        creating node: starting node: getting active deals: scanning deal row: sql: Scan error on column index 1, name "TransferSize": converting NULL to uint64 is unsupported

Implement manual dummy deal

Update the dummy deal CLI to create a signed deal proposal (currently the deal proposal is not signed)
Uncomment the code in the deals state manager that verifies the signature

Fixed by #63

Refactor: be consistent about error logging

In some places we log with the key "err": err, in other places with "error": err

Integrate Boost with existing markets process

Support for existing graphsync transfer
#293

Estuary client

dagstore: destroy shard on deal expiration or slashing

Context

Currently we don't delete shards from the dagstore when deals expire or are slashed. This is currently inoffensive, as we don't have a lot of expiration or slashing activity going on. But it must be resolved.

Proposal

Destroy shards when deals are slashed or expired, after N confirmations. The main complexity item is dealing with the fact that shard are associated with piece CID, and there could be more than one deal for the same piece CID, but not all of them expire at once. Some form of refcounting may be needed.

Stress test Boost

Introspection - Journal / Logging / pprof

Improve SM logging

Request to expand miner filter hooks

A request from Murmuration Labs is to expand the set of hook slots for external programs. Currently lotus provides Filter which runs before storage-deal acceptance, and determines whether to accept the deals bytes based on wallet/peerid/payload-cid/other criteria. Request is to expand these hooks as follows:

~~StorageNegotiationFilter : same as current Filter: receives the JSON of the deal on STDIN and indicates with exit0/1 whether the payment/data for the deal should be accepted~~ implemented as Filter
StorageContentFilter: when specified, upon receipt of the entire DAG from the client, but before inserting it into a WaitDeals sector, an external program receives the DAG determinisitcally on STDIN as a .car stream, and indicates with exit0/1 whether to proceed
RetrievalNegotiationFilter: same as StorageNegotiationFilter but for retrieval deals ( allows blocking of known-unwelcome wallets/peerids, or commP/payloadCIDs that became classified as problamtic material post-sealing ) implemened as RetrievalFilter
RetrievalContentFilter: same as StorageContentFilter, the dag is fed to on STDIN prior to graphsyncing the first byte to the client.

It is likely that the 4 hooks will invoke portions of the same suite, which will coordinate caching and known-block-list updates out of band.

Transfers page: Change the min on the graph to 0 and continue to adjust for the max.

Refactor

Listing down parts of the code that need to be refactored before the prototype:

Deal Detail Page: Cancel and retry buttons should be disabled when they do not apply

Write tests for Boost SM

Storage Manager

Fix races between DB state and in-memory deal handlers

Race on new deal

When a deal is executed:

The deal state is written to the database
The deal handler is created in memory in a go routine

This can cause a race if for example the UX requests to list all deals from the database, then subscribes to updates for the deal, because the deal handler keeps subscription information and it may not have been created when the UX makes the subscription.

Race on restart

When boost starts up, it starts the Provider:

All active deals are read from the database
A deal handler is created for each

If the graphql server starts before the Provider completes startup, a similar race to above can occur.

filecoin-project / boost Goto Github PK

boost's People

Contributors

Stargazers

Watchers

Forkers

boost's Issues

Storage Deals

Deal Detail Page

Storage Space

Funds

Deal Publish

General

Current Proposal

ASKv2 Protocol

Checklist

Lotus component

Improvement Suggestion

Checklist

Lotus component

Improvement Suggestion

Checklist

Lotus component

Lotus Version

Describe the Bug

Logging Information

Repo Steps

Context

Proposal

Race on new deal

Race on restart

Recommend Projects

Recommend Topics

Recommend Org