This problem was created when bootstrap nodes were added.
Apparently peers cannot see each other without a WebRTC listening server.
A user can upload and download files, but CID queries do nothing because of the peers blindness.

#30 obviates the need for that entire top box on the Options tab (Rendezvous IP and Rendezvous Port), so this issue is to remove that from options.

We need to also remove the code related to those options. This would include the Save button handler, turning the rendezvous addr+port into a hardcoded constant instead of a user-settable variable, completely remove it from the config data model, etc.

Also there is some cleanup around line 33 of src/background/Node.js:

// ToDo: remove these lines at cleanup

// const rendezvousProtocol = /^\d+\.\d+\.\d+\.\d+$/.test(rendezvousIp) ? 'ip4' : 'dns4';
// const rendezvousWsProtocol = `${rendezvousPort}` === '443' ? 'wss' : 'ws';

// const rendezvousAddress = `/${rendezvousProtocol}/${rendezvousIp}/tcp/${rendezvousPort}/${rendezvousWsProtocol}/p2p-webrtc-star`;

First, look at the Query CID box history. You can see I have never queried for the CID QmTWK8JbyyBsP9U5qgJxNH8iQQeHF8S5WkMu2wi3L95hNe

But below there, I have nine offers to buy that CID. Browser extension was untouched and the offers just appeared.

Goal of this issue:

• Find the cause
• Fix it

The basic idea is to add a status column to box showing transfers in progress:

Here are some additional requirements:

• Write ports.setDealStatus(dealId,"Message...."). The status column is updated by calling a new function that needs to be written called ports.setDealStatus. Why? The status will be updated from the background primarily, and ports.js is the current mechanism we have for the background to send notifications to the UI.

• Each row in the transfers table may need an invisible dealId value. There are a lot of ways to approach this, but the point is that there may be multiple transfers going on simultaneously. A call like ports.setDealStatus(72389001284393,"Message for deal 72389001284393....") should only cause the status field to be updated for the row with dealId 72389001284393. The unique identifier has to be dealId (not CID) because there could be multiple downloads of the same CID at the same time.

• have to confirm the nonce value from another node

For this one, might require a second cloud lotus instance, just so we have a nonce reference.

• have to have funds in the To&From wallets so they exist on chain
• check the exit code on "null" receipts returned

Let's just put all exit codes in the log (even if its 0 == success exit)

This will include:

• redesign the project structure

• bundle the web app using parcel

• convert the frontend side from extension to web app

• adapt the communication between frontend and background

• adapt the background side to be run by the web app

• upgrade libp2p and its dependencies to latest

MpoolEstimateGasPrice(ctx context.Context, nblocksincl uint64, sender address.Address, gaslimit int64, tsk types.TipSetKey) (types.BigInt, error)

Use nblocksincl = 2
tsk can be empty string

Observed: It is not possible to configure the extension via copy and paste of parameters because the popup closes when you tab out to another window to copy another configuration parameter and loses in progress changes
Expected: Popup should not close, or remember the in progress configuration changes
Environment: Firefox on PopOS 20.04 (Ubuntu 20.04 based)
Steps to reproduce:

1. Install extension on firefox
2. Copy one of the configuration parameters into the clipboard (e.g. lotus endpoint)
3. Click "Filecoin" icon
4. Click "Options"
5. Paste the configuration parameter in the clipboard into a text box on the configuration screen
6. Tab to another application (e.g. slack) to copy the next configuration parameter (e.g. JWT Wallet)
7. Tab back to browser, observe that popup is no longer visible
8. Click "Filecoin" icon
9. Click "Options", observe that the first configuration item pasted on step 5 is gone

After starting the extension in Firefox I try to query for a file I have uploaded in Chromium. The logs show that the node is being restarted every time I query for it.

Logs:

DEBUG: creating peer id
DEBUG: creating libp2p node
DEBUG: creating lotus client
DEBUG: creating datastore
DEBUG: creating retrieval market client
DEBUG: creating retrieval market provider
DEBUG: starting libp2p node
DEBUG: adding listeners to node
DEBUG: getting node info
DEBUG: node created
DEBUG: Enter query function
DEBUG: Offers cleared
INFO: querying for QmdPvu6HvQcxPSbbQzvn6WbFAfvpZBKox6qM1dhrxNZ1FL , minerID:  t0102
DEBUG: publishing message {"messageType":"RetrievalQuery","cid":"QmdPvu6HvQcxPSbbQzvn6WbFAfvpZBKox6qM1dhrxNZ1FL"}
INFO: restarting
DEBUG: stopping ipfs node
DEBUG: received message {"messageType":"RetrievalQueryResponse","cid":"QmdPvu6HvQcxPSbbQzvn6WbFAfvpZBKox6qM1dhrxNZ1FL","multiaddrs":["/dns4/jsrc-bootstrap.goelzer.io/tcp/443/wss/p2p-webrtc-star/p2p/QmetSas2E8JCyFYY3UriqH6YTiqkNvjTWJsnUU9B6v9JAQ"],"params":{"wallet":"t137sjdbgunloi7couiy4l5nc7pd6k2jmq32vizpy","size":57978,"pricePerByte":1000,"paymentInterval":1048576,"paymentIntervalIncrease":1048576}}
DEBUG: Handle query response
INFO: restarting
DEBUG: stopping ipfs node
ERROR: stop node failed: repo is already closed
DEBUG: creating peer id
DEBUG: creating libp2p node
DEBUG: creating lotus client
DEBUG: creating datastore
DEBUG: creating retrieval market client
DEBUG: creating retrieval market provider
DEBUG: starting libp2p node
DEBUG: creating peer id
DEBUG: creating libp2p node
DEBUG: creating lotus client
DEBUG: creating datastore
ERROR: start node failed: Lock already being held for file: filecoinretrieval/repo.lock
DEBUG: adding listeners to node
DEBUG: getting node info
DEBUG: node created

NOTE: I have added some more DEBUG message to follow the process.

Looks like it is hapening when this function is being called : https://github.com/filecoin-shipyard/browser-retrieval/blob/master/src/background/Node.js#L194-L204

Could it trigger the hook (https://github.com/filecoin-shipyard/browser-retrieval/blob/master/src/background/background.js#L53-L68) ? and then restart the browser...

For big numbers math, pretty much anything price-related

MetaMask opens a new tab when the extension icon is clicked. Let's do the same, all browsers.

Currently, we're thinking this would happen any time the icon is clicked in the tool bar.

(Question: can we think of any scenarios where having multiple instances of the extension open simultaneously would cause harm or race conditions, etc? Answer: no, not a problem, there is a single application state for all tabs running the extension.)

ENS.domains was interested in a Filecoin-enabled content resolver.
This relates to the general Epic of being able to retrieve data from the Filecoin Network in a browser.

Ideal flow (including ENS):

1. User requests a domain like filecoin.shipyard.eth that is mapped to an IPFS content hash in an ENS registry
   (https://app.ens.domains/name/brantly.xyz provides an example mapping and EthDNS can be used to get the hash)

2. The gateway attempts to retrieve the content hash from IPFS

3. If not available in IPFS it checks the FIlecoin network and if available prompts the user to initiate and pay for retrieval

4. The user either downloads the data or displays it in a website gateway.
   (A related workflow is discussed for web torrents using MetaMask here: https://blog.statechannels.org/introducing-web3torrent/)

Please see demonstration video here: https://drive.google.com/drive/folders/1RNYKnP1-pR1xwyyjEJmuarHC945IwGp_?usp=sharing

• The buy button should show a transfer of $CID1 starting, a progress bar and completion.
• If there is an error, it should be shown in the main UI (not only in the log). #64
• User should still be able to click other buy buttons to initiate additional buys of $CID1
• Only once the user queries for a different CID ($CID2), or all transfers for $CID1 have completed, should the box showing Offers for CID {$CID1} disappear.
• Even if the box disappears, transfers should continue.

When the user presses "Query," these things should happen:

• Whitespace around the CID in the query field should be trimmed (both left trim and right trim)
• Search conducted as it is now
• A panel drops down[*] showing all the offers found by the search.
• The user can select one. He can also have his javascript override, which will receive the list of offers as a JSON array of {"cid":"fdadsfadsfasdfasd", "priceAttofil":"832832", "sizeBytes":"1024", "proivderMultiAddr":"/ip4/1.2.3.4/tcp/p2p/k12jkdsjdsnksdksdk"}, select one offer according to an algorithm.

• Here's a really crude illustration of what I mean by "a panel drops down":

What

• when an error occurs show an "error" div with the message and a button to dismiss

Why

Users can only see error if they go to the logs tab, it's not clear if things are working on not unless they analyze logs

Additional Details

• Here's a sketch of roughly how I envision this looking

• Note that there can be 0 or 1 or 2 or more error bars being displayed at the same time. They persist until "Dismiss" is pressed.

• We need to be able to post these errors from background. Thus, my suggestion is to modify ports.js so that we can make a call like this from anywhere:

ports.addErrorBar("Error:  There is a chicken on the roof.");

To be added on the pymt-chan branch:

• verify voucher (used only by provider)
• update PCH (used only by provider)
• settle PCH (used by client in closeDeal)
• collect PCH (used by client and by provider)

On Options, Rendezvous IP works correctly with jsrc-bootstrap.goelzer.io but not with webrtc-star-1.browser-retrieval.filecoin.io. The latter stays stuck at "Trying to connect to rendezvous server" and no peers are ever found. Both resolve to the same IP address.

We need to fix webrtc-star-1.browser-retrieval.filecoin.io so it can work.

I assume this is just an nginx or similar proxy, but we should document the steps somewhere.

Our current strategy for browser retrieval clients is to be able to direct retrieve from Storage Miners is to put both on the same libp2p network.

Goal of this issue is:

• Put browser retrieval clients on same p2p network as Storage Miners (this means using same bootstrapper nodes for each - you can look these up in the go code or ask for help in #shared-pl-digitalmob-retrieval)
• Verify that browser clients can filter out Storage Miner <-> Storage Miner gossip messages (may require a header on browser gossip messages)
• Verify that Storage Miners are not harmed by browser <-> browser gossip messages
• Verify that one browser node can still retrieve from another browser node successfully (see demo of how extension is supposed to work here (starts at 2:36:54))

Zondax has a lot of expertise here, and I expect them to deliver something working by end of this week. If that lands sooner than filecoin-rs supports the full PCH flow (PCH creation, voucher creation, voucher verification, PCH settle, chain waitch for PCH settle, PCH update with voucher (within 24 hours), PCH Close and funds transferred by specs-actors), I'd like to just use the wasm code in the interest of time.

Blocked on Zondax having workable code (expect end of this week).

This is temporary, until we support BLS wallets in #14

Related: filecoin-shipyard/proxy-retrieval#27

Browser:

1. request a CID
   • if NOT available: gets available: false and stops
2. sends a message indicating that funds have been sent
   • insufficient funds: stops
   • price changed: stops
3. QueryRetrievalStatus
4. Chunk
5. ChunkReceived
6. ChunkResend

QueryCid / CidAvailability

• send
  • send CID from query form
  • send miner from query form

{
  message
  cid
  miner
}

• handle
  • if available
    • continue connection
    • send funds (#97)
    • send FundsSent message
  • if NOT available
    • end connection
    • display some feedback message
  • save client token for future messages

{
  available
  cid
  client_token
  message
  payment_wallet
  price_attofil
}

FundsSent / FundsConfirmed

• send

{
  message
  client_token
  payment_wallet
}

• handle
  • funds_confirmed: continue
  • funds_confirmed_error_insufficient_funds: x
  • funds_confirmed_error_price_changed: x

{
  message
  cid
  payment_wallet
  remaining_balance
}

QueryRetrievalStatus

• send
  • every N seconds check status ???

{
  message
  cid
  client_token
}

• handle
  • display ???

{
  message
  cid
  status
}

Chunk

• send with clientToken
  • on success ChunkReceived
  • on error ChunkResend N times
    • define and set N
    • invalid sha256
    • invalid length
    • any error

{
  message
  client_token
  cid
  id
}

• handle
  • validate sha256
  • validate length

{
  chunk_data
  chunk_len_bytes
  chunk_sha256
  cid
  full_data_len_bytes
  full_data_sha256
  id
  message
}

Options:

1. Put browsers on the same p2p network as the storage miners and let them request directly
2. Have cloud lotus function as a proxy => does retrieval from the storage miner => passes complete data back to the browser client

Get information on what the protocol is by which storage miners send data base on lotus client retrieve. Will Lotus tolerate with a non-Lotus peer on its peer-to-peer network?

Add minerID form and send the minerID to background service.

This could be done in lotus.closePaymentChannel or Client.closeDeal:

• call Settle on the PCH
• pend an operation that calls Collect to the pending operations queue. Return value is blank (success) or an error message.
• Do the same in Provider.js's analog of Client.closeDeal (or maybe both call into lotus.closePaymentChannel?)

(On the pymt-chan branch.)

After a payment channel is no longer in use, there is a 12-hour waiting period before you can call "Collect" on the channel and receive the funds. The goal of this issue is to create a cron-like feature where we can add pending operations with a deadline when they execute.

This should be more robust and easier to debug than using window.setTimeout() every time we have an operation that must run 12 hour later.

Description:

• Create or find a preexisting cron-like class for browser apps. (I'm sure there are open source ones - e.g., from Node.) For now, let's call this class PendingOperations.
• PendingOperations exposes an API to add objects to a Set<> or similar. Each object represents an operation the will run in the future. These objects could look like this:

     op = {
        label: "some label, e.g., Collect payment channel from deal 7623923 (refund balance 1000 attoFil)",
        f: func.bind(user-specified args),  // func is some function def'd in Lotus.closeDeal probably
        invokeAt: datetime, // this is when the PendingOperations class should invoke pendingOp 
     }

• The class should wake up once a second and do these things:

  • invokes any pending operations where currentTime>=invokeAt and deletes those objects from the Set
  • update a new, always-visible section beneath "Known Cids" on the Home tab that displays all pending operations:

         +-----------------------------------------------------------------------------------------+
         | Pending Operations                                                                      |
         +---------------------------+-------------------------------------------------------------+
         |  HH:MM:SS until invokeAt  | Label                                                       |
         |  Executing                | Label                                                       |
         |  Done                     | ${ String returned by f() }                                 |
         +---------------------------+-------------------------------------------------------------+

If there are no pending operations, display this:

         +-----------------------------------------------------------------------------------------+
         | Pending Operations                                                                      |
         +-----------------------------------------------------------------------------------------+
         | None                                                                                    |
         +-----------------------------------------------------------------------------------------+

• The class should persist itself to some browser storage area just like the Options currently do. This ensures that closing/reopening the browser, or even refreshing the extension, will not lose any of the objects in the Set.

If it's not possible to serialize a function in JS, help me think of another solution. For example, a string that is eval'd at the correct time (though it will need to run in the context of Lotus.js).

Get feedback on UI

tl;dr When a Buy button is clicked, we need to initiate a proper retrieval deal with the other browser using Client.retrieve().

When a Buy button is clicked:

• it looks like it calls Offers.downloadFile() in src/popup/App/Home/Offers/Offers.js
• That function sends a message to the background like this: chrome.runtime.sendMessage({ messageType: messageTypes.downloadFile, msg });
• messageTypes.downloadFile calls Node.downloadFile(), which uses chrome.downloads.download(). That transfers the file for free.

What we want instead is to use the class Client interface in src/background/retrieval-market/Client.js to kick off a retrieval deal. Since Node object already constructs the Client class and saves it as this.client, we should just add a Node method like described here.

We need to undo #30 because FC bootstrap peers don't have a wss endpoint and can't add one until mid-October at the earliest.

Mike will ask Gui and/or inspect SSL cert on rendezvous machine

• Do you update channel after receiving each voucher? NO
• Can you only keep the highest verified voucher received? YES
• Is Submit/Update before or after Settle? AFTER

Need to solve #47 before making these changes.

Let's remove these options from the UI and hardcode them as consts in the source code:

• Rendezvous IP - use webrtc-star-1.browser-retrieval.filecoin.io
• Rendezvous Port - use 443
• Lotus endpoint - use http://cloud-lotus-1.browser-retrieval.filecoin.io:80/rpc/v0
• JWT Token - see this private doc
• Payment interval (which is in bytes, no ms) - use 1048576
• Interval Increase - 1048576

Uploading a file to the extension fails in Firefox, but works correctly in Chrome. Maybe be a side effect of #29.

Here is a video showing how it fails in Firefox.

Here is a video showing the desired behavior.

This issue covers two related bugs:

1. When you open the extension in Firefox, and then Alt+Tab away from it to a different window, and then Alt+Tab back to Firefox, then extension has disappeared! The behavior I want is for the Extension to stay open across Alt+Tab window changes. Video demonstrating the problem: https://drive.google.com/file/d/1ZROnpZBJJ4wzeFyJsXSXh2ZZ3_plQslR/view?usp=sharing

This bug occurs on Firefox for Mac. It does not occur in Chrome.

2. Even if you ignore the problem in #1 above, there is still a problem with Options values disappearing within the extension. See video description here: https://drive.google.com/file/d/1U_yDJE_FKvY5BB2-FM0aW4-t5YJFx4kz/view?usp=sharing

The desired behavior is that when you Alt+Tab away from the extension without having saved the values, they should still remain populated in the Options fields. After all 4 values are filled in, the user should then press Save.

If the user tries to move away from the Options tab without pressing Save, a message (like a modal dialog) should pop up warning that "You have unsaved values in the Options fields. Please press Save to set those values." The fields that are filled in but not yet saved should be highlighted, and pressing Save should remove the highlight.

This bug occurs in both Firefox and Chrome (on Mac).

We want to give the user a user-defined function that gets executed inside the JS code. An example use case might be if the user knows that a certain software download website has been updated recently, he or she could download it and offer it as a CID on the retrieval market. We call these "custom JS strategies."

The user should have a tab with a code editor text box, where he/she can input any JS code and the browser-based application will execute that code every ten minutes (as long as the browser tab stays open). If running the code produces an error, it should be printed in the log. Note that the user may edit the code many times. After each edit + save button press, the code should run immediately, and then again 10 minutes later, again 10 minutes after that, etc.

The default context in the editor :

async function userLoop(_this) {
  ////////////////////////////////////////////////////////////////////////////////////
  // Here's how to query the storage network for a CID
  //let cid = "bafk2bzacedgizbdbiiji5rohflf47ax2zgmnkbl2kx3nezrr2ygb7mxdoc3x6"
  //let asks = _this.queryStorageMiners(cid)  // dummy func for now
  //for ask in asks {
  //  console.log(ask["price"], ask["minerId"])
  //}

  ////////////////////////////////////////////////////////////////////////////////////
  // Here's how to retrieve a CID from a storage miner
  // let amt = asks[0]["price"];
  // let miner = ask[0]["minerId"];
  // _this.retrieveFromStorageMiner(cid, miner, amt); //dummy for now

  ////////////////////////////////////////////////////////////////////////////////////
  // Here's how to change the price of a CID your node is offering
  // let cid = "bafk2bzacedgizbdbiiji5rohflf47ax2zgmnkbl2kx3nezrr2ygb7mxdoc3x6"
  // let priceInFil = "0.000000005"; // total price for the CID
  // cids.updatePrice(cid, priceInFil);
  
  let f = function { userLoop(_this); } 
  window.setTimeout("f()", 600);
}

It should go in a code-editor widget under a tab like "automation".

The word "Required" here is an indicator on the top box (Query CID), so it should appear directly beneath that box when it's empty.

The miner id stuff should appear below that.

Move repo and change name

The goal of this task is to get rid of the extension and replace it with a regular webpage server by a normal HTTP server in the cloud.