filecoin-shipyard / browser-retrieval Goto Github PK
View Code? Open in Web Editor NEWA p2p network of Filecoin content retrievers
A p2p network of Filecoin content retrievers
This problem was created when bootstrap nodes were added.
Apparently peers cannot see each other without a WebRTC listening server.
A user can upload and download files, but CID queries do nothing because of the peers blindness.
#30 obviates the need for that entire top box on the Options tab (Rendezvous IP and Rendezvous Port), so this issue is to remove that from options.
We need to also remove the code related to those options. This would include the Save button handler, turning the rendezvous addr+port into a hardcoded constant instead of a user-settable variable, completely remove it from the config data model, etc.
Also there is some cleanup around line 33 of src/background/Node.js
:
// ToDo: remove these lines at cleanup
// const rendezvousProtocol = /^\d+\.\d+\.\d+\.\d+$/.test(rendezvousIp) ? 'ip4' : 'dns4';
// const rendezvousWsProtocol = `${rendezvousPort}` === '443' ? 'wss' : 'ws';
// const rendezvousAddress = `/${rendezvousProtocol}/${rendezvousIp}/tcp/${rendezvousPort}/${rendezvousWsProtocol}/p2p-webrtc-star`;
The basic idea is to add a status column to box showing transfers in progress:
Here are some additional requirements:
Write ports.setDealStatus(dealId,"Message....")
. The status column is updated by calling a new function that needs to be written called ports.setDealStatus
. Why? The status will be updated from the background primarily, and ports.js is the current mechanism we have for the background to send notifications to the UI.
Each row in the transfers table may need an invisible dealId
value. There are a lot of ways to approach this, but the point is that there may be multiple transfers going on simultaneously. A call like ports.setDealStatus(72389001284393,"Message for deal 72389001284393....")
should only cause the status field to be updated for the row with dealId 72389001284393
. The unique identifier has to be dealId (not CID) because there could be multiple downloads of the same CID at the same time.
For this one, might require a second cloud lotus instance, just so we have a nonce reference.
Let's just put all exit codes in the log (even if its 0 == success exit)
This will include:
redesign the project structure
bundle the web app using parcel
convert the frontend side from extension to web app
adapt the communication between frontend and background
adapt the background side to be run by the web app
upgrade libp2p and its dependencies to latest
MpoolEstimateGasPrice(ctx context.Context, nblocksincl uint64, sender address.Address, gaslimit int64, tsk types.TipSetKey) (types.BigInt, error)
Use nblocksincl = 2
tsk can be empty string
Observed: It is not possible to configure the extension via copy and paste of parameters because the popup closes when you tab out to another window to copy another configuration parameter and loses in progress changes
Expected: Popup should not close, or remember the in progress configuration changes
Environment: Firefox on PopOS 20.04 (Ubuntu 20.04 based)
Steps to reproduce:
After starting the extension in Firefox I try to query for a file I have uploaded in Chromium. The logs show that the node is being restarted every time I query for it.
Logs:
DEBUG: creating peer id
DEBUG: creating libp2p node
DEBUG: creating lotus client
DEBUG: creating datastore
DEBUG: creating retrieval market client
DEBUG: creating retrieval market provider
DEBUG: starting libp2p node
DEBUG: adding listeners to node
DEBUG: getting node info
DEBUG: node created
DEBUG: Enter query function
DEBUG: Offers cleared
INFO: querying for QmdPvu6HvQcxPSbbQzvn6WbFAfvpZBKox6qM1dhrxNZ1FL , minerID: t0102
DEBUG: publishing message {"messageType":"RetrievalQuery","cid":"QmdPvu6HvQcxPSbbQzvn6WbFAfvpZBKox6qM1dhrxNZ1FL"}
INFO: restarting
DEBUG: stopping ipfs node
DEBUG: received message {"messageType":"RetrievalQueryResponse","cid":"QmdPvu6HvQcxPSbbQzvn6WbFAfvpZBKox6qM1dhrxNZ1FL","multiaddrs":["/dns4/jsrc-bootstrap.goelzer.io/tcp/443/wss/p2p-webrtc-star/p2p/QmetSas2E8JCyFYY3UriqH6YTiqkNvjTWJsnUU9B6v9JAQ"],"params":{"wallet":"t137sjdbgunloi7couiy4l5nc7pd6k2jmq32vizpy","size":57978,"pricePerByte":1000,"paymentInterval":1048576,"paymentIntervalIncrease":1048576}}
DEBUG: Handle query response
INFO: restarting
DEBUG: stopping ipfs node
ERROR: stop node failed: repo is already closed
DEBUG: creating peer id
DEBUG: creating libp2p node
DEBUG: creating lotus client
DEBUG: creating datastore
DEBUG: creating retrieval market client
DEBUG: creating retrieval market provider
DEBUG: starting libp2p node
DEBUG: creating peer id
DEBUG: creating libp2p node
DEBUG: creating lotus client
DEBUG: creating datastore
ERROR: start node failed: Lock already being held for file: filecoinretrieval/repo.lock
DEBUG: adding listeners to node
DEBUG: getting node info
DEBUG: node created
NOTE: I have added some more DEBUG
message to follow the process.
Looks like it is hapening when this function is being called : https://github.com/filecoin-shipyard/browser-retrieval/blob/master/src/background/Node.js#L194-L204
Could it trigger the hook (https://github.com/filecoin-shipyard/browser-retrieval/blob/master/src/background/background.js#L53-L68) ? and then restart the browser...
For big numbers math, pretty much anything price-related
MetaMask opens a new tab when the extension icon is clicked. Let's do the same, all browsers.
Currently, we're thinking this would happen any time the icon is clicked in the tool bar.
(Question: can we think of any scenarios where having multiple instances of the extension open simultaneously would cause harm or race conditions, etc? Answer: no, not a problem, there is a single application state for all tabs running the extension.)
ENS.domains was interested in a Filecoin-enabled content resolver.
This relates to the general Epic of being able to retrieve data from the Filecoin Network in a browser.
Ideal flow (including ENS):
User requests a domain like filecoin.shipyard.eth
that is mapped to an IPFS content hash in an ENS registry
(https://app.ens.domains/name/brantly.xyz provides an example mapping and EthDNS can be used to get the hash)
The gateway attempts to retrieve the content hash from IPFS
If not available in IPFS it checks the FIlecoin network and if available prompts the user to initiate and pay for retrieval
The user either downloads the data or displays it in a website gateway.
(A related workflow is discussed for web torrents using MetaMask here: https://blog.statechannels.org/introducing-web3torrent/)
Please see demonstration video here: https://drive.google.com/drive/folders/1RNYKnP1-pR1xwyyjEJmuarHC945IwGp_?usp=sharing
Offers for CID {$CID1}
disappear.When the user presses "Query," these things should happen:
{"cid":"fdadsfadsfasdfasd", "priceAttofil":"832832", "sizeBytes":"1024", "proivderMultiAddr":"/ip4/1.2.3.4/tcp/p2p/k12jkdsjdsnksdksdk"}
, select one offer according to an algorithm.Users can only see error if they go to the logs tab, it's not clear if things are working on not unless they analyze logs
Note that there can be 0 or 1 or 2 or more error bars being displayed at the same time. They persist until "Dismiss" is pressed.
We need to be able to post these errors from background. Thus, my suggestion is to modify ports.js
so that we can make a call like this from anywhere:
ports.addErrorBar("Error: There is a chicken on the roof.");
To be added on the pymt-chan
branch:
On Options, Rendezvous IP works correctly with jsrc-bootstrap.goelzer.io
but not with webrtc-star-1.browser-retrieval.filecoin.io
. The latter stays stuck at "Trying to connect to rendezvous server" and no peers are ever found. Both resolve to the same IP address.
We need to fix webrtc-star-1.browser-retrieval.filecoin.io
so it can work.
I assume this is just an nginx or similar proxy, but we should document the steps somewhere.
Our current strategy for browser retrieval clients is to be able to direct retrieve from Storage Miners is to put both on the same libp2p network.
Goal of this issue is:
Zondax has a lot of expertise here, and I expect them to deliver something working by end of this week. If that lands sooner than filecoin-rs supports the full PCH flow (PCH creation, voucher creation, voucher verification, PCH settle, chain waitch for PCH settle, PCH update with voucher (within 24 hours), PCH Close and funds transferred by specs-actors), I'd like to just use the wasm code in the interest of time.
Blocked on Zondax having workable code (expect end of this week).
This is temporary, until we support BLS wallets in #14
Related: filecoin-shipyard/proxy-retrieval#27
Browser:
available: false
and stops{
message
cid
miner
}
FundsSent
message{
available
cid
client_token
message
payment_wallet
price_attofil
}
{
message
client_token
payment_wallet
}
funds_confirmed
: continuefunds_confirmed_error_insufficient_funds
: xfunds_confirmed_error_price_changed
: x{
message
cid
payment_wallet
remaining_balance
}
{
message
cid
client_token
}
{
message
cid
status
}
clientToken
ChunkReceived
ChunkResend
N times
{
message
client_token
cid
id
}
sha256
length
{
chunk_data
chunk_len_bytes
chunk_sha256
cid
full_data_len_bytes
full_data_sha256
id
message
}
Options:
Get information on what the protocol is by which storage miners send data base on lotus client retrieve
. Will Lotus tolerate with a non-Lotus peer on its peer-to-peer network?
Add minerID form and send the minerID to background service.
This could be done in lotus.closePaymentChannel
or Client.closeDeal
:
lotus.closePaymentChannel
?)(On the pymt-chan
branch.)
After a payment channel is no longer in use, there is a 12-hour waiting period before you can call "Collect" on the channel and receive the funds. The goal of this issue is to create a cron-like feature where we can add pending operations with a deadline when they execute.
This should be more robust and easier to debug than using window.setTimeout()
every time we have an operation that must run 12 hour later.
Description:
op = {
label: "some label, e.g., Collect payment channel from deal 7623923 (refund balance 1000 attoFil)",
f: func.bind(user-specified args), // func is some function def'd in Lotus.closeDeal probably
invokeAt: datetime, // this is when the PendingOperations class should invoke pendingOp
}
The class should wake up once a second and do these things:
+-----------------------------------------------------------------------------------------+
| Pending Operations |
+---------------------------+-------------------------------------------------------------+
| HH:MM:SS until invokeAt | Label |
| Executing | Label |
| Done | ${ String returned by f() } |
+---------------------------+-------------------------------------------------------------+
If there are no pending operations, display this:
+-----------------------------------------------------------------------------------------+
| Pending Operations |
+-----------------------------------------------------------------------------------------+
| None |
+-----------------------------------------------------------------------------------------+
If it's not possible to serialize a function in JS, help me think of another solution. For example, a string that is eval'd at the correct time (though it will need to run in the context of Lotus.js).
Get feedback on UI
tl;dr When a Buy button is clicked, we need to initiate a proper retrieval deal with the other browser using Client.retrieve().
When a Buy button is clicked:
src/popup/App/Home/Offers/Offers.js
chrome.runtime.sendMessage({ messageType: messageTypes.downloadFile, msg });
chrome.downloads.download()
. That transfers the file for free.What we want instead is to use the class Client
interface in src/background/retrieval-market/Client.js
to kick off a retrieval deal. Since Node object already constructs the Client class and saves it as this.client
, we should just add a Node method like described here.
We need to undo #30 because FC bootstrap peers don't have a wss endpoint and can't add one until mid-October at the earliest.
Mike will ask Gui and/or inspect SSL cert on rendezvous machine
Need to solve #47 before making these changes.
Let's remove these options from the UI and hardcode them as consts in the source code:
http://cloud-lotus-1.browser-retrieval.filecoin.io:80/rpc/v0
Uploading a file to the extension fails in Firefox, but works correctly in Chrome. Maybe be a side effect of #29.
This issue covers two related bugs:
This bug occurs on Firefox for Mac. It does not occur in Chrome.
The desired behavior is that when you Alt+Tab away from the extension without having saved the values, they should still remain populated in the Options fields. After all 4 values are filled in, the user should then press Save.
If the user tries to move away from the Options tab without pressing Save, a message (like a modal dialog) should pop up warning that "You have unsaved values in the Options fields. Please press Save to set those values." The fields that are filled in but not yet saved should be highlighted, and pressing Save should remove the highlight.
This bug occurs in both Firefox and Chrome (on Mac).
We want to give the user a user-defined function that gets executed inside the JS code. An example use case might be if the user knows that a certain software download website has been updated recently, he or she could download it and offer it as a CID on the retrieval market. We call these "custom JS strategies."
The user should have a tab with a code editor text box, where he/she can input any JS code and the browser-based application will execute that code every ten minutes (as long as the browser tab stays open). If running the code produces an error, it should be printed in the log. Note that the user may edit the code many times. After each edit + save button press, the code should run immediately, and then again 10 minutes later, again 10 minutes after that, etc.
The default context in the editor :
async function userLoop(_this) {
////////////////////////////////////////////////////////////////////////////////////
// Here's how to query the storage network for a CID
//let cid = "bafk2bzacedgizbdbiiji5rohflf47ax2zgmnkbl2kx3nezrr2ygb7mxdoc3x6"
//let asks = _this.queryStorageMiners(cid) // dummy func for now
//for ask in asks {
// console.log(ask["price"], ask["minerId"])
//}
////////////////////////////////////////////////////////////////////////////////////
// Here's how to retrieve a CID from a storage miner
// let amt = asks[0]["price"];
// let miner = ask[0]["minerId"];
// _this.retrieveFromStorageMiner(cid, miner, amt); //dummy for now
////////////////////////////////////////////////////////////////////////////////////
// Here's how to change the price of a CID your node is offering
// let cid = "bafk2bzacedgizbdbiiji5rohflf47ax2zgmnkbl2kx3nezrr2ygb7mxdoc3x6"
// let priceInFil = "0.000000005"; // total price for the CID
// cids.updatePrice(cid, priceInFil);
let f = function { userLoop(_this); }
window.setTimeout("f()", 600);
}
It should go in a code-editor widget under a tab like "automation".
Move repo and change name
The goal of this task is to get rid of the extension and replace it with a regular webpage server by a normal HTTP server in the cloud.
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.