Comments (31)
No problem! I don't think I could confidently comment on that since I'm not involved with estuary-www atm, but I think jim should be able to answer when he's back. :)
Oh amazing thank you so much! I really appreciate everything! I will tag Jim so he can confirm that when he gets back thanks again! @jimmylee
from estuary.
Sorry for the delayed responses here, as a team we've been mostly communicating over slack and havent kept up on our issue tracker.
@elijaharita We should probably just default estuary to using api.chain.love, respect whatever values users set, but if its not there, default to something that works.
from estuary.
Hey! Not sure if you were able to get it running already, but for future visitors:
Estuary currently requires a connection to a Lotus node. Estuary will search for a locally hosted node by default, but if you don't have one (which I think is your case), you can tell it to connect to a remote node by setting the
FULLNODE_API_INFO
environment variable (wss://api.chain.love
is what we use).To generate the root API key, run
estuary setup
and save the resulting key somewhere. After that, you should be good to go.
Oh awesome! This info is perfect thank you so much. It appears that I was able to get the node running.
from estuary.
from estuary.
Hey! Not sure if you were able to get it running already, but for future visitors:
Estuary currently requires a connection to a Lotus node. Estuary will search for a locally hosted node by default, but if you don't have one (which I think is your case), you can tell it to connect to a remote node by setting the FULLNODE_API_INFO
environment variable (wss://api.chain.love
is what we use).
To generate the root API key, run estuary setup
and save the resulting key somewhere. After that, you should be good to go.
from estuary.
Sorry for the delayed responses here, as a team we've been mostly communicating over slack and havent kept up on our issue tracker.
@elijaharita We should probably just default estuary to using api.chain.love, respect whatever values users set, but if its not there, default to something that works.
No problem and thanks for this reply!
from estuary.
Btw one more question...
How do I run the front end to connect to the node locally? I'm under the impression that the node can't really operate without the front in displayed through the web-browser. Do I just run the estuary-www repo and input the API key and information from my local node somewhere in a config file in the estuary-www repo and do npm start
or something like that?
Like for example:
git clone https://github.com/application-research/estuary-www.git
cd estuary-www
npm install
nano path/to/some/env/file
- add line
api=localhost:whatever-port-estuary-node-runs-on
Idk something like that?
from estuary.
Btw one more question...
How do I run the front end to connect to the node locally? I'm under the impression that the node can't really operate without the front in displayed through the web-browser. Do I just run the estuary-www repo and input the API key and information from my local node somewhere in a config file in the estuary-www repo and donpm start
or something like that?Like for example:
1. `git clone https://github.com/application-research/estuary-www.git` 2. `cd estuary-www` 3. `npm install` 4. `nano path/to/some/env/file` 5. add line `api=localhost:whatever-port-estuary-node-runs-on`
Idk something like that?
from estuary.
Was actually looking into how to do this just now! So normally @jimmylee would be able to answer this, but he is away ATM.
The solution I came up with is a bit janky, but for now you can nano common/constants.ts
and change the value of the host
variable on the second to last line to your own Estuary address (for a local Estuary node it'll be http://localhost:3004
). After that, you can go to the webpage in your browser (http://localhost:4444
), and should be able to log in using the API key you generated with estuary setup
.
from estuary.
btw, I just tested this myself and found the output of estuary setup
a bit messed up with an unnecessary error message and missing newline. Just merged a fix for that. If you want to redo the API key generation step, you can run git pull
, delete the database file estuary.db
, and estuary setup
again.
from estuary.
Was actually looking into how to do this just now! So normally @jimmylee would be able to answer this, but he is away ATM.
The solution I came up with is a bit janky, but for now you can
nano common/constants.ts
and change the value of thehost
variable on the second to last line to your own Estuary address (for a local Estuary node it'll behttp://localhost:3004
). After that, you can go to the webpage in your browser (http://localhost:4444
), and should be able to log in using the API key you generated withestuary setup
.
Thank you so much for the help. I am still having some trouble signing in though. Do I use the new Estuary token I created in the ./estuary setup
in export const auth = 'ESTUARY_TOKEN';
replacing ESTURAY_KEY with the token? Or do I use the token as the Invite Key or Password in the create new user page? What do I put for the username @elijaharita
from estuary.
No need to change any files except that one line in constants.ts. With the token, you can go to the normal log in page, and underneath the user/password page, there is a section that allows you to log in directly using an API key.
from estuary.
No need to change any files except that one line in constants.ts. With the token, you can go to the normal log in page, and underneath the user/password page, there is a section that allows you to log in directly using an API key.
Ohhhh haha I see it thanks! This is amazing thank you so much for the help!
One more minor note... If I want to set this up in a production environment should I just change the salt
variable in constraints.ts to be unique, long, and random and make the auth
variable in constraints.ts also unique, long, and random? Would that make it secure?
from estuary.
No problem! I don't think I could confidently comment on that since I'm not involved with estuary-www atm, but I think jim should be able to answer when he's back. :)
from estuary.
(he should be back tomorrow)
from estuary.
I'm back @ChristianOConnor are you still having trouble?
from estuary.
I'm back @ChristianOConnor are you still having trouble?
Hi Jimmy! Huge fan of you and your code! Anyway I am trying to run my own estuary node that's completely self contained meaning no API keys and completely permission-less. I am running an estuary node via the repo on Github and connecting it to api.chain.love and then in a new vscode window, running the estuary-www page and connecting it to the estuary node by changing the host
variable in common/constraints.ts
from https://api.estuary.tech
to http://localhost:3004
.
My first question is if I change the salt
variable in common/constraints.ts
to a unique and lengthy new random string and change the auth
variable to a unique and lengthy new random string, is that secure enough for production?
Also I noticed there are still some connections to the estuary.tech website which therefore make this node not self contained. Again my goal is to make this node completely self contained and therefore only relying on the api.chain.love connection. These are the only estuary.tech references I could find.
add-file.sh - EST_HOST="https://shuttle-2.estuary.tech"
handlers.go - if !s.CM.localContentAddingDisabled { out = append(out, "https://api.estuary.tech/content/add") }
main.go - var loginCmd = &cli.Command{ Name: "login", Flags: []cli.Flag{ &cli.StringFlag{ Name: "host", Value: "https://api.estuary.tech", }, },
main.go - var benchAddFileCmd = &cli.Command{ Name: "add-file", Flags: []cli.Flag{ &cli.StringFlag{ Name: "host", Value: "api.estuary.tech", },
main.go - &cli.StringFlag{ Name: "estuary-api", Usage: "api endpoint for master estuary node", Value: "api.estuary.tech",
Is there any way to get rid of these web connections without breaking the node?
from estuary.
Hey there! Happy to help with the estuary-www
questions, thanks a bunch for kicking the tires on our work, its super helpful.
My first question is if I change the salt variable in common/constraints.ts to a unique and lengthy new random string and change the auth variable to a unique and lengthy new random string, is that secure enough for production?
This is correct, and by design. Our authentication strategy never touches a "middleman" server. For us that means our servers will never see a plain text password, for you that means you will never see a plain text password. We believe for most threat models this is appropriate, but I have also suggested having another set of secrets and hashing again on the server in case more of your infrastructure is compromised, but it shouldn't matter, bcrypt
is only uni-directional.
add-file.sh - EST_HOST="https://shuttle-2.estuary.tech"
handlers.go - if !s.CM.localContentAddingDisabled { out = append(out, "https://api.estuary.tech/content/add") }
main.go - var loginCmd = &cli.Command{ Name: "login", Flags: []cli.Flag{ &cli.StringFlag{ Name: "host", Value: "https://api.estuary.tech", }, },
main.go - var benchAddFileCmd = &cli.Command{ Name: "add-file", Flags: []cli.Flag{ &cli.StringFlag{ Name: "host", Value: "api.estuary.tech", },
main.go - &cli.StringFlag{ Name: "estuary-api", Usage: "api endpoint for master estuary node", Value: "api.estuary.tech",
Is there any way to get rid of these web connections without breaking the node?
This is a great question for @whyrusleeping and @elijaharita, I'm sure they can make this configurable.
from estuary.
@ChristianOConnor Most of those references to estuary.tech are just default values in config flags, so you just specify your own values there.
The add-file.sh script is mostly a debugging script i use to wrap around curl, I can change this to check for an existing value for EST_HOST
and use that if found.
The value in handlers.go is definitely a bad hardcoding, I will fix that one up now.
from estuary.
Hey there! Happy to help with the
estuary-www
questions, thanks a bunch for kicking the tires on our work, its super helpful.My first question is if I change the salt variable in common/constraints.ts to a unique and lengthy new random string and change the auth variable to a unique and lengthy new random string, is that secure enough for production?
This is correct, and by design. Our authentication strategy never touches a "middleman" server. For us that means our servers will never see a plain text password, for you that means you will never see a plain text password. We believe for most threat models this is appropriate, but I have also suggested having another set of secrets and hashing again on the server in case more of your infrastructure is compromised, but it shouldn't matter,
bcrypt
is only uni-directional.add-file.sh - EST_HOST="https://shuttle-2.estuary.tech"
handlers.go - if !s.CM.localContentAddingDisabled { out = append(out, "https://api.estuary.tech/content/add") }
main.go - var loginCmd = &cli.Command{ Name: "login", Flags: []cli.Flag{ &cli.StringFlag{ Name: "host", Value: "https://api.estuary.tech", }, },
main.go - var benchAddFileCmd = &cli.Command{ Name: "add-file", Flags: []cli.Flag{ &cli.StringFlag{ Name: "host", Value: "api.estuary.tech", },
main.go - &cli.StringFlag{ Name: "estuary-api", Usage: "api endpoint for master estuary node", Value: "api.estuary.tech",Is there any way to get rid of these web connections without breaking the node?
This is a great question for @whyrusleeping and @elijaharita, I'm sure they can make this configurable.
Oh awesome! Thanks for the info! Glad I could provide some feedback!
from estuary.
@ChristianOConnor Most of those references to estuary.tech are just default values in config flags, so you just specify your own values there.
The add-file.sh script is mostly a debugging script i use to wrap around curl, I can change this to check for an existing value forEST_HOST
and use that if found.
The value in handlers.go is definitely a bad hardcoding, I will fix that one up now.
Oh great thanks!
from estuary.
@ChristianOConnor, I am very interested to see how this works out for you. please report back.
from estuary.
@ChristianOConnor how did things go?
from estuary.
So sorry for the delay @sheriflouis and @jimmylee .
Anyway below are some notes from my using the latest version of the app stack aka estuary
and estuary-www
.
I noticed that I can only authenticate 1 time in the estuary-www webpage in my browser connected to my estuary node. I get this when I try to use my API key to autheticate in the sign in page a second time:
<myusername>@<mycomputername> estuary-www % npm run dev
> [email protected] dev
> next dev -p 4444
ready - started server on 0.0.0.0:4444, url: http://localhost:4444
info - Using webpack 5. Reason: Enabled by default https://nextjs.org/docs/messages/webpack5
event - compiled successfully
event - build page: /
wait - compiling...
event - build page: /404
{ error: 'ERR_INVALID_TOKEN' }
event - build page: /sign-in
event - compiled successfully
wait - compiling...
{ error: 'ERR_INVALID_TOKEN' }
Could not find files for /sign-in in .next/build-manifest.json
Could not find files for /sign-in in .next/build-manifest.json
event - compiled successfully
{ error: 'ERR_INVALID_TOKEN' }
{ error: 'ERR_INVALID_TOKEN' }
wait - compiling...
event - compiled successfully
And in the node terminal I get this:
____ __
/ __/___/ / ___
/ _// __/ _ \/ _ \
/___/\__/_//_/\___/ v4.5.0
High performance, minimalist Go web framework
https://echo.labstack.com
____________________________________O/_______
O\
⇨ http server started on [::]:3004
2021-09-30T05:15:54.367-0400 INFO estuary estuary/main.go:278 running key provider func
2021-09-30T05:15:54.369-0400 INFO estuary estuary/main.go:288 key provider func returning 0 values
2021/09/30 05:16:24 /Users/<myusername>/Desktop/Coding/EstuaryLotusStackStart092921/estuary/handlers.go:2066 record not found
[1.255ms] [rows:0] SELECT * FROM `auth_tokens` WHERE token = "<My Correct Auth Token>" AND `auth_tokens`.`deleted_at` IS NULL ORDER BY `auth_tokens`.`id` LIMIT 1
2021-09-30T05:16:24.084-0400 ERROR estuary estuary/handlers.go:76 handler error: ERR_INVALID_TOKEN
{"time":"2021-09-30T05:16:24.08507-04:00","id":"","remote_ip":"127.0.0.1","host":"localhost:3004","method":"GET","uri":"/viewer","user_agent":"node-fetch/1.0 (+https://github.com/bitinn/node-fetch)","status":403,"error":"ERR_INVALID_TOKEN","latency":1812791,"latency_human":"1.812791ms","bytes_in":0,"bytes_out":30}
2021/09/30 05:16:55 /Users/<myusername>/Desktop/Coding/EstuaryLotusStackStart092921/estuary/handlers.go:2066 record not found
[0.208ms] [rows:0] SELECT * FROM `auth_tokens` WHERE token = "<My Correct Auth Token>" AND `auth_tokens`.`deleted_at` IS NULL ORDER BY `auth_tokens`.`id` LIMIT 1
2021-09-30T05:16:55.383-0400 ERROR estuary estuary/handlers.go:76 handler error: ERR_INVALID_TOKEN
{"time":"2021-09-30T05:16:55.383773-04:00","id":"","remote_ip":"127.0.0.1","host":"localhost:3004","method":"GET","uri":"/viewer","user_agent":"node-fetch/1.0 (+https://github.com/bitinn/node-fetch)","status":403,"error":"ERR_INVALID_TOKEN","latency":2135415,"latency_human":"2.135415ms","bytes_in":0,"bytes_out":30}
{"time":"2021-09-30T05:17:01.688751-04:00","id":"","remote_ip":"127.0.0.1","host":"localhost:3004","method":"OPTIONS","uri":"/public/metrics/deals-on-chain","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:92.0) Gecko/20100101 Firefox/92.0","status":204,"error":"","latency":36679,"latency_human":"36.679µs","bytes_in":0,"bytes_out":0}
{"time":"2021-09-30T05:17:01.693286-04:00","id":"","remote_ip":"127.0.0.1","host":"localhost:3004","method":"OPTIONS","uri":"/public/miners","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:92.0) Gecko/20100101 Firefox/92.0","status":204,"error":"","latency":33417,"latency_human":"33.417µs","bytes_in":0,"bytes_out":0}
{"time":"2021-09-30T05:17:01.712033-04:00","id":"","remote_ip":"127.0.0.1","host":"localhost:3004","method":"GET","uri":"/public/miners","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:92.0) Gecko/20100101 Firefox/92.0","status":200,"error":"","latency":3312711,"latency_human":"3.312711ms","bytes_in":0,"bytes_out":2208}
{"time":"2021-09-30T05:17:01.720217-04:00","id":"","remote_ip":"127.0.0.1","host":"localhost:3004","method":"GET","uri":"/public/metrics/deals-on-chain","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:92.0) Gecko/20100101 Firefox/92.0","status":200,"error":"","latency":2288949,"latency_human":"2.288949ms","bytes_in":0,"bytes_out":5}
{"time":"2021-09-30T05:17:01.888258-04:00","id":"","remote_ip":"127.0.0.1","host":"localhost:3004","method":"GET","uri":"/public/metrics/deals-on-chain","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:92.0) Gecko/20100101 Firefox/92.0","status":200,"error":"","latency":889427,"latency_human":"889.427µs","bytes_in":0,"bytes_out":5}
{"time":"2021-09-30T05:17:01.919043-04:00","id":"","remote_ip":"127.0.0.1","host":"localhost:3004","method":"OPTIONS","uri":"/public/stats","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:92.0) Gecko/20100101 Firefox/92.0","status":204,"error":"","latency":14138,"latency_human":"14.138µs","bytes_in":0,"bytes_out":0}
{"time":"2021-09-30T05:17:01.923683-04:00","id":"","remote_ip":"127.0.0.1","host":"localhost:3004","method":"GET","uri":"/public/stats","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:92.0) Gecko/20100101 Firefox/92.0","status":200,"error":"","latency":835785,"latency_human":"835.785µs","bytes_in":0,"bytes_out":51}
2021/09/30 05:17:04 /Users/<myusername>/Desktop/Coding/EstuaryLotusStackStart092921/estuary/handlers.go:2066 record not found
[0.457ms] [rows:0] SELECT * FROM `auth_tokens` WHERE token = "<My Correct Auth Token>" AND `auth_tokens`.`deleted_at` IS NULL ORDER BY `auth_tokens`.`id` LIMIT 1
2021-09-30T05:17:04.812-0400 ERROR estuary estuary/handlers.go:76 handler error: ERR_INVALID_TOKEN
{"time":"2021-09-30T05:17:04.812249-04:00","id":"","remote_ip":"127.0.0.1","host":"localhost:3004","method":"GET","uri":"/viewer","user_agent":"node-fetch/1.0 (+https://github.com/bitinn/node-fetch)","status":403,"error":"ERR_INVALID_TOKEN","latency":658475,"latency_human":"658.475µs","bytes_in":0,"bytes_out":30}
2021/09/30 05:17:36 /Users/<myusername>/Desktop/Coding/EstuaryLotusStackStart092921/estuary/handlers.go:2066 record not found
[0.217ms] [rows:0] SELECT * FROM `auth_tokens` WHERE token = "<My Correct Auth Token>" AND `auth_tokens`.`deleted_at` IS NULL ORDER BY `auth_tokens`.`id` LIMIT 1
2021-09-30T05:17:36.980-0400 ERROR estuary estuary/handlers.go:76 handler error: ERR_INVALID_TOKEN
{"time":"2021-09-30T05:17:36.980249-04:00","id":"","remote_ip":"127.0.0.1","host":"localhost:3004","method":"GET","uri":"/viewer","user_agent":"node-fetch/1.0 (+https://github.com/bitinn/node-fetch)","status":403,"error":"ERR_INVALID_TOKEN","latency":3622810,"latency_human":"3.62281ms","bytes_in":0,"bytes_out":30}
2021-09-30T05:19:54.604-0400 INFO estuary estuary/replication.go:396 content check queue {"length": 0, "nextEvent": "0001-01-01T00:00:00.000Z"}
I did the following to try to make sure I get my node completely disconnected from Estuary's APIs. Did I do that correctly?
In estuary-www, I changed:
- changed
const curl =
curl \n-X POST https://api.estuary.tech/content/add \n-H "Authorization: Bearer YOUR_API_KEY" \n-H "Accept: application/json" \n-H "Content-Type: multipart/form-data" \n-F "data=@PATH_TO_FILE";
toconst curl =
curl \n-X POST http://localhost:3004/content/add \n-H "Authorization: Bearer YOUR_API_KEY" \n-H "Accept: application/json" \n-H "Content-Type: multipart/form-data" \n-F "data=@PATH_TO_FILE";
in index.tsx - In experience.txt I changed
curl -X GET -H "Authorization: Bearer $ESTUARY_TOKEN" https://api.estuary.tech/content/staging-zones | jq
tocurl -X GET -H "Authorization: Bearer $ESTUARY_TOKEN" https://http://localhost:3004/content/staging-zones | jq
(I know this does nothing but I wanted to be really thorough. - In constant.ts I changed
export const auth = 'ESTUARY_TOKEN';
to contain my generated Estuary token.
In estuary, I changed:
- In add-file.sh, I commented out line 14, then uncommented line 11 and replaced
whysaccesstoken3
with my created Estuary token. I uncommented line 12 which is#EST_HOST="http://localhost:3004"
- In cmd/barge/main.go, cmd/benchtest/main.go, and cmd/benchtest/main.go I replaced every instance of
https://api.estuary.tech
withhttp://localhost:3004
the one time it appeared in each file.
In order to authenticate again I have to delete my storage directory and do ./estuary setup again. Also, Am I truly disconnected from estuary.tech as far as all of the file-uploading/ipfs/filecoin actions are concerned?
Btw, is this the problem? It looks like it deleted my Auth token?
https://imgur.com/a/S7hiENo
UPDATE
So I was able to get around this issue by redoing the Estuary setup.
- I deleted estuary/estuary.db
- I deleted everything in the storage directory EstuaryStorOne in the parallel directory to estuary.
- then I ran
./estuary setup
and restarted the node. - Next time when I first authenticated I MADE SURE I SET A PASSWORD IN THE ACCOUNT TAB.
https://imgur.com/a/PakllGZ
https://imgur.com/a/KJb1InD
I noticed that I can't see the Filecoin address in the Balance tab. I get my address from the terminal when I start my estuary node but I can't actually see my Filecoin address or any of the stats. Also what's my escrow address? Does a third party handle the escrow account or am I still in control of the private key for my escrow account?
from estuary.
You control the private key for your escrow account, its simply just to separate the Filecoin you're willing to use for deals and the Filecoin you're not willing to use for deals.
from estuary.
Thank you for sharing your notes!
from estuary.
@jimmylee Oh awesome thanks! How can I get my Filecoin Address private keys for my Deposit account and escrow account?
And is that normal that none of my Filecoin Address stats are available in the balance tab? I can't see the balance or address of the accounts. Maybe that's normal because I didn't deposit the Filecoin yet?
from estuary.
And is that normal that none of my Filecoin Address stats are available in the balance tab? I can't see the balance or address of the accounts. Maybe that's normal because I didn't deposit the Filecoin yet?
Yes once you deposit some funds you will see it in the UI
from estuary.
And is that normal that none of my Filecoin Address stats are available in the balance tab? I can't see the balance or address of the accounts. Maybe that's normal because I didn't deposit the Filecoin yet?
Yes once you deposit some funds you will see it in the UI
Oh awesome thanks!
from estuary.
@wfschrec can you make sure this feedback is incorporated in your docs/instructions/setup script? thank you!!
from estuary.
from estuary.
Related Issues (20)
- Estuary performance and deals pages don't load HOT 3
- Retrieval not working HOT 4
- /public/miners - chain_info is null
- Content can be added to staging zones located in dead shuttles
- PeerPingManager: Multiple sequential pings to average out jitter
- Add Storage Provider Deal Filtration
- GET /collections/:coluuid directory search is not working as intended HOT 1
- Files uploaded using FormData placed incorrectly within collection. HOT 2
- Revamp EstuaryV1 to use Delta/EV2
- unable to make filecoin deals HOT 1
- SSL Certificate error when trying to access uploaded files
- 502 when trying to upload CAR
- Broken links in Slate
- Estuary landing page failure HOT 4
- Patch documentation to reflect proper use of Python when uploading data
- Repeatedly getting 504 Gateway Timeout error on uploads HOT 1
- Metamask auth not working on Estuary
- PinManager not able to keep up with backlog
- Parameters missing in OpenAPI spec at `/pinning/pins`
- Pins queued for long time and deleting old pins
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from estuary.