Seeking Intructions on How to Run My Own Node about estuary HOT 31 CLOSED

No problem! I don't think I could confidently comment on that since I'm not involved with estuary-www atm, but I think jim should be able to answer when he's back. :)

Oh amazing thank you so much! I really appreciate everything! I will tag Jim so he can confirm that when he gets back thanks again! @jimmylee

from estuary.

Sorry for the delayed responses here, as a team we've been mostly communicating over slack and havent kept up on our issue tracker.

@elijaharita We should probably just default estuary to using api.chain.love, respect whatever values users set, but if its not there, default to something that works.

from estuary.

Hey! Not sure if you were able to get it running already, but for future visitors:

Estuary currently requires a connection to a Lotus node. Estuary will search for a locally hosted node by default, but if you don't have one (which I think is your case), you can tell it to connect to a remote node by setting the FULLNODE_API_INFO environment variable (wss://api.chain.love is what we use).

To generate the root API key, run estuary setup and save the resulting key somewhere. After that, you should be good to go.

Oh awesome! This info is perfect thank you so much. It appears that I was able to get the node running.

from estuary.

@whyrusleeping

from estuary.

Hey! Not sure if you were able to get it running already, but for future visitors:

Estuary currently requires a connection to a Lotus node. Estuary will search for a locally hosted node by default, but if you don't have one (which I think is your case), you can tell it to connect to a remote node by setting the FULLNODE_API_INFO environment variable (wss://api.chain.love is what we use).

To generate the root API key, run estuary setup and save the resulting key somewhere. After that, you should be good to go.

from estuary.

Sorry for the delayed responses here, as a team we've been mostly communicating over slack and havent kept up on our issue tracker.

@elijaharita We should probably just default estuary to using api.chain.love, respect whatever values users set, but if its not there, default to something that works.

No problem and thanks for this reply!

from estuary.

Btw one more question...
How do I run the front end to connect to the node locally? I'm under the impression that the node can't really operate without the front in displayed through the web-browser. Do I just run the estuary-www repo and input the API key and information from my local node somewhere in a config file in the estuary-www repo and do npm start or something like that?

Like for example:

1. git clone https://github.com/application-research/estuary-www.git
2. cd estuary-www
3. npm install
4. nano path/to/some/env/file
5. add line api=localhost:whatever-port-estuary-node-runs-on

Idk something like that?

from estuary.

Btw one more question...
How do I run the front end to connect to the node locally? I'm under the impression that the node can't really operate without the front in displayed through the web-browser. Do I just run the estuary-www repo and input the API key and information from my local node somewhere in a config file in the estuary-www repo and do npm start or something like that?

Like for example:

1. `git clone https://github.com/application-research/estuary-www.git`

2. `cd estuary-www`

3. `npm install`

4. `nano path/to/some/env/file`

5. add line `api=localhost:whatever-port-estuary-node-runs-on`

Idk something like that?

@elijaharita @whyrusleeping

from estuary.

Was actually looking into how to do this just now! So normally @jimmylee would be able to answer this, but he is away ATM.

The solution I came up with is a bit janky, but for now you can nano common/constants.ts and change the value of the host variable on the second to last line to your own Estuary address (for a local Estuary node it'll be http://localhost:3004). After that, you can go to the webpage in your browser (http://localhost:4444), and should be able to log in using the API key you generated with estuary setup.

from estuary.

btw, I just tested this myself and found the output of estuary setup a bit messed up with an unnecessary error message and missing newline. Just merged a fix for that. If you want to redo the API key generation step, you can run git pull, delete the database file estuary.db, and estuary setup again.

from estuary.

Was actually looking into how to do this just now! So normally @jimmylee would be able to answer this, but he is away ATM.

The solution I came up with is a bit janky, but for now you can nano common/constants.ts and change the value of the host variable on the second to last line to your own Estuary address (for a local Estuary node it'll be http://localhost:3004). After that, you can go to the webpage in your browser (http://localhost:4444), and should be able to log in using the API key you generated with estuary setup.

Thank you so much for the help. I am still having some trouble signing in though. Do I use the new Estuary token I created in the ./estuary setup in export const auth = 'ESTUARY_TOKEN'; replacing ESTURAY_KEY with the token? Or do I use the token as the Invite Key or Password in the create new user page? What do I put for the username @elijaharita

from estuary.

No need to change any files except that one line in constants.ts. With the token, you can go to the normal log in page, and underneath the user/password page, there is a section that allows you to log in directly using an API key.

from estuary.

No need to change any files except that one line in constants.ts. With the token, you can go to the normal log in page, and underneath the user/password page, there is a section that allows you to log in directly using an API key.

@elijaharita

Ohhhh haha I see it thanks! This is amazing thank you so much for the help!

One more minor note... If I want to set this up in a production environment should I just change the salt variable in constraints.ts to be unique, long, and random and make the auth variable in constraints.ts also unique, long, and random? Would that make it secure?

from estuary.

No problem! I don't think I could confidently comment on that since I'm not involved with estuary-www atm, but I think jim should be able to answer when he's back. :)

from estuary.

(he should be back tomorrow)

from estuary.

I'm back @ChristianOConnor are you still having trouble?

from estuary.

I'm back @ChristianOConnor are you still having trouble?

@jimmylee

Hi Jimmy! Huge fan of you and your code! Anyway I am trying to run my own estuary node that's completely self contained meaning no API keys and completely permission-less. I am running an estuary node via the repo on Github and connecting it to api.chain.love and then in a new vscode window, running the estuary-www page and connecting it to the estuary node by changing the host variable in common/constraints.ts from https://api.estuary.tech to http://localhost:3004.

My first question is if I change the salt variable in common/constraints.ts to a unique and lengthy new random string and change the auth variable to a unique and lengthy new random string, is that secure enough for production?

Also I noticed there are still some connections to the estuary.tech website which therefore make this node not self contained. Again my goal is to make this node completely self contained and therefore only relying on the api.chain.love connection. These are the only estuary.tech references I could find.
add-file.sh - EST_HOST="https://shuttle-2.estuary.tech"
handlers.go - if !s.CM.localContentAddingDisabled { out = append(out, "https://api.estuary.tech/content/add") }
main.go - var loginCmd = &cli.Command{ Name: "login", Flags: []cli.Flag{ &cli.StringFlag{ Name: "host", Value: "https://api.estuary.tech", }, },
main.go - var benchAddFileCmd = &cli.Command{ Name: "add-file", Flags: []cli.Flag{ &cli.StringFlag{ Name: "host", Value: "api.estuary.tech", },
main.go - &cli.StringFlag{ Name: "estuary-api", Usage: "api endpoint for master estuary node", Value: "api.estuary.tech",

Is there any way to get rid of these web connections without breaking the node?

from estuary.

Hey there! Happy to help with the estuary-www questions, thanks a bunch for kicking the tires on our work, its super helpful.

My first question is if I change the salt variable in common/constraints.ts to a unique and lengthy new random string and change the auth variable to a unique and lengthy new random string, is that secure enough for production?

This is correct, and by design. Our authentication strategy never touches a "middleman" server. For us that means our servers will never see a plain text password, for you that means you will never see a plain text password. We believe for most threat models this is appropriate, but I have also suggested having another set of secrets and hashing again on the server in case more of your infrastructure is compromised, but it shouldn't matter, bcrypt is only uni-directional.

add-file.sh - EST_HOST="https://shuttle-2.estuary.tech"
handlers.go - if !s.CM.localContentAddingDisabled { out = append(out, "https://api.estuary.tech/content/add") }
main.go - var loginCmd = &cli.Command{ Name: "login", Flags: []cli.Flag{ &cli.StringFlag{ Name: "host", Value: "https://api.estuary.tech", }, },
main.go - var benchAddFileCmd = &cli.Command{ Name: "add-file", Flags: []cli.Flag{ &cli.StringFlag{ Name: "host", Value: "api.estuary.tech", },
main.go - &cli.StringFlag{ Name: "estuary-api", Usage: "api endpoint for master estuary node", Value: "api.estuary.tech",

Is there any way to get rid of these web connections without breaking the node?

This is a great question for @whyrusleeping and @elijaharita, I'm sure they can make this configurable.

from estuary.

@ChristianOConnor Most of those references to estuary.tech are just default values in config flags, so you just specify your own values there.
The add-file.sh script is mostly a debugging script i use to wrap around curl, I can change this to check for an existing value for EST_HOST and use that if found.
The value in handlers.go is definitely a bad hardcoding, I will fix that one up now.

from estuary.

Hey there! Happy to help with the estuary-www questions, thanks a bunch for kicking the tires on our work, its super helpful.

My first question is if I change the salt variable in common/constraints.ts to a unique and lengthy new random string and change the auth variable to a unique and lengthy new random string, is that secure enough for production?

This is correct, and by design. Our authentication strategy never touches a "middleman" server. For us that means our servers will never see a plain text password, for you that means you will never see a plain text password. We believe for most threat models this is appropriate, but I have also suggested having another set of secrets and hashing again on the server in case more of your infrastructure is compromised, but it shouldn't matter, bcrypt is only uni-directional.

add-file.sh - EST_HOST="https://shuttle-2.estuary.tech"
handlers.go - if !s.CM.localContentAddingDisabled { out = append(out, "https://api.estuary.tech/content/add") }
main.go - var loginCmd = &cli.Command{ Name: "login", Flags: []cli.Flag{ &cli.StringFlag{ Name: "host", Value: "https://api.estuary.tech", }, },
main.go - var benchAddFileCmd = &cli.Command{ Name: "add-file", Flags: []cli.Flag{ &cli.StringFlag{ Name: "host", Value: "api.estuary.tech", },
main.go - &cli.StringFlag{ Name: "estuary-api", Usage: "api endpoint for master estuary node", Value: "api.estuary.tech",

Is there any way to get rid of these web connections without breaking the node?

This is a great question for @whyrusleeping and @elijaharita, I'm sure they can make this configurable.

Oh awesome! Thanks for the info! Glad I could provide some feedback!

from estuary.

@ChristianOConnor Most of those references to estuary.tech are just default values in config flags, so you just specify your own values there.
The add-file.sh script is mostly a debugging script i use to wrap around curl, I can change this to check for an existing value for EST_HOST and use that if found.
The value in handlers.go is definitely a bad hardcoding, I will fix that one up now.

Oh great thanks!

from estuary.

@ChristianOConnor, I am very interested to see how this works out for you. please report back.

from estuary.

@ChristianOConnor how did things go?

from estuary.

So sorry for the delay @sheriflouis and @jimmylee .

Anyway below are some notes from my using the latest version of the app stack aka estuary and estuary-www.

I noticed that I can only authenticate 1 time in the estuary-www webpage in my browser connected to my estuary node. I get this when I try to use my API key to autheticate in the sign in page a second time:

<myusername>@<mycomputername> estuary-www % npm run dev   

> [email protected] dev
> next dev -p 4444

ready - started server on 0.0.0.0:4444, url: http://localhost:4444
info  - Using webpack 5. Reason: Enabled by default https://nextjs.org/docs/messages/webpack5
event - compiled successfully
event - build page: /
wait  - compiling...
event - build page: /404
{ error: 'ERR_INVALID_TOKEN' }
event - build page: /sign-in
event - compiled successfully
wait  - compiling...
{ error: 'ERR_INVALID_TOKEN' }
Could not find files for /sign-in in .next/build-manifest.json
Could not find files for /sign-in in .next/build-manifest.json
event - compiled successfully
{ error: 'ERR_INVALID_TOKEN' }
{ error: 'ERR_INVALID_TOKEN' }
wait  - compiling...
event - compiled successfully

And in the node terminal I get this:

   ____    __
  / __/___/ /  ___
 / _// __/ _ \/ _ \
/___/\__/_//_/\___/ v4.5.0
High performance, minimalist Go web framework
https://echo.labstack.com
____________________________________O/_______
                                    O\
⇨ http server started on [::]:3004
2021-09-30T05:15:54.367-0400    INFO    estuary estuary/main.go:278     running key provider func
2021-09-30T05:15:54.369-0400    INFO    estuary estuary/main.go:288     key provider func returning 0 values

2021/09/30 05:16:24 /Users/<myusername>/Desktop/Coding/EstuaryLotusStackStart092921/estuary/handlers.go:2066 record not found
[1.255ms] [rows:0] SELECT * FROM `auth_tokens` WHERE token = "<My Correct Auth Token>" AND `auth_tokens`.`deleted_at` IS NULL ORDER BY `auth_tokens`.`id` LIMIT 1
2021-09-30T05:16:24.084-0400    ERROR   estuary estuary/handlers.go:76  handler error: ERR_INVALID_TOKEN
{"time":"2021-09-30T05:16:24.08507-04:00","id":"","remote_ip":"127.0.0.1","host":"localhost:3004","method":"GET","uri":"/viewer","user_agent":"node-fetch/1.0 (+https://github.com/bitinn/node-fetch)","status":403,"error":"ERR_INVALID_TOKEN","latency":1812791,"latency_human":"1.812791ms","bytes_in":0,"bytes_out":30}

2021/09/30 05:16:55 /Users/<myusername>/Desktop/Coding/EstuaryLotusStackStart092921/estuary/handlers.go:2066 record not found
[0.208ms] [rows:0] SELECT * FROM `auth_tokens` WHERE token = "<My Correct Auth Token>" AND `auth_tokens`.`deleted_at` IS NULL ORDER BY `auth_tokens`.`id` LIMIT 1
2021-09-30T05:16:55.383-0400    ERROR   estuary estuary/handlers.go:76  handler error: ERR_INVALID_TOKEN
{"time":"2021-09-30T05:16:55.383773-04:00","id":"","remote_ip":"127.0.0.1","host":"localhost:3004","method":"GET","uri":"/viewer","user_agent":"node-fetch/1.0 (+https://github.com/bitinn/node-fetch)","status":403,"error":"ERR_INVALID_TOKEN","latency":2135415,"latency_human":"2.135415ms","bytes_in":0,"bytes_out":30}
{"time":"2021-09-30T05:17:01.688751-04:00","id":"","remote_ip":"127.0.0.1","host":"localhost:3004","method":"OPTIONS","uri":"/public/metrics/deals-on-chain","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:92.0) Gecko/20100101 Firefox/92.0","status":204,"error":"","latency":36679,"latency_human":"36.679µs","bytes_in":0,"bytes_out":0}
{"time":"2021-09-30T05:17:01.693286-04:00","id":"","remote_ip":"127.0.0.1","host":"localhost:3004","method":"OPTIONS","uri":"/public/miners","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:92.0) Gecko/20100101 Firefox/92.0","status":204,"error":"","latency":33417,"latency_human":"33.417µs","bytes_in":0,"bytes_out":0}
{"time":"2021-09-30T05:17:01.712033-04:00","id":"","remote_ip":"127.0.0.1","host":"localhost:3004","method":"GET","uri":"/public/miners","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:92.0) Gecko/20100101 Firefox/92.0","status":200,"error":"","latency":3312711,"latency_human":"3.312711ms","bytes_in":0,"bytes_out":2208}
{"time":"2021-09-30T05:17:01.720217-04:00","id":"","remote_ip":"127.0.0.1","host":"localhost:3004","method":"GET","uri":"/public/metrics/deals-on-chain","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:92.0) Gecko/20100101 Firefox/92.0","status":200,"error":"","latency":2288949,"latency_human":"2.288949ms","bytes_in":0,"bytes_out":5}
{"time":"2021-09-30T05:17:01.888258-04:00","id":"","remote_ip":"127.0.0.1","host":"localhost:3004","method":"GET","uri":"/public/metrics/deals-on-chain","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:92.0) Gecko/20100101 Firefox/92.0","status":200,"error":"","latency":889427,"latency_human":"889.427µs","bytes_in":0,"bytes_out":5}
{"time":"2021-09-30T05:17:01.919043-04:00","id":"","remote_ip":"127.0.0.1","host":"localhost:3004","method":"OPTIONS","uri":"/public/stats","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:92.0) Gecko/20100101 Firefox/92.0","status":204,"error":"","latency":14138,"latency_human":"14.138µs","bytes_in":0,"bytes_out":0}
{"time":"2021-09-30T05:17:01.923683-04:00","id":"","remote_ip":"127.0.0.1","host":"localhost:3004","method":"GET","uri":"/public/stats","user_agent":"Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:92.0) Gecko/20100101 Firefox/92.0","status":200,"error":"","latency":835785,"latency_human":"835.785µs","bytes_in":0,"bytes_out":51}

2021/09/30 05:17:04 /Users/<myusername>/Desktop/Coding/EstuaryLotusStackStart092921/estuary/handlers.go:2066 record not found
[0.457ms] [rows:0] SELECT * FROM `auth_tokens` WHERE token = "<My Correct Auth Token>" AND `auth_tokens`.`deleted_at` IS NULL ORDER BY `auth_tokens`.`id` LIMIT 1
2021-09-30T05:17:04.812-0400    ERROR   estuary estuary/handlers.go:76  handler error: ERR_INVALID_TOKEN
{"time":"2021-09-30T05:17:04.812249-04:00","id":"","remote_ip":"127.0.0.1","host":"localhost:3004","method":"GET","uri":"/viewer","user_agent":"node-fetch/1.0 (+https://github.com/bitinn/node-fetch)","status":403,"error":"ERR_INVALID_TOKEN","latency":658475,"latency_human":"658.475µs","bytes_in":0,"bytes_out":30}

2021/09/30 05:17:36 /Users/<myusername>/Desktop/Coding/EstuaryLotusStackStart092921/estuary/handlers.go:2066 record not found
[0.217ms] [rows:0] SELECT * FROM `auth_tokens` WHERE token = "<My Correct Auth Token>" AND `auth_tokens`.`deleted_at` IS NULL ORDER BY `auth_tokens`.`id` LIMIT 1
2021-09-30T05:17:36.980-0400    ERROR   estuary estuary/handlers.go:76  handler error: ERR_INVALID_TOKEN
{"time":"2021-09-30T05:17:36.980249-04:00","id":"","remote_ip":"127.0.0.1","host":"localhost:3004","method":"GET","uri":"/viewer","user_agent":"node-fetch/1.0 (+https://github.com/bitinn/node-fetch)","status":403,"error":"ERR_INVALID_TOKEN","latency":3622810,"latency_human":"3.62281ms","bytes_in":0,"bytes_out":30}
2021-09-30T05:19:54.604-0400    INFO    estuary estuary/replication.go:396      content check queue     {"length": 0, "nextEvent": "0001-01-01T00:00:00.000Z"}

I did the following to try to make sure I get my node completely disconnected from Estuary's APIs. Did I do that correctly?

In estuary-www, I changed:

1. changed const curl = curl \n-X POST https://api.estuary.tech/content/add \n-H "Authorization: Bearer YOUR_API_KEY" \n-H "Accept: application/json" \n-H "Content-Type: multipart/form-data" \n-F "data=@PATH_TO_FILE"; to const curl = curl \n-X POST http://localhost:3004/content/add \n-H "Authorization: Bearer YOUR_API_KEY" \n-H "Accept: application/json" \n-H "Content-Type: multipart/form-data" \n-F "data=@PATH_TO_FILE"; in index.tsx
2. In experience.txt I changed curl -X GET -H "Authorization: Bearer $ESTUARY_TOKEN" https://api.estuary.tech/content/staging-zones | jq to curl -X GET -H "Authorization: Bearer $ESTUARY_TOKEN" https://http://localhost:3004/content/staging-zones | jq (I know this does nothing but I wanted to be really thorough.
3. In constant.ts I changed export const auth = 'ESTUARY_TOKEN'; to contain my generated Estuary token.

In estuary, I changed:

1. In add-file.sh, I commented out line 14, then uncommented line 11 and replaced whysaccesstoken3 with my created Estuary token. I uncommented line 12 which is #EST_HOST="http://localhost:3004"
2. In cmd/barge/main.go, cmd/benchtest/main.go, and cmd/benchtest/main.go I replaced every instance of https://api.estuary.tech with http://localhost:3004 the one time it appeared in each file.

In order to authenticate again I have to delete my storage directory and do ./estuary setup again. Also, Am I truly disconnected from estuary.tech as far as all of the file-uploading/ipfs/filecoin actions are concerned?

Btw, is this the problem? It looks like it deleted my Auth token?
https://imgur.com/a/S7hiENo

UPDATE

So I was able to get around this issue by redoing the Estuary setup.

1. I deleted estuary/estuary.db
2. I deleted everything in the storage directory EstuaryStorOne in the parallel directory to estuary.
3. then I ran ./estuary setup and restarted the node.
4. Next time when I first authenticated I MADE SURE I SET A PASSWORD IN THE ACCOUNT TAB.

https://imgur.com/a/PakllGZ
https://imgur.com/a/KJb1InD

I noticed that I can't see the Filecoin address in the Balance tab. I get my address from the terminal when I start my estuary node but I can't actually see my Filecoin address or any of the stats. Also what's my escrow address? Does a third party handle the escrow account or am I still in control of the private key for my escrow account?

from estuary.

You control the private key for your escrow account, its simply just to separate the Filecoin you're willing to use for deals and the Filecoin you're not willing to use for deals.

from estuary.

Thank you for sharing your notes!

from estuary.

@jimmylee Oh awesome thanks! How can I get my Filecoin Address private keys for my Deposit account and escrow account?

And is that normal that none of my Filecoin Address stats are available in the balance tab? I can't see the balance or address of the accounts. Maybe that's normal because I didn't deposit the Filecoin yet?

from estuary.

And is that normal that none of my Filecoin Address stats are available in the balance tab? I can't see the balance or address of the accounts. Maybe that's normal because I didn't deposit the Filecoin yet?

Yes once you deposit some funds you will see it in the UI

from estuary.

And is that normal that none of my Filecoin Address stats are available in the balance tab? I can't see the balance or address of the accounts. Maybe that's normal because I didn't deposit the Filecoin yet?

Yes once you deposit some funds you will see it in the UI

Oh awesome thanks!

from estuary.

@wfschrec can you make sure this feedback is incorporated in your docs/instructions/setup script? thank you!!

from estuary.

from estuary.