How to replicate databases? about field-manual HOT 11 OPEN

Thanks @Andlinks-Yourturn for opening the issue! I'm not sure what exactly you mean by "have nodes synced by default", perhaps you can elaborate what problems you're having?

As for getting peers to connect and sync, I would recommend to try out the PR here: orbitdb/orbit-db-cli#8. It contains a fix that makes peers to automatically start syncing when they connect. We're working on getting that PR merged into master and released soon.

As for how to replicate (sync) a database between peers, see https://github.com/orbitdb/orbit-db-cli#replicate.

Let me know if this doesn't solve your problem.

from field-manual.

Thx, @haadcode
Unfortunately, orbitdb replicate didn't work for me
I started my add two nodes together (following github.com/Kubuxu/go-ipfs-swarm-key-gen/ipfs-swarm-key-gen ) to create a private swarm.
ipfs swarm peers shows that they are connected,
then I followed the instruction
orbitdb create hello feed orbitdb add /orbitdb/QmXtU2Jo3HA7rTZgAC8na4XJbvfvQK56RE9SxUGTRpQxUe/hello "world" orbitdb replicate /orbitdb/QmXtU2Jo3HA7rTZgAC8na4XJbvfvQK56RE9SxUGTRpQxUe/hello --progress --dashboard
the output is
Swarm listening on /ip4/127.0.0.1/tcp/42731/ipfs/QmXtU2Jo3HA7rTZgAC8na4XJbvfvQK56RE9SxUGTRpQxUe Swarm listening on /ip4/192.168.1.63/tcp/42731/ipfs/QmXtU2Jo3HA7rTZgAC8na4XJbvfvQK56RE9SxUGTRpQxUe Loading '/orbitdb/QmXtU2Jo3HA7rTZgAC8na4XJbvfvQK56RE9SxUGTRpQxUe/hello' (feed) Loading '/orbitdb/QmXtU2Jo3HA7rTZgAC8na4XJbvfvQK56RE9SxUGTRpQxUe/hello' ░░░░░░░░░░░░░░░ 0/1 | 0.0% | 00:00:00 Loading '/orbitdb/QmXtU2Jo3HA7rTZgAC8na4XJbvfvQK56RE9SxUGTRpQxUe/hello' ░░░░░░░░░░░░░░░ 0/0 | 0.0% | 00:00:00 Loading '/orbitdb/QmXtU2Jo3HA7rTZgAC8na4XJbvfvQK56RE9SxUGTRpQxUe/hello' ░░░░░░░░░░░░░░░ 0/0 | 0.0% | 00:00:00 ███████████████████████████████████████████████████████████████████████████████████████ 1/1 | 100.0% | 00:00:00 Replicating '/orbitdb/QmXtU2Jo3HA7rTZgAC8na4XJbvfvQK56RE9SxUGTRpQxUe/hello' ███████████████████████████████████████████████████████████████████████████████████████ 1/1 | 100.0% | 00:00:00 ┌─────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │O │ └─────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ Peers: 0 | Tasks running: 0 | Queued: 0 | Replicated: 0 Bytes

I checked the other node
orbitdb get /orbitdb/QmXtU2Jo3HA7rTZgAC8na4XJbvfvQK56RE9SxUGTRpQxUe/hello Database '/orbitdb/QmXtU2Jo3HA7rTZgAC8na4XJbvfvQK56RE9SxUGTRpQxUe/hello' is empty!

It seems the dababase is not replicated.

Can you point out where's wrong?

Thanks so much ^^

from field-manual.

Got it. So there's couple of things you need to do:

First, we currently don't have a support for running a separate IPFS daemon to be used with OrbitDB, but we will in the future. What this means is that you don't need to start the go-ipfs daemon to run OrbitDB (and thus can't use private network, yet). In short: OrbitDB will start IPFS for you.

Second, the reason why your second node say "database is empty" is that by default it'll only try to load the database locally and since it's not replicated yet, it's considered empty. To make it work, you can pass --sync parameter to the second peer, eg. orbitdb get /orbitdb/QmXtU2Jo3HA7rTZgAC8na4XJbvfvQK56RE9SxUGTRpQxUe/hello --sync. What it'll do is that the second peer will wait until it's connected to the first peer (this may take up to 30sec), replicate the database and display the results of the database.

Third, note that all of the above will work smoothly only with the aforementioned PR, so make sure you're using that. And note that when using that PR, you need to run node src/bin.js <command> as orbitdb as a command will only be available when installed from npm.

Fourth, if you don't want to use the PR above, you can still get the replication working but it won't start automatically, so the first peer needs to be connected to the second peer and then issue an update to the database (ie. add). To do this, you can run the following on the first peer:

orbitdb add <address> --interactive --sync

And on the second peer:

orbitdb replicate <address> --progress --dashboard

The first peer will then wait and connect to the second peer and in the dashboard output of the second peer you'll see "Peers: 1 / 1" when they are connected. Once you see that, the first peer will display a prompt where you can write text and upon hitting enter, the first peer will add the text to the database and it'll be replicated to the second peer. You can confirm this by looking at the dashboard and see that there's one entry in the database ("Replicating ... 1 / 1 | 100%"). When you see the entry on the second peer, you can close it with ctrl-c (twice) and then run:

orbitdb get <address>

...and it'll display the entry you just added.

I understand this is a bit cumbersome with the version in master/npm and as such I highly recommend to use the PR as it fixes these problems and makes everything "automatic" and easy in terms of the replication.

Let me know if this helps or if you have more questions. Feedback in generally is also highly appreciated!

Edited the issue description to better describe what's been discussed here

from field-manual.

Hey, @haadcode
Thanks for all these helpful instructions, it's been very helpful.
I succeed with Master branch,
orbitdb add /orbitdb/QmbLR9y91cE1r5Mti4D72L2hLEdF9PvsqrJLweVnRjiFjJ/test --interactive --sync
(I strangely got an entry of "--sync" when querying the feed.)

I ran with PR branch
node ./src/bin get /orbitdb/QmWk3d4oPKyt2ycHLni2C3LpruNyeVEL89G4meMyM8fN3h/andlinks --progress --sync
I waited for some time, but there's no output

And I'd love to know if it's possible for Orbitdb to run on a privae swarm. Look forward to contribute on this subject.

from field-manual.

@Andlinks-Yourturn - I am doing my research on this as well. But looking at the source, and @haadcode 's comment, I think this is still WIP.
Were you able use go-ipfs privately and connect the orbit-db binary from PR branch? Let me know.

from field-manual.

Where can I find the settings to resolve options.create=false? I tried to manually adjust the config.js either before compiling and after... always the same error.

$ orbitdb add a feed --interactive --sync
Swarm listening on /ip4/127.0.0.1/tcp/55401/ipfs/QmZNc6cH9D1dV8af6z6ZtXrabRSTJW6wkMxeDW19xQjN8u
Swarm listening on /ip4/192.52.3.142/tcp/55401/ipfs/QmZNc6cH9D1dV8af6z6ZtXrabRSTJW6wkMxeDW19xQjN8u
Error: 'options.create' set to 'false'. If you want to create a database, set 'options.create' to 'true'.

from field-manual.

@iivooo I'm getting this error too. Not sure what the problem is.

Edit: got it! When you create your db, it spits out a directory with a hash in it. Use that as the directory of your database.

from field-manual.

Error during replication. I've started the first peer with 'orbitdb replicate -p --dashboard' and then started second with 'orbitdb add -r --sync'. It starts replicating but on the last item the second peer throws an error:

 /usr/local/lib/node_modules/orbit-db-cli/node_modules/ipfs-pubsub-room/src/connection.js:50
      this._ipfs._libp2pNode.dialProtocol(peerAddresses[0], PROTOCOL, (err, conn) => {
                             ^

TypeError: this._ipfs._libp2pNode.dialProtocol is not a function
    at _getPeerAddresses (/usr/local/lib/node_modules/orbit-db-cli/node_modules/ipfs-pubsub-room/src/connection.js:50:30)

It replicates all but last item to the peer. The item is added on the second peer but not replicated.

from field-manual.

@dibu28 thanks for reporting the issue! This should now be fixed in master and release 0.0.15 on npm. Latest ipfs and orbit-db contains the fix.

Re. the other error mentioned by @iivooo and @lpulley, indeed use the full address that orbit-db gives out as the result of a 'create' as noted by @lpulley. All commands except 'create' should require the full orbit-db address.

from field-manual.

Hello, I have questions regarding the similar topic.
In many places I read do this "...." in peer 1 and do this "..." in peer 2.
My questions are:
if the app is the same how will it know if is peer1 or peer2?
if the app is running each peer will try to create a new database?
how does the app running peer2 will receive the db address from peer1?

from field-manual.

Moving to the Field Manual for deeper discussion

from field-manual.