zerotier / lf Goto Github PK

I read through the readme and I'm fascinated by this project. I've been interested in decentralization for a long time. One of the biggest problems facing web-based decentralization is the need for centralized signaling servers to allow browsers to speak with each other directly over WebRTC. LF has serious potential to solve that problem.

Given the speed of puts and gets on LF, and the way data for a topic is chained, LF could potentially serve as a fully distributed WebRTC signaling server to connect peers together without the need for centralized signaling servers. That is a truly powerful prospect.

@feross
@mafintosh
@DiegoRBaquero

make

go build -trimpath -ldflags -s -o ./lf cmd/lf/lf.go
flag provided but not defined: -trimpath
usage: go build [-o output] [-i] [build flags] [packages]
Run 'go help build' for details.

if removed -trimpath at Makefile

go build -ldflags -s -o ./lf cmd/lf/lf.go
# lf/pkg/lf
pkg/lf/clientconfig.go:79:14: undefined: strings.ReplaceAll
note: module requires Go 1.12make: *** [Makefile:32: lf] Error 2

go version go1.11.13 linux/amd64

Is LF (possibly in combination with conflict resolution mechanisms) able to provide monotic eventual consistency? In particular:

• nodes receiving same updates in any order will have same values for all keys
• a read from the same node will not return an older value for a key after receiving additional updates having previously returned a newer value (no rollback)

Hey, sounds like a cool project.
Can you elaborate more on the encryption?
When I store values are they encrypted?
Are there REST apis? or just the go command line? other languages apis?

Private ZeroTier network requires private LF, and private LF network requires ZeroTier. I'm wondering how to keep both LF and ZeroTier private ?

According to README.md:

LF peers will not talk to one another if they aren't members of the same network. This is accomplished by cryptographic means using the network's unique 256-bit ID as a pre-shared key. Beyond this simple mechanism there is no system built into LF to control node access over the network. It's the responsibility of those running private networks to secure them by (for example) running them only over ZeroTier virtual networks instead of over the public Internet.

Private ZeroTier network needs a working root which need LF.
Private LF network need ZeroTier to protect it from public internet.

???

Hi there!

Super cool project and great solution to decentralize ZeroTier (and lots of other things) 💖 🚀 I wonder about incentives for users to host a lf node to increase decentralization. Are they simply supposed to see the intrinsic value of doing this? Seems some incentive solution would drive decentralization much further 🤔

Also, are you looking for people to host lf nodes and join lf.zerotier.com as a peer?

Regards,
Asbjorn

As soon as I bootstrap a node from an existing parent and try to connect to it using P2P I receive the following error:

2020/03/04 17:24:14 P2P attempting to connect to 172.19.2.30 9908 brightfish
2020/03/04 17:24:14 P2P connection to 172.19.2.30:9908 closed: remote identity (public key) does not match expected identity

https://github.com/zerotier/lf#building-and-running

It's mostly written in Go (1.11+ required) with some C for performance critical bits.

On another note: This looks really neat! Excited to see where this goes.

https://lf.zerotier.com/ https certificate has expired 3 days ago.

➜ lf git:(master) ./lf status
ERROR: status query failed: Get "https://lf.zerotier.com/status": dial tcp 50.7.73.34:443: connect: connection refused

You guys should port this to Flutter with ICE-based NAT traversal capabilities. Which presumably would result in a cross platform (Android + iOS) capability for mobile handsets to communicate without using the mobile telecom infrastructure, which would in turn open the door for completely private decentralized P2P mobile handset coms etc.

For a fully decentralized system, blockchain is not a referential model as it proved not censorship-resistant in China. The Great Firewall(GFW) system can easily block those seed nodes and/or sniff further variations like Tor was totally crippled there.

What's the real unblockable model? It's worth of debate in depth.

Hi. Really excited to start playing with LF. I've been looking for a decentralized KV replication framework for a long time. I'm hoping you've nailed it. 👍

I'm looking to run KV replication between multiple servers to share some basic configuration data (without the limitations of distributed hashing or centralized approaches) but the private network instructions included in this project are a bit to basic.

So far I've gotten this far:

$ mkdir ~/lf_data
$ lf makegenesis
$ lf -path /home/user/lf_data/ -json node-start

Then in another term (note these are test keys/certs):

$ ls -path /home/user/lf_data status
{
  "Software": "ZeroTier LF Reference",
  "Version": [0, 9, 24, 0],
  "APIVersion": 1,
  "MinAPIVersion": 1,
  "MaxAPIVersion": 1,
  "Uptime": 254,
  "Clock": 1566569903,
  "RecordCount": 2,
  "DataSize": 1286,
  "FullySynchronized": true,
  "GenesisRecords": "\bHaRsnbEKxzWOIWIwSi5J14B06EqWCRRRNvDLHAxfXn9wNvZkxbGa3E7ggRyBS8U0i8gmiaD7BmDIrpXa4g69DNvJAcr9VkhKpG1hB1WeWGVPxj2AGD0Y30EIQ48b0pNBiX5t6SeiJXbjpdSm31ahqme9xv97s1KTjnL2CQVGiNzTzNU6wV53FdA42dt69agXBcb83v6la4pQVCz8hPXMadV78hdxJPG3lI0cKbSuMkD4P2F8Azujxvpnm8slv2rwCEtTNyUMUsActlfSljfkToFqGIwo2bmLu7GdavflSyqZNaLwQUNrXmyQCKIVBeahJE09HEZRgFPrWxySr4yV2IfXO8ROcQfDtGdDB6CSq9TuuslGtzmV7GT57kVbyXB8bMIAMGJogYLnn4h1CRl3FNJLsxW5cR1ESF52bIAuXSJTe1mymKy1bYx7t7pWB5jM4I7Qx6g8V2VbxBW4E0cQxVUKisQ2qM9LNTvKugHfrLlyEEfturhcpdwdZb2yPWC7M93R09CI5QmwFl6x8Vefn6krLuFNLOqOgzzkzIwhgvOIRg3zKIvR6Zf9tXizcL6YdXwqPABGVMQM2zATDORr4BIueujZWziWTm2u6PrvRcqfzqsqgETT3kUxHoJqdgmuCvQdX6fQj2MSeFFGrnpmov4OGEfM7AyrZmA2BIVH6wIx1jJqPGosRqeNUdnL2Bz2kdbLuGXXH3UrJEQKfZYUSXzo9ZPQhEto871sdicwJdDiBNGSPEAZllYso6ZIryGxq5fZIBfpLLYad35085I5knvn7iwLl1tpIpmty7uMaBG24RJReEO9DfDKbtg483Y3pHTWamC0uXcEXz8QDp2bIbZ171AHCxhJmFvS1TzjkmAotLXDlfurtmDbv4mtxAONkm2pT8jHbqgI20rO8HvrO5aUXWEd1m9WTo9w5IgZ698YGTwXcWqmvLPbGquhcyU40t9KRGzj2ZHfyK8GqWL3zhswnDY8Z91aDwMklPCWVnRPos670MgY0E5wKawWoakJwFLVZrZd2mvDM1aB3rzNgkXz9cFnBslWTOzvCj04HdAuFxDTdAH0foCIkkz20ByaansGOjxcWQseMrarHWh78BXk6jQrTydbv2KBiqPhX3vkRlQSa3foozyMXVhygIpD3ln4XQ3S67vMhieyAtMYgGE9DQDWLBOrNTXPAd9KQDjF9GRM3tBMy4QsKSB1NfOCMNKiH1N1SuTpkXiCaMhmIjjkLVwVX3J1FwKemohqDbNnxvTlygTFB05bOoXeScXZNV4rrlwKNTMDygwKreZmA0AeHaxGFehtHmhN71gkbb7VARnaV0FZvWkjPkNmAMDlyklGJNI5fn8JpgIEbE8QBgmoRw5MJcTJJ9rMheYWkJacXYOj31JW58reUrZCQpwSWbpSXRzQ7SRc83DFC42qEQzKf6QAyCvo0bjPcVbdz7snV47YxSZIFtTPZNm6yHzxKPi78H3ndp5hACciNH9Jt51TafwbQh9d1JWifwTyRQ12OhZugufarZ7oPetR3zsQrBQl2eXvWRlY011aXrMYWjVvRCUs3SclW3M3cxd3cNencVNr25vYTKSa9nkNXI5WVsq6hzCvI8Nu0dcrt4PF6nZZz8oH81iA2Q2nM8VvBBWY1qxYL3hPbmpxYY4Dyf0HebXKvljfBNGN3ra70TxBeyNOSnMPIfKvUia16gIte6mzvS63xX1VSqkGz5X64bLF",
  "GenesisParameters": {
    "ID": [153, 53, 94, 241, 154, 32, 192, 98, 227, 92, 205, 131, 213, 7, 66, 102, 96, 158, 12, 138, 84, 27, 67, 103, 227, 228, 163, 134, 106, 95, 57, 195],
    "AmendableFields": ["authcertificates"],
    "Name": "unity",
    "Contact": "[email protected]",
    "AuthCertificates": "\bidKH8qMHjgeh4pfSaC7wSlwZS6oq40lp5pTT2rwGvPzkCdetlF0Z8oMwR8ijL36jyQO74VhcpEUJlksVWOlLImAeb1Xf5ZZvK451x7gVNakSzkfopX7pLlt7SqfDk2QIbY3HUnMDSSaj21mMiAqVk6fGzdDtuANTDp9A3hDGv7TCTzelX5JgMmmimOX6xDjotHr7M0Q7I1lOvprNVBkYkxm9gg4KzhjqG30X8iooBtCGXwTCx72zfRRw5YcW1onMVPektGysG0XKTQlLpyCxdLIcnHCjIGWNTMSt8P0bUjznExR9V2yKyeff8pRw4DEWVktgLAqZxt0ja14ozmPVi9HnTPS3IJ2HYfSHgiixepd0W3IZxEahgLLyKaB7dvlbtV0ceNpJjy9FmokwVau3IRYcGHNxGeXgeGOnG5dGP9SRzQJEXKBhPztT51JHjfaY9VzHVNM942sPpOFRuwQcoh8sK7vLbRQtFd2NJLznAy0QmJs9OxzQQsPOOJmtKNbfehd3XG5xAi6ITQdyDZ6j7LWcVltPbykrd2L5GzadaDGckQI3RGPmkN7MsH5M6d12LbwuyeCHlzk1vYmRpcIbLidyuxbOJ5upXsiLJavnrzMskm4ajOAWD1yYOQHzLCGTQf39SEXGDVRgO7vCyMJzusYgbD4sCYWlALWpcFYrZyV1kvVgXqbISqp2X846fd1xBPfj3wcqPAIUA4zUeeBpKqvunrTdZQf7FUcQVwIuU2D6Dw2JXg61ZSQvM7HwFDMM9I6VOHrm1AXOZiruAjZJdYVx2f6cN2Ay2QOl26mI9WZYUO6aKAX43ZtOC5RqbyljLF9wy1q1fPOM7R5leaX4eadtKS2SR2bQ3Y2Wz38wPJd3IMMh7FYItmhghjstIixTWcQNWGWHXldeu0fBJxxgtPqTWTOOS0jEpZ6V9eAY9P1DyV3L3LFhWifJJboBZ8L",
    "AuthRequired": false,
    "RecordMinLinks": 2,
    "RecordMaxValueSize": 1024,
    "RecordMaxTimeDrift": 60
  },
  "P2PPort": 9908,
  "LocalTestMode": false,
  "Identity": "\bLfCyQ09gydhx9ASP7UsqEqWSY8ajvMfOs3XN9vjpBgjiGqjuMZwX2KfjREaXSHkPV"
}

What's next? How do I setup and peer another node correctly? Do I copy some of the genesis files over to other nodes? Or some lf usage? Is the owner command part of the equation? If so, when making a genesis db, where and how is the CA key/cert part of it?

Also, for the private network and cert generation, what key parts of the form should be linked to the network/server?

Can makegenesis be automated? e.g. pass in the form values up front so it's not necessary to re-type during testing, etc...

Thanks and I'm looking forward to testing lf!

Hello! I just found this great peace of software and I’m playing with it. It’s great and it seems to work perfectly fine! I created a private test network and so far so good.

However I’ve a question with duplicated records. I read your documentation and I understand how it works (mostly). But why don’t just discard those records that are not set/updated by the original owner? Won’t this fix the problem at all? Surely there is a reason for that but I don’t see it.

FYI - EdgeFS, scale-out, multi-cloud scalable S3/NFS/iSCSI distributed storage system for Edge/IoT Computing

https://github.com/Nexenta/edgefs

EdgeFS is capable of spanning unlimited number of geographically distributed sites (Geo-site), connected with each other as one global name space data fabric running on top of Kubernetes platform, providing persistent, fault-tolerant and high-performance fully compatible S3 Object API and CSI volumes for stateful Kubernetes Applications.

At each Geo-site, EdgeFS segment nodes deployed as containers (Kubernetes StatefulSet or Docker Compose) on physical or virtual nodes, pooling available storage capacity and presenting it via compatible S3/NFS/iSCSI/etc storage emulated protocols for cloud-native applications running on the same or dedicated servers.

OS: Ubuntu 18.04 x64

$ go version
go version go1.12.6 linux/amd64

$ ./lf owner new username
panic: runtime error: index out of range

goroutine 1 [running]:
main.doOwner(0xc000020540, 0x9376ab, 0xb, 0xc0000200e0, 0x2, 0x2, 0xc000024431)
        /root/tmp/lf/cmd/lf/lf.go:967 +0x58e1
main.main()
        /root/tmp/lf/cmd/lf/lf.go:1818 +0x724