@whyrusleeping here's a rough spec of ipns / the namespace:

• Make an NSResolver interface.

type NSResolver interface {
  Resolve(name string) (string, error)
}

• Implement DNS, proquint NSResolvers (which @Bren2010 started).
• Implement a RoutingResolver (the meat of IPNS) which does:

// RoutingName is the de-serialized name structure that is stored (serialized) 
// in the routing system. Basically, a hash + a digital signature. (serialization can be
// protobuf, or a simple binary format)
type RoutingName {
  Hash        u.Key
  Signature []byte
}

// RoutingResolver implements NSResolver for the main IPFS SFS-like naming
type RoutingResolver {
  routing routing.IpfsRouting
  dag      mdag.DAGService
}

func (r *RoutingResolver) Resolve (name string) (string, error) {

  // name should be a multihash. if it isn't, error out here.

  // use the routing system to get the name.
  // /ipns/<name>
  key := u.Key(name) // or whatever
  val, err := r.routing.Get(key)
  if err != nil {
    return "", err
  }

  // name should be a public key retrievable from ipfs
  // /ipfs/<name>
  node, err :=  r.dag.Get(key)
  if err != nil {
    return "", err
  }

  // get PublicKey from node.Data
  pk := PublicKey(node.Data)

  // check sig with pk
  if !pk.Verify(val.Hash, val.Signature) {
    return "", errors.New("Invalid value. Not signed by PrivateKey corresponding to %v", pk)
  }

  // ok sig checks out. this is a valid name.
  return string(val.Hash)
}

• implement NameService or NameSystem object:

// NameSystem is the object implementing IPNS. It adheres to the `NSResolver` 
// interface, and uses child `NSResolvers`, depending on what the name looks like.
// 
// hashes -> routing, domains -> dns, proquints -> proquint
// we can add `.bit` and `.onion` resolution here in the future
// 
// in non-routing cases, the values returned are hashes to lookup in the routing system.
// (most of the time, if not ALL of the time. still deciding on this). meaning that 
// a dns record: `DNS TXT ipns=<some-hash>` needs to then resolve `<some-hash>`
// via the routing system.
type NameSystem struct {
  routing   RoutingResolver
  dns        DNSResovler
  proquint ProquintResolver
}

• add mounting of /ipns to ipfs mount cmd (maybe have options for each i.e. -n, --ipns and -f, --ipfs to toggle each on/off)

My goal with the diagnostic messages is to provide a means to gather data about the network, number of nodes, and a map of the connections between them. Having this will allow us to analyze performance bottlenecks and other interesting issues that may arise as we move towards a larger ipfs deployment.

Peers behind nats need a way for other peers in the network to connect to them. It would be nice to also have webrtc, but until a go library for that exists, this will have to do.

running a command like ipfs ls or ipfs refs locally feels a ~300ms slower than we could get it. is not doing much beyond reading a few files and printing out stuff, leveldb is pretty fast, so not sure what's up.

=== RUN TestProvides
Malformed public key.
--- FAIL: TestProvides (0.19 seconds)
    dht_test.go:201: Negotiation failed.

commit: f03afaf
test log: https://s3.amazonaws.com/archive.travis-ci.org/jobs/35251330/log.txt
CI: https://travis-ci.org/jbenet/go-ipfs/jobs/35251330#L208

currently go get github.com/jbenet/go-ipfs/cmd/ipfs doesnt work because of breaking changes in gogoprotobuf, so maybe put deps in a vendor-directory?

https://botbot.me/freenode/ipfs/2014-08-15/?msg=19828336&page=1

I would like to update all the logging in ipfs (util.DOut, and friends) to instead use go-logging. This will help us out immensely once we start doing larger deloys and things start failing.

This project has a lot of graph-based data structures involved and the word Node is used to refer to completely different concepts in the codebase. To avoid confusion, should we rename the MerkelDAG nodes to Objects and stick to this notation?

Let me know and I will make the refactoring PR.

can't load package: package .: no buildable Go source files in /home/bussiere/go-ipfs

crazy project but go :)

Solved have to go in cli and build everything and get all the depencies.
regards

dht.go links to https://github.com/jbenet/node-ipfs/blob/master/submodules/ipfs-dht/index.js
which I'm currently porting from JS to Go but there are some quirks on the JS code.

May I ask you what is supposed to be a DHTStream? Currently is not implemented and the closest thing I could find is swarm.Conn.

Also there's issues with peer.ID being a multihash.Multihash which it's underlying type is []byte which can't be used as map keys and you are trying to use it that way, I already informed about this on #8.

You should focus on just one implementation until is finished so other people can port to other languages. Preferably the Go one because it's a static one.

Presently, for a key |k|, concurrent calls to bitswap.GetBlock(k) result in multiple requests (get providers then fan out and get blocks).

Need to ensure only one active request per key.

In-progress.

2<<32 overflows int on a 32 bit system (arm processor best processor!!)

As noted in: https://github.com/jbenet/go-ipfs/pull/38/files#r17404647

I wish to propose a pattern for logging the non-fatal errors that occur in asynchronous operations.

Context exposes an error logging method:

func (ctx *u.IPFSContext) LogError(err error) {
// may be implemented synchronously
}

The interface extends the golang Context:

type IPFSContext interface {
  context.Context
  LogError(err error)
}

Now async actors can get information back out to parent without modifying control flow or confusing the caller/callee contract!

func asyncWork(ctx u.IPFSContext, in <-chan rawData) {
  for raw := <- in {
    val, err := transform(raw)
    if err != nil {
      ctx.LogError(err)
      continue
    }
    doThingsWith(val)
  }
}

the context creator can listen for errors:

func (ctx *context.IPFSContext) Errs() <-chan error {
  // return a channel of errors
}

The pattern is analogous to the way children listen on Done()

  ctx := context.New(context.Background())
  go asyncWork(ctx, ch)
  for {
    select {
      case err <-ctx.Errs():
        u.DOut(err)
    }
  }

Thoughts? @jbenet @whyrusleeping

dht keeps Peer's latency, but right now it's only the last value. this should probably be an EWMA.

Need dev.md explaining current state of src

One thing I found missing from the current Go source is I didn't find a document explaining the current state of the repository. So to figure out what is working you have to go searching.

I'll get on this asap :)

Having UDP as a protocol for swarm to use will make ipfs MUCH more flexible.

Some considerations will need to be taken for packet loss though, in some cases lost packets are not okay. Implementing a basic packet-wise ACK system might be useful.

Continuing discussion from #69 (comment)

@perfmode good point, that doesn't make much sense. See https://github.com/jbenet/go-ipfs/blob/net/net/service/service.go#L183-L188 Think it should just return a message, and handle/log/whatever its own error?

Yeah.

If something goes wrong when handling a message, trttd might be to propagate the error upward. ie. an error channel or an error recorder (à la vitess) used by the service (DHT, BitSwap)

Hi - as already mentioned in #8 (comment) it appears that the fuse project will not compile on Windows. I get the following error when attempting go get ./...:

  # bazil.org/fuse
  ..\..\..\..\..\bazil.org\fuse\fuse.go:985: undefined: attr
  ..\..\..\..\..\bazil.org\fuse\fuse_kernel.go:314: undefined: attr

This comment at the fuse project from last year seems to indicate that windows is not supported: bazil/fuse#5 (comment)

Windows x64, via cygwin if that is relevant.

• Check OSXFUSE is installed
• Make sure to have OSXFUSE >= 2.7 (2.3.4 kernel panics...)
• Check mountpoint exists
• "Mount server running -- ^C to exit."
• when mounted leveldb is locked.
• ls on root should error (unlistable)

Some datastores are not thread-safe. Eg. Datastore implementations backed by the Go map.

Proposal: Introduce datastore.go's new ThreadsafeDatastore interface.

Original: #85 (comment)

BitSwap is not currently implemented in the go-ipfs repo. This issue can serve as a contact point for work and discussion around it.

in-progress

? github.com/jbenet/go-ipfs/routing [no test files]
--- FAIL: TestLayeredGet (0.16 seconds)
dht_test.go:324: Error: Not Found.
FAIL

at 676d9c5

=== RUN TestLayeredGet
Failed to listen on: &{������} [listen tcp 127.0.0.1:5000: bind: address already in use]--- FAIL: TestLayeredGet (0.25 seconds)
    dht_test.go:51: 0: listen tcp 127.0.0.1:5000: bind: address already in use

=== RUN TestFindPeer
Failed to listen on: &{������} [listen tcp 127.0.0.1:5000: bind: address already in use]--- FAIL: TestFindPeer (0.10 seconds)
    dht_test.go:51: 0: listen tcp 127.0.0.1:5000: bind: address already in use

commit: f03afaf
test log: https://s3.amazonaws.com/archive.travis-ci.org/jobs/35251330/log.txt
ci: https://travis-ci.org/jbenet/go-ipfs/jobs/35251330#L208

https://gist.github.com/perfmode/3498c372c5ffde915406

Add a bitswap daemon to periodically sweep the wantlist and send requests for the keys contained therein.

We are using pointers to multiaddrs in quite a few places, its pretty redundant to do so since they are just byte arrays, i feel that fixing this will clean up a good deal of code.

If there are multiple open PRs and one is merged, the remaining PRs fall behind master and create hard-to-bisect, overlapping merges.

I'm happy to perform merges locally so they apply cleanly, but I am sensitive to the extra work that a "merge-locally" policy creates. Thoughts?

https://gist.github.com/jbenet/ee6c9ac48068889b0912

ipfs command should check for updates from github and warn the user. This should be able to be changed in the config. Maybe:

• update.check ignore Do not check.
• update.check warn (default) Warn the user on every run of the ipfs command.
• update.check error Warn the user on every run of the ipfs command, and exit.

(So, changed with ipfs config update.check ignore. )

It is my opinion that we dont have enough tests. This issue will serve as a discussion for creating new tests and tracking code coverage over time.

the codebase had this comment:

    // pretend to encrypt key, then store it unencrypted
    skbytes, err := sk.Bytes()
    if err != nil {
        return err
    }
    cfg.Identity.PrivKey = base64.StdEncoding.EncodeToString(skbytes)

Right now we're cutting a lot of corners and doing things not securely yet. But it's important to flag them as such for people who might be just browsing the codebase. If I saw that on a random codebase i'd be alarmed, even knowing the code is changing.

bug located at: https://github.com/jbenet/go-ipfs/blob/7968b45e58e6dfee3e732276a6cb7eab889a4333/bitswap/bitswap.go#L182

see: https://code.google.com/p/go-wiki/wiki/CommonMistakes#Using_Closures_with_Goroutines

Some Gophers might be turned off by the code not following golint rules. Most Go projects largely adhere to golint's rules. Maybe try running golint on the code?

The most common thing I saw was docstrings not starting with the name of the definition they are documenting. E.g., use:

// Foo is a...
func Foo() {}

instead of

// fetch the baz...
func Foo() {}

the config flag should work globally, but it seems it doesn't:

% ipfs init -c="."
flag provided but not defined: -c
Usage: ipfs init

Initialize ipfs local configuration


Options:
  -b=4096: number of bits for keypair
  -f=false: force overwrite of existing config
  -p="": passphrase for encrypting keys

(on net / 128b6bb)

Hey, Just creating a discussion space for figuring out how the filesystem interface for ipns is going to feel!

The Golang toolchain allows you to generate nifty html test coverage reports. Proposal:

1. Push to go-ipfs origin
2. CI server runs tests
3. CI server generates html coverage reports
4. CI server pushes the html coverage reports to an IPFS node. Node publishes the files.
5. Developer http://www.ipfs-dev.io/coverage/
6. On a VPS, HTTP server proxies request to local IPFS node. Node queries swarm and returns results.
7. Coverage reports are displayed in the browser.

The benefits are three-fold:

1. improve test coverage -> improve software quality
2. Dogfood an interesting use-case for IPFS as a web datastore. -> improve toolchain, api
3. Be able to detect problems in the network early and often (on every commit!) -> improve network reliability

Our current file chunking scheme is just splitting on blocks of a certain size. This is bad for a variety of reasons. @jbenet and I think it will be best to use rabin fingerprinting as our default method for file chunking. The downside is that, other than a brief description on wikipedia, there isnt much to go on for the algorithm itself, so its gonna be a bit of fun figuring this out.

Things like this are weird for code reviewing.

// Copied from core/core.go

import (
    "fmt"
    ds "github.com/jbenet/datastore.go"
    blocks "github.com/jbenet/go-ipfs/blocks"
    config "github.com/jbenet/go-ipfs/config"
    merkledag "github.com/jbenet/go-ipfs/merkledag"
    path "github.com/jbenet/go-ipfs/path"
    peer "github.com/jbenet/go-ipfs/peer"
)

Most of those named imports are useless (same name as package name) and unsorted.

gofmt will deal with import sorting, separating std from local package and other (non-std) packages would be the best way to go (it doesn't violate any code conventions and it's better for readers/reviewers).

import (
    "fmt"

    "github.com/jbenet/go-ipfs/blocks"
    "github.com/jbenet/go-ipfs/config"
    "github.com/jbenet/go-ipfs/merkledag"
    "github.com/jbenet/go-ipfs/path"
    "github.com/jbenet/go-ipfs/peer"

    ds "github.com/jbenet/datastore.go"
)

I browsed the source code and all the local storage stubs refer to the datastore library github.com/jbenet/datastore.go. The README and the references describe it as an abstraction layer that would allow to bind it to any database format there is.

And then there is a link to level_db as one of the 'datastore' types in config/config.go:db.

I am really not sure if that abstraction layer is needed at this stage. Specific implementation of the local storage is not that important for minimum viable version. More than that, git, that other very successful DAG-on-top-of-data blobs design, implements and stores objects in a very opinionated way and seems to be successful in spite of it.

Is using that datastore library an intentional design decision? Can we just use something like github.com/syndtr/goleveldb at this stage?

Loving the project and the design, keep up the amazing work!

I want to make sure all code pushed is run through go fmt. We probably want:

• a git pre commit hook
• a github integration that checks PRs (go fmt, go test)

This post (http://peter.bourgon.org/go-in-production/) also recommends:

on Save: go fmt (or goimports)
on Build: go vet, golint, and maybe go test
on Deploy: go test -tags=integration

Hi,

I've created a package for the Arch User Repository. If you'd like, I can fork and write some instructions on the README. It is available here: https://aur.archlinux.org/packages/go-ipfs-git/

The package git clones from this exact repo and go gets the dependencies as per the README file. I hope this helps users try out IPFS more easily.

wget https://aur.archlinux.org/packages/go/go-ipfs-git/go-ipfs-git.tar.gz
tar -xvf go-ipfs-git.tar.gz
cd go-ipfs-git
makepkg -i