ipfs / ipget Goto Github PK

Retrieve files over IPFS and save them locally.

License: MIT License

Go 29.03% Makefile 6.44% Shell 64.54%

ipget's Introduction

ipget

wget for IPFS: retrieve files over IPFS and save them locally.

ipget is a standalone program analogous to GNU wget. Unlike wget though, ipget specializes in downloading files and directory structures from the IPFS network.

ipget includes its own IPFS node, so you don't need IPFS installed on your system. This makes it ideal for users and projects that want a simple utility for whenever they want to retrieve files from IPFS.

Install

Download a binary for your platform from IPFS Distributions.

Install From Source

$ go install github.com/ipfs/ipget@latest

Example

Find a fun IPFS address and ipget away!

$ ipget QmQ2r6iMNpky5f1m4cnm3Yqw8VSvjuKpTcK1X7dBR1LkJF/cat.gif -o nyan.gif

or with an /ipfs prefix:

$ ipget -o nyan.gif /ipfs/QmQ2r6iMNpky5f1m4cnm3Yqw8VSvjuKpTcK1X7dBR1LkJF/cat.gif

or even IPNS addresses (note this is an IPNS address example and does not work):

$ ipget /ipns/QmQG1kwx91YQsGcsa9Z1p6BPJ3amdiSLLmsmAoEMwbX61b/files/cat.gif

Usage

NAME:
   ipget - Retrieve and save IPFS objects.

USAGE:
   ipget [global options] command [command options] [arguments...]

VERSION:
   see version.json

COMMANDS:
   help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --output value, -o value  specify output location
   --node value, -n value    specify ipfs node strategy ("local", "spawn", "temp" or "fallback") (default: "fallback")
   --peers value, -p value   specify a set of IPFS peers to connect to
   --progress                show a progress bar (default: false)
   --help, -h                show help (default: false)
   --version, -v             print the version (default: false)

Contribute

Feel free to join in. All welcome. Open an issue!

This repository falls under the IPFS Code of Conduct.

License

MIT

ipget's People

Contributors

Stargazers

Watchers

ipget's Issues

Use the last part of the path, if no output path is given

i.e. ipget QmTJHuzG3mjgmvcfvTU4ykWXwD4QjA5aCk6QsU4BPaD8Hh/cat2.gif should save to cat2.gif

`temp` sets up plugins in default path but then it calls tmpNode which creates another ephemeral repo

ipget/node.go

Lines 77 to 89 in 70497d7

 func temp(ctx context.Context) (iface.CoreAPI, error) { 

 defaultPath, err := config.PathRoot() 

 if err != nil { 

 // shouldn't be possible 

 return nil, err 

 } 

 if err := setupPlugins(defaultPath); err != nil { 

 return nil, err 

 } 

 return tmpNode(ctx) 

 }

sets the plugins on the defaultPath, but then calls tempNode that goes on and creates a repo on a new temporary directory

ipget/node.go

Line 92 in 70497d7

dir, err := ioutil.TempDir("", "ipfs-shell")

Doesn't seem right, but I be missing a goism or two :)

Feature: report version and commit

Either in --help or --version, ipget should report the commit it was built from, and its version string.

ipget does not work on Windows 10 ...

I'm running Windows 10. I also have IPFS Desktop installed. After installing ipget, and then invoking ipget, it hangs and after 30 seconds or so, returns complaining:

C:\Users\username\Desktop\ipget.exe --peers /ip4/10.101.25.16/tcp/4001 --progress QmbjXfUs1wpQyp1hEmFCzdSjHtjZvwsK7ABz8oziq8RXy5
2021-03-02T12:17:40.092-0500   ERROR  reprovider.simple simple/reprovide.go:108 failed to reprovide: failed to find any peer in table

ipget --help long description

This should include a description of the program and a short example of its use.

I think that:

the current help output (--help) should be the new default output if no arguments (or wrong arguments are given
if --help or -h are given, the tool description and usage example should be shown (and the usage output should note that --help gives "full help"

maybe

ipget is a tool for non-interactive downloading of files or objects from IPFS to the 
user's filesystem, similar to wget or curl. It supports /ipfs and /ipns paths. ipget
is meant to be used in scripts and pipelines.
(via jbenet)

doesn't build on Windows

Any Windows experts here who feel like helping out, I'd love to get ipget building and working!

Here's a log from trying to build from distributions:

==> building for windows 386
    output to ../../releases/ipget/v0.0.1/windows-386
../gopath/src/github.com/noffle/ipget/Godeps/_workspace/src/github.com/cheggaaa/pb/pb_win.go:6:2: cannot find package "github.com/olekukonko/ts" in any of:
    /usr/lib/go/src/github.com/olekukonko/ts (from $GOROOT)
    /home/stephen/Projects/Forks/distributions/dists/ipget/gopath/src/github.com/olekukonko/ts (from $GOPATH)
    failed.
==> building for windows amd64
    output to ../../releases/ipget/v0.0.1/windows-amd64
../gopath/src/github.com/noffle/ipget/Godeps/_workspace/src/github.com/cheggaaa/pb/pb_win.go:6:2: cannot find package "github.com/olekukonko/ts" in any of:
    /usr/lib/go/src/github.com/olekukonko/ts (from $GOROOT)
    /home/stephen/Projects/Forks/distributions/dists/ipget/gopath/src/github.com/olekukonko/ts (from $GOPATH)
    failed.

Clean shutdown on Ctrl+C (and other OS signals)

As per @cryptix's IRC notes, it looks like https://godoc.org/os/signal#example-Notify provides the mechanism for capturing OS signals. From there, canceling the IPFS node's context should suffice.

SIGINT doesn't interrupt the embeded node

./ipget QmRrPRCRT3KEkQVBoA2f6X2jpbBEV7vgypbYJpVaoag4ia                               
^C^C^C^C^C^C^C^C^C^C^C^C^C

Prefer to bootstrap to nodes on localhost first

This will let the user take advantage of whatever is already cached on the local nodes.

--quiet|-q option

When --quiet is passed, only the resultant filename / path should be reported (as per #21):

$ ipget /ipfs/QmWrUZ42cb34SezHLA4xZkPGGTfdcKpnMuAxEXD9yJAeiw/go-multihash.git
./go-multihash.git

Fall back to public gateway after trying local API

If I run ipget <hash> I'd expect to (in order):

connect to the IPFS node running on my local machine
ask the public gateway via HTTPS

We could also try spawning an ephemeral node, but the gateway is already well-connected and should give a faster response. Lots of switches and flags for controlling this flow can come in after.

Feature idea: --seed

It might be nice to be able to run

$ ipget /ipfs/Qmfoobar --seed

and have ipget hang on completion, keeping the data around and available to the network.

v0.3.0 is not available as a release

There's a tag for v0.3.0, but it's not available on the release pages, so it's harder to get the code (packaging for a distro).

Update to go-ipfs 0.5.1

What needs to be done to update to go-ipfs to 0.5.1?

What does good CLI output look like?

ipget should aim to be transparent: exposing its inner workings as needed to users. (but only as needed)

usage

All good programs dump a block of helpful text when invoked without arguments:

Usage: ipget IPFS_PATH [-o]

ipget - retrieve and save IPFS objects

ipget is a tool for non-interactive downloading of files or objects from IPFS to the 
user's filesystem, similar to wget or curl. It supports /ipfs and /ipns paths. ipget
is meant to be used in scripts and pipelines.

Arguments:
  IPFS_PATH=""   the IPFS object path

Options:
  -o, --output=""   output file path

necessary information

Output MUST always include, at the minimum (--quiet):

whether the download succeeded
the resultant fs path to the downloaded file(s), if success

The next tier of verbosity should include

the size of the downloaded file(s)
the transport (ephemeral node, local daemon, public gateway, bluetooth)

Finally, maximum verbosity (--verbose) could much much more (how many peers we bootstrapped to, the resolved /ipfs path, resolved IP of the public gateway, progress bar of the download). The goal here is to make human troubleshooting easier.

typical use

Here's what an average successful invocation of ipget ought to look like:

$ ipget /ipfs/QmWrUZ42cb34SezHLA4xZkPGGTfdcKpnMuAxEXD9yJAeiw/go-multihash.git
Downloading from IPFS.. ..succeeded
Saved to QmWrUZ42cb34SezHLA4xZkPGGTfdcKpnMuAxEXD9yJAeiw/go-multihash.git [519kb]

A really common UX pain is when the IPFS node can't find the hash. It should timeout after some reasonable time (user configurable), and let the user know it's fallen back to another option:

$ ipget /ipfs/QmWrUZ42cb34SezHLA4xZkPGGTfdcKpnMuAxEXD9yJAeiw/go-multihash.git
Downloading from local daemon.. ..failed
Downloading from IPFS.. ..failed
Downloading from public gateway (ipfs.io).. ..succeeded
Saved to QmWrUZ42cb34SezHLA4xZkPGGTfdcKpnMuAxEXD9yJAeiw/go-multihash.git [519kb]

And of course, sometimes your hash just can't be found:

$ ipget /ipfs/QmWrUZ42cb34SezHLA4xZkPGGTfdcKpnMuAxEXD9yJAeiw/go-multihash.git
Downloading from local daemon.. ..failed
Downloading from IPFS.. ..failed
Downloading from public gateway (ipfs.io).. ..failed
Unable to locate QmWrUZ42cb34SezHLA4xZkPGGTfdcKpnMuAxEXD9yJAeiw/go-multihash.git

verbose / quiet output

It's *really not fun when you can't debug a failure any further than this. Unlike HTTP, there's a lot more that can go wrong. Having a -v/--verbose flag will let savvy users get deeper insight into possible failure causes.

$ ipget -v /ipfs/QmWrUZ42cb34SezHLA4xZkPGGTfdcKpnMuAxEXD9yJAeiw/go-multihash git
Downloading via IPFS..
Bootstrapping           DONE (9/9)
Resolving path          DONE (/ipfs/QmV4XiZEWAL6JDNxfP6PNtGm2MtxviPaGnaGhP9yV5p2ik)
Downloading             TIMEOUT (300kb/519kb)
..failed

Downloading from public gateway (ipfs.io)..
Resolving               DONE (104.236.176.52)
Downloading             NOT FOUND (404)
..failed

Unable to locate QmWrUZ42cb34SezHLA4xZkPGGTfdcKpnMuAxEXD9yJAeiw/go-multihash.git

Inversely, like most ipfs commands, there should be a -q/--quiet flag for terseness:

$ ipget /ipfs/QmWrUZ42cb34SezHLA4xZkPGGTfdcKpnMuAxEXD9yJAeiw/go-multihash.git
QmWrUZ42cb34SezHLA4xZkPGGTfdcKpnMuAxEXD9yJAeiw/go-multihash.git

This is optimal for scripts: they can use POSIX exit codes to determine whether the download succeeded.

cc @jbenet @whyrusleeping @lgierth
cc @substack too, who may have some unixy program design wisdom

--public flag to decide whether to boostrap to public nodes

--public use the public ipfs network for bootstrap (default: true)

Currently this is the default. Add a boolean flag that lets users disable this behaviour.

Build failure due to missing godep

It seems that the vendoring of some of IPFS's dependencies with GoDep broke here:

package github.com/noffle/ipget/Godeps/_workspace/src/github.com/Sirupsen/logrus: cannot find package "github.com/noffle/ipget/Godeps/_workspace/src/github.com/Sirupsen/logrus" in any of:

[...]

        /home/jesse/stuff/src/github.com/noffle/ipget/Godeps/_workspace/src/github.com/Sirupsen/logrus (from $GOPATH)

I tracked the reference down to several source files in:
Godeps/_workspace/src/github.com/ipfs/go-ipfs/vendor/QmXJkcEXB6C9h6Ytb6rrUTFU56Ro62zxgrbxTT3dgjQGA8/go-log/

Support for retrieving directories

Right now ipget bails with this dag node is a directory.

Ensure example hashes are always available

The README refers to a delightful cat gif, QmQ2r6iMNpky5f1m4cnm3Yqw8VSvjuKpTcK1X7dBR1LkJF. It would be nice if we made sure it was pinned on Solar (if it isn't already) so that when people try the example it works.

The other challenge is ensuring the IPNS example continues to work. I'm not sure what options exist for keeping an IPNS entry fresh in Solar indefinitely is -- @lgierth?

Reach feature parity with `ipfs get`

This just means compression -- do we want this? (@jbenet)

-a, --archive           bool   - Output a TAR archive
-C, --compress          bool   - Compress the output with GZIP compression
-l, --compression-level int    - The level of compression (1-9)

Use public gateway for small files

To offset the cost of starting up a new ephemeral node (high), first ping a public gateway (e.g. ipfs.io) to get the size of the requested object. If it's over some size (recursively), start up the node. Otherwise perform a GET over the gateway (and validate the hash).

use `go get` to install

i think should be able to run just

go get github.com/noffle/ipget

to install

Fallback to ephemeral shell on "get" failure, too

Right now ipget tries to connect to a local IPFS node and does a get on the desired hash. If connecting to the local node fails, an ephemeral node is spawned. However, if the local node is connected to successfully but the get fails due to the hash not being found, the code doesn't fall back to creating an ephemeral node.

This is usually fine, since the local node is generally connected to the Solar bootstrap nodes. However, in cases like ipfs running in --offline mode, this won't be true. What is the expected behaviour?

mdns flag

--mdns enable mdns discovery (default: true)

Boolean flag defaulting to true.

GX error, with Go version 1.11

Hi,
I'm the AUR ipget maintainer. When trying to fix the build, I got this error:

gx/ipfs/QmaPHkZLbQQbvcyavn8q1GFHg6o6yeceyHFSJ3Pjf3p3TQ/go-crypto/blake2b.supportAVX2: relocation target runtime.support_avx2 not defined
make: *** [Makefile:38: install] Error 2

This is all related to this issue: holochain/holochain-proto#771
Hope you can fix this soon.
Greetings,
Oskar

normal-verbosity output

As per #21, this should include which methods are used (local daemon, ephemeral node, ...), the resultant filename, and its size:

$ ipget /ipfs/QmWrUZ42cb34SezHLA4xZkPGGTfdcKpnMuAxEXD9yJAeiw/go-multihash.git
Downloading from local daemon.. ..failed
Downloading from IPFS.. ..failed
Downloading from public gateway (ipfs.io).. ..succeeded
Saved to go-multihash.git [519kb]

Update to go-ipfs 0.6

go-ipfs has been on 0.6 for a while now. Time to update the ipget!

1.0.0 Roadmap

roadmap

Full list: Milestone 1.0.0

goals

Primary goals for 1.0.0 release:

it works reliably, and doesn't hang indefinitely when pieces cannot be found
pleasant UX story for the first-time user: good usage docs, output that offers transparency (progress bars, good error messages)
available on all major linux and osx package managers

non-goals

additional features (mdns, custom bootstrap list, optimizations)

Depends on #24
Depends on #26
Depends on #20
Depends on #21

Clean cli termination

Right now sending SIGTERM or SIGINT to ipget results in the process being exited immediately, with no clean up. We should have fallback-ipfs-shell's API methods accept a Context that lets us cancel the shell(s) at the ipget level once a termination signal is encountered.

nicer help?

help should have a description of what this tool is used for, and an example. look at the man pages for wget and curl. so instead of:

Retrieve and save IPFS objects.

maybe:

ipget - retrieve and save IPFS objects

ipget is a tool for non-interactive downloading of files or objects from IPFS to the 
user's filesystem, similar to wget or curl. It supports /ipfs and /ipns paths. ipget
is meant to be used in scripts and pipelines.

or something.

use codegangsta/cli

Most of our other go cli apps use this. Its a great lib and having consistency across the code in our projects is important.

ipget panics when run

> ipget QmT78zSuBmuS4z925WZfrqQ1qHaJ56DQaTfyMUF7F8ff5o
2016/04/05 16:53:45 Reuse of exported var name: utpAckSkippedResends
panic: Reuse of exported var name: utpAckSkippedResends


goroutine 1 [running]:
log.Panicln(0xc8200d3da0, 0x2, 0x2)
    /home/whyrusleeping/go/src/log/log.go:334 +0xc1
expvar.Publish(0xdc58e0, 0x14, 0x7fe83dc4e978, 0xc820075bb0)
    /home/whyrusleeping/go/src/expvar/expvar.go:254 +0x233
expvar.NewInt(0xdc58e0, 0x14, 0xb)
    /home/whyrusleeping/go/src/expvar/expvar.go:272 +0x75
gx/ipfs/QmadkZhaLw1AhXKyBiPmydRuTexhj6WkiZPpo5Uks6WUVq/utp.init()
    /home/whyrusleeping/gopkg/src/gx/ipfs/QmadkZhaLw1AhXKyBiPmydRuTexhj6WkiZPpo5Uks6WUVq/utp/utp.go:52 +0x97
gx/ipfs/QmYVqhVfbK4BKvbW88Lhm26b3ud14sTBvcm1H7uWUx1Fkp/go-multiaddr-net/utp.init()
    /home/whyrusleeping/gopkg/src/gx/ipfs/QmYVqhVfbK4BKvbW88Lhm26b3ud14sTBvcm1H7uWUx1Fkp/go-multiaddr-net/utp/utp_util.go:105 +0x45
gx/ipfs/QmYVqhVfbK4BKvbW88Lhm26b3ud14sTBvcm1H7uWUx1Fkp/go-multiaddr-net.init()
    /home/whyrusleeping/gopkg/src/gx/ipfs/QmYVqhVfbK4BKvbW88Lhm26b3ud14sTBvcm1H7uWUx1Fkp/go-multiaddr-net/net.go:298 +0x56
gx/ipfs/QmSN2ELGRp4T9kjqiSsSNJRUeR9JKXzQEgwe1HH3tdSGbC/go-libp2p/p2p/net/conn.init()
    /home/whyrusleeping/gopkg/src/gx/ipfs/QmSN2ELGRp4T9kjqiSsSNJRUeR9JKXzQEgwe1HH3tdSGbC/go-libp2p/p2p/net/conn/secure_conn.go:124 +0x6f
gx/ipfs/QmSN2ELGRp4T9kjqiSsSNJRUeR9JKXzQEgwe1HH3tdSGbC/go-libp2p/p2p/net.init()
    /home/whyrusleeping/gopkg/src/gx/ipfs/QmSN2ELGRp4T9kjqiSsSNJRUeR9JKXzQEgwe1HH3tdSGbC/go-libp2p/p2p/net/interface.go:155 +0x40
gx/ipfs/QmSN2ELGRp4T9kjqiSsSNJRUeR9JKXzQEgwe1HH3tdSGbC/go-libp2p/p2p/host.init()
    /home/whyrusleeping/gopkg/src/gx/ipfs/QmSN2ELGRp4T9kjqiSsSNJRUeR9JKXzQEgwe1HH3tdSGbC/go-libp2p/p2p/host/host.go:64 +0x4c
github.com/ipfs/go-ipfs/core.init()
    /home/whyrusleeping/gopkg/src/github.com/ipfs/go-ipfs/core/pathresolver.go:91 +0x6a
github.com/noffle/ipfs-embedded-shell.init()
    /home/whyrusleeping/gopkg/src/github.com/noffle/ipfs-embedded-shell/shell.go:57 +0x5d
github.com/noffle/fallback-ipfs-shell.init()
    /home/whyrusleeping/gopkg/src/github.com/noffle/fallback-ipfs-shell/shell.go:22 +0x54
main.init()
    /home/whyrusleeping/gopkg/src/github.com/ipfs/ipget/main.go:57 +0x4f

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /home/whyrusleeping/go/src/runtime/asm_amd64.s:1721 +0x1

goroutine 20 [syscall]:
os/signal.loop()
    /home/whyrusleeping/go/src/os/signal/signal_unix.go:22 +0x18
created by os/signal.init.1
    /home/whyrusleeping/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 21 [select]:
gx/ipfs/Qmazh5oNUVsDZTs2g59rq8aYQqwpss8tcUWQzor5sCCEuH/go-log.(*MirrorWriter).logRoutine(0xc820070400)
    /home/whyrusleeping/gopkg/src/gx/ipfs/Qmazh5oNUVsDZTs2g59rq8aYQqwpss8tcUWQzor5sCCEuH/go-log/writer.go:71 +0x33c
created by gx/ipfs/Qmazh5oNUVsDZTs2g59rq8aYQqwpss8tcUWQzor5sCCEuH/go-log.NewMirrorWriter
    /home/whyrusleeping/gopkg/src/gx/ipfs/Qmazh5oNUVsDZTs2g59rq8aYQqwpss8tcUWQzor5sCCEuH/go-log/writer.go:38 +0xe2

goroutine 22 [chan receive]:
github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/codahale/metrics.init.1.func2()
    /home/whyrusleeping/gopkg/src/github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/codahale/metrics/metrics.go:321 +0x73
created by github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/codahale/metrics.init.1
    /home/whyrusleeping/gopkg/src/github.com/ipfs/go-ipfs/Godeps/_workspace/src/github.com/codahale/metrics/metrics.go:328 +0x7b

Update to go-ipfs 0.9.1

Please update IPFS again.

panic: Reuse of exported var name: utpAckSkippedResends

On Ubuntu 16.04 (64Bit) I installed from source as described in the README.
Whe executing ipget I get a runtime error:

$ ipget
2016/06/14 15:49:16 Reuse of exported var name: utpAckSkippedResends
panic: Reuse of exported var name: utpAckSkippedResends


goroutine 1 [running]:
panic(0xb2cb40, 0xc820077ee0)
    /home/oli/.linuxbrew/Cellar/go/1.6.2/libexec/src/runtime/panic.go:481 +0x3e6
log.Panicln(0xc8200d1da8, 0x2, 0x2)
    /home/oli/.linuxbrew/Cellar/go/1.6.2/libexec/src/log/log.go:334 +0xc1
expvar.Publish(0xea12a0, 0x14, 0x7f1d18330cb0, 0xc820077e88)
    /home/oli/.linuxbrew/Cellar/go/1.6.2/libexec/src/expvar/expvar.go:254 +0x224
expvar.NewInt(0xea12a0, 0x14, 0xc)
    /home/oli/.linuxbrew/Cellar/go/1.6.2/libexec/src/expvar/expvar.go:272 +0x75
gx/ipfs/QmPSR1zYgWRmTSt6nTGXdeeUhWkq9AdvK4sva4Y2H5daPi/utp.init()
    /home/oli/.go/src/gx/ipfs/QmPSR1zYgWRmTSt6nTGXdeeUhWkq9AdvK4sva4Y2H5daPi/utp/expvar.go:6 +0xab
gx/ipfs/Qme5FbqN4BteaBcpbsj51fjzvqJhr54DigbqWAdXS5zgEe/go-libp2p-transport.init()
    /home/oli/.go/src/gx/ipfs/Qme5FbqN4BteaBcpbsj51fjzvqJhr54DigbqWAdXS5zgEe/go-libp2p-transport/utp.go:203 +0x4c
gx/ipfs/QmQkQP7WmeT9FRJDsEzAaGYDparttDiB6mCpVBrq2MuWQS/go-libp2p/p2p/net/conn.init()
    /home/oli/.go/src/gx/ipfs/QmQkQP7WmeT9FRJDsEzAaGYDparttDiB6mCpVBrq2MuWQS/go-libp2p/p2p/net/conn/secure_conn.go:124 +0x97
gx/ipfs/QmQkQP7WmeT9FRJDsEzAaGYDparttDiB6mCpVBrq2MuWQS/go-libp2p/p2p/net.init()
    /home/oli/.go/src/gx/ipfs/QmQkQP7WmeT9FRJDsEzAaGYDparttDiB6mCpVBrq2MuWQS/go-libp2p/p2p/net/interface.go:155 +0x45
gx/ipfs/QmQkQP7WmeT9FRJDsEzAaGYDparttDiB6mCpVBrq2MuWQS/go-libp2p/p2p/host.init()
    /home/oli/.go/src/gx/ipfs/QmQkQP7WmeT9FRJDsEzAaGYDparttDiB6mCpVBrq2MuWQS/go-libp2p/p2p/host/host.go:65 +0x51
github.com/ipfs/go-ipfs/core.init()
    /home/oli/.go/src/github.com/ipfs/go-ipfs/core/pathresolver.go:91 +0x6f
gx/ipfs/QmTrvhAM8734GYEY7JCtAgJrqUWkEvM7va9Qysq2u2o1LA/ipfs-embedded-shell.init()
    /home/oli/.go/src/gx/ipfs/QmTrvhAM8734GYEY7JCtAgJrqUWkEvM7va9Qysq2u2o1LA/ipfs-embedded-shell/shell.go:57 +0x5d
gx/ipfs/QmTgXW4YdmqzesTvAZzSBWawoc1MdBoXLtYM1VY1WkLUxg/fallback-ipfs-shell.init()
    /home/oli/.go/src/gx/ipfs/QmTgXW4YdmqzesTvAZzSBWawoc1MdBoXLtYM1VY1WkLUxg/fallback-ipfs-shell/shell.go:22 +0x59
main.init()
    /home/oli/.go/src/github.com/ipfs/ipget/ipget/main.go:60 +0x4f

404 when trying to build

I get a 404 and subsequent build failure when following the instructions for installing from source.

david@t400:~/go/src/github.com/ipfs/ipget$ make install 
installing gx v0.11.0
==> Using wget to download "https://ipfs.io/ipfs/QmNZL8wNsvAGdVYr8uGeUE9aGfHjFpHegAWywQFEdSaJbp/gx/v0.11.0/gx_v0.11.0_linux-amd64.tar.gz" to "bin/tmp/gx.tar.gz"
--2017-05-09 21:59:57--  https://ipfs.io/ipfs/QmNZL8wNsvAGdVYr8uGeUE9aGfHjFpHegAWywQFEdSaJbp/gx/v0.11.0/gx_v0.11.0_linux-amd64.tar.gz
Resolving ipfs.io (ipfs.io)... 104.236.151.122, 178.62.61.185, 104.236.176.52, ...
Connecting to ipfs.io (ipfs.io)|104.236.151.122|:443... connected.
HTTP request sent, awaiting response... 404 Not Found
2017-05-09 21:59:58 ERROR 404: Not Found.

failed to download https://ipfs.io/ipfs/QmNZL8wNsvAGdVYr8uGeUE9aGfHjFpHegAWywQFEdSaJbp/gx/v0.11.0/gx_v0.11.0_linux-amd64.tar.gz to bin/tmp/gx.tar.gz
Makefile:18: recipe for target 'bin/gx-v0.11.0' failed
make: *** [bin/gx-v0.11.0] Error 1

Go get fails with unrecognized import path

Following https://github.com/ipfs/ipget#install-from-source on a mac:

go get -d github.com/ipfs/ipget
package gx/ipfs/QmQa2wf1sLFKkjHCVEbna8y5qhdMjL8vtTJSAc48vZGTer/go-ipfs/path: unrecognized import path "gx/ipfs/QmQa2wf1sLFKkjHCVEbna8y5qhdMjL8vtTJSAc48vZGTer/go-ipfs/path" (import path does not begin with hostname)
package gx/ipfs/QmVcLF2CgjQb5BWmYFWsDfxDjbzBfcChfdHRedxeL3dV4K/cli: unrecognized import path "gx/ipfs/QmVcLF2CgjQb5BWmYFWsDfxDjbzBfcChfdHRedxeL3dV4K/cli" (import path does not begin with hostname)
package gx/ipfs/Qmdk8Ea9GkbwHr7UNKVYaLRwwHSt69xBXuSvRVyNWZ9sZE/fallback-ipfs-shell: unrecognized import path "gx/ipfs/Qmdk8Ea9GkbwHr7UNKVYaLRwwHSt69xBXuSvRVyNWZ9sZE/fallback-ipfs-shell" (import path does not begin with hostname)

Fetching data quite slow, even with direct connections

Over in filecoin land we are using ipget to download snark parameters. These files are (currently) around 1.5 GB, and take quite a long time (often in the 20 minute range) using the following invocation:

ipget --node=temp --progress -p=/ip4/138.201.67.219/tcp/4002/ws/ipfs/QmUd6zHcbkbcs7SMxwLs48qZVX3vpcM8errYS7xEczwRMA -p=/ip4/138.201.67.218/tcp/4002/ws/ipfs/QmbVWZQhCGrS7DhgLqWbgvdmKN7JueKCREVanfnVpgyq8x -p=/ip4/94.130.135.167/tcp/4002/ws/ipfs/QmUEMvxS2e7iDrereVYc5SWPauXPyNwxcy9BXZrC1QTcHE -p=/ip4/138.201.68.74/tcp/4001/ipfs/QmdnXwLrC8p1ueiq2Qya8joNvk3TVVDAut7PrikmZwubtR -p=/ip4/138.201.67.220/tcp/4001/ipfs/QmNSYxZAiJHeLdkBg38roksAR9So7Y5eojks1yjEcUtZ7i QmNSuxq15JPFCTehxVpgJydNZ79rpLoNwnLzQMGA9EziXg

Realistically, directly connecting to the peers with the data and pulling it from them on my connection should take ~30 seconds, i'm fine with a small amount of overhead, so if it took a minute it would be fine.

These files are going to get much bigger in the nearish future, even up to 100GB, and people will be required to download them. We need this process to be as fast as possible.

Option to connect to an API node

e.g. -a, --api <addr> delegate to node api at given address

Using this flag will forego local ephemeral node creation.

Tests + TravisCI

travis github integration
basic test coverage using sharness
test against local node and fallback ephemeral node separately

How to bootstrap ipget without ipfs?

$ go get github.com/noffle/ipget package gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess: unrecognized import path "gx/ipfs/QmQopLATEYMNg7dVqZRNDfeE2S1yKy8zrRh5xnYiuqeZBn/goprocess" (import path does not begin with hostname) package gx/ipfs/QmT8rehPR3F6bmwL6zjUN8XpiDBFFpMP2myPdC6ApsWfJf/go-base58: unrecognized import path "gx/ipfs/QmT8rehPR3F6bmwL6zjUN8XpiDBFFpMP2myPdC6ApsWfJf/go-base58" (import path does not begin with hostname) package gx/ipfs/QmYf7ng2hG5XBtJA3tN34DQ2GUN5HNksEw1rLDkmr6vGku/go-multihash: unrecognized import path "gx/ipfs/QmYf7ng2hG5XBtJA3tN34DQ2GUN5HNksEw1rLDkmr6vGku/go-multihash" (import path does not begin with hostname) package gx/ipfs/QmZNVWh8LLjAavuQ2JXuFmuYH3C11xo988vSgp7UQrTRj1/go-ipfs-util: unrecognized import path "gx/ipfs/QmZNVWh8LLjAavuQ2JXuFmuYH3C11xo988vSgp7UQrTRj1/go-ipfs-util" (import path does not begin with hostname) package gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context: unrecognized import path "gx/ipfs/QmZy2y8t9zQH2a1b8q2ZSLKp17ATuJoCNxxyMFG5qFExpt/go-net/context" (import path does not begin with hostname) package gx/ipfs/Qmazh5oNUVsDZTs2g59rq8aYQqwpss8tcUWQzor5sCCEuH/go-log: unrecognized import path "gx/ipfs/Qmazh5oNUVsDZTs2g59rq8aYQqwpss8tcUWQzor5sCCEuH/go-log" (import path does not begin with hostname) package gx/ipfs/QmZ4Qi3GaRbjcx28Sme5eMH7RQjGkt8wHxt2a65oLaeFEV/gogo-protobuf/proto: unrecognized import path "gx/ipfs/QmZ4Qi3GaRbjcx28Sme5eMH7RQjGkt8wHxt2a65oLaeFEV/gogo-protobuf/proto" (import path does not begin with hostname)

And so on. Do I need a newer Go version? I'm having issues with the main ipfs client and wanted to try a simpler means of getting into the network.

Request: install.sh be added to ipget linux files.

Request: install.sh be added to the ipget_vX.X.X_linux-amd64.tar.gz file to be consistent with go-ipfs_vX.X.X_linux-amd64.tar.gz and ipfs-update_vX.X.X_linux-amd64.tar.gz files.

Usage as a library

osmosis-labs/osmosis#416

https://github.com/osmosis-labs/osmosis/blob/303ba00e14110e5e82c99841154743758411a89f/cmd/osmosisd/cmd/init.go#L70-L165

I want to use ipget to make our users lives easier-- or IPFS.

Some things:

We should introduce as few dependencies as possible
I don't want to write ipfs configuration to disk
I don't want to think about "ipfs node"
I do want to grab genesis state-- and possibly later the full state of the chain -- from IPFS, to reduce our launch steps to

osmosisd start

from

osmosisd init
wget -O ~/.osmosisd/genesisstate genisis
osmosisd start

Update for go-ipfs 0.4.20

Anyone want to take this @ipfs/go-team?

move to ipfs org?

Since this is an official distribution we should move it to the org (not sure if we discussed this before and reached a different conclusion?)

ipget failed: unexpected EOF

when running ipget on a hash that contains a lot of other files and directories, I get the ipget failed: unexpected EOF error without any other details.

Timeout fetching dependencies

I am unable to build with make install. It seems to time out fetching the dependencies. I am new to IPFS, so it's possible of course that the fault may lie in another component.

bin/gx-v0.11.0 --verbose install --global
installing package: ipget-0.2.5
  - Qmdk8Ea9GkbwHr7UNKVYaLRwwHSt69xBXuSvRVyNWZ9sZE not found locally, fetching into /home/iain/programming/gopath/src/gx/ipfs/Qmdk8Ea9GkbwHr7UNKVYaLRwwHSt69xBXuSvRVyNWZ9sZE
  - fetching Qmdk8Ea9GkbwHr7UNKVYaLRwwHSt69xBXuSvRVyNWZ9sZE via ipfs api
  - fetch finished in 1.063073511s
  - fetch Qmdk8Ea9GkbwHr7UNKVYaLRwwHSt69xBXuSvRVyNWZ9sZE complete!
[1 / 1] fetched dep: fallback-ipfs-shell
successfully found all deps for ipget
  - ipget depends on fallback-ipfs-shell (Qmdk8Ea9GkbwHr7UNKVYaLRwwHSt69xBXuSvRVyNWZ9sZE)
installing package: fallback-ipfs-shell-3.2.4
  - QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG not found locally, fetching into /home/iain/programming/gopath/src/gx/ipfs/QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG
  - QmQ1TuGrcSXjsGXEhdWPJTyfkrgMHCu2pcENRxgHQQ59wL not found locally, fetching into /home/iain/programming/gopath/src/gx/ipfs/QmQ1TuGrcSXjsGXEhdWPJTyfkrgMHCu2pcENRxgHQQ59wL
  - fetching QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG via ipfs api
  - fetching QmQ1TuGrcSXjsGXEhdWPJTyfkrgMHCu2pcENRxgHQQ59wL via ipfs api
  - QmdidTt1eZtEAn1Qv1rF9kHPusm57RkfNnsZsrQndsm8TT not found locally, fetching into /home/iain/programming/gopath/src/gx/ipfs/QmdidTt1eZtEAn1Qv1rF9kHPusm57RkfNnsZsrQndsm8TT
  - fetching QmdidTt1eZtEAn1Qv1rF9kHPusm57RkfNnsZsrQndsm8TT via ipfs api
  - fetch finished in 435.753791ms
  - fetch QmSWLfmj5frN9xVLMMN846dMDriy5wN5jeghUm7aTW3DAG complete!
[1 / 3] fetched dep: go-multiaddr
  - fetch finished in 521.122854ms
  - fetch QmdidTt1eZtEAn1Qv1rF9kHPusm57RkfNnsZsrQndsm8TT complete!
[2 / 3] fetched dep: go-ipfs-api
ERROR: from shell.Get(): Post http://v04x.ipfs.io/api/v0/get?arg=QmQ1TuGrcSXjsGXEhdWPJTyfkrgMHCu2pcENRxgHQQ59wL&encoding=json&stream-channels=true: net/http: request canceled (Client.Timeout exceeded while awaiting headers)
retrying fetch QmQ1TuGrcSXjsGXEhdWPJTyfkrgMHCu2pcENRxgHQQ59wL after a second...

manual

wget and curl do a great job in most regards. To be invoked on man ipget or ipget --manual:

a great man page that gives background on IPFS, what multiaddresses are, includes full step-by-step examples, and any other context a brand new user would need to understand the tool (curl --manual does a great job; talks about what URLs are, etc)
should offer hooks throughout that invite the user to learn more about IPFS; act as a gateway into the IPFS community / suite of tools

Accept a custom bootstrap list

--bootstrap <file> use custom bootstrap list

With a format matching that of ipfs bootstrap.

Postfix -o option is ignored

ipget uses urfave/cli, which ignores post-argument options. This is an issue with the core Go flag package. They also seem interested in addressing this, so let's see if they get to fixing it soon. If not, we can either make the PR ourselves or look at other CLI packages.

Separate into lib and CLI

Break this repo apart into ipget-cli and ipget-lib. Projects like gx can take advantage of ipget's functionality, while users can continue to benefit via the CLI app.

	func temp(ctx context.Context) (iface.CoreAPI, error) {
	defaultPath, err := config.PathRoot()
	if err != nil {
	// shouldn't be possible
	return nil, err
	}

	if err := setupPlugins(defaultPath); err != nil {
	return nil, err
	}

	return tmpNode(ctx)
	}