anacrolix / torrent Goto Github PK

Full-featured BitTorrent client package and utilities

License: Mozilla Public License 2.0

Go 99.74% Shell 0.12% Dockerfile 0.11% Just 0.02%

torrent go magnet-link bittorrent p2p streaming tracker

torrent's Introduction

torrent

This repository implements BitTorrent-related packages and command-line utilities in Go. The emphasis is on use as a library from other projects. It's been used 24/7 in production by downstream services since late 2014. The implementation was specifically created to explore Go's concurrency capabilities, and to include the ability to stream data directly from the BitTorrent network. To this end it supports seeking, readaheads and other features exposing torrents and their files with the various Go idiomatic io package interfaces. This is also demonstrated through torrentfs.

There is support for protocol encryption, DHT, PEX, uTP, and various extensions. There are several data storage backends provided: blob, file, bolt, mmap, and sqlite, to name a few. You can write your own to store data for example on S3, or in a database.

Some noteworthy package dependencies that can be used for other purposes include:

Installation

Install the library package with go get github.com/anacrolix/torrent, or the provided cmds with go install github.com/anacrolix/torrent/cmd/...@latest.

Library examples

There are some small examples in the package documentation.

Mentions

@anacrolix is interviewed about this repo in Console 32.

Downstream projects

There are several web-frontends, sites, Android clients, storage backends and supporting services among the known public projects:

cove: Personal torrent browser with streaming, DHT search, video transcoding and casting.
confluence: torrent client as a HTTP service
Gopeed: Gopeed (full name Go Speed), a high-speed downloader developed by Golang + Flutter, supports (HTTP, BitTorrent, Magnet) protocol, and supports all platforms.
Erigon: an implementation of Ethereum (execution layer with embeddable consensus layer), on the efficiency frontier.
exatorrent: Elegant self-hostable torrent client
bitmagnet: A self-hosted BitTorrent indexer, DHT crawler, content classifier and torrent search engine with web UI, GraphQL API and Servarr stack integration.
TorrServer: Torrent streaming server over http
distribyted: Distribyted is an alternative torrent client. It can expose torrent files as a standard FUSE, webDAV or HTTP endpoint and download them on demand, allowing random reads using a fixed amount of disk space.
Mangayomi: Cross-platform app that allows users to read manga and stream anime from a variety of sources including BitTorrent.
Simple Torrent: self-hosted HTTP remote torrent client
autobrr: autobrr redefines download automation for torrents and Usenet, drawing inspiration from tools like trackarr, autodl-irssi, and flexget.
mabel: Fancy BitTorrent client for the terminal
webtor.io: free cloud BitTorrent-client
Android Torrent Client: Android torrent client
libtorrent: gomobile wrapper
Go-PeersToHTTP: Simple torrent proxy to http stream controlled over REST-like api
CortexFoundation/torrentfs: Independent HTTP service for file seeding and P2P file system of cortex full node
Torrent WebDAV Client: Automatic torrent download, streaming, WebDAV server and client.
goTorrent: torrenting server with a React web frontend
Go Peerflix: Start watching the movie while your torrent is still downloading!
hTorrent: HTTP to BitTorrent gateway with seeking support.
Remote-Torrent: Download Remotely and Retrieve Files Over HTTP
Trickl: torrent client for android
ANT-Downloader: ANT Downloader is a BitTorrent Client developed by golang, angular 7, and electron
Elementum (up to version 0.0.71)

Help

Communication about the project is primarily through Discussions and the issue tracker.

Command packages

Here I'll describe what some of the packages in ./cmd do. See installation to make them available.

`torrent`

`torrent download`

Downloads torrents from the command-line.

$ torrent download 'magnet:?xt=urn:btih:KRWPCX3SJUM4IMM4YF5RPHL6ANPYTQPU'
... lots of jibber jabber ...
downloading "ubuntu-14.04.2-desktop-amd64.iso": 1.0 GB/1.0 GB, 1989/1992 pieces completed (1 partial)
2015/04/01 02:08:20 main.go:137: downloaded ALL the torrents
$ md5sum ubuntu-14.04.2-desktop-amd64.iso
1b305d585b1918f297164add46784116  ubuntu-14.04.2-desktop-amd64.iso
$ echo such amaze
wow

`torrent metainfo magnet`

Creates a magnet link from a torrent file. Note the extracted trackers, display name, and info hash.

$ torrent metainfo testdata/debian-10.8.0-amd64-netinst.iso.torrent magnet
magnet:?xt=urn:btih:4090c3c2a394a49974dfbbf2ce7ad0db3cdeddd7&dn=debian-10.8.0-amd64-netinst.iso&tr=http%3A%2F%2Fbttracker.debian.org%3A6969%2Fannounce

See torrent metainfo --help for other metainfo related commands.

`torrentfs`

torrentfs mounts a FUSE filesystem at -mountDir. The contents are the torrents described by the torrent files and magnet links at -metainfoDir. Data for read requests is fetched only as required from the torrent network, and stored at -downloadDir.

$ mkdir mnt torrents
$ torrentfs -mountDir=mnt -metainfoDir=torrents &
$ cd torrents
$ wget http://releases.ubuntu.com/14.04.2/ubuntu-14.04.2-desktop-amd64.iso.torrent
$ cd ..
$ ls mnt
ubuntu-14.04.2-desktop-amd64.iso
$ pv mnt/ubuntu-14.04.2-desktop-amd64.iso | md5sum
996MB 0:04:40 [3.55MB/s] [========================================>] 100%
1b305d585b1918f297164add46784116  -

torrent's People

Contributors

Stargazers

Watchers

Forkers

dborzov godeep neuroradiology prodigeni sarvex grimmkull pdivitantonio zeratel ankibahuguna techend gitter-badger zofuthan milosgajdos sashabaranov achillesa insionng kerasking mewbak gbjk scollison lclarkmichalek oldfaust therewillbecode ngaut vignestion kupsef soul9 ssi379 zlzlovezl pombredanne jpillora robaman ponychicken alash3al shoenig lampgit alidzapp mmshamada digideskio vertis srsman klaudit bigflood jakop345 zeronetscript samvicky26 soulsu mozii a0i0a gvsurenderreddy zhoupan hustcat ct1n vbatts tomzhang dinkin nowucme duyanghao billyprice1old maniacs-satm vaginessa vishal44ful seabyte7 worldboss notriddle ravindra050 mrswiss vk2298 bravechou2009 kustomzone piyush9798 arranlomas ii0 mengzhuo binstreamtech awesome-archive thomas-maurice lamproae teran whalehulk gyga8k self20 evanson wlmbz kanulau shamanoid thelinker duckee25 alexvanc tejasmanohar barsanuphe kokizzu wxfpdc univrs firetamer verdureorange thegamer123 aa6my pavrorov cathalgarvey

torrent's Issues

Implement SkipRegion(off, len int64)

Hey matt, based off this, I was about to send a PR to add a:

func (t Torrent) SetRegionPriorityTo(off, len int64, priority piecePriority)

To allow certain regions to be skipped. Though I noticed you've got a noprio branch going. Does torrent not support changes of priorities on running torrents?

Along with:

func (t Torrent) DownloadRegion(off, len int64)

Is it possible to implement:

func (t Torrent) SkipRegion(off, len int64)

Remove C dependencies

Hey matt,

Cross compiling fails due to C dependencies. Currently attempting to compile github.com/anacrolix/torrent/cmd/torrent on http://gox.jpillora.com/ yields:

../../../../spacemonkeygo/monotime/mono_fallback.go:21:2: no buildable Go source files in /app/.heroku/go/src/github.com/spacemonkeygo/monotime/_cgo

If it's not practical to remove the C dependencies, feel free to close this.

This is for https://github.com/jpillora/cloud-torrent where I'd like to have auto-releases working for all platforms (currently only native - linux amd64 - works)

panic: listeners exist on different addresses

My code:

package main

import (
    "fmt"
    "github.com/anacrolix/torrent"
)

func main() {

    c, e := torrent.NewClient(&torrent.Config{})
    defer c.Close()
    if e != nil {
        fmt.Printf("torrent.NewClient errors:%s", e)
    }

    t, e := c.AddMagnet("magnet:?xt=urn:btih:KRWPCX3SJUM4IMM4YF5RPHL6ANPYTQPU")
    if e != nil {
        fmt.Printf("AddMagnet errors:%s", e)
    }
    fmt.Println(t)
    fmt.Println(t.GotInfo)
    fmt.Println(t.DisplayName)
    t.DownloadAll()
    c.WaitAll()
    fmt.Println("ermahgerd, torrent downloaded")
}

i got this errors:

panic: listeners exist on different addresses

goroutine 1 [running]:
github.com/anacrolix/torrent.(*Client).ListenAddr(0xc08202e000, 0x1421f0, 0xc082
066ba0)
E:/gopath/src/github.com/anacrolix/torrent/client.go:164 +0x220
github.com/anacrolix/torrent.func路003(0x0, 0x0)
E:/gopath/src/github.com/anacrolix/torrent/client.go:486 +0x4c
github.com/anacrolix/torrent.NewClient(0xc08202a100, 0xc08202e000, 0x0, 0x0)
E:/gopath/src/github.com/anacrolix/torrent/client.go:519 +0xc68
main.main()
E:/magnetor/tor.go:10 +0x5b

goroutine 5 [runnable]:
github.com/anacrolix/torrent.(*Client).acceptConnections(0xc08202e000, 0x142220,
0xc082042038, 0x0)
E:/gopath/src/github.com/anacrolix/torrent/client.go:589
created by github.com/anacrolix/torrent.NewClient
E:/gopath/src/github.com/anacrolix/torrent/client.go:503 +0x984

goroutine 6 [runnable]:
github.com/anacrolix/utp.(*Socket).reader(0xc082074300)
E:/gopath/src/github.com/anacrolix/utp/utp.go:416
created by github.com/anacrolix/utp.NewSocket
E:/gopath/src/github.com/anacrolix/utp/utp.go:407 +0x2e6

goroutine 7 [runnable]:
github.com/anacrolix/utp.(*Socket).dispatcher(0xc082074300)
E:/gopath/src/github.com/anacrolix/utp/utp.go:470
created by github.com/anacrolix/utp.NewSocket
E:/gopath/src/github.com/anacrolix/utp/utp.go:408 +0x303

goroutine 8 [runnable]:
github.com/anacrolix/torrent.(*Client).acceptConnections(0xc08202e000, 0x143398,
0xc082074300, 0x1)
E:/gopath/src/github.com/anacrolix/torrent/client.go:589
created by github.com/anacrolix/torrent.NewClient
E:/gopath/src/github.com/anacrolix/torrent/client.go:511 +0xbf2

How do I get list of peers on a given swarm?

I'm trying to create ui for toy torrent client and so far everything is good, except I cannot locate where peers live.

*metainfo.Info to torrent.InfoHash

How do I get from a https://godoc.org/github.com/anacrolix/torrent/metainfo#Info to a https://godoc.org/github.com/anacrolix/torrent#InfoHash?

Here's what I came up with though its not matching...

package main

import (
    "crypto/sha1"
    "log"
    "os"

    "github.com/anacrolix/torrent/bencode"
    "github.com/anacrolix/torrent/metainfo"
)

func main() {
    f, err := os.Open("myfile.torrent")
    if err != nil {
        log.Fatal(err)
    }

    info, err := metainfo.Load(f)
    if err != nil {
        log.Fatal(err)
    }
    f.Close()
    i := info.Info.Info
    b, err := bencode.Marshal(&i)
    if err != nil {
        log.Fatal(err)
    }

    ifx := metainfo.InfoEx{}
    err = ifx.UnmarshalBencode(b)
    if err != nil {
        log.Fatal(err)
    }
    result := ifx.Hash

    // h := sha1.New()
    // h.Write(b)
    // result := h.Sum(nil)

    if bytes.Compare(info.Info.Hash, result) != 0 {
        log.Printf("infohash mismatch\n%x\n%x", info.Info.Hash, result)
    }
}

How to bootstrap seeding after creating the torrent file?

I have run two track servers(using opentrack, the same as the piratebay), and also know how to download files from the DHT network, but I want to distribute my own files(created locally) for private sharing, how to add (downloaded) files list to client or torrent to indicate downloading is finished and should only seeding(do the upload part)?
I can see the client will not re-download the files in current directory after the files has been download from the DHT before using torrent files or magnet link.

fatal error: invalid stack pointer in dht.go

I have been running into another panic. It seems to be intermittent and might only be happening on one file. The full stack trace was over 3000 lines of code, but I think the relevant bit is this:

runtime: bad pointer in frame github.com/anacrolix/torrent/dht.(*Server).processPacket at 0xc82384be38: 0x10
fatal error: invalid stack pointer

trace

goroutine 10 [copystack]:
runtime.assertE2T2(0x7f93e0, 0x7f93e0, 0xc8207071b0, 0xc8239cde48, 0xc823952bf8)
    /usr/local/go/src/runtime/iface.go:233 fp=0xc82384bda0 sp=0xc82384bd98
github.com/anacrolix/torrent/dht.(*Server).processPacket(0xc8200f20f0, 0xc820104000, 0x45, 0x10000, 0x7f3af3c1e158, 0xc823a66f80)
    /home/craig/go/src/github.com/anacrolix/torrent/dht/dht.go:639 +0x278 fp=0xc82384be88 sp=0xc82384bda0
github.com/anacrolix/torrent/dht.(*Server).serve(0xc8200f20f0, 0x0, 0x0)
    /home/craig/go/src/github.com/anacrolix/torrent/dht/dht.go:675 +0x312 fp=0xc82384bf60 sp=0xc82384be88
github.com/anacrolix/torrent/dht.NewServer.func1(0xc8200240d0)
    /home/craig/go/src/github.com/anacrolix/torrent/dht/dht.go:148 +0x28 fp=0xc82384bf98 sp=0xc82384bf60
runtime.goexit()
    /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1 fp=0xc82384bfa0 sp=0xc82384bf98
created by github.com/anacrolix/torrent/dht.NewServer
    /home/craig/go/src/github.com/anacrolix/torrent/dht/dht.go:157 +0x283

It could totally be me doing something dumb on my end again, but I'm not sure. I'd be happy to provide additional info if it will help.

New Feature:Download the specified file

Hello, How do I download the specified file

Minimising disk usage via TorrentDataOpener

Hey Matt, I'm wanting to stream individual torrent files either a cloud storage or directly to the user. This will result in varying degrees of outgoing bandwidth. The restriction is I want to use as little disk as possible. For example, say I have a 1.5GB ubuntu distribution that I want to stream to a user. Could this be done with say 50MB of disk?

I know I can grab a Reader from the torrent and just stream out the file, though wouldn't this consume the disk as the download progresses? Can I delete the section that's already been streamed? I imagine trimming off the front of a file would produce invalid offsets...
My next idea to save disk was to implement my own TorrentDataOpener to store the torrent's "working files" in the cloud. I'm even sure if this is possible because it writing a single piece, like 32KB, in the middle of a file might be difficult...
As I was writing this, I came up with a 3rd way. Using pure memory implementation of TorrentDataOpener which I can tightly control. For example, keep up-to 100MB of the torrent (how ever many pieces) and then free pieces as this goes over. I saw torrent/mmap though this seems to memory map over file/s.

I'm guessing 3 would be easiest, though what would you suggest?

How to find out if a client is currently uploading data of a specific torrent?

The need of this is to know when we can drop a torrent from a client safely.

Some downloading status information

Is there way to know progress, download speed and time left info? In your cli downloader i see only progress state and it's calculated by separate goroutine wich takes currently downloaded bytes, total bytes and converts it to percentage, here.

I can calculate speed and time left by myself, but, maybe, calculating this values in Client or in Torrent is better, than multiple realizations in all based on your package tools?

Crash in latest version

I know you just closed out one of my previous ones. I am now getting a new one. It is consistently reproducible with certain magnet urls.

panic: lame
    panic: sync: Unlock of unlocked RWMutex

goroutine 176 [running]:
sync.(*RWMutex).Unlock(0xc82012a238)
    /usr/local/Cellar/go/1.5/libexec/src/sync/rwmutex.go:114 +0xa3
github.com/anacrolix/sync.(*RWMutex).Unlock(0xc82012a108)
    /Users/craig/go/src/github.com/anacrolix/sync/rwmutex.go:26 +0x57
github.com/anacrolix/torrent.(*torrent).hashPiece(0xc8202f8b00, 0x21, 0x0, 0x0, 0x0)
    /Users/craig/go/src/github.com/anacrolix/torrent/torrent.go:676 +0x49f
github.com/anacrolix/torrent.(*Client).verifyPiece(0xc82012a000, 0xc8202f8b00, 0x21)
    /Users/craig/go/src/github.com/anacrolix/torrent/client.go:2716 +0x1c2
created by github.com/anacrolix/torrent.(*Client).queuePieceCheck
    /Users/craig/go/src/github.com/anacrolix/torrent/client.go:119 +0xb6

Is there a straightforward way to deal with edge pieces?

When I'm downloading a file from a multi-file torrent I don't want edge pieces of the file to be split and stored separately.

Is there a way, might be a flag, to simply download those pieces to fill beginning and the end of the requested file and then simply removes them from the piece list without asking backed to create extra files?

go-flags prevents blockfile use with cmd/torrent

https://github.com/anacrolix/torrent/blob/master/cmd/torrent/main.go#L80 seems to fill rootGroup.Client.IPBlocklist with &{[]}.

This causes if cfg.IPBlocklist != nil to always resolve true, which then prevents the call to cl.setEnvBlocklist()

Allow configuring the number of concurrent connections

Hey

As per our discussion, some users would like to configure the number of open connections for performance reasons.

The connection limit is currently a constant . Would be nice if we could configure it from the config struct.

Thanks! =)

dht implementation details & code completion status?

Hello,
I would like to use your library in a project of mine, but I'm not sure as to the completion status of your project. You say it is running in production, but the DHT library seems to be incomplete. It is not capable, for example, of announcing itself as a possible source for a torrent. I am not sure how much more is missing from it.
Would you be interested if I tried to implement announce_peer so the DHT client can be completed? how much do you think is missing? It seems there is also some code missing for infohash storage, and infohash to peerid mapping.
I am stressing on the DHT implementation since I can not rely on external DHT nodes for this project.

When is the torrent ready?

I want to read the files inside a torrent but it seems that something like this will work only with a sleep involved

torrentEngine, err := client.AddMagnet(magnetLink)
    defer torrentEngine.Drop()
    log.Println(torrentEngine.String())
    time.Sleep(1* time.Second)
    if err != nil {
        log.Println("Error when adding magnet link", err)
    }
    for index, file := range torrentEngine.Files() {
        log.Printf("File %d: +v", index, file)
    }

disable pex and dht

there's a cli flag for disabling trackers; it would be super useful to get some to disable pex and dht

help

how can i add more magnet url for download meta-info,
im try new a global client and add many magnet url
but out of memory pprof tip me utp use many memory

Fail to connect to track servers after upgrade from 0256508 to 6b27e14

Before upgrading, the code works, but break after upgrade as mentioned in title, I have logged the announce response:

2015/08/14 20:05:22 udp.go:127: binary.Read:&{[0 0 6 153 0 0 0 5 0 0 0 2 10 10 41 1 133 142 10 10 41 1 152 222 10 10 41 1 154 199 10 10 41 1 233 229 10 10 41 65 195 87 10 10 41 1 222 74 10 10 41 1 172 1] 12 -1}
client.go:2438: big: error announcing to udp://10.10.41.65:6969: error announcing: binary.Read: invalid type *util.CompactPeer

This is the related change for CompactPeer struct in util/types.go:

 type CompactPeer struct {
-       IP   [4]byte
+       IP   net.IP
        Port uint16
 }

The problem is not resolved even after DisableIPV6 is set to true in client.

dht-ping crashes when attempting to ping a non-Bittorrent server address

Attempting to ping a non-server, for example with dht-ping google.com:80 crushes the program after the timeout:

> dht-ping.exe google.com:80
2015/12/04 15:24:50 main.go:34: dht server on [::]:64335
panic: runtime error: invalid memory address or nil pointer dereference
[signal 0xc0000005 code=0x1 addr=0x0 pc=0x401a65]

goroutine 1 [running]:
main.main()
        C:/Work/src/github.com/anacrolix/torrent/cmd/dht-ping/main.go:72 +0xa25

goroutine 5 [IO wait]:
net.runtime_pollWait(0x2335620, 0x72, 0xc0820ae520)
        c:/go/src/runtime/netpoll.go:157 +0x67
net.(*pollDesc).Wait(0xc082022d70, 0x72, 0x0, 0x0)
        c:/go/src/net/fd_poll_runtime.go:73 +0x41
net.(*ioSrv).ExecIO(0xc08202c038, 0xc082022c60, 0x7d24a0, 0xb, 0x866880, 0x557e90, 0x0, 0x0)
        c:/go/src/net/fd_windows.go:182 +0x177
net.(*netFD).readFrom(0xc082022c00, 0xc082094000, 0x10000, 0x10000, 0x0, 0x0, 0x0, 0x0, 0x0)
        c:/go/src/net/fd_windows.go:490 +0x1ee
net.(*UDPConn).ReadFromUDP(0xc08202c048, 0xc082094000, 0x10000, 0x10000, 0x0, 0x0, 0x0, 0x0)
        c:/go/src/net/udpsock_posix.go:61 +0x11e
net.(*UDPConn).ReadFrom(0xc08202c048, 0xc082094000, 0x10000, 0x10000, 0x23357a0, 0x0, 0x0, 0x0, 0x0)
        c:/go/src/net/udpsock_posix.go:79 +0x11d
github.com/anacrolix/torrent/dht.(*Server).serve(0xc082054900, 0x0, 0x0)
        C:/Work/src/github.com/anacrolix/torrent/dht/dht.go:574 +0xc4
github.com/anacrolix/torrent/dht.NewServer.func1(0xc08202c030)
        C:/Work/src/github.com/anacrolix/torrent/dht/dht.go:149 +0x2f
created by github.com/anacrolix/torrent/dht.NewServer
        C:/Work/src/github.com/anacrolix/torrent/dht/dht.go:158 +0x28f

goroutine 6 [select]:
github.com/anacrolix/torrent/dht.(*Server).bootstrap(0xc082054900, 0x0, 0x0)
        C:/Work/src/github.com/anacrolix/torrent/dht/dht.go:1030 +0x59f
github.com/anacrolix/torrent/dht.NewServer.func2(0xc08202c030)
        C:/Work/src/github.com/anacrolix/torrent/dht/dht.go:160 +0x2f
created by github.com/anacrolix/torrent/dht.NewServer
        C:/Work/src/github.com/anacrolix/torrent/dht/dht.go:168 +0x2b1

goroutine 18 [semacquire]:
sync.runtime_Semacquire(0xc0820d0e6c)
        c:/go/src/runtime/sema.go:43 +0x2d
sync.(*WaitGroup).Wait(0xc0820d0e60)
        c:/go/src/sync/waitgroup.go:126 +0xbb
github.com/anacrolix/torrent/dht.(*Server).bootstrap.func2(0xc0820d0e60, 0xc0820db740)
        C:/Work/src/github.com/anacrolix/torrent/dht/dht.go:1026 +0x28
created by github.com/anacrolix/torrent/dht.(*Server).bootstrap
        C:/Work/src/github.com/anacrolix/torrent/dht/dht.go:1028 +0x3ed

goroutine 58 [runnable]:
github.com/anacrolix/torrent/dht.(*Transaction).(github.com/anacrolix/torrent/dht.timerCallback)-fm()
        C:/Work/src/github.com/anacrolix/torrent/dht/dht.go:319
created by time.goFunc
        c:/go/src/time/sleep.go:129 +0x41

File descriptor use by UDP trackers

It seems every UDP tracker present when a torrent is first added to a client consumes a socket for the lifetime of the client. This should be fixed so that sockets are only in use for the duration of an announce on a given tracker.

Mocking Torrent internals

Hi. I writing tests for my downloader and i don't want to download anything there. So, i ran into troubles with mocking Torrent.

You define Torrent struct as

type Torrent struct {
    cl *Client
    *torrent
}

I undestand, that you want to encapsulate it's logic from the outside code. But it creates some barrier for changing it's internals from the client tests. Your tests is defined in the same package as library, so you can do anything in tests. It is impossible to do it from my tests.

The good practice for testing libraries is defining it's tests in separate package torrent_test for example.

Is there any way(interfaces magic or something else) to totally mock downloading and write tests without network access?

Is there a way to controll upload\download speed and ratio?

Or uploading\downloading speed, per torrent or global?

Is there any config option to disable public network connection?

I want to do file sharing in the local private network, and don't want to even receive any piece of data from the public network, is there any config to do this?

Download progress

Is there a way to get at the download progress of a torrent?
Looks like you could add a method on torrent that locks t.stateMu and divides the number of completed pieces by the total number of pieces.

Is there anyway to config a peer to continue seeding for a certain time after downloaded the whole file?

The point of this is to avoid high traffic load on the original seeder when there are a lot of peers need to download the same file.

Open/Close performance

os: windows8 x64
go: master
code: https://github.com/anacrolix/torrent/blob/master/data/file/file.go#L73-L80
pprof: http://svgur.com/i/3X.svg
same torrent without Open/Close for each write:
http://svgur.com/i/2s.svg

panic: sync: unlock of unlocked mutex

I'm cross posting Sioro-Neoku/go-peerflix#16 as it seems this issue also happens on torrent.

I am building/running torrent on alpine linux which is based on musl libc.

 [--------------------------------------------------------------------]   0% getting info
�[0A�[2K
 [--------------------------------------------------------------------]   0% getting info
�[0A�[2K
 [--------------------------------------------------------------------]   0% getting info
�[0A�[2K
 [--------------------------------------------------------------------]   0% getting info
�[0A�[2K
 [--------------------------------------------------------------------]   0% getting info
�[0A�[2K
 [--------------------------------------------------------------------]   0% getting info
�[0A�[2K
 [--------------------------------------------------------------------]   0% getting info
�[0A�[2K
 [--------------------------------------------------------------------]   0% getting info
�[0A�[2K
 [--------------------------------------------------------------------]   0% getting info
�[0A�[2K
 [--------------------------------------------------------------------]   0% getting info
�[0A�[2K
 [--------------------------------------------------------------------]   0% getting info
�[0A�[2K
 [--------------------------------------------------------------------]   0% getting info
�[0A�[2K
 [--------------------------------------------------------------------]   0% getting info
�[0A�[2K
 [--------------------------------------------------------------------]   0% getting info
�[0A�[2K
 [--------------------------------------------------------------------]   0% getting info
�[0A�[2K
 [--------------------------------------------------------------------]   0% getting info
panic: sync: unlock of unlocked mutex

goroutine 73 [running]:
sync.(*Mutex).Unlock(0xc8200856b8)
    /usr/lib/go/src/sync/mutex.go:108 +0x9f
github.com/anacrolix/torrent/dht.(*Announce).maybeAnnouncePeer(0xc8201fc000, 0x7f50d7a0e000, 0xc8201de0c0, 0x0, 0x0, 0xc82021ab20, 0x14)
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/dht/announce.go:162 +0x2f1
github.com/anacrolix/torrent/dht.(*Announce).getPeers.func1(0x0, 0x0, 0x0, 0xc820219f40, 0x1, 0xc820219fe0, 0x1, 0xc8201d85a0, 0x0, 0xc820219650, ...)
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/dht/announce.go:199 +0x47b
github.com/anacrolix/torrent/dht.(*Transaction).tryHandleResponse(0xc8201e8240)
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/dht/transaction.go:40 +0xc9
github.com/anacrolix/torrent/dht.(*Transaction).handleResponse(0xc8201e8240, 0x0, 0x0, 0x0, 0xc820219f40, 0x1, 0xc820219fe0, 0x1, 0xc8201d85a0, 0x0, ...)
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/dht/transaction.go:147 +0x199
created by github.com/anacrolix/torrent/dht.(*Server).processPacket
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/dht/server.go:197 +0x3e2

goroutine 1 [semacquire]:
sync.runtime_Syncsemacquire(0xc8200b47e0)
    /usr/lib/go/src/runtime/sema.go:237 +0x201
sync.(*Cond).Wait(0xc8200b47d0)
    /usr/lib/go/src/sync/cond.go:62 +0x9b
github.com/anacrolix/torrent.(*Client).WaitAll(0xc8200b4580, 0x0)
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/client.go:2479 +0xd6
main.main()
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/cmd/torrent/main.go:143 +0x2d6

goroutine 17 [syscall, locked to thread]:
runtime.goexit()
    /usr/lib/go/src/runtime/asm_amd64.s:1721 +0x1

goroutine 6 [semacquire]:
sync.runtime_Syncsemacquire(0xc8200b47e0)
    /usr/lib/go/src/runtime/sema.go:237 +0x201
sync.(*Cond).Wait(0xc8200b47d0)
    /usr/lib/go/src/sync/cond.go:62 +0x9b
github.com/anacrolix/torrent.(*Client).waitAccept(0xc8200b4580)
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/client.go:616 +0x18b
github.com/anacrolix/torrent.(*Client).acceptConnections(0xc8200b4580, 0x7f50d7b54a70, 0xc820038070, 0xc820011600)
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/client.go:622 +0x33
created by github.com/anacrolix/torrent.NewClient
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/client.go:511 +0x863

goroutine 7 [IO wait]:
net.runtime_pollWait(0x7f50d7b55930, 0x72, 0xc8200101a0)
    /usr/lib/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc820067870, 0x72, 0x0, 0x0)
    /usr/lib/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitRead(0xc820067870, 0x0, 0x0)
    /usr/lib/go/src/net/fd_poll_runtime.go:78 +0x36
net.(*netFD).readFrom(0xc820067810, 0xc820122000, 0x2000, 0x2000, 0x0, 0x0, 0x0, 0x7f50d7b50050, 0xc8200101a0)
    /usr/lib/go/src/net/fd_unix.go:259 +0x2a5
net.(*UDPConn).ReadFromUDP(0xc820038078, 0xc820122000, 0x2000, 0x2000, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/go/src/net/udpsock_posix.go:61 +0x117
net.(*UDPConn).ReadFrom(0xc820038078, 0xc820122000, 0x2000, 0x2000, 0x1e5, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/go/src/net/udpsock_posix.go:79 +0x116
github.com/anacrolix/utp.(*Socket).reader(0xc820114000)
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/utp/utp.go:458 +0xf1
created by github.com/anacrolix/utp.NewSocketFromPacketConn
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/utp/utp.go:432 +0x24b

goroutine 8 [chan receive]:
github.com/anacrolix/utp.(*Socket).dispatcher(0xc820114000)
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/utp/utp.go:509 +0x6e
created by github.com/anacrolix/utp.NewSocketFromPacketConn
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/utp/utp.go:433 +0x270

goroutine 9 [semacquire]:
sync.runtime_Syncsemacquire(0xc8200b47e0)
    /usr/lib/go/src/runtime/sema.go:237 +0x201
sync.(*Cond).Wait(0xc8200b47d0)
    /usr/lib/go/src/sync/cond.go:62 +0x9b
github.com/anacrolix/torrent.(*Client).waitAccept(0xc8200b4580)
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/client.go:616 +0x18b
github.com/anacrolix/torrent.(*Client).acceptConnections(0xc8200b4580, 0x7f50d7b55bb0, 0xc820114000, 0xc82000b401)
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/client.go:622 +0x33
created by github.com/anacrolix/torrent.NewClient
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/client.go:525 +0xa37

goroutine 10 [running]:
    goroutine running on other thread; stack unavailable
created by github.com/anacrolix/torrent/dht.NewServer
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/dht/server.go:111 +0x3aa

goroutine 11 [select]:
github.com/anacrolix/torrent/dht.(*Server).bootstrap(0xc820085680, 0x0, 0x0)
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/dht/server.go:586 +0x5a5
github.com/anacrolix/torrent/dht.NewServer.func2(0xc820038088)
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/dht/server.go:113 +0x28
created by github.com/anacrolix/torrent/dht.NewServer
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/dht/server.go:121 +0x3cc

goroutine 12 [sleep]:
time.Sleep(0x989680)
    /usr/lib/go/src/runtime/time.go:59 +0xf9
github.com/gosuri/uiprogress.(*Progress).Listen(0xc820012640)
    /home/clandmeter/go-torrent-orig/src/github.com/gosuri/uiprogress/progress.go:98 +0xbc
created by github.com/gosuri/uiprogress.(*Progress).Start
    /home/clandmeter/go-torrent-orig/src/github.com/gosuri/uiprogress/progress.go:112 +0x35

goroutine 13 [chan receive]:
github.com/anacrolix/torrent.(*Client).announceTorrentTrackersFastStart(0xc8200b4580, 0xc820016840, 0xc82000b6a0, 0x1, 0x1, 0xc820088dc0, 0x0)
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/client.go:2396 +0x20f
github.com/anacrolix/torrent.(*Client).announceTorrentTrackers(0xc8200b4580, 0xc820088dc0)
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/client.go:2421 +0x20c
created by github.com/anacrolix/torrent.(*Client).AddTorrentSpec
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/client.go:2217 +0x7f7

goroutine 14 [select]:
github.com/anacrolix/torrent.(*Client).announceTorrentDHT(0xc8200b4580, 0xc820088dc0, 0x1)
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/client.go:2293 +0x811
created by github.com/anacrolix/torrent.(*Client).AddTorrentSpec
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/client.go:2220 +0x839

goroutine 15 [chan receive]:
main.torrentBar.func3(0xc8200b4580, 0xc820088dc0, 0xc8200a8500)
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/cmd/torrent/main.go:60 +0x59
created by main.torrentBar
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/cmd/torrent/main.go:67 +0x12b

goroutine 16 [chan receive]:
main.addTorrents.func3(0xc8200b4580, 0xc820088dc0)
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/cmd/torrent/main.go:106 +0x55
created by main.addTorrents
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/cmd/torrent/main.go:108 +0x23a

goroutine 19 [select]:
net/http.(*Transport).getConn(0xc820082090, 0xc8201ae000, 0x0, 0xc8201aa000, 0x4, 0xc8201aa007, 0x15, 0x0, 0x0, 0x0)
    /usr/lib/go/src/net/http/transport.go:554 +0x663
net/http.(*Transport).RoundTrip(0xc820082090, 0xc8201ae000, 0x150, 0x0, 0x0)
    /usr/lib/go/src/net/http/transport.go:228 +0x4ac
net/http.send(0xc8201ae000, 0x7f50d7b544d8, 0xc820082090, 0xc8201aa100, 0x0, 0x0)
    /usr/lib/go/src/net/http/client.go:220 +0x52c
net/http.(*Client).send(0xb65c20, 0xc8201ae000, 0xf6, 0x0, 0x0)
    /usr/lib/go/src/net/http/client.go:143 +0x15a
net/http.(*Client).doFollowingRedirects(0xb65c20, 0xc8201ae000, 0x9d3cf0, 0x0, 0x0, 0x0)
    /usr/lib/go/src/net/http/client.go:380 +0xbc0
net/http.(*Client).Get(0xb65c20, 0xc8201aa000, 0xf6, 0xc8201a6000, 0x0, 0x0)
    /usr/lib/go/src/net/http/client.go:306 +0xaa
net/http.Get(0xc8201aa000, 0xf6, 0xf6, 0x0, 0x0)
    /usr/lib/go/src/net/http/client.go:281 +0x45
github.com/anacrolix/torrent/tracker.(*client).Announce(0xc8200a8480, 0xc820016840, 0x0, 0xc800000000, 0x0, 0x0, 0x0, 0x0, 0x0)
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/tracker/http.go:74 +0xd43
github.com/anacrolix/torrent.(*Client).announceTorrentSingleTracker(0xc8200b4580, 0x7f50d7b55d08, 0xc8200a8480, 0xc820016840, 0xc820088dc0, 0x0, 0x0)
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/client.go:2362 +0x3ed
github.com/anacrolix/torrent.(*Client).announceTorrentTrackersFastStart.func1(0xc8200b4580, 0xc820016840, 0xc820088dc0, 0xc8200168a0, 0x7f50d7b55d08, 0xc8200a8480)
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/client.go:2390 +0x50
created by github.com/anacrolix/torrent.(*Client).announceTorrentTrackersFastStart
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/client.go:2392 +0x195

goroutine 72 [semacquire]:
sync.runtime_Semacquire(0xc8202187fc)
    /usr/lib/go/src/runtime/sema.go:43 +0x26
sync.(*WaitGroup).Wait(0xc8202187f0)
    /usr/lib/go/src/sync/waitgroup.go:126 +0xb4
github.com/anacrolix/torrent/dht.(*Server).bootstrap.func2(0xc8202187f0, 0xc820226660)
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/dht/server.go:582 +0x21
created by github.com/anacrolix/torrent/dht.(*Server).bootstrap
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/dht/server.go:584 +0x3f6

goroutine 66 [IO wait]:
net.runtime_pollWait(0x7f50d7b55630, 0x77, 0x453980)
    /usr/lib/go/src/runtime/netpoll.go:157 +0x60
net.(*pollDesc).Wait(0xc82019e220, 0x77, 0x0, 0x0)
    /usr/lib/go/src/net/fd_poll_runtime.go:73 +0x3a
net.(*pollDesc).WaitWrite(0xc82019e220, 0x0, 0x0)
    /usr/lib/go/src/net/fd_poll_runtime.go:82 +0x36
net.(*netFD).connect(0xc82019e1c0, 0x0, 0x0, 0x7f50d7b55d48, 0xc820190500, 0xece0b1cc5, 0x397e26a5, 0xb65ce0, 0x0, 0x0)
    /usr/lib/go/src/net/fd_unix.go:114 +0x1f6
net.(*netFD).dial(0xc82019e1c0, 0x7f50d7b54aa8, 0x0, 0x7f50d7b54aa8, 0xc82018e780, 0xece0b1cc5, 0x397e26a5, 0xb65ce0, 0x0, 0x0)
    /usr/lib/go/src/net/sock_posix.go:137 +0x351
net.socket(0x903060, 0x3, 0x2, 0x1, 0x0, 0xc82018e700, 0x7f50d7b54aa8, 0x0, 0x7f50d7b54aa8, 0xc82018e780, ...)
    /usr/lib/go/src/net/sock_posix.go:89 +0x411
net.internetSocket(0x903060, 0x3, 0x7f50d7b54aa8, 0x0, 0x7f50d7b54aa8, 0xc82018e780, 0xece0b1cc5, 0x5b397e26a5, 0xb65ce0, 0x1, ...)
    /usr/lib/go/src/net/ipsock_posix.go:160 +0x141
net.dialTCP(0x903060, 0x3, 0x0, 0xc82018e780, 0xece0b1cc5, 0xc8397e26a5, 0xb65ce0, 0x1, 0x0, 0x0)
    /usr/lib/go/src/net/tcpsock_posix.go:171 +0x11e
net.dialSingle(0xc820188280, 0x7f50d7b54a40, 0xc82018e780, 0xece0b1cc5, 0x7fff397e26a5, 0xb65ce0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/go/src/net/dial.go:364 +0x3f5
net.dialSerial.func1(0xece0b1cc5, 0x397e26a5, 0xb65ce0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/go/src/net/dial.go:336 +0x75
net.dial(0x903060, 0x3, 0x7f50d7b54a40, 0xc82018e780, 0xc8201596e8, 0xece0b1cc5, 0x397e26a5, 0xb65ce0, 0x0, 0x0, ...)
    /usr/lib/go/src/net/fd_unix.go:40 +0x60
net.dialSerial(0xc820188280, 0xc82018c670, 0x1, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/go/src/net/dial.go:338 +0x760
net.(*Dialer).Dial(0xc820012410, 0x903060, 0x3, 0xc8201aa007, 0x15, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/go/src/net/dial.go:232 +0x50f
net.(*Dialer).Dial-fm(0x903060, 0x3, 0xc8201aa007, 0x15, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/go/src/net/http/transport.go:38 +0x6e
net/http.(*Transport).dial(0xc820082090, 0x903060, 0x3, 0xc8201aa007, 0x15, 0x0, 0x0, 0x0, 0x0)
    /usr/lib/go/src/net/http/transport.go:499 +0x79
net/http.(*Transport).dialConn(0xc820082090, 0x0, 0xc8201aa000, 0x4, 0xc8201aa007, 0x15, 0x0, 0x0, 0x0)
    /usr/lib/go/src/net/http/transport.go:596 +0x19a9
net/http.(*Transport).getConn.func4(0xc820082090, 0x0, 0xc8201aa000, 0x4, 0xc8201aa007, 0x15, 0xc8201bc000)
    /usr/lib/go/src/net/http/transport.go:549 +0x66
created by net/http.(*Transport).getConn
    /usr/lib/go/src/net/http/transport.go:551 +0x265

goroutine 83 [select]:
github.com/anacrolix/torrent/dht.(*Server).Announce.func2(0xc8201fc000)
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/dht/announce.go:76 +0x244
created by github.com/anacrolix/torrent/dht.(*Server).Announce
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/dht/announce.go:87 +0x474
panic: sync: inconsistent mutex state

goroutine 10 [running]:
sync.(*Mutex).Lock(0xc8200856b8)
    /usr/lib/go/src/sync/mutex.go:74 +0x16d
github.com/anacrolix/torrent/dht.(*Server).serve(0xc820085680, 0x0, 0x0)
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/dht/server.go:213 +0x1a7
github.com/anacrolix/torrent/dht.NewServer.func1(0xc820038088)
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/dht/server.go:102 +0x28
created by github.com/anacrolix/torrent/dht.NewServer
    /home/clandmeter/go-torrent-orig/src/github.com/anacrolix/torrent/dht/server.go:111 +0x3aa

Panic in golang 1.5

I am still debugging, but after updating to go 1.5 I am getting this panic

2015/08/22 13:08:11 http: panic serving 127.0.0.1:51786: sync.Cond is copied
goroutine 50 [running]:
net/http.(*conn).serve.func1(0xc8201a6000, 0x4fa5e80, 0xc820026090)
    /usr/local/Cellar/go/1.5/libexec/src/net/http/server.go:1287 +0xb5
sync.(*copyChecker).check(0x4805440)
    /usr/local/Cellar/go/1.5/libexec/src/sync/cond.go:116 +0xa8
sync.(*Cond).signalImpl(0x4805410, 0x1)
    /usr/local/Cellar/go/1.5/libexec/src/sync/cond.go:83 +0x35
sync.(*Cond).Broadcast(0x4805410)
    /usr/local/Cellar/go/1.5/libexec/src/sync/cond.go:79 +0x26
github.com/anacrolix/torrent.(*Client).setMetaData(0x4805300, 0xc8200a68c0, 0xc820174a20, 0xc820216000, 0xeafb, 0xeafb, 0x0, 0x0)
    /Users/craig/go/src/github.com/anacrolix/torrent/client.go:1927 +0x28c
github.com/anacrolix/torrent.(*Client).AddTorrentSpec(0x4805300, 0xc820085270, 0x4805300, 0x0, 0x1, 0x0, 0x0)
    /Users/craig/go/src/github.com/anacrolix/torrent/client.go:2183 +0xa23

I did notice this in the 1.5 release notes

The WaitGroup implementation in package sync now diagnoses code that races a call to Add against a return from Wait. If it detects this condition, the implementation panics.

License

Hope you'll use MIT, BSD, or one of these other permissive licenses - http://copyfree.org/standard/licenses

Ability to prioritize blocks/pieces/chunks/files when downloading

It would be very beneficial in many applications if anacrolix/torrent had a method for setting the download priority of individual blocks/pieces/chunks (BitTorrent terminology can be somewhat confusing, if not poorly defined in the first place).

While I understand the objections many people have related to selective or "streaming" as there can be a negative effect to the health of the greater swarm as a whole. The thing is, in many cases BitTorrent is used in controlled environments where it's known in advance that such an tactic is perfectly acceptable as it's already been established that there are enough clients with complete copies of the files being shared to sustain a healthy swarm.

anacrolix/torrent isn't a mainstream file sharing client in the same way that µTorrent or Vuze are; it's a library to be incorporated into larger projects where the developers are (hopefully) fully capable of understanding the ramifications of granular prioritization such as this.

HTTP unknown tracker scheme

steps to reproduce:

$ wget http://releases.ubuntu.com/14.04.2/ubuntu-14.04.2-desktop-amd64.iso.torrent
$ torrent ubuntu-14.0.4-2-desktop-amd64.iso.torrent
2015/03/24 20:35:16 client.go:1964: error creating tracker client for "http://torrent.ubuntu.com:6969/announce": unknown scheme
2015/03/24 20:35:16 client.go:1964: error creating tracker client for "http://ipv6.torrent.ubuntu.com:6969/announce": unknown scheme

DHT-Sec is supposed to limit data-hosts, not clients, on DHT

So, while exploring torrent for #44, I noted that the security handling was essentially excluding insecure nodeIds entirely from the routing table. Further, that some KRPC handlers were heading towards excluding insecure nodeIds entirely from KRPC responses, too.

The original intent of the DHT-Sec extension was to limit the ability for an attacker to host targeted data by limiting their ability to create valid secure IDs. A given external IP can have at most seven valid secure IDs, and cannot choose the ID prefix, making it infeasible for all but the most IP-rich attackers to deliberately target a specific torrent.

However, it was not intended to limit participation in the DHT, and doing so would make the DHT less useful for many applications. An argument could be made that it could optionally extend to barring insecure NodeIds from offering routing information, but it makes no sense to refuse to allow insecure NodeIds to request or even to post information to secure nodes.

The reason being that all the Secure ID guarantees is difficulty to choose a prefix; it has no other effect, and pretending that it offers any other sort of guarantee is pointless. As the prefix is only relevant to hosting data on the DHT, applying the secure ID requirement to anything else doesn't add any security, but does impose annoyance on developers wishing to build DHT-centric applications. It doesn't make identifying bad actors easier, because DHT-Sec does not prevent IDs from being changed rapidly, it only limits alteration of the immediate first bytes. It doesn't limit the number of nodes an actor can deploy either, only where in the DHT those nodes must localise themselves.

I propose that torrent's implementation of DHT-Sec be limited to the specification; that non-local IPs be required to have a Secure ID before torrent will store data on them or consider them a valid endpoint to request ids from, but that torrent will not refuse service to insecure IDs that come to it requesting or announcing Nodes.

Additional thoughts and considerations:

Refusing service and excluding nodes entirely because of DHT-Sec would make impossible many alternative uses of Mainline, such as for secure messaging applications which use NodeId as a key fingerprint. Nodes of this sort that are not "Secure" in the libtorrent sense should not host torrent data, but are still valid DHT participants.
Refusing service will kill many low-end or 'mini' routers that implement minimal DHT traversal to locate resources. These are useful for innovative prototypes into DHT-DNS's, aforementioned secure messaging apps, IOT/Embedded routing, and simply for exploration or monitoring. Requiring them to implement DHT-Sec, which uses an unusual CRC32 that may not be preimplemented on their platform or language, would be harmful to development.
torrent could be more suspicious of insecure Ids in its routing table retention policy, and could choose to drop insecure nodes first to make room for secure nodes. As routing is generally to locate a place to host a torrent, this would be a valid, if unnecessary, policy to adopt.

I hope you reconsider the exclusionary policy on DHT-Sec and look instead at only implementing the basic spec. I'm happy to do the legwork there, seeing how there's not much DHT-Sec here yet, and it's early enough to change direction easily.

Thanks,
Cathal

Background: LibTorrent DHT-Sec overview.

validateInfo in misc.go is offbyone

ValidateInfo should work perfectly fine except for a -1 that slipped in the length sanity verification somehow here https://github.com/anacrolix/torrent/blob/master/misc.go#L119

Unable to build on Linux

Hey,

I received this error regarding your package:

go get github.com/Sioro-Neoku/go-peerflix
# github.com/anacrolix/torrent/dht
/root/go/src/github.com/anacrolix/torrent/dht/addr.go:6: import /root/go/pkg/linux_arm/github.com/anacrolix/missinggo.a: object is [linux arm go1.2.1 X:none] expected [linux arm go1.4.2 X:precisestack]
# github.com/anacrolix/utp
/root/go/src/github.com/anacrolix/utp/utp.go:29: import /root/go/pkg/linux_arm/github.com/anacrolix/jitter.a: object is [linux arm go1.2.1 X:none] expected [linux arm go1.4.2 X:precisestack]

Issues with internal packages when sending pull request

golang/go#11307

  imports github.com/anacrolix/torrent/internal/pieceordering: use of internal package not allowed

Usage: Quick example regarding piece backend

Hi, can you give me a quick example on how the piece data backend should work?

New Feature: Rate limiting

Are there plans to add rate limiting? alternatively, a connection hook might work, where one could implement rate limiting themselves. I don't know enough about the Bittorrent protocol though to know if this is feasible.

New Feature: Custom torrent output

How much of a change would it be to provide an io.Writer as the destination for in-progress torrent files:

client.SetOutput(func(file File) io.Writer {
  //file encrypter
  //network write (HTTP req starts torrent -> response streams it back?)
})

This way I wouldn't have to buffer files on disk before modifying them / sending them out again. Not sure if this is practical though.

Torrent file parsing error

This package sometimes fails to parse a torrent file.
The problem turns out to be a type mismatch error:
"Nodes" field of "MetaInfo" struct is defined as [][]string type,
expecting both host and port are of type string.
However, torrent spec defines port value as of integer type.

http://bittorrent.org/beps/bep_0005.html

download/upload chunk number constrait goes to config

I know this line is here to prevent some bad peers, but any plan to move this (the magic number 6) to client's config : https://github.com/anacrolix/torrent/blob/master/client.go#L1431

for seeding || c.chunksSent < c.UsefulChunksReceived+6 {

I also want to do some network feedback to the client(or client's torrents), so I get more control of the usage of network bandwidth(not use to much of it), any suggestion on this or download/upload rate-limit?

tracker server package

Do you have any plans for torrent server package ?

Question: Download without touching the disk

I'd like to stream torrents using a configurable amount of RAM. Assuming I'm not uploading, my plan is create a custom TorrentDataOpener which keeps chunks around just long enough to ship them off again. Is this safe to delete a chunk after it has been verified?

Improve start-up efficiency when hashing with file-based storage backends

Hi, i built some torrent downloader based on you package.

When i tryed to download big torrent (about 15GB), i noticed, that memory usage is really great (about 700MB), CPU usage about 80-90% on one core. htop shows me large number of downloader threads (50 at least) and it's increases.

At about 15% of downloading i see that messages in logs:
2015/12/23 08:03:00 error writing chunk: open <path to one of files>: too many open files

My wrapper around your package is small and thin and can't produce described problems. You can find it here

It is your bugs or it is my fault and i should play with packge config/tuning go runtime?

For example, transmission-web uses about 100MB of memory and 20-30% of one CPU core with 50 seeding torrents and 1-2 active downloads.

I am newbie in go development and not familiar with golang's profilers, but i want to learn it. I will try to debug this problems in the next few days.

Thanks and sorry for my english!

Query parameters in Announce URL not retained

It looks like the tracker package doesn't include original query parameters from the announce URL in the HTTP announce requests.

For example, a torrent with Announce URL: http://tracker.domain.com/announce?passkey=abc123

This code makes the request look like: http://tracker.domain.com/announce?info_hash=x&peer_id=z

Transmission (http://www.transmissionbt.com/) for example constructs the following announce request:
http://tracker.domain.com/announce?passkey=abc123&info_hash=x&peer_id=z

It retains the parameter passkey in the announce request from the announce url.

A lot of private BT trackers uses this to add a passkey parameter to the announce requests to be able to associate the bittorrent annonce request with a user in the system.

What do you think about this @anacrolix ? You want me to try and get a PR together?

Thanks!

How to know path of my torrent files?

It's me again=)

I can't find Torrent's method wich allows me to get it's root path.

So, i use helper like this
path.Join(downloadPath, t.Info().Name) Where downloadPath is root which i pass to Client, but i'm not sure it is right for all torrents.

Maybe, finction like Torrent.path() that returns actual path should be added?

Counter for chunks uploaded

I've been using this for a while now with some tweaks, one thing I've found most useful was to have a counter for the chunks uploaded, but my implementation leaves a lot to be desired and is incredibly error prone. An official implementation would be excellent and think might help solve some issues people have with seeding to a certain ratio.

Ctrl+C wont terminate in ubuntu

Hello,
I've tried terminating a torrent process by using ctrl+c and it wont kill the process. closing the terminal also results in the torrent process continuing.

anacrolix / torrent Goto Github PK

torrent's Introduction

torrent

Installation

Library examples

Mentions

Downstream projects

Help

Command packages

torrent

torrent download

torrent metainfo magnet

torrentfs

torrent's People

Contributors

Stargazers

Watchers

Forkers

torrent's Issues

i got this errors:

Recommend Projects

Recommend Topics

Recommend Org

`torrent`

`torrent download`

`torrent metainfo magnet`

`torrentfs`