Tracker Clustering ability about chihaya HOT 46 OPEN

I'd like to see some good discussion on design before anyone touches a line of code for this reason.

from chihaya.

For us personally, we would love to be able to simply do something along the lines of a config change as:

cluster_nodes: ["2.2.2.2", "3.2.1.2", "dead::beef"],

This will set to which other servers (trackers) the current tracker will listen to updates from and also will send updates to.

This however will have limitations in terms of scalability, lets say you have a cluster of ~100 servers for each newly connected peer that means the server receiving this connections has to send 99 updates.
The way OpenTracker solves this is to enforce the use of Multicast within a network. Meaning sending the update once to the multicast address and all other trackers inside the same network receive updates. however this is truly a pain in the ass especially if you want redundancy across datacenters. it means you have to set up a VPN network with multicast support which seems to be really a hard thing to achieve properly .

On another side note: I dont think the clustering replication needs to be absolute live. it can be done in bursts too. I strongly think that if say the cluster gets updated every say 2-5 mins it would be ok due to the nature of BitTorrent and trackers. just my thoughts.

from chihaya.

Like you said, replication delay isn't an issue, so I think a ring or otherwise loosely connected topology would be perfect. Something like each tracker connecting to 2 others would make log_2(n) hop broadcasts possible.

from chihaya.

Yeah that does sound as a basic to start with, with a setup like that, how would you layout the config file?

Would you have the trackers figure out themselves on boot which 2 other trackers to talk to?

What would happen if say one (or two) badly places trackers go down, would this result in one tracker trying to push its updates to two dead trackers and thus the data not propagating properly throughout the cluster?

from chihaya.

An array of cluster nodes sounds perfect for the config. However, I think that being able to specify a subset and doing autodiscovery would be nice as well, maybe as a thing to implement after getting basic clustering working.

To coordinate the network, something like Raft or SWIM (there are Go implementations of both) would work well. To mitigate replication error when trackers stop, there could be a (configurable) replication overlap.

from chihaya.

How about a consistent distribution scheme based on the hash of the torrent on question?

In my head in envisage this working well something like this...
Each machine is set up with the exact same "clusterNodes" configuration. For example, node 172.22.0.1 has the configuration line clusterNodes: {"172.22.0.1", "172.22.0.2"} to connect to this example two node cluster. The nodes then divide the available hash space evenly between all the nodes. In this example, hex hashes starting with 1-8 get delegated to 172.22.0.1, while the cluster puts 172.22.0.2 in charge of hashes starting with 9-F. This could easily work even with 32 or more cluster nodes.
Requests come in, distributed through DNS round robin or geoip etc, when they get to the node chosen it first looks up within that node to see if it knows about the torrent. If not, it asks the node that's deemed to be "in charge" of that hash. If it is the node deemed in charge of that hash, it adds it to its database as a new torrent.

The advantage of this is that the storage requirement is divided as evenly as possible between all the nodes, while any of the nodes can answer any query.
However, it relies on there being a fast and hopefully direct connection between nodes. It also means that any of the nodes rebooting dumps that section of knowledge if it's only stored in RAM.
It would also require careful thinking about how the cluster should deal with one node unexpectedly going down.

from chihaya.

@johncave

As you said, your proposed solution solves a storage issue, thing is, with high traffic trackers storage is by no means a concern. Its the network traffic/kernal limitations on the IP stack that usually is the first to be bottle necked. Storage would be one of the last resources to become an issue I would think.

In your proposed solution the a user has 33% chance to hit the correct tracker node meaning that network traffic and CPU cycles are minimized.

Just my opinion.

from chihaya.

How about something more simple: a Cassandra storage implementation, and then you can just use any load balancer in front of the tracker instances to round robin requests?

from chihaya.

@jzelinskie I have little to no knowledge on how Cassandra clusters, however as I mentioned earlier. The first thing that will go (become a bottleneck) is the networking part of the server on a high traffic tracker. Either max out the normal 100mbit given by your provider or max out the linux kenral IP stack limits.

Meanings that putting a load balancer to push all traffic to different backends will defeat the purpose of this. But in reality simple using DNS round robin will solve this issue, again assuming that the multiple servers are in sync (or close to it)

from chihaya.

It feels a bit unclear what exactly clustering should solve and in what setup.

• WAN or LAN (e.g. how much latency and packet loss can we expect)
• What problem is replication solving (e.g. stability, lack of memory, lack of bandwidth, lack of other resources)

Ring replication
The solution proposed by @jzelinskie and @johncave is basically the same (segmenting the keyspace / using Cassandra ring) and it solves the problem with memory and space usage - and with partition replication it also solves stability problems.

The problem is that it'll work terribly over a network with latency as it'll have a huge impact on response times. It's probably the solution that'll best with the problem @dtrackd has.

It'll also require 3+ servers to make sense (in the face of unexpected downtime).

Fully replicated

Replicating the peers to every tracker. This one probably doesn't scale far beyond 10 servers but I'm not sure that's a problem at all. As decribed by @dtrackd it'll work by flooding updates to every other tracker. Assuming a bittorrent tracker does not have any action ordering requirements for the shared state, updates can just be flooded to other trackers in the cluster.

This solution is my personal favourite because I can't see a situation where you'd need 100 trackers or have any consistency requirements. It also ignores any problems that distributed systems come with as sharing peers between trackers become "best effort", can handle n-1 trackers offline and does not require any special precaution or changes to other parts of the system. I.e. it can be a simple middleware that sends packets of peer-changes to every other tracker in the cluster. The format can be optimized for fasr parsing.

A bit of napkin math with these assumptions:

• 50.000.000 peers announcing every 30 minutes (biggest number I could find)
• 6 trackers handling the requests
• Requests evenly distributed across trackers and time
• Announce size is 300 bytes incl. HTTP headers.
• Only things that'll change the availability of peers matter, e.g. scraping is ignored.
• Only IPv4
• Ignoring overhead

With those in mind:
Each tracker has 50.000.000 peers / 6 trackers / 30 minutes / 60 seconds = 4629 requests per second which totals to 300 bytes * 4629rps = 1.388.700 bytes per second.

Some of the announced state has to be shared with the other trackers: info_hash (20 bytes) + peer_id (20 bytes) + port (2 bytes) + ip (4 bytes) + event_type (1 byte) = 47 bytes - The total inter-tracker traffic is then 47bytes * 4629rps * 5 trackers = 1.087.815 bytes per second both ways, i.e. around 2MB/s.
The inter-tracker format can probably be optimized a bit by combining several announces to same info_hash or from same peer.

To sum up this solution:

• It'll work WAN and is resistant to packetloss and latency - of course, peers aren't shared if the peer packet is lost. As announcements happen fairly often, I'm not sure that'll pose as a major problem.
• Only lose data if all trackers offline
• Works only if bandwidth and memory isn't the bottleneck

What I'd love is some real-world bottleneck information.

from chihaya.

@JohnDoee Thanks for the well thought out response.

I think @dtrackd is out of the game now, but from talking to him, I think we can be confident that the largest of installs will fit on one beefy server. However, I'm personally used to writing software to scale well and run on many low-end machines and don't know if beefy machines are feasible or popular for people operating trackers.

from chihaya.

Just to throw that in here: @rarbg might be able to provide real-world data.

from chihaya.

The question is how you want to move forward from here and what the goal should be. If a single server is beefy enough to sustain even the biggest tracker nowadays it's probably more interesting to support 100 RPi's clustered and spread across the world ?

Anyways, I'd be happy to outline how I think a Cassandra-esq/Dynamo-ish clustering would look like with all the optimizations I think you can do with a Bittorrent Tracker. It will require a new peer store that'll send requests to remote instances of Chihaya or the local memory store.

Of course, it's only if this is a feature you see valuable for Chihaya and something you want in it. The price is latency but the payoff is lots of scaling (something competing products lack :) )

from chihaya.

Our current software can sustain ~50 mln peers on a 8 core @ 3.5ghz with 32gb ram.
The question of replication/clustering is not about performance and it never was as much as its about redundancy.

from chihaya.

Oh and keep in mind that the 100 RPIs cluster example is very bad because

1. Moving traffic through continents is expensive
2. RPIs nics are shit and dont provide any offload
3. RPIs memory is not big enough to sustain a big tracker. And having a partial "state" of peers is not a working option

from chihaya.

So the way I see it we're talking about two goals of clustering:

1. Replicating all of the peer data to achieve redundancy
2. Replicating a subset of the peer data on each node, for load balancing/data distribution/other reasons/no single point of failure, although we still always have DNS

We could achieve redundancy relatively easily by building store implementations that occasionally synchronize. It doesn't have to be live due to the nature of BitTorrent. If we replicate all stores, instead of just the peer store, we also automagically replicate blacklists/whitelists and whatnot.

The second solution is interesting as well: You could run chihaya at home, use that as tracker and use a DHT backend, or something similar.

from chihaya.

@rarbg Thanks for the feedback, it's very helpful with some real-world data. The RPi example was more a thought example using crap hardware to share the burden.

@mrd0ll4r Regarding DNS, I've had great success using short TTLs and lots of A records. And it seems like others have too. Normal web browsers will actually try multiple IPs if multiple A records are found, until the browser finds a connectable one. Although I'm not sure that's a behaviour exhibited by torrent clients.

While the consistency requirements for the peer store are very easy to map, it might be harder to do such with the other stores as middlewares can use them differently. It might be wise to cut this ticket down to just distribute the peer store and leave the others as an exercise for the reader / the future?

One design choice is a superset of the other. With the Cassandra model it will be possible to run setup like:

• Three nodes in Datacenter A each storing 66% of all peers
• Two nodes in Datacenter B each storing 100% of all peers

That'll be a total of five trackers with 200% peers stored in each datacenter.

I'm not sure how to move this forward though, someone has to make a choice or more info should be gathered.

from chihaya.

@JohnDoee I agree that we should try the peer store first, as it is the most performance-critical part as well.

I never worked with Cassandra, but I still believe we can't achieve both ideas: What I imagined is a completely distributed tracker to run at home, to eliminate the tracker as a central instance for BitTorrent. But I realize this issue is about scaling/clustering/achieving redundancy for a single deployment of chihaya, so let's focus on that.

I'm not sure if Cassandra provides the necessary performance. I feel like as soon as someone switched from the in-memory peer store to a Cassandra-based peer store they'd need much more resources to achieve the same performance they had with a single in-memory instance before.
I don't know if this is realistic, but I believe we'd be better off building our own replication around the memory implementation - how hard can it be? Especially for peers: we just need to add to other stores and take the newest LastAction in case of already-present entries. We could then just GC after receiving replication data and clean up the peers.

from chihaya.

My thoughts :
DNS with short ttl/rr A will never work

1. not all clients have perfect implementation of the torrent protocol
2. there are clients that are using only the first returned record
3. there are clients that do 5 requests for 5 records ,
4. with 50 million peers on 7200 ttl you get ~7.5-10k dns queries / sec to your dns servers. Setting a low ttl will move your expenses to boxes for dns ( unless you decide to go with CF's dns but thats your only other option that will handle shit load of requests )

About replication

1. As i said before there should be permanent storage to keep your database in case of restarts ( no having a backup node running just to keep the db stats online is not an option , its a waste :) )
2. Again , moving the load from the tracker itself to Cassandra/anydb for replication purposes is just waste of resources ...
   Best possible replication will be if its a custom protocol , highly compressed , made in batches , being inserts of new peers , updates of existing or GC of old... the trackers can talk to each other via simple sockets , batch the state updates of peers/hashes and then snappy compress them to each other via these sockets

from chihaya.

@rarbg About replication 1: I guess you need the permanent storage for blacklists/whitelists etc., right? The peer state should be restored after one announce interval in case of a restart, afaik. I totally agree with the way you described the replication of the peer store.

Now the synchronization of the (string|ip) store is not as easy as the peer store. The synchronization here has to be live and fully replicated/fully accessible from every node. From what I've read Cassandra sounds like a good choice for that, @JohnDoee ?

@jzelinskie and I talked about persistent storage in general and came up with something like an in-memory implementation that mirrors inserts/updates/deletes to a leveldb/boltdb and restores from that, but does all lookups in memory. It's not replicated at all, but it works for a single node and should be very fast.
The reason I'm bringing this up is this: We could develop a replication/synchronization mechanism that is not implemented as a store driver. We could then use the leveldb-backed driver and replicate to other nodes. Following the same idea, we could just use the memory implementation of the peer store and replicate/synchronize independently of that. Thoughts?

@jzelinskie - I feel like the store should make registered drivers available somehow, that'd make writing a leveldb-wrapper very easy. Thoughts?

from chihaya.

I dont see @mrd0ll4r's email reply here so im going to paste it
@rarbg About replication 1: I guess you need the permanent storage for blacklists/whitelists etc., right? The peer state should be restored after one announce interval in case of a restart, afaik.

Now the synchronization of the (string|ip) store is not as easy as the peer store. The synchronization here has to be live and fully replicated/fully accessible from every node. From what I've read Cassandra sounds like a good choice for that, @JohnDoee ?

@jzelinskie and I talked about persistent storage in general and came up with something like an in-memory implementation that mirrors inserts/updates/deletes to a leveldb/boltdb and restores from that, but does all lookups in memory. It's not replicated at all, but it works for a single node and should be very fast.
The reason I'm bringing this up is this: We could develop a replication/synchronization mechanism that is not implemented as a store driver. We could then use the leveldb-backed driver and replicate to other nodes. Following the same idea, we could just use the memory implementation of the peer store and replicate/synchronize independently of that. Thoughts?

@jzelinskie - I feel like the store should make registered drivers available somehow, that'd make writing a leveldb-wrapper very easy. Thoughts?

from chihaya.

I need permanent storage for blacklists , whitelists and the completed column on each hash , thats all.
No idea what you are talking about (string|ip) ?

from chihaya.

I'm not sure why you can't see my earlier reply, but here's some info about our architecture:

We have split up our storage into three interfaces and three corresponding drivers:

• The peer store stores peer data
• The string store stores arbitrary strings (for example infohashes or client IDs for blacklists/whitelists)
• The IP store stores IPs and networks (for example for blacklists/whitelists)

That's why I said we need to synchronize the IP and string store in a different way than the peer store - peer data can be updated in batch and asynchronously, but IPs, networks and strings must be kept synchronized and must be persistent in your case.

In light of the rewrite we just did, we don't store completed at all right now, but that can be achieved relatively easily, we just have to figure out where to put it.

I think we'll need to discuss this in depth, might take a while. Stay subscribed to the issue for updates :)

from chihaya.

@mrd0ll4r did anyone request IP store for blacklisting/whitelisting ips ? Because thats usually done at firewall level

from chihaya.

@rarbg nope, not directly. Do you firewall-block based on the IP contained in an announce as well?

from chihaya.

@mrd0ll4r public trackers ignore the ip contained in the announce ... so no...

from chihaya.

@rarbg IP blacklisting and whitelisting is optional middleware. The usefulness is probably more in that you can use the tracker's API to dynamically manipulate these lists.

storage There needs to be two solutions: one for the eventually consistent peer data and another fully consistent store for everything else.

snapshotting I like the idea of using memory and then having asynchronous snapshots to leveldb/boltdb. This is the latest architecture that the etcd database uses (we may even be able to reuse some code).

replication The problem with hand-rolling replication is that you somehow need some consistent view of the cluster aka service discovery. You can either do this at the load balancer level or using a distributed lock service like etcd or zookeeper. Either way, it drastically increases complexity. I'd like to leverage a distributed database this rather than reimplementing it from scratch. We might be able to reuse the mechanism for snapshotting also for synchronizing with said distributed database. That way we can still serve best effort from memory, but eventually converge on the same results when trackers synchronize with the database.

from chihaya.

@jzelinskie What replication are you talking about ? for the peer storage ? that one doesn't have any consistency requirements that can cause conflicts (as in, there's no need for locks).

I think this issue needs to be split into two, is a bit confusing when talking about two different things.

from chihaya.

@JohnDoee you still need a consistent view of who is in the cluster of trackers in order to rebalance data across the cluster. This is the crux of distributed systems.

from chihaya.

@jzelinskie That's not really required for such static cluster. It feels like you think it won't work or it'll be too much work ?

from chihaya.

@jzelinskie your consistent view is assembled on boot and you dont need rebalancing , data is replicated consistently 1:1 to all parts of the cluster , there is no locking involved what-so-ever.

from chihaya.

Yes, the cluster can be static, but then you can't take advantage of autoscaling. Seems silly to put in all the effort of making a distributed replication scheme, but not have it be dynamically scalable.

from chihaya.

@jzelinskie what you just said just shows that the last 4 days you didnt read a single line of what i said :)

from chihaya.

@rarbg please understand that chihaya aims to be a very flexible and configurable tracker. When making a design choice of this magnitude we consider small and large, private and public trackers.

Having a static cluster is just a subset of having a dynamic cluster, so we aim to implement clustering for a dynamic cluster.

from chihaya.

@mrd0ll4r so why dont you ask one of those private trackers for input then ?

from chihaya.

@rarbg we'd love to gather input from them as well!

I simply want to avoid personally implementing my own a homegrown, naive distributed key-value store. Chihaya is already pluggable that others choose the proper database implementations for their own installations and write implementations of our storage interface. There are a wide variety of ways for trackers to be deployed and I think allowing the user to configure the database should be just as key as configuring the middleware functionality.

Be aware that literally everything in chihaya is now pluggable. If you don't want to use any of the middleware for the store server, you could implement all of your own middleware and storage abstractions. As per your idea of the wrapper store, that is totally possible without making any code changes to the store package: you just simply take the name and configuration of the second store as configuration for your wrapper store and then call store.OpenPeerStore(name, configuration) again and you can delegate to that store implementation within your wrapper however you please. The config would end up looking something like this:

...
servers:
  - name: store
    config:
      peer_store:
        name: memory_wrapper
        config:
          wrapped_peer_store_name: cassandra
          wrapped_peer_store_config:
             ...

from chihaya.

I will go forth and do this:

• implement a persistency-wrapper for the string and IP stores (using boldtb/leveldb/rocksdb)
• look into implementing a cache for the peer store

from chihaya.

@mrd0ll4r "Having a static cluster is just a subset of having a dynamic cluster"
That is not true. You have to CHOOSE which knob you want to be able to turn here. At least if we're talking about the same stuff which is how to decide where tokens are stored.

When I am talking about dynamic, I'm talking about adjusting the replication factor, i.e. how many copies a given cluster have of a given piece of data. Adding new trackers to the cluster here will decrease the amount of tokens a given tracker instance controls and therefore reduce the data controlled. For reference see https://docs.datastax.com/en/cassandra/3.x/cassandra/architecture/archDataDistributeReplication.html

When I am talking about static, I'm talking about each Chihaya configuration file have a defined set of tokens stored by each tracker, e.g. Tracker1 stores token 0 to 127. Here you have full control over replication and can adjust what a given tracker stores. This one is excellent for the usecase @rarbg has because you can just add more 100% token nodes.

from chihaya.

@JohnDoee I think I explained that poorly. What I was talking about is something like this:

I'm generally working with the idea of an async, batch-like, add-or-update-only, but full replication of peer data, so we don't need to worry about what part(s) of the whole peer data is stored on a node.

Suppose we do develop our own synchronization for the peer store. Suppose we have n machines where n will not change. We can then just provide each node with n-1 addresses of all other nodes (or with two, creating a ring-like structure). The important part is: We know all nodes' addresses when we start the cluster. That's what I was referring to as a static cluster.

Now suppose we don't know all other nodes' addresses when we start the cluster: Now we need some way of discovering the other nodes. @jzelinskie's idea of using (if present) the load balancer for this works, assuming the load balancer knows about all nodes. We could implement this in various other ways, but the point is: We allow adding and removing nodes from the cluster at runtime. This is what I was referring to as a dynamic cluster.

Now my point was this: If we implement it in a way that we can discover new nodes at runtime, we also automatically support static clusters.

from chihaya.

Hello everyone!
I have good news to report: I have written a peer store that replicates its data to a set of other nodes periodically.

It works like this:

• Each node listens on some port via TCP
• Each node has a list of other nodes to replicate to (config)
• Each node does, after a timer (config) elapses:
  • Open TCP connections to all reachable nodes in the list of nodes to replicate to
  • Iterate over all infohashes and take a certain amount (config) and marshal them into a package (not a network packet, just a bunch of bytes). This requires an RLock on one shard for each batch.
  • Send that package to connected clients
  • Repeat until all infohashes have been sent
• If a node receives an incoming connection:
  • Receive package
  • Decode package, for each infohash: (This requires a Lock on one shard for each infohash)
    • If leechers: add-or-update own leechers
    • If seeders: add-or-update own seeders and remove eventual leechers
  • Repeat until all packages have been received

A few things to note:
The main issues are network speed (see below) and lock contention. What does that mean?
The peer store is divided into a bunch of shards that equally divide the key space of all infohashes. A certain infohash has exactly one shard where it is stored. When creating packages for replication, we do the following:

• RLock the shard
• Collect all infohashes tracked by this shard (this temporarily creates a somewhat-large-ish array in-memory, but that's typically not too bad). We could, theoretically, iterate over the shard and obtain an RLock for every iteration, but that creates some other things to worry about.
• Release RLock, schedule other goroutines
• Take a bunch (one batch size) of the infohashes and
• RLock the shard
• Collect peer data for all these infohashes and marshal them to wire format
• RUnlock the shard
• Schedule other goroutines, then continue with more infohashes

I have included timings and measurements for how long this takes in a real-world scenario. The big issue, as mentioned before, is lock contention. We hold an RLock for the amount of time it takes to collect peer data for one batch of infohashes. We could subdivide this and process a batch in parts, but then again we can just configure batches to be smaller.

But don't worry! Luckily we don't have to RLock the whole peer store! We just RLock one shard. So if we divide our key space into more shards, lock contention becomes less of an issue.

Receiving a replication package has a similar issue, but this time we have to Lock, e.g. obtain an exclusive write-lock, one shard per infohash. This isn't as bad as the whole batch when sending.

By giving the user the choice of defining a set of nodes to replicate to, the user decides the topology of the network. That could be a ring, a mesh or something else entirely.

I got my hands on a bunch of beefy machines and performed a test with two of them. One generated a bunch of random infohashes and populated them with a bunch of random peers, the other was empty.

Some general data:
2500 infohashes with 10k IPv4 peers each (5k seeders + 5k leechers) = 25M peers take about 14 to 15 GB of memory. (see edit at the end)

Serializing 100 infohashes with 5k IPv4 peers each yields a package of approx. 40 MiB, or about 85 bytes per peer, and takes about a second. The size per peer goes down the more peers a single infohash has and goes up vice versa. I see some room for improvement here, as IPv4 peers could be as small as 26 bytes. To support IPv4 and IPv6 we have to include at least one marker byte. Maybe just using a stream compression helps to reduce this. The current solution uses protocol buffers that are written in a readable way. It is actually remarkable that we can encode 1M peers in one second and I'm very happy with that.

Garbage-collecting 25M peers only takes a few seconds. This didn't delete any peers or infohashes, but it's good to know that checking 25M peers takes a few seconds on this hardware, which is very fast if you ask me.

The tests were performed on two identical machines with this hardware, connected via 1G ethernet:
RAM: 32GB DDR4 @ 2667 MHz
CPU: i7 5820K
MoBo: ASUS Sabertooth X99
NIC: Onboard, Intel Corporation Ethernet Connection (2) I218-V (rev 05)

Nothing apart from chihaya was running on the servers during these tests. Also no operations on the peer store were performed during these tests, i.e. lock contention effects did not exist.

These are the configurations used:

"Generating" node:

peer_store:
  name: synced
  config:
    gc_interval: 5m
    gc_cutoff: 15m
    shards: 3
    sync_every: 2m
    sync_addr: 0.0.0.0:6263
    batch_size: 100
    generate: true #testing-related
    generate_infohashes: 2500 #testing-related
    generate_peers: 10000 #testing-related
    neighbours:
      - 10.150.7.2:6262

Empty node:

peer_store:
  name: synced
  config:
    gc_interval: 5m
    gc_cutoff: 15m
    shards: 4
    sync_every: 2m
    sync_addr: 0.0.0.0:6262
    batch_size: 100
    neighbours:
      - 10.150.7.6:6263

These are the logs of the current state:

"Generating" node:

2016/06/29 04:31:03 Store API: Registered: PUT /clients/:client
2016/06/29 04:31:03 Store API: Registered: DELETE /clients/:client
2016/06/29 04:31:03 Store API: Registered: GET /clients/:client
2016/06/29 04:31:03 Store API: Registered: PUT /infohashes/*infohash
2016/06/29 04:31:03 Store API: Registered: DELETE /infohashes/*infohash
2016/06/29 04:31:03 Store API: Registered: GET /infohashes/*infohash
2016/06/29 04:31:03 synced: generating 2500 infohashes with 10000 seeders and leechers each
2016/06/29 04:32:13 synced: waiting for next sync...
2016/06/29 04:32:13 synced: listening on [::]:6263
2016/06/29 04:34:13 synced: starting synchronization. GCing first...
2016/06/29 04:34:13 synced: collecting garbage. Cutoff time: 2016-06-29 04:19:13.686214618 +0000 UTC
2016/06/29 04:34:18 synced: attempting to connect to neighbors
2016/06/29 04:34:18 synced: connecting to 10.150.7.2:6262
2016/06/29 04:34:18 synced: connected to 10.150.7.2:6262
2016/06/29 04:34:18 synced: connected to 1 nodes. Starting sync...
2016/06/29 04:34:18 synced: marshaled a packet with 100 infohashes to 42683700 bytes
2016/06/29 04:34:19 synced: marshaled a packet with 100 infohashes to 42684114 bytes
2016/06/29 04:34:20 synced: marshaled a packet with 100 infohashes to 42682429 bytes
2016/06/29 04:34:21 synced: marshaled a packet with 100 infohashes to 42683758 bytes
2016/06/29 04:34:23 synced: marshaled a packet with 100 infohashes to 42683786 bytes
2016/06/29 04:34:24 synced: marshaled a packet with 100 infohashes to 42683406 bytes
2016/06/29 04:34:26 synced: marshaled a packet with 100 infohashes to 42683055 bytes
2016/06/29 04:34:27 synced: marshaled a packet with 100 infohashes to 42683724 bytes
2016/06/29 04:34:28 synced: marshaled a packet with 100 infohashes to 42683693 bytes
2016/06/29 04:34:29 synced: marshaled a packet with 100 infohashes to 42684145 bytes
2016/06/29 04:34:30 synced: marshaled a packet with 100 infohashes to 42682769 bytes
2016/06/29 04:34:31 synced: marshaled a packet with 100 infohashes to 42683236 bytes
2016/06/29 04:34:33 synced: marshaled a packet with 100 infohashes to 42684344 bytes
2016/06/29 04:34:34 synced: marshaled a packet with 100 infohashes to 42682901 bytes
2016/06/29 04:34:36 synced: marshaled a packet with 100 infohashes to 42682971 bytes
2016/06/29 04:34:37 synced: marshaled a packet with 100 infohashes to 42683260 bytes
2016/06/29 04:34:38 synced: marshaled a packet with 96 infohashes to 40975957 bytes
2016/06/29 04:34:39 synced: marshaled a packet with 100 infohashes to 42683810 bytes
2016/06/29 04:34:40 synced: marshaled a packet with 100 infohashes to 42683482 bytes
2016/06/29 04:34:41 synced: marshaled a packet with 100 infohashes to 42683234 bytes
2016/06/29 04:34:43 synced: marshaled a packet with 100 infohashes to 42683973 bytes
2016/06/29 04:34:44 synced: marshaled a packet with 100 infohashes to 42683113 bytes
2016/06/29 04:34:45 synced: marshaled a packet with 100 infohashes to 42683747 bytes
2016/06/29 04:34:46 synced: marshaled a packet with 100 infohashes to 42683387 bytes
2016/06/29 04:34:48 synced: marshaled a packet with 100 infohashes to 42683120 bytes
2016/06/29 04:34:49 synced: marshaled a packet with 100 infohashes to 42684125 bytes
2016/06/29 04:34:51 synced: marshaled a packet with 100 infohashes to 42683998 bytes
2016/06/29 04:34:52 synced: marshaled a packet with 100 infohashes to 42682532 bytes
2016/06/29 04:34:53 synced: marshaled a packet with 100 infohashes to 42683872 bytes
2016/06/29 04:34:54 synced: marshaled a packet with 100 infohashes to 42683145 bytes
2016/06/29 04:34:55 synced: marshaled a packet with 100 infohashes to 42683957 bytes
2016/06/29 04:34:56 synced: marshaled a packet with 100 infohashes to 42684102 bytes
2016/06/29 04:34:57 synced: marshaled a packet with 100 infohashes to 42684048 bytes
2016/06/29 04:34:58 synced: marshaled a packet with 42 infohashes to 17927701 bytes
2016/06/29 04:34:59 synced: marshaled a packet with 100 infohashes to 42683785 bytes
2016/06/29 04:35:00 synced: marshaled a packet with 100 infohashes to 42683743 bytes
2016/06/29 04:35:01 synced: marshaled a packet with 100 infohashes to 42683481 bytes
2016/06/29 04:35:04 synced: marshaled a packet with 100 infohashes to 42682988 bytes
2016/06/29 04:35:05 synced: marshaled a packet with 100 infohashes to 42683280 bytes
2016/06/29 04:35:06 synced: marshaled a packet with 100 infohashes to 42683837 bytes
2016/06/29 04:35:07 synced: marshaled a packet with 100 infohashes to 42683179 bytes
2016/06/29 04:35:08 synced: marshaled a packet with 100 infohashes to 42683568 bytes
2016/06/29 04:35:09 synced: marshaled a packet with 100 infohashes to 42682969 bytes
2016/06/29 04:35:10 synced: marshaled a packet with 100 infohashes to 42684000 bytes
2016/06/29 04:35:11 synced: marshaled a packet with 100 infohashes to 42684037 bytes
2016/06/29 04:35:14 synced: marshaled a packet with 100 infohashes to 42682148 bytes
2016/06/29 04:35:15 synced: marshaled a packet with 100 infohashes to 42683605 bytes
2016/06/29 04:35:16 synced: marshaled a packet with 100 infohashes to 42683451 bytes
2016/06/29 04:35:17 synced: marshaled a packet with 100 infohashes to 42683131 bytes
2016/06/29 04:35:21 synced: marshaled a packet with 100 infohashes to 42683284 bytes
2016/06/29 04:35:22 synced: marshaled a packet with 62 infohashes to 26463495 bytes
2016/06/29 04:35:23 synced: sync finished
2016/06/29 04:35:23 synced: waiting for next sync...
2016/06/29 04:35:26 synced: incoming sync from 10.150.7.2:40144
2016/06/29 04:35:27 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:35:28 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:35:29 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:35:30 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:35:31 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:35:32 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:35:33 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:35:34 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:35:35 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:35:36 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:35:37 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:35:38 synced: received sync packet with 86 infohashes from 10.150.7.2:40144
2016/06/29 04:35:39 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:35:40 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:35:41 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:35:42 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:35:43 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:35:44 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:35:45 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:35:47 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:35:48 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:35:49 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:35:50 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:35:51 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:35:52 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:35:53 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:35:54 synced: received sync packet with 26 infohashes from 10.150.7.2:40144
2016/06/29 04:35:55 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:35:56 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:35:57 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:35:58 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:35:59 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:36:00 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:36:01 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:36:02 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:36:03 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:36:04 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:36:05 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:36:06 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:36:08 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:36:08 synced: received sync packet with 8 infohashes from 10.150.7.2:40144
2016/06/29 04:36:09 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:36:10 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:36:11 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:36:12 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:36:13 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:36:14 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:36:15 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:36:16 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:36:17 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:36:18 synced: received sync packet with 100 infohashes from 10.150.7.2:40144
2016/06/29 04:36:19 synced: received sync packet with 80 infohashes from 10.150.7.2:40144
2016/06/29 04:36:20 synced: received EOF from 10.150.7.2:40144 - probably finished
2016/06/29 04:36:20 synced: incoming sync from 10.150.7.2:40144 finished, collecting garbage...
2016/06/29 04:36:20 synced: collecting garbage. Cutoff time: 2016-06-29 04:21:20.004022001 +0000 UTC
2016/06/29 04:36:21 synced: finished collecting garbage after incoming sync
^C2016/06/29 04:36:26 synced: unable to accept connection: accept tcp [::]:6263: use of closed network connection
2016/06/29 04:36:26 shutdown receive loop
2016/06/29 04:36:26 shutdown sync loop
2016/06/29 04:36:26 PeerStore: shutdown GC
2016/06/29 04:36:26 Store server shut down cleanly

Empty node:

2016/06/29 04:34:12 Store API: Registered: PUT /clients/:client
2016/06/29 04:34:12 Store API: Registered: DELETE /clients/:client
2016/06/29 04:34:12 Store API: Registered: GET /clients/:client
2016/06/29 04:34:12 Store API: Registered: PUT /infohashes/*infohash
2016/06/29 04:34:12 Store API: Registered: DELETE /infohashes/*infohash
2016/06/29 04:34:12 Store API: Registered: GET /infohashes/*infohash
2016/06/29 04:34:12 synced: waiting for next sync...
2016/06/29 04:34:12 synced: listening on [::]:6262
2016/06/29 04:35:05 synced: incoming sync from 10.150.7.6:55860
2016/06/29 04:35:06 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:08 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:09 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:11 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:12 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:13 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:14 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:15 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:17 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:18 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:19 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:20 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:22 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:23 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:24 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:26 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:27 synced: received sync packet with 96 infohashes from 10.150.7.6:55860
2016/06/29 04:35:28 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:29 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:31 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:32 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:33 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:34 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:36 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:37 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:38 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:39 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:40 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:41 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:43 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:44 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:45 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:46 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:47 synced: received sync packet with 42 infohashes from 10.150.7.6:55860
2016/06/29 04:35:47 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:49 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:51 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:52 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:53 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:54 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:55 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:56 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:35:57 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:36:01 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:36:02 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:36:03 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:36:04 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:36:05 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:36:06 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:36:09 synced: received sync packet with 100 infohashes from 10.150.7.6:55860
2016/06/29 04:36:10 synced: received sync packet with 62 infohashes from 10.150.7.6:55860
2016/06/29 04:36:10 synced: received EOF from 10.150.7.6:55860 - probably finished
2016/06/29 04:36:10 synced: incoming sync from 10.150.7.6:55860 finished, collecting garbage...
2016/06/29 04:36:10 synced: collecting garbage. Cutoff time: 2016-06-29 04:21:10.794645918 +0000 UTC
2016/06/29 04:36:12 synced: finished collecting garbage after incoming sync
2016/06/29 04:36:12 synced: starting synchronization. GCing first...
2016/06/29 04:36:12 synced: collecting garbage. Cutoff time: 2016-06-29 04:21:12.381844026 +0000 UTC
2016/06/29 04:36:13 synced: attempting to connect to neighbors
2016/06/29 04:36:13 synced: connecting to 10.150.7.6:6263
2016/06/29 04:36:13 synced: connected to 10.150.7.6:6263
2016/06/29 04:36:13 synced: connected to 1 nodes. Starting sync...
2016/06/29 04:36:14 synced: marshaled a packet with 100 infohashes to 42683741 bytes
2016/06/29 04:36:14 synced: marshaled a packet with 100 infohashes to 42683785 bytes
2016/06/29 04:36:15 synced: marshaled a packet with 100 infohashes to 42684153 bytes
2016/06/29 04:36:16 synced: marshaled a packet with 100 infohashes to 42683395 bytes
2016/06/29 04:36:17 synced: marshaled a packet with 100 infohashes to 42683015 bytes
2016/06/29 04:36:19 synced: marshaled a packet with 100 infohashes to 42683441 bytes
2016/06/29 04:36:19 synced: marshaled a packet with 100 infohashes to 42684087 bytes
2016/06/29 04:36:20 synced: marshaled a packet with 100 infohashes to 42684082 bytes
2016/06/29 04:36:21 synced: marshaled a packet with 100 infohashes to 42683569 bytes
2016/06/29 04:36:22 synced: marshaled a packet with 100 infohashes to 42683485 bytes
2016/06/29 04:36:24 synced: marshaled a packet with 100 infohashes to 42682791 bytes
2016/06/29 04:36:24 synced: marshaled a packet with 86 infohashes to 36707609 bytes
2016/06/29 04:36:25 synced: marshaled a packet with 100 infohashes to 42683083 bytes
2016/06/29 04:36:26 synced: marshaled a packet with 100 infohashes to 42683362 bytes
2016/06/29 04:36:27 synced: marshaled a packet with 100 infohashes to 42683841 bytes
2016/06/29 04:36:28 synced: marshaled a packet with 100 infohashes to 42682926 bytes
2016/06/29 04:36:29 synced: marshaled a packet with 100 infohashes to 42683217 bytes
2016/06/29 04:36:30 synced: marshaled a packet with 100 infohashes to 42683522 bytes
2016/06/29 04:36:31 synced: marshaled a packet with 100 infohashes to 42682739 bytes
2016/06/29 04:36:33 synced: marshaled a packet with 100 infohashes to 42683682 bytes
2016/06/29 04:36:34 synced: marshaled a packet with 100 infohashes to 42683427 bytes
2016/06/29 04:36:35 synced: marshaled a packet with 100 infohashes to 42683166 bytes
2016/06/29 04:36:36 synced: marshaled a packet with 100 infohashes to 42683606 bytes
2016/06/29 04:36:37 synced: marshaled a packet with 100 infohashes to 42683232 bytes
2016/06/29 04:36:38 synced: marshaled a packet with 100 infohashes to 42683307 bytes
2016/06/29 04:36:39 synced: marshaled a packet with 100 infohashes to 42682889 bytes
2016/06/29 04:36:40 synced: marshaled a packet with 26 infohashes to 11097728 bytes
2016/06/29 04:36:41 synced: marshaled a packet with 100 infohashes to 42682820 bytes
2016/06/29 04:36:42 synced: marshaled a packet with 100 infohashes to 42683926 bytes
2016/06/29 04:36:43 synced: marshaled a packet with 100 infohashes to 42683592 bytes
2016/06/29 04:36:44 synced: marshaled a packet with 100 infohashes to 42683542 bytes
2016/06/29 04:36:45 synced: marshaled a packet with 100 infohashes to 42683575 bytes
2016/06/29 04:36:46 synced: marshaled a packet with 100 infohashes to 42683812 bytes
2016/06/29 04:36:47 synced: marshaled a packet with 100 infohashes to 42683964 bytes
2016/06/29 04:36:48 synced: marshaled a packet with 100 infohashes to 42683516 bytes
2016/06/29 04:36:49 synced: marshaled a packet with 100 infohashes to 42683143 bytes
2016/06/29 04:36:50 synced: marshaled a packet with 100 infohashes to 42684410 bytes
2016/06/29 04:36:51 synced: marshaled a packet with 100 infohashes to 42683747 bytes
2016/06/29 04:36:52 synced: marshaled a packet with 100 infohashes to 42682827 bytes
2016/06/29 04:36:53 synced: marshaled a packet with 100 infohashes to 42684078 bytes
2016/06/29 04:36:54 synced: marshaled a packet with 8 infohashes to 3414641 bytes
2016/06/29 04:36:54 synced: marshaled a packet with 100 infohashes to 42683119 bytes
2016/06/29 04:36:56 synced: marshaled a packet with 100 infohashes to 42683571 bytes
2016/06/29 04:36:57 synced: marshaled a packet with 100 infohashes to 42683535 bytes
2016/06/29 04:36:58 synced: marshaled a packet with 100 infohashes to 42684437 bytes
2016/06/29 04:36:59 synced: marshaled a packet with 100 infohashes to 42683622 bytes
2016/06/29 04:37:00 synced: marshaled a packet with 100 infohashes to 42683334 bytes
2016/06/29 04:37:01 synced: marshaled a packet with 100 infohashes to 42683548 bytes
2016/06/29 04:37:02 synced: marshaled a packet with 100 infohashes to 42683473 bytes
2016/06/29 04:37:03 synced: marshaled a packet with 100 infohashes to 42682949 bytes
2016/06/29 04:37:04 synced: marshaled a packet with 100 infohashes to 42683725 bytes
2016/06/29 04:37:05 synced: marshaled a packet with 80 infohashes to 34146789 bytes
2016/06/29 04:37:06 synced: sync finished
2016/06/29 04:37:06 synced: waiting for next sync...
^C2016/06/29 04:37:18 synced: unable to accept connection: accept tcp [::]:6262: use of closed network connection
2016/06/29 04:37:18 shutdown receive loop
2016/06/29 04:37:18 shutdown sync loop
2016/06/29 04:37:18 PeerStore: shutdown GC
2016/06/29 04:37:18 Store server shut down cleanly

What I need now is @rarbg to provide me with a few real-world numbers. How many infohashes do you track? How many peers on average does each infohash have? What's the maximum/minimum amount of peers per infohash? How many announces do you get per second on average? How many of those are seeders/leechers?

Generally: Please comment on the idea and the numbers. The implementation still has a few rough edges and is not open source yet until I figured some more things out.

Ping @jzelinskie @pushrax @JohnDoee

EDIT: After some calculations the 14 GB for 25M peers seems horrible. I already have a few ideas on how to reduce this significantly.

from chihaya.

@mrd0ll4r

1. Can you downclock your memory to 2133 ? There arent going to be any servers running on higher than that clock in the upcoming ~6-12 months. Not until next-gen intel. Your GC depends on memory performance a lot , so if you can test that it will be great
2. Can you also test the GC with actual deletion of hashes/peers on performance ? GC should run every 120 seconds else the tracker will run out of memory on a 32gb ram in 4 hours
3. IPV6 - i dont think any respectable public tracker will ever start using ipv6 due to dmca agencies , so i cant comment on that
4. Real numbers
   hashes - 30 mil
   peers total - 50 mil
   average - :D 50/30 but thats not a reliable number...
   There are torrents like game of thrones with 500k peers and there are torrents with 1 peer
   announces per second - average 23k/s since start, peaks/burst at 80k-85k/s
   seeders/leechers = 62%/37%
5. Idea - i have no use for any of what you implemented , as i said multiple times :D

from chihaya.

About 1 and 2: I might be able to do something tomorrow.
About 3: What's the deal with IPv6 and DMCA?
About 4: I'll refine my tests...
About 5: May I ask what you're currently running? Opentracker or something homegrown?

from chihaya.

@mrd0ll4r
3 ipv6 has 3.4×10^38 of addresses! they can easily monitor peers without you able to stop them or rate-limit them , get it ?
5 homegrown , if you want i can list again the features i actually need

from chihaya.

General update: I had my numbers wrong yesterday. I was testing with 2500 infohashes, 10k seeders and 10k leechers each, i.e. 50M peers in total. Those took 14 to 15 GB yesterday.

I improved the way we store peers, memory usage for 50M peers is now at ~6GB. Operations on the peer store are also about 20%-30% faster in the benchmarks. I also improved the wire format, takes about 39 bytes for an IPv4 peer now.

@rarbg huh, never thought about IPv6 like that, but makes sense. Yes, please list the features again! Open a new issue unless they are related to clustering please :)

I also did a GC of all 50M peers a few times, takes anywhere between 2 and 10 seconds without other load. This is still on 2667 MHz memory, not sure if I get to downclock it today.

from chihaya.

@mrd0ll4r hehe no point of opening a new issue , considering nothing was done for probably more than 1 year on what i can use from chihaya , i was just asking if you want to know , i dont expect chihaya to be compatible for public use anytime soon :D

from chihaya.

I'm sorry to hear that, but yeah I'd love to know! Still, please either open another issue or join us on IRC (
#chihaya on freenode) to discuss things - I don't want to clutter this issue.

from chihaya.