simplecrypto / powerpool Goto Github PK

Powerpool doesn't close the connection on clients that have hung...

https://github.com/simplecrypto/powerpool/blob/master/powerpool/manager.py#L206
https://github.com/simplecrypto/powerpool/blob/master/powerpool/stats.py#L21
https://github.com/simplecrypto/powerpool/blob/master/powerpool/stats.py#L38
https://github.com/simplecrypto/powerpool/blob/master/powerpool/stratum_server.py#L191

And then test carefully...

Would prevent all those unpayable balances.

Traceback (most recent call last):
File "/home/isaac/programming/powerpool/powerpool/netmon.py", line 32, in monitor_nodes
conn.getinfo()
File "/home/isaac/.virtualenvs/pp/local/lib/python2.7/site-packages/bitcoinrpc/authproxy.py", line 115, in call
'Content-type': 'application/json'})
File "/usr/lib/python2.7/httplib.py", line 973, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python2.7/httplib.py", line 1007, in _send_request
self.endheaders(body)
File "/usr/lib/python2.7/httplib.py", line 969, in endheaders
self._send_output(message_body)
File "/usr/lib/python2.7/httplib.py", line 829, in _send_output
self.send(msg)
File "/usr/lib/python2.7/httplib.py", line 791, in send
self.connect()
File "/usr/lib/python2.7/httplib.py", line 772, in connect
self.timeout, self.source_address)
File "/home/isaac/.virtualenvs/pp/local/lib/python2.7/site-packages/gevent/socket.py", line 591, in create_connection
raise err
error: [Errno 111] Connection refused
2014-02-22 12:21:48,951 [INFO] Couldn't connect to any RPC servers, sleeping for 1
2014-02-22 12:21:49,951 [INFO] Couldn't connect to any RPC servers, sleeping for 1
Traceback (most recent call last):
File "/home/isaac/.virtualenvs/pp/local/lib/python2.7/site-packages/gevent/greenlet.py", line 327, in run
result = self._run(_self.args, *_self.kwargs)
File "/home/isaac/programming/powerpool/powerpool/netmon.py", line 32, in monitor_nodes
conn.getinfo()
File "/home/isaac/.virtualenvs/pp/local/lib/python2.7/site-packages/bitcoinrpc/authproxy.py", line 115, in call
'Content-type': 'application/json'})
File "/usr/lib/python2.7/httplib.py", line 973, in request
self._send_request(method, url, body, headers)
File "/usr/lib/python2.7/httplib.py", line 1001, in _send_request
self.putrequest(method, url, **skips)
File "/usr/lib/python2.7/httplib.py", line 871, in putrequest
raise CannotSendRequest()
CannotSendRequest

Returned when failing a request, needs to catch the errror as a diconnected node and continue.

Most likely caused by this: zone117x/node-stratum-pool#9

2014-03-09 18:01:26,490 [stratum_server] [INFO] Recieving stratum connection from addr ('24.124.87.128', 38615) on sock
Traceback (most recent call last):
File "/home/dogecoin/venv/local/lib/python2.7/site-packages/gevent/event.py", line 114, in _notify_links
link(self)
File "/home/dogecoin/powerpool/powerpool/stratum_server.py", line 109, in new_block_call
push_job(flush=True)
File "/home/dogecoin/powerpool/powerpool/stratum_server.py", line 167, in push_job
fp.flush()
File "/usr/lib/python2.7/socket.py", line 303, in flush
self._sock.sendall(view[write_offset:write_offset+buffer_size])
File "/home/dogecoin/venv/local/lib/python2.7/site-packages/gevent/socket.py", line 458, in sendall
data_sent += self.send(_get_memory(data, data_sent), flags)
File "/home/dogecoin/venv/local/lib/python2.7/site-packages/gevent/socket.py", line 435, in send
return sock.send(data, flags)
error: [Errno 32] Broken pipe
(<function new_block_call at 0x2eea0c8>,
<gevent.event.Event object at 0x300dcd0>) failed with error

Check to confirm that all cleanup is being performed properly. This should still be fine tho.

404968-2014-03-30 11:06:16,853 [stratum_server] [INFO] Sending job id f5210000 to worker None.

this shouldn't happen. It's unlikely to be causing a problem, but should be fixed either way.

Currently the configuration options allow you to specify multiples, but only the first one is actually used...

Occasionally (0.2%) of the time we try to push a new block right when the main loop is using the socket. This causes new block notification to fail, thus increasing stale shares slightly. While efficiency is consistently 99%+ this should still be resolved. Will likely require rebuilding the main stratum loop to use read and write queues.

Est time 6+ hours.

This may also be indicative of a bleeding file descriptor problem with the server.

Dump from incident.
https://gist.github.com/icook/9885402#file-gistfile1-txt

Becomming disconnected from certain services should cause all clients to become disconnected until communication is restored. For example, loosing connection to all coinservers will cause no new block announcements to be sent out, silently generating 100% stale shares. It would be highly preferable to disconnect all peers so they can switch to a backup pool.

Not 100% sure why, but sometimes a worker will think the difficulty is different than it is and can be seen by submitting a bunch of low diff shares. Seems to happen very infrequently.

Likely a race condition in which a job is generated based on one difficulty from the new block notification command and at the same time vardiff pushes a new difficulty. If the new difficulty notification arrives out of order with the job the client will think it's the wrong diff. This should be solved by job submission locking vardiff calculation function. The other way around will make gevent grumpy because it adds a blocking operation to an event call.

Currently we count n1 accepted share to estimate hashrate, but it would be nice to keep track of shares/minute acceptance on a server.

Currently we implement vardiff on a per-connection level. This is somewhat undesirable since people with large numbers of workers start hitting DOS limits at 20 shares/second with like 10-15 workers. A better solution would be setting difficulty on a per-address or per-ip basis. This would scale much better as well.

This is a simple fix for all phantom connection issues.

Mainly RPC servers, although some other configuration values could easily be re-loaded as well.

When running multiple power pool servers only one should report the block notifications, therefore we need a way to disable them.

This would allow a mining server to become disconnected from rabbitmq and not loose any shares.

It seems that many sockets in 'CLOSE_WAIT' condition don't get removed from the client index for some reason, and client count will continue to grow for certain types of connections.

We don't want these to turn out to be wrong at block submission time, safer to check on startup

New aux work on multiple auxiliary chains can trigger a send of the same job multiple times, which is wasteful. It would be a good idea to record the last job and have async job pushes only send jobs that hadn't been sent.

Again, more bandwidth reduction.

http://www.simplevert.com/Vfmiz3ZVZfXFvpZTtsLnvHXJCRmsZiaVFH

Right now any pending stats on a worker are dropped when they disconnect, meaning the stats are off for workers that connect/disconnect a lot. Also means stats don't quite reflect actual shares.

Some batch submission times are too grouped if a server reset has happened recently which causes peaks and valleys in disk load, etc. Staggering the interval for batch submission would largely fix this.

Currently it's quite common for jobs to be sent out 3 times when a main network block is solved (once for each merge mined block...). This could be prevented somewhat by recording changes in last_gbt and merged values and only actually recomputing and pushing when needed.

This will allow you to view when blocks where solved back to x number of blocks. Very handy. Would be great for both merged and non-merged blocks.

Since pushing of new jobs is asynchronous then it would be a good idea to store a last sent time, and have the timeout refer to last sent time instead of blindly sending. This will save some bandwidth and avoid spamming new job notifs.

Since workers stats will take 24 hours to clear to zero they will show up on the status page for that long. Would be nice to be able to hide them or have them disappear sooner.

People are reporting that 'untracked share rejected from stratum' shouldn't be returned. Reasons for this are unknown, but should be investigated.

This is honestly a waste of time because scheduling overhead and no real advantages. In addition sending them all synchronously will allow easy benchmarking of transmission speed.