Comments (11)
That’s odd, if the nounce difference between the last and the most recent message is different it would trigger a resync. Can you try comparing using an inspector instead of the what they show as the current orders. Sometimes there is a slowdown in the connection from the browser and an order passes during that and doesn’t get shown while it’s shown in the socket.
from python-bittrex-websocket.
Another solution is to set the order book depth to a higher number, I think it’s currently 10 or 20. Try with 100 and report back. Check the documentation to see how that is done. Thanks!
from python-bittrex-websocket.
tried both, it looks like the problem arises when enough order book entries are filled/deleted and there's not enough data points due to the orderbook depth to fill the freed up spots so they get filled in by new data points but there are other data points missing in between
when using the default depth the orderbook falls out of sync faster (5-10 minutes), when I set the order book depth to 9999 it took about 6-7 hours (probably also highly depends on the volatility of the pair)
Is there a way to hold the full orderbook?
I also looked at the inspector data, the nounces seemed to go in order, no holes, no delays and as far as i could see data from them was incorporated correctly, it really looked like the problem is always the missing order_book_depth+1 data point and the hole in data propagates to the top of the orderbook in time
what did seem strange though was that when I let run both orderbook and orderbook_update the nounces for the orderbook update messages were not the same as those of orderbook. The orderbook nounces where the same that I saw through the inspector. The orderbook_update nounces were higher, the timestamp was ok so this was no delay. For example
Tue Jan 30 16:23:36 2018 : orderbook_update msg = {'ticker': 'BTC-ETH', 'nounce': 1678116, 'timestamp': 1517325816.700767, 'bids': [{'Type': 1, 'Rate': 0.10682001, ' ....
Tue Jan 30 16:23:37 2018 : orderbook msg = {'MarketName': 'BTC-ETH', 'Nounce': 1685995, 'Buys': [{'Quantity': 0.89374148, 'Rate': 0.10745493}, {'Quantity': 0.103 ...... 'Rate': 0.11382648}], 'timestamp': 1517325817.122321}
from python-bittrex-websocket.
tried both, it looks like the problem arises when enough order book entries are filled/deleted and there's not enough data points due to the orderbook depth to fill the freed up spots so they get filled in by new data points but there are other data points missing in between
when using the default depth the orderbook falls out of sync faster (5-10 minutes), when I set the order book depth to 9999 it took about 6-7 hours (probably also highly depends on the volatility of the pair)
Is there a way to hold the full orderbook?
Yes, remove the index, i.e omit the [0:index]
part in _sync_order_book
. This is the part:
if made_change:
# Sort by price, with respect to BUY(desc) or SELL(asc)
self.order_books[ticker][side[0]] = sorted(
self.order_books[ticker][side[0]],
key=lambda k: k['Rate'],
reverse=side[1])
# Put depth to 10
self.order_books[ticker][side[0]] = \
self.order_books[ticker][side[0]][
0:book_depth]
# Add nounce unix timestamp
self.order_books[ticker]['timestamp'] = time()
The main reason I put an index initially was that if we are syncing a lot of order books, the dict that holds the order book would get big and slow down performance. So perhaps we can switch to some other container?
I also looked at the inspector data, the nounces seemed to go in order, no holes, no delays and as far as i could see data from them was incorporated correctly, it really looked like the problem is always the missing order_book_depth+1 data point and the hole in data propagates to the top of the orderbook in time
Let's see how it goes without the index part.
what did seem strange though was that when I let run both orderbook and orderbook_update the nounces for the orderbook update messages were not the same as those of orderbook. The orderbook nounces where the same that I saw through the inspector. The orderbook_update nounces were higher, the timestamp was ok so this was no delay. For example
Tue Jan 30 16:23:36 2018 : orderbook_update msg = {'ticker': 'BTC-ETH', 'nounce': 1678116, 'timestamp': 1517325816.700767, 'bids': [{'Type': 1, 'Rate': 0.10682001, ' ....
Tue Jan 30 16:23:37 2018 : orderbook msg = {'MarketName': 'BTC-ETH', 'Nounce': 1685995, 'Buys': [{'Quantity': 0.89374148, 'Rate': 0.10745493}, {'Quantity': 0.103 ...... 'Rate': 0.11382648}], 'timestamp': 1517325817.122321}
This is not a bug. Most certainly orderbook
and orderbook_update
are made through connections from different threads. In general each connection gets a different nounce for the specific ticker. If you think about it, if the difference was more than 1, it would trigger a resync as per the code logic.
As a whole the code logic for the order book sync needs lots of improvements, the good thing is that I know what the problems are, the bad thing is that I lack the time :).
Anyways, thanks for helping and contributing!
from python-bittrex-websocket.
thanks a lot, removing the index helped now as long as I am getting data its in sync
now I am testing it by subscribing to all tickers and it seems to work fine for a while, not sure if this is connected to the performance dip because of holding more data in memory but after some times (usually hours) I do get an error message and the orderbook stops being updated
the error messages look like this:
Traceback (most recent call last):
File "D:\Python\lib\site-packages\gevent\greenlet.py", line 536, in run
result = self._run(*self.args, **self.kwargs)
File "D:\Python\lib\site-packages\signalr_connection.py", line 53, in wrapped_listener
listener()
File "D:\Python\lib\site-packages\signalr\transports_ws_transport.py", line 42, in receive
for notification in self.ws:
File "D:\Python\lib\site-packages\websocket_core.py", line 107, in iter
yield self.recv()
File "D:\Python\lib\site-packages\websocket_core.py", line 296, in recv
opcode, data = self.recv_data()
File "D:\Python\lib\site-packages\websocket_core.py", line 313, in recv_data
opcode, frame = self.recv_data_frame(control_frame)
File "D:\Python\lib\site-packages\websocket_core.py", line 326, in recv_data_frame
frame = self.recv_frame()
File "D:\Python\lib\site-packages\websocket_core.py", line 360, in recv_frame
return self.frame_buffer.recv_frame()
File "D:\Python\lib\site-packages\websocket_abnf.py", line 340, in recv_frame
self.recv_header()
File "D:\Python\lib\site-packages\websocket_abnf.py", line 288, in recv_header
header = self.recv_strict(2)
File "D:\Python\lib\site-packages\websocket_abnf.py", line 375, in recv_strict
bytes = self.recv(min(16384, shortage))
File "D:\Python\lib\site-packages\websocket_core.py", line 430, in _recv
return recv(self.sock, bufsize)
File "D:\Python\lib\site-packages\websocket_socket.py", line 94, in recv
"Connection is already closed.")
websocket._exceptions.WebSocketConnectionClosedException: Connection is already closed.
Mon Feb 5 18:19:13 2018 <Greenlet at 0x2463301d5a0: wrapped_listener> failed with WebSocketConnectionClosedException
or this
Traceback (most recent call last):
File "D:\Python\lib\site-packages\gevent\greenlet.py", line 536, in run
result = self._run(*self.args, **self.kwargs)
File "D:\Python\lib\site-packages\signalr_connection.py", line 53, in wrapped_listener
listener()
File "D:\Python\lib\site-packages\signalr\transports_ws_transport.py", line 42, in receive
for notification in self.ws:
File "D:\Python\lib\site-packages\websocket_core.py", line 107, in iter
yield self.recv()
File "D:\Python\lib\site-packages\websocket_core.py", line 296, in recv
opcode, data = self.recv_data()
File "D:\Python\lib\site-packages\websocket_core.py", line 313, in recv_data
opcode, frame = self.recv_data_frame(control_frame)
File "D:\Python\lib\site-packages\websocket_core.py", line 326, in recv_data_frame
frame = self.recv_frame()
File "D:\Python\lib\site-packages\websocket_core.py", line 360, in recv_frame
return self.frame_buffer.recv_frame()
File "D:\Python\lib\site-packages\websocket_abnf.py", line 340, in recv_frame
self.recv_header()
File "D:\Python\lib\site-packages\websocket_abnf.py", line 288, in recv_header
header = self.recv_strict(2)
File "D:\Python\lib\site-packages\websocket_abnf.py", line 375, in recv_strict
bytes = self.recv(min(16384, shortage))
File "D:\Python\lib\site-packages\websocket_core.py", line 430, in recv
return recv(self.sock, bufsize)
File "D:\Python\lib\site-packages\websocket_socket.py", line 81, in recv
bytes = sock.recv(bufsize)
File "D:\Python\lib\site-packages\gevent_ssl3.py", line 457, in recv
return self.read(buflen)
File "D:\Python\lib\site-packages\gevent_ssl3.py", line 293, in read
return self._sslobj.read(len or 1024)
BlockingIOError: [WinError 10035] A non-blocking socket operation could not be completed immediately
Mon Feb 5 19:18:55 2018 <Greenlet at 0x2271c34d898: wrapped_listener> failed with BlockingIOError
or this
Exception in thread Thread-3:
Traceback (most recent call last):
File "D:\Eclipse\workspace\HuBot 3.0\src\bittrex_websocket\websocket_client.py", line 321, in _init_connection
conn.start()
File "D:\Python\lib\site-packages\signalr_connection.py", line 50, in start
listener = self.__transport.start()
File "D:\Python\lib\site-packages\signalr\transports_auto_transport.py", line 28, in start
return self.__transport.start()
File "D:\Python\lib\site-packages\signalr\transports_ws_transport.py", line 38, in start
enable_multithread=True)
File "D:\Python\lib\site-packages\websocket_core.py", line 490, in create_connection
websock.connect(url, **options)
File "D:\Python\lib\site-packages\websocket_core.py", line 216, in connect
self.handshake_response = handshake(self.sock, *addrs, **options)
File "D:\Python\lib\site-packages\websocket_handshake.py", line 69, in handshake
status, resp = _get_resp_headers(sock)
File "D:\Python\lib\site-packages\websocket_handshake.py", line 135, in _get_resp_headers
raise WebSocketBadStatusException("Handshake status %d", status)
websocket._exceptions.WebSocketBadStatusException: Handshake status 504During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "D:\Python\lib\threading.py", line 916, in _bootstrap_inner
self.run()
File "D:\Python\lib\threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "D:\Eclipse\workspace\HuBot 3.0\src\bittrex_websocket\websocket_client.py", line 349, in _init_connection
raise ImportWarning(MSG_ERROR_SOCKET_WEBSOCKETBADSTATUS)
ImportWarning: Please report error to https://github.com/slazarov/python-bittrex-websocket, Error:WebSocketBadStatusException
instead of restarting it dies after these errors occur, I have not looked into them deeper yet but I plan to and will let you know what I find out
from python-bittrex-websocket.
I am under impression that there is no deletion going on, each call to on_orderbook delivers a new copy of orderbook in the msg?
from python-bittrex-websocket.
No.
from python-bittrex-websocket.
Apologies then, I'll go RTFS.
from python-bittrex-websocket.
Follow up question, if you don't mind me using the ticket for this: I have my own implementation of orderbooks and code that builds on them. Which do you think would be a better strategy:
- Wrap your orderbooks in an interface that behaves like my orderbooks
- Override your methods and write to my orderbook instances instead
- Load the entire orderbook contents from your instances to mine once in a while?
from python-bittrex-websocket.
Hi, would be a better strategy for what :)?
In my opinion do whatever is easier in order to get something working. The best strategy if you want to contribute is to work together on the project and improve it. I don’t have a lot of free time to work on it constantly so I am happy to implement better solutions. Best!
from python-bittrex-websocket.
Closing due to inactivity.
from python-bittrex-websocket.
Related Issues (20)
- QueryExchangeState Market Name is None. HOT 9
- Nonetype object not subscriptable HOT 1
- SubscribeToSummaryDeltas error HOT 2
- SSL Fail HOT 27
- Example not finished properly on Ubuntu HOT 4
- Are updates to Order Book provided in real-time or using a polling interval? HOT 3
- Authentication and trade executions HOT 2
- authenticate callback exception HOT 9
- SSL: CERTIFICATE_VERIFY_FAILED HOT 7
- SubscribeToSummaryDeltas read operation timed out
- ImportError: cannot import name 'six' HOT 1
- on monitoring new listings HOT 2
- Lastest price bug HOT 2
- Error messages not being thrown HOT 4
- Unhandled Exception <[Errno 104] Connection reset by peer> HOT 2
- Unhandled Exception Message = <'PassthroughUuid'> HOT 1
- AttributeError: 'ReconnectEvent' object has no attribute 'invoke' HOT 1
- Issue with IndexError after authenticate() HOT 1
- Auto-reconnection is disabled for unhandled exceptions when calling ws.query_summary_state()
- Websockets code to get tickers doesn't work anymore HOT 1
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from python-bittrex-websocket.