orderbook validity about btfxwss HOT 17 CLOSED

Hey there,
great to hear there's someone else working on the same problem (I'm working on a framework for a cross-platform API for bitcoin exchanges ).

I've ran into this issue as well - however, for both exchanges there is a solution available:

Bitfinex offers a configuration endpoint (for v2 of the WebSocket API). This allows you to enable sequencing, hence giving you an option to validate the correct order of messages.
It's not perfect (sequencing is local (it starts anew each session), as opposed to sequencing at Poloniex, which is global and allows you to compare websocket trades with REST Books), but it's certainly better than nothing.

Gemini offers timestamps in each payload (in both in ms and s since epoch). This can be used to ensure correctness of order, as the timestamps are created by the trading engine (not the API), as far as I was able to confirm.

Best of luck,

Nils

from btfxwss.

Hi Nils,

Thank you so much for your kind reply.
It is true that timestamp and event_id can help verify the order of the messages, but they still cannot tell dropped (missing) messages. GDAX does not have such problem because the sequence number is increased by 1 at each step. But timestamp and event_id do not offer such property. Any suggestions for identify missing messages?
BTW your cross-platform API implementation is AWESOME!

Blessings,
Seratna

from btfxwss.

Thanks, I'm glad you appreciate it! It still needs a lot of work, but I'm getting there.

The only approach would be cross-referencing your built order book via a REST API query to the order book endpoint - however, this requires a server-side timestamp on the book, which isn't alway present unfortunately. You would then have to check a snapshot of your built order book - which would have to reflect the state of the book at the REST query's timestamp - against this query.

So, in short, you will have to keep a history of messages and a built order book in memory to get it to work. But since this is regressive, it ultimately means that there is always a chance of missing an update message in your book. But this is the same for GDAX's sequence number (albeit their API is truly top-notch, and the number of out-of-order messages has been next to 0 for me).

from btfxwss.

Thanks Nils,

I guess cross-referencing is the best we could do now.
Currently we I did is to have a hash function to hash the local order book and store 50 hashed result as my order book history, and compare the hashed result of REST order book snap shot. If there's a discrepancy, I restart the websocket.
Regarding GDAX, I currently run it 24/7 and every one or two days I would have a out-of-order or missing message. I guess the point is not this won't happen, but is that when there's a missing/misplaced message, we can identify it and restart.
It's really nice talking to you. And I hope we have more chance to discuss these exchanges.

Blessings,
Seratna

from btfxwss.

keeping the last 50 hashes seems excessive, no ?
Afterall, once you notice it's out of place, all you would have to do is add all new messages older then the correct snapshot's time stamp. That way, you'd only ever have to keep, say, 1 snapshot and messages for the past 30 seconds or whatever request lag you have for the REST API (plus some buffer).

So, in Pseudo code this would be..

wss = WebSocketConnection()
wss.recv()  # Version number message

wss.subscribe()  # Subscribe to whatever book

 # Magic order book class that keeps track of your ob 
# and the last time it was updated
orderbook = OrderBook() 
orderbook.update(json.loads(wss.recv()))  # Snapshot

> In a separate Thread, keep track of the messages and add it to the message list

    wss_message_q = []
    while True:
        data = json.loads(wss.recv())
        wss_message_q.append(data)  # Store messages
        orderbook.update(data) 

> In another thread, every minute, compare the snapshot and if your own is out of sync? Discard
> the old one, use the new one, and check if there's a message in the q younger than the snapshot's 
> timestamp and add that.

    while time.time() - start < 60:
        # Now get a fresh snapshot from the REST API
       fresh_orderbook = requests.get(orderbook) 
       if fresh_orderbook != orderbook:
           orderbook = OrderBook(fresh_orderbook)
           for msg in wss_message_q:
               if msg.age > orderbook.age():
                   # Old, discard it
                  continue
               else:
                   orderbook.update(msg)
          # Reset the message list, since we're done
         wss_message_q = []

Of course for this to work you need to take into account thread-saftey etc., but ultimately this should work just as fine and be a bit more performant (I hope that code made sense to you ;D)

If you have further questions or discussions, feel free to do so!

from btfxwss.

Thank you very much!
I will have a look

from btfxwss.

How about writing to every exchange, that they should offer an optional ws orderbook, like the one from therocktrading:
https://api.therocktrading.com/doc/#api-websocket
"Get first 20 orderbook rows on order book changes".

This way you always have best 20 bids/asks without any problems. And usually this is enough to trade.
I already wrote to alot of exchanges, that they should add such a thing, but it seems I'm the only one requesting this...
If everyone would do this, no one would have this stupid problems anymore.
So please also write to them :)

from btfxwss.

What would be the point of that? You can get an order book from the REST API, or if you want it real-time you should build it yourself. Having a WS endpoint for this sort of thing does not make much sense to me.

from btfxwss.

I always need the latest ~20 bids asks to see if there is a good trading opportunity.
Most exchanges, like bitfinex have very low REST api call limit. If you want to trade every currency pair, it is just impossible to always make the REST API call, to stay up to date with the orderbooks.
I do not need the whole orderbook, just ~best 20. And for this it is the perfect solution.

from btfxwss.

@Serpens66 while this could be something nice to have, it is hard for the exchange to actually implement it. Some trader need best 20 bids asks, some need 100; for less liquid coins, some may need 200. That's why exchanges offer the whole L2 book.

@nlsdfnbch Hi Nils, I tried the Bitfinex websocket v2 with sequencing and it worked. And Gemini told me they are adding sequencing to the API but no ETA yet.

from btfxwss.

@Serpens66 Well, but how would you request it? A wss channel that broadcasts the top 20 on every order book change ? You essentially have that already, by building your own order book. It would simply be a convenience method, so I can understand that this is a low priority for API devs.
It's already implemented in the client as well.

@Seratna , thanks for the update!

from btfxwss.

yes, that is how it works at therocktrading.

The problem with update-orderbook is, that it is inconsistent after minutes/hours (I do a bid>ask test to find out). This happens for GDAX and Bitstamp quite often, although I already update the orderbook with REST API every 1 hour. With Cex ws this problem is only very rare, so it is the only exchange I use the realtime orderbook.
Therefore I even do not know, if it is a problem with connection (missing updates), or if it is a problem with my code. I tried my own code and also this one: https://github.com/danpaquin/gdax-python/blob/master/gdax/order_book.py
But both have same issue with bid>ask.
And if you trigger trades on a faulty orderbook, this can be deadly, especiall if you don't know it is faulty.

Bitfinex is now the next on my list to test, that's why I came here.
I will try your solution to update the orderbook. I only used lists for the orderbook yet, but I see you make it a dicitionary. This way it is much simpler to add and remove an entry (do not have to loop over entries to find the one to remove), so it sounds good.
I will also take a look at your code about other exchanges. Thanks for sharing.

But how do you convert this raw-orderbook dictionary into something useable? I mean how do you order it by price, since dictionaries are not ordered? If I request the orderbook ~every 1-2 seconds from my websocket script, and all the time I have to convert it to an ordered list, I can imagine this is not the right way to go?

from btfxwss.

@Serpens66 To (efficiently) get the bid/ask or the ordered prices from the order book, you need store the order book in a self-balancing tree (AVL tree or red-black tree) or in a priority queue. I am using priority queue. I am doing some final test on the implementations and I will share the code when it is finished.

from btfxwss.

@Serpens66 , have a look here. The Orders Class is what takes care of sorting the keys.

@Seratna , that's an interesting approach! Please do share- I'd love to implement it into this project if it provides better performance than my dict-based approach.

from btfxwss.

@Serpens66 @nlsdfnbch Here's my code of a working implementation of the GDAX API.
https://github.com/Seratna/GDAX-python
You put in the API key, secret and passphrase, you should be able to run it. The order book was built on top of a priority queue and a dict. Updates of the order book can be done in O(lgn) time and getting the actual bid/ask can be done in O(1). (my system only needs the best prices, but it is very easy to return more price levels given the priority queue). I would say red-black tree is also a good candidate, but it takes O(lgn) to get the quote.
The code is part of a bigger project, so you may find some of the methods confusing. If you have any questions, just let me know.

from btfxwss.

@Seratna , have you compared its performance ? And how do you get the largest item from your book? I only see a heappop() call, which should return the smallest item on the heap, no?

Also, if you discard out-of-order messages, shouldnt you also request a new book? Afterall, you would have to keep a history of your messages and rebuild the orderbook with the new out-of-order message in the correct order instead, since your book is missing data otherwise.

An interesting alternative might also be using a Skiplist (I'm trying that out atm) - because while my Order book implementation using a dict has O(1) update speed, returning it still requires to sort keys first. Also, it requires much more memory (since the hash table is duplicated every time too many of its buckets are filled, but memory never released).

from btfxwss.

@nlsdfnbch ,

1. Regarding the performance, there's a significant improvement especially for large books, for example the order book (L3) of GDAX. Because priority queue takes O(lgn), sorting the dict takes O(nlgn), and iterating over the dict (for example max()) takes O(n).
   One priority queue is used for bids and another one for asks. keys of the asks are prices, and keys of bids are negative prices. So the smallest ask would be on the top of the its heap and largest bid would be on the top of the other heap. And the top of the heap is just the first element, so no sorting needed and it takes O(1).

2. And for out-of-order messages, it would raise DataError (line 339 and 347). This DataError would be caught in websocket module (line 74) and websocket would re-connect and build a new book.

3. I would still prefer heap queue and red-black tree over Skiplist as they introduce less overhead (IMO). But yes it is a very interesting data structure.

from btfxwss.