pmaji / crypto-whale-watching-app Goto Github PK
View Code? Open in Web Editor NEWPython Dash app that tracks whale activity in cryptocurrency markets.
License: MIT License
Python Dash app that tracks whale activity in cryptocurrency markets.
License: MIT License
Hi all,
I have been trying out the app and I think is awesome, although I am getting a lot of websocket empty messages. This is the same issue that @pmaji open in danpaquin/coinbasepro-python#361 but I think is specific to whale-watching-app hence I open it here. Taking a look at the code I think the problem is although the code is trying to use websockets, is still performing polling requests (for each websocket message) which are rate limited by CB. The line 25 in gdax_book is doing this:
res = self._client.get_product_order_book(self.product,level=3)
for each message that is received on the websocket.
CB docs mention this:
Level 3 is only recommended for users wishing to maintain a full real-time order book using the websocket stream. Abuse of Level 3 via polling will cause your access to be limited or blocked.
At the moment I only see one way of fixing this:
-Get the full order book ONCE (when app is launched) and store in database
-update it for each message we get on the websocket
Let me know your thoughts please, happy to help with the project.
Cheers
it's all in the title, it would be nice to stop refreshing when zoomed in, or better refresh but keep the zoom as it is!
Hi @pmaji,
When I try to launch app.py then I got some exception as below,
127.0.0.1 - - [16/Oct/2019 01:16:10] "POST /_dash-update-component HTTP/1.1" 200 -
Exception in thread Thread-65:
Traceback (most recent call last):
File "C:\Users\Alex\AppData\Local\Programs\Python\Python37-32\lib\threading.py", line 917, in _bootstrap_inner
self.run()
File "C:\Users\Alex\AppData\Local\Programs\Python\Python37-32\lib\threading.py", line 865, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\Alex\AppData\Local\Programs\Python\Python37-32\lib\site-packages\cbpro\websocket_client.py", line 41, in _go
self._listen()
File "C:\Users\Alex\AppData\Local\Programs\Python\Python37-32\lib\site-packages\cbpro\websocket_client.py", line 91, in _listen
self.on_message(msg)
File "C:\Users\Alex\Desktop\crypto-whale-watching-app-master\gdax_book.py", line 21, in on_message
sequence = message['sequence']
**KeyError: 'sequence'**
Then I set a break point at
def on_message(self, message):
sequence = message['sequence']
to figure out what happens, then I got the error as below,
<class 'dict'>: {'type': 'error', 'message': 'Failed to subscribe', 'reason': '**No channels provided**'}
Could you give me guidance on how to solve this problem? Thank you.
am i the only one not able to build this?? downloaded dash and still can't find out how to build this at all?
Looking for long term and speedy options for hosting this. I tried Heroku to no avail. The present plan now is to move to AWS long-term, but I'm open to other suggestions in this thread.
I've been wondering how to fix this myself but I don't know enough about these libraries to figure it out. When you load the app it looks like it's doing nothing for 10 seconds until it makes the first requests to the server to get the data.
It seems to be because of the interval being set to 10 seconds, it waits for the first 10 seconds to tick and then it loads the data. What would be better is if it loaded the graphs immediately on the page's first load and then updated them every 10 seconds, but I can't see how to do this from the library being used.
Python 3.6.0 (v3.6.0:41df79263a11, Dec 23 2016, 08:06:12) [MSC v.1900 64 bit (AMD64)] on win32
pip install -r D:\_installers\crypto-whale-watching-app-master\requirements.txt
^
SyntaxError: invalid syntax
Is this open source and what is the license?
Hello team,
needed Tasks:
Thanks and Greets
Theimo
So a fan / supporter reached out and mentioned that they were wondering if we could integrate OKEX into our present methodology. They had already written some python code to hit the OKEX websocket API, so some of the work has already been done for us!
See this link here for their first crack at some websocket code. I'm thinking how we might want to add this. At the moment, I'm thinking of maybe demo'ing a separate instance of the app dropping the coinbase api info and replacing it with OKEX info. If it works, we can then think of a solution that combines both--either via a menu option to select the exchange, etc.
I welcome any and all thoughts at how to proceed on this matter!
Since there are a couple of dependencies you should do a requirements file so people can install all the dependencies with pip. To create a requirements file all you need to do is setup a virtual env and only install the required python packages then do "pip freeze > requirements.txt". With that it will output all the names of the dependencies along with the version you were using.
Hello,
what do you guys think about changing current dark/ bright to an algorithm based color.
This way you could see a difference between 1, 4, 6 and 100 unique Orders.
Currently 1=4 and 6=100.
Additionally if we touch this, we should think about making it readable for people with red-green issue
Thanks and Greets
Theimo
Edit: Needed Tasks
Hi, users are unable to run crypto-whale-watching-app due to dependency conflict with urllib3 package.
As shown in the following full dependency graph of crypto-whale-watching-app, crypto-whale-watching-app requires urllib3,while requests==2.21.0 requires urllib3>=1.21.1,<1.25.
According to pip’s “first found wins” installation strategy, urllib3==1.25.3 is the actually installed version.
However, urllib3==1.25.3 does not satisfy urllib3>=1.21.1,<1.25.
crypto-whale-watching-app(version range:)
| +-attrs(version range:==19.1.0)
| +-bintrees(version range:==2.0.7)
| +-cbpro(version range:==1.1.4)
| +-certifi(version range:==2018.1.18)
| +-chardet(version range:==3.0.4)
| +-click(version range:==6.7)
| +-colorama(version range:==0.3.9)
| +-dash(version range:==0.30.0)
| +-dash-core-components(version range:==0.38.0)
| +-dash-html-components(version range:==0.13.2)
| +-dash-renderer(version range:==0.15.0)
| +-dash-table(version range:==3.6.0)
| +-datetime(version range:==4.3)
| +-decorator(version range:==4.2.1)
| +-flask(version range:==1.0.2)
| +-flask-compress(version range:==1.4.0)
| +-gdax(version range:==1.0.6)
| +-idna(version range:==2.6)
| +-ipython-genutils(version range:==0.2.0)
| +-itsdangerous(version range:==0.24)
| +-jinja2(version range:==2.10.1)
| +-jsonschema(version range:==2.6.0)
| +-jupyter-core(version range:==4.4.0)
| +-markupsafe(version range:==1.0)
| +-nbformat(version range:==4.4.0)
| +-numpy(version range:==1.16.2)
| +-pandas(version range:==0.24.2)
| +-plotly(version range:==2.7.0)
| | +-decorator(version range:>=4.0.6)
| | +-nbformat(version range:>=4.2)
| | +-pytz(version range:)
| | +-requests(version range:)
| | | +-chardet(version range:>=3.0.2,<3.1.0)
| | | +-idna(version range:>=2.5,<2.9)
| | | +-urllib3(version range:>=1.21.1,<1.26)
| | | +-certifi(version range:>=2017.4.17)
| | +-six(version range:)
| +-pymongo(version range:==3.5.1)
| +-pyrsistent(version range:==0.14.11)
| | +-six(version range:)
| +-python-dateutil(version range:==2.8.0)
| +-pytz(version range:==2019.1)
| +-requests(version range:==2.21.0)
| | +-chardet(version range:>=3.0.2,<3.1.0)
| | +-idna(version range:>=2.5,<2.9)
| | +-urllib3(version range:>=1.21.1,<1.25)
| | +-certifi(version range:>=2017.4.17)
| +-retrying(version range:==1.3.3)
| | +-six(version range:>=1.7.0)
| +-six(version range:==1.10.0)
| +-sortedcontainers(version range:==2.1.0)
| +-traitlets(version range:==4.3.2)
| +-urllib3(version range:>=1.24.2)
| +-websocket-client(version range:==0.40.0)
| | +-six(version range:)
| +-werkzeug(version range:==0.15.2)
| +-zope.interface(version range:==4.6.0)
Thanks for your help.
Best,
Neolith
I wanted to see if its possible in order to grab the results from the graphs and store them in a variable i can use to perform other tasks for example i want to get the prices and total btc that is in the orderbook that a whale has placed when i run dash it prints everything to the console but i would like to print the data from the app or store them in a variable any way of doing this?
This line incorrectly takes first_ask, as opposed to first_bid. Additionally, first_bid is never initialized.
See:
https://i.imgur.com/0UuGiWF.png
https://i.imgur.com/pquAdI3.png
These should all be "ladder whales", 8.000 LTC ladder and 450 BTC ladder, but are 7 and 9 bubbles instead, all for the same amount (1000 LTC and 50 BTC)
Taken from a reddit comment; should be able to do something like this:
"Can you put perhaps a thin black stroke around the circles that are controlled by say under 5 addresses and perhaps a thick one around 1 address? Also a small written text of the price really close would be great - hoovering the mouse and it going in less than 2 seconds is hard to keep track of when they’re moving. Thanks excellent tool."
In the bottom current code, wich groups by price and then filters all less than 1% of Volume in Range.
In the top I first filter all orders < 1% of Volume in Range and then group by prize.
This results in much better data. e.g. the sell order at 0.083 (yellow) is 1 50 ETH order plus 5 additional orders (summed up to 6 ETH).
Top graph warns about a whale. Bottom graph say "Nah, it´s okay, there are 6 unique orders"
Wrote it as an issue to post picture.
Is it just me?
Server should be running now
Initializing order Book websocket for ETH-USD
-- Subscribed! --Initializing order Book websocket for ETH-BTC
-- Subscribed! --Initializing order Book websocket for BTC-USD
-- Subscribed! --Initializing order Book websocket for LTC-USD
-- Subscribed! --Initializing order Book websocket for LTC-BTC
-- Subscribed! --Initializing order Book websocket for ETH-EUR
-- Subscribed! --Initializing order Book websocket for BTC-EUR
-- Subscribed! --Initializing order Book websocket for LTC-EUR
-- Subscribed! --Initializing order Book websocket for BCH-USD
-- Subscribed! --Initializing order Book websocket for BCH-BTC
-- Subscribed! --Initializing order Book websocket for BCH-EUR
-- Subscribed! --Web sockets up
Exception in thread Thread-2:
Traceback (most recent call last):
File "C:\Python\lib\threading.py", line 916, in _bootstrap_inner
self.run()
File "C:\Python\lib\threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\mifune\Desktop\app.py", line 592, in sendPrepareThread
sendCache = prepare_send()
File "C:\Users\mifune\Desktop\app.py", line 471, in prepare_send
figure=cData[exchange + ticker]
KeyError: 'GDAXETH-USD'ReCalc up
Everything should be running now, starting Watchdog, to control the herd
Watchdog detected dead Preparer, restarting
Watchdog got some bad sheeps back to group
Exception in thread Thread-47:
Traceback (most recent call last):
File "C:\Python\lib\threading.py", line 916, in _bootstrap_inner
self.run()
File "C:\Python\lib\threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\mifune\Desktop\app.py", line 592, in sendPrepareThread
sendCache = prepare_send()
File "C:\Users\mifune\Desktop\app.py", line 471, in prepare_send
figure=cData[exchange + ticker]
KeyError: 'GDAXLTC-EUR'
Right now when you hover over a dot, the tooltip disappears before you have a chance to fully read what it says. It should stay up until you move the mouse off of it.
Hi guys, the syncing stopping problem is still an issue. When looking at the logs I discovered that they often print errors for different graphs like this:
127.0.0.1 - - [20/Feb/2018 04:07:42] "POST /_dash-update-component HTTP/1.0" 500 -
[2018-02-20 04:07:42,794] ERROR in app: Exception on /_dash-update-component [POST]
Traceback (most recent call last):
File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1982, in wsgi_app
response = self.full_dispatch_request()
File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1614, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1517, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/local/lib/python3.5/dist-packages/flask/_compat.py", line 33, in reraise
raise value
File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1612, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/local/lib/python3.5/dist-packages/flask/app.py", line 1598, in dispatch_request
return self.view_functions[rule.endpoint](**req.view_args)
File "/usr/local/lib/python3.5/dist-packages/dash/dash.py", line 537, in dispatch
for component_registration in self.callback_map[target_id]['inputs']:
KeyError: 'live-graph-ethbtc.figure'
Plotly 3.0 Changes some of its data presenting code. If anyone hits attribute append error, please check this thread.
https://community.plot.ly/t/attributeerror-tuple-object-has-no-attribute-append/11782/4
Hi,
I really appreciate this tool and it's fantastic. The only thing really bugs me is that the tool tips when you hover over an order block disappears too fast without you getting a chance to digest the information. Is there any way you could possibly make an option to just always keep the hover tooltips on if for example you clicked the order block.
Thanks
I installed at
Site missing CSS, shows no graph.
On console, i get error "No graphs loaded yet".
Any idea why it is not wokring ?
I was wondering if you plan on adding ETC to the app now that's ETC is listed on Coinbase Pro.
Doesn't plot any graph, doesn't load any data.
Error here:
File "C:\Python\lib\threading.py", line 916, in _bootstrap_inner
self.run()
File "C:\Python\lib\threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "C:\Python\lib\site-packages\cbpro\websocket_client.py", line 41, in _go
self._listen()
File "C:\Python\lib\site-packages\cbpro\websocket_client.py", line 91, in _listen
self.on_message(msg)
File "C:\test\gdax_book.py", line 21, in on_message
sequence = message['sequence']
KeyError: 'sequence'
And doesn't load any data:
2020-11-25 10:45:59.950112 [Errors]: Empty data for GDAXETH-EUR Will wait 3s
2020-11-25 10:46:00.125238 [Errors]: Empty data for GDAXETH-BTC Will wait 3s
2020-11-25 10:46:00.412958 [Errors]: Empty data for GDAXBTC-USD Will wait 3s
Edit: I see this is also mentioned in bugs #112 and #113
I tested both on Windows and Linux, same issue. Maybe related to danpaquin/coinbasepro-python#365 ?
Should be 30, 40, 50, ... 120, 130...
Or is this intentional to not clog the space up?
@pmaji also unrelated, but ladder_tooltip_screenshot.JPG and bubble_tooltip_screenshot.JPG should be updated to show the new info available in the tooltip (and I guess the readme as well).
Given the treasure trove of data that we now are developing, as well the growing community of traders / coders that are involved in this project, I wanted to open an issue where people can periodically contribute questions that they want explored further.
My aim to open up a Python Notebook separate from this app in which we can dig into deeper (perhaps more long-form analytic questions) that this app or curiosity in general may create. Comment below with any hypotheses you want tested or theories you want explored, etc., and I will add them to the master list of topics to add to the exploratory Python Notebook that will be created.
List of subjects to investigate:
I found a few instances of what looks to be no-longer-used code in the present app.py. I'll do a full clean shortly.
error message below:
File "C:\Users\test\Downloads\crypto-whale-watching-app-master\crypto-whale-watching-app-master\gdax_book.py", line 23, in on_message
sequence = message['sequence']
KeyError: 'sequence'
can help?
Trying to install with python 3.6.0 but got the following error. Modifying the version manually creates a lot more errors...
ERROR: Cannot install -r requirements.txt (line 3) and requests==2.21.0 because these package versions have conflicting dependencies.
The conflict is caused by:
The user requested requests==2.21.0
cbpro 1.1.4 depends on requests==2.13.0
To fix this you could try to:
1. loosen the range of package versions you've specified
2. remove package versions to allow pip attempt to solve the dependency conflict
ERROR: ResolutionImpossible: for help visit https://pip.pypa.io/en/latest/user_guide/#fixing-conflicting-dependencies
With colorblind on, tooltips on ask side flash green/blue.
If Freeze is toggled, it would be great if the button changed to "Unfreeze"
See horizontal line & annotation: https://i.imgur.com/UQ6qyS2.png
An issue to collect data and tasks needed for adding binance
Decrease Callback/ Load Amount to 1 in #40
Analyze compatibility
Needed changes in Data call?
Needed changes in Data storage (Multiple times same pair e.g. "ETH-USD")
Discuss form of presentation
Original:Send all data to client, and hide/ show selection on clientside with js?
Update: Sending all data should be no problem due to changes in #40
Maybe rework show/ hide in the future, to show/ hide complete exchanges.
Add Dropdown to select exchange
Update Show/ Hide Button is much easier
define format for get_data function
Goal is to keep get_data modular for future.
exclude api calls from get_data function
When we change this we should change #28 aswell
Thanks and Greets
Theimo
As suggested here #31 (comment) it'd be great to have a hover text on Plotly lines ("ladder whales") as with the bubbles instead of annotations, but Plotly does not support that. A workaround might be https://stackoverflow.com/questions/46037897/line-hover-text-in-plotly/46039229#46039229
E.g. ETH-BTC: 0.07564 BTC not 0.0756.
Hello,
in my Opinion it would be nice to see the total size of order-book by a line in the background.
Due to the fact, we already have the data, it would be little problematic.
I just have the problem I´m not familiar with Plotty....
Thanks and Greets
Theimo
Edit:
Needed Tasks:
@theimo1221 you can see in my updated README that I have split our definition of whale into two types. There is a bit of a methodological blind spot that I identified though when it comes to what I called "ladder price-point whales", i.e. those that we mark with the linebar chart instead of the bubble. Namely, the problem is that we ALSO pick up on psychological modal points.
What I mean by this is that, for example, 10 ETH is a very popular volume amount. So is 5 ETH. If many people put orders for 10 ETH along the span of say pricepoints of $950 to $1000, our present methodology would flag this as a potential whale. That said, if we saw a bunch of orders for 10.0085 ETH spanning that same range of prices, THAT is more likely to be a whale, because it is a rarer volume number (and thus less likely to just be a psychological modal point that many people might pick).
It's very hard to control for this, but I think one way to do it might be to exclude from our ladder price-point whale-identification logic any span of orders that is obviously a popular order number (i.e. 10 / 5 / 1 / and maybe a few others). The problem is that this logic is a lot more based on assumptions than the single price-point whale identification logic, but we can make it a bit better I think (given the logic I laid out here). Let me know your thoughts.
You guys have a great tool here! I haven't seen any work lately. I'd like to promote you on my segments that I do on Alt Coin Buzz. It's a TA segment along with the news. Are you still working on this? If so can I promote, or would it be better to wait?
Also wondering if you guys are working on a time stamp for the trades to be included in the roll-over. Certainly would be helpful to see if the larger orders are set up to affect the price.
Thank you in advance!
Hi,
I get this error on my MacOS 10.13.3 (17D102) when running the installation and hence cannot continue:
Found existing installation: six 1.4.1
DEPRECATION: Uninstalling a distutils installed project (six) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
Uninstalling six-1.4.1:
Exception:
Traceback (most recent call last):
File "/Library/Python/2.7/site-packages/pip/basecommand.py", line 215, in main
status = self.run(options, args)
File "/Library/Python/2.7/site-packages/pip/commands/install.py", line 342, in run
prefix=options.prefix_path,
File "/Library/Python/2.7/site-packages/pip/req/req_set.py", line 778, in install
requirement.uninstall(auto_confirm=True)
File "/Library/Python/2.7/site-packages/pip/req/req_install.py", line 754, in uninstall
paths_to_remove.remove(auto_confirm)
File "/Library/Python/2.7/site-packages/pip/req/req_uninstall.py", line 115, in remove
renames(path, new_path)
File "/Library/Python/2.7/site-packages/pip/utils/init.py", line 267, in renames
shutil.move(old, new)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 302, in move
copy2(src, real_dst)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 131, in copy2
copystat(src, dst)
File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 103, in copystat
os.chflags(dst, st.st_flags)
OSError: [Errno 1] Operation not permitted: '/tmp/pip-3DElGw-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six-1.4.1-py2.7.egg-info'
Q.
Thanks!
Exception in thread Thread-66:
Traceback (most recent call last):
File "C:\Python\lib\threading.py", line 916, in _bootstrap_inner
self.run()
File "C:\Python\lib\threading.py", line 864, in run
self._target(*self._args, **self._kwargs)
File "C:\Users\mifune\Desktop\app.py", line 647, in websockThread
kill = 5 / pair.webSocketKill
ZeroDivisionError: division by zero
Now with #66 merged, hovering over a bubble shows the total price of the order/s. I tried to add the same to the ladder whale orders, and if I play with a simple test dataframe it works, but when I try with app.py, I get an error on:
vol_grp_bid = vol_grp_bid[
((vol_grp_bid[TBL_VOLUME] >= minVolume) & (vol_grp_bid['count'] >= 2.0) & (vol_grp_bid['count'] < 70.0))]
for some reason. Could someone with more experience implement this?
Had a nice and good discussion with Paul. We found many good and some little points.
Pull #41 did a lot of changes towards original goal of detecting whales and fake Resistances/ Supports.
Currrently we spot fakes by grouping price.
An additional approach would be grouping by volume (e.g. 50 orders of 1.53 ETH from $962.65 to $984.42.
At current state we thing adding a bar at the position of summed up volume with the bot beeing at the start and top beeing at the end of price range could be best.
Needed steps to do this:
I installed modules with pip, when i run, i get error
[root@sultryserver crypto-whale-watching-app]# python app.py
File "app.py", line 35
SyntaxError: Non-ASCII character '\xe2' in file app.py on line 35, but no encoding declared; see http://www.python.org/peps/pep-0263.html for details
[root@sultryserver crypto-whale-watching-app]#
Is there any instruction to run this ? Any specific Python version required ?
Hi, I'm no way near a coder, just a sysadmin debugging perl/python/bash code sometimes. I was playing around to see if I was able to add the ethbtc chart, I think I got it.( Don't really understand the details, it's all copy pasta from your code ;) )
Feel free to take a look and adapt it if you want to merge in the code!
Thanks again for sharing!
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.