f1lt3r / bitcoin-scraper Goto Github PK

View Code? Open in Web Editor NEW

82.0 5.0 29.0 115.75 MB

💲 bitcoin chart history scraper

Home Page: https://f1lt3r.io

License: MIT License

JavaScript 92.15% HTML 7.85%

bitcoin chart data graph moore logarithmic javascript bitcoin-scraper json scrapes-bitcoin-data

bitcoin-scraper's Introduction

Bitcoin Chart Scraper

Scrape the entire Bitcoin chart history to JSON.

Install & Scrape

git clone https://github.com/F1LT3R/bitcoin-scraper.git
cd bitcoin-scraper
npm install
mkdir data
node scrape.js

Output Example

  _                 _              _
 /_)._/__  _  ._   / `/_ _  __/_  /_`_  __  _  _  _
/_)/ / /_ /_/// / /_,/ //_|/ /   ._//_ //_|/_//_'/
                                          /
Running with maxstreams=8

Fetching: 2017-8-9
Fetching: 2017-8-8
Fetching: 2017-8-7
Fetching: 2017-8-6
Fetching: 2017-8-5
Fetching: 2017-8-4
Fetching: 2017-8-3
Fetching: 2017-8-2
Recevied: 2017-8-3
Saved to: data/bitstampUSD-2017-8-3.json
Recevied: 2017-8-9
Saved to: data/bitstampUSD-2017-8-9.json
Recevied: 2017-8-6
Saved to: data/bitstampUSD-2017-8-6.json
Recevied: 2017-8-7
Saved to: data/bitstampUSD-2017-8-7.json
...
DONE!

Data Structure

[
  [
    1419033600,
    318.58,
    318.58,
    318.58,
    318.58,
    0.01719605,
    5.478317609,
    318.58
  ],
  ...

Columns

Timestamp	Open	High	Low	Close	Volume (BTC)	Volume (Currency)	Weighted Price (USD)
1419033600	318.58	318.58	318.58	318.58	0.01719605	5.478317609	318.58

Le Combine...

Combine all data into one file sequentially.

node combine.js data/bitcoin-history.json

Chart...

Charting Bitcoin against the Moore's Law theory of Bitcoin growth. Logarithmic Y axis = dollars.

npm install http-server -g
http-server
# point your browser to http://localhost:8080/chart.html

2020 May 30

2017 August 9

bitcoin-scraper's People

Contributors

Stargazers

Watchers

bitcoin-scraper's Issues

url or parameters changed

Hi Alistair,

Your job is really good but I think bitcoincharts.com have changed the url or parameters...

In this moment, if you execute your code every data file get the same content (last day datas).

I have tried with differents urls:

Received "Undefined" data for 2021-2-20. Too many streams?

For some reasons scraper is not working anymore for me. I reseive the following. Tuninx down max streams does not help. A manual copy paste of the Url works.
All files in data are empty.

Any idears ?

Fetching: 2021-2-18 - http://bitcoincharts.com/charts/chart.json?m=bitstampUSD&SubmitButton=Draw&r=60&i=1-min&c=1&s=2021-02-18&e=2021-02-18&Prev=&Next=&t=S&b=&a1=&m1=10&a2=&m2=25&x=0&i1=&i2=&i3=&i4=&v=1&cv=0&ps=0&l=0&p=0&
Fetching: 2021-2-19 - http://bitcoincharts.com/charts/chart.json?m=bitstampUSD&SubmitButton=Draw&r=60&i=1-min&c=1&s=2021-02-19&e=2021-02-19&Prev=&Next=&t=S&b=&a1=&m1=10&a2=&m2=25&x=0&i1=&i2=&i3=&i4=&v=1&cv=0&ps=0&l=0&p=0&
Fetching: 2021-2-20 - http://bitcoincharts.com/charts/chart.json?m=bitstampUSD&SubmitButton=Draw&r=60&i=1-min&c=1&s=2021-02-20&e=2021-02-20&Prev=&Next=&t=S&b=&a1=&m1=10&a2=&m2=25&x=0&i1=&i2=&i3=&i4=&v=1&cv=0&ps=0&l=0&p=0&
Fetching: 2021-2-21 - http://bitcoincharts.com/charts/chart.json?m=bitstampUSD&SubmitButton=Draw&r=60&i=1-min&c=1&s=2021-02-21&e=2021-02-21&Prev=&Next=&t=S&b=&a1=&m1=10&a2=&m2=25&x=0&i1=&i2=&i3=&i4=&v=1&cv=0&ps=0&l=0&p=0&
Received "Undefined" data for 2021-2-19. Too many streams? (Data for ome dates are not available, eg: '2011-10-1')
Saved to: data/bitstampUSD-2021-02-19.json
Received "Undefined" data for 2021-2-18. Too many streams? (Data for ome dates are not available, eg: '2011-10-1')
Saved to: data/bitstampUSD-2021-02-18.json
Received "Undefined" data for 2021-2-21. Too many streams? (Data for ome dates are not available, eg: '2011-10-1')
Received "Undefined" data for 2021-2-20. Too many streams? (Data for ome dates are not available, eg: '2011-10-1')
Saved to: data/bitstampUSD-2021-02-21.json
Saved to: data/bitstampUSD-2021-02-20.json

1.7e+308 what is it!?

When I scrape the data this (1.7e+308) shows why? , what mean and how to avoid it
thank you

font is missing

Program gave me errors saying that it could not find a font at bitcoin-scraper/node_modules/figlet/lib/../fonts/

a "wget http://www.textfiles.com/art/pepper.flf" in that folder fixed for me the problem

doesnt go past august 9 2017

How can I get it to scrape past that point?

data dir is missing

I installed bitcoin-scrapper from scratch:

I get an error saying that it cannot open the bitstamp.log file wich it want to create. reason for that is that there is no "data" dir. an mkdir data fixed the problem for me