Giter VIP home page Giter VIP logo

asx_options's Introduction

Web scraper built with python, scrapy and beautifulsoup

What is ASX Scraper?

This scraper creates a CSV file that contains the publicly available options data for all ASX traded options. Essentially, it generates a snapshot-in-time spreadsheet of all ASX traded options, along with all the relevant information.

How to use ASX Scraper

  1. Clone the repository onto your computer
  2. Install Scrapy with pip install Scrapy
  3. Install bs4 with pip install beautifulsoup4
  4. cd into /asx_options-master and run python -m scrapy crawl asx_spider. This will create a .html file in /asx_options-master for every company that has ASX traded options. About 100 .html files will be generated.
  5. Run python parser.py from /asx_options-master. parser.py will create a new CSV file, parse through the .html files one by one, extract the options data, and place the data into the CSV file. This will take ~20 seconds.
  6. Copy-paste the data from the CSV file into a MS Excel spreadsheet.

Screenshot of the output - showing the the first ~50 of 43,837 lines in the csv file

First fifty lines of the CSV file

asx_options's People

Contributors

maxitron93 avatar

Stargazers

 avatar

Watchers

 avatar

asx_options's Issues

appears discontinued. Would this still work in 2020? worth resurrecting?

Found this in March 2020. [era of the Corona Virus quarantine :) ]

Before I spend too much time on this, did you find an alternative source of data for options or some other reason why this was discontinued?
brief examination suggests the asx web page structure has changed, hence the failure. idk.

If I tackle, this, likely will use selenium as I've found it easier to work with. (if less computationally efficient)

$ python parser.py

Traceback (most recent call last): File "parser.py", line 12, in options_data = options_table.find_all("td") AttributeError: 'NoneType' object has no attribute 'find_all'

12: options_data = options_table.find_all("td")

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.