Giter VIP home page Giter VIP logo

currency-rates-api-scraping's Introduction

Currency-API-Scraping

A webscraper that scrapes Malaysia eRemit Companies (Wise, Instaremit, MoneyMatch, SunwayMoney) via APIs and stores them in a comma-separated-value (.csv) file.

Some applications for webscraping rates is storing them for future rates predictions and machine learning applications/projects.

Disclaimer:

This repository is solely for learning purposes on web/api scraping.
- Please respect a website's terms or robots.txt on webscraping.
Ethics in Web Scraping | Towards Data Science

Prerequisites

Ensure you have installed the following packages requests and pandas.

pip install requests
pip install pandas

Clone the Repository

git clone https://github.com/lester-liam/weather-forecast-email.git

Update Configuration File


In `config.csv`, ensure you added the currencies / rates you want.

Columns Info:

# Remittance Site to Scrape
provider:list = ["Wise", "Instarem", "MoneyMatch", "SunwayMoney"]

country:str     # Country Name (non-standard for output)
currency:str    # Currency Code as ISO 4217 Format
amount:int      # Amount to Convert (integer only)
param1:str      # Additional Parameter for Sunway Money

For Sunway Money:
Input param1 as Currency Code ISO 4217 format + Country Code as ISO 3166 format.
Example: Singapore Dollar = SGD-SG

Running the Code

The code should run out of the box. However you can also configure the file output locations.

Final CSV Output and Configuration File

# Stores the Currency/Countries to Query
configFilename:str = "config.csv" 
# Final Output Filename
output_filename:str = f"rates_history/rates_history_{datetime.now().strftime("%d%b%y-%H%M")}.csv"

Updating File Outputs

main.py: Line 30:

# Read CSV Outputs
wise_rates = pd.read_csv("./output/wise.csv", ...)
instarem = pd.read_csv("./output/instarem.csv", ...)
sunway_money = pd.read_csv("./output/sunwaymoney.csv", ...)
money_match = pd.read_csv("./output/moneymatch.csv", ...)

rates_funcs.py: Line 323

# Reset CSV Files:
filenames:list = ["./output/instarem.csv", "./output/moneymatch.csv", "./output/wise.csv", "./output/sunwaymoney.csv"]
for file in filenames:
    with open(file, 'w', newline="") as output:
        output.write('')

rates_funcs.py: Line 332:

# Define processes for each provider
processes = [
    Process(..., "./output/instarem.csv")),
    Process(..., "./output/sunwaymoney.csv")),
    Process(..., "./output/moneymatch.csv"))
]

Credits / References

Remittance Websites used in the code.
MoneyMatch
Instarem
Wise
Sunway Money

Licenses

Unlicense

currency-rates-api-scraping's People

Contributors

lester-liam avatar

Watchers

 avatar

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    ๐Ÿ–– Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. ๐Ÿ“Š๐Ÿ“ˆ๐ŸŽ‰

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google โค๏ธ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.