eveprosper / prosperapi Goto Github PK

View Code? Open in Web Editor NEW

13.0 3.0 4.0 39.81 MB

API endpoints for working with EVE Prosper data sources

License: MIT License

Python 100.00%

eveonline rest-api python-3 flask-restful forecasting dh-virtualenv crest tinydb

prosperapi's Introduction

ProsperAPI

A collection of API resources for the general public to leverage EVE Prosper’s EVE Online market data.

These services are designed to be easy to deploy and maintain for any member of the community. Though some private resources will require data from our master host, this Flask-Restful project should be easy to distribute.

Getting Started

Though testing/developent is platform agnostic, the following resources are required for production:

Debian Linux (Ubuntu16)
Python >3.5
dh-virtualenv >1.0

For Developers

Getting started is easy for developers. Just spin up a virtualenv and make sure cython and numpy are installed

pip install cython pip install numpy

then install with

pip install . --extra-index-url=https://repo.fury.io/lockefox/

Testing is designed to be easy too! Just run tests from setup.py

python setup.py test

NOTES

On Windows, numerical libraries are touchy. Have included manually compiled wheels in project
fbprophet is a complicated dedpendency. See build documentation for more notes
Mac is special, fbprophet is not supported, but everything else is

For Sysadmins

Please refer to our build documentation for step by step

What Is Offered

PublicAPI - Transformative API Endpoints

All endpoints provided in publicAPI are open and require no special databases.

OHLC: transform in-game history feed into a traditional OHLC format
Prophet: provide forecasts on future prices, given previous history

Legal Stuff

EVE Online and the EVE logo are the registered trademarks of CCP hf. All rights are reserved worldwide. All other trademarks are the property of their respective owners. EVE Online, the EVE logo, EVE and all associated logos and designs are the intellectual property of CCP hf. All artwork, screenshots, characters, vehicles, storylines, world facts or other recognizable features of the intellectual property relating to these trademarks are likewise the intellectual property of CCP hf. CCP hf. has granted permission to EVE-Prosper and John Purcell to use EVE Online and all associated logos and designs for promotional and information purposes on its website but does not endorse, and is not in any way affiliated with, EVE-Prosper. CCP is in no way responsible for the content on or functioning of this website, nor can it be liable for any damage arising from the use of this websi

prosperapi's People

Contributors

Stargazers

Watchers

Forkers

kanelarrete xiantra-eve arandel2007

prosperapi's Issues

Add pytest-pylint test coverage

https://pypi.python.org/pypi/pytest-pylint/0.7.0

Add pylint validation to pytest test suite.

PLEX Split compatability

PLEX 29668 is splitting on May 9th

Database will need to backfill the split, and handle the type conversion across split-date.

Expected behavior

typeid=29668: newplex * 500
typeid=44992: oldplex / 500

Add tinydb records on endpoint access

Right after logger.info() at the start of each get(), a record should be saved of the following:

Which endpoint was requested
Time requested
args of request

This should be saved somewhere other than cache to avoid steamrolling when installing. /var/ or /opt/ might be the place. In testmode, could save to ROOT level of project. Add path to app.cfg

ProsperCommon v0.3.x integration

Config/Logger utilities were overhauled in v0.2.x to v0.3.x. Need to upgrade/update calls to config and how logger is created

add robots.txt to endpoint

In an effort to subdue crawlers, add a robots.txt to publicAPI/static

http://stackoverflow.com/questions/14048779/with-flask-how-can-i-serve-robots-txt-and-sitemap-xml-as-static-files

Notes:

need to add publicAPI/static to MANIFEST.in
may need to add publicAPI/static/* to package_data in setup.py
Any new endpoints will need to add tests to test_crest_endpoint.py

Switch all TinyDB connections to tinymongo ones
Add pymongo connection
Default to tiny if/when db connection fails

ProsperCommon v1.0.0 Integration

Update system to take advantage of new packaging on ProsperCommon.

end-to-end ujson integration

Improve performance by making sure ujson is integrated everywhere in the stack it should be.

incoming requests
tinydb connections
pandas to_json
- pandas.io.json.dumps = ujson.dumps
- pandas.io.json.loads = usjon.loads + precise_float=True in pd.read_json() args
skip Flask.jsonify by building response directly from pandas

Better HTTP error capture

Though we are covered for exceptions with req.raise_for_status() we are not capturing all the error information.

Add header/req.text capture to log messages
- Be careful of 1000 char limit on WebHook log messages
Fix double-counted HTTP exceptions spamming discord

Fix EMD requests

EMD requests are causing major issues:

self.chunk_left = int(line, 16)
E           ValueError: invalid literal for int() with base 16: b''

https://travis-ci.org/EVEprosper/ProsperAPI/builds/482103441#L703

Add retry to ESI calls

ESI calls are failing ~1%. Add retry

[ERROR:crest_utils.py--fetch_market_history:354]
ERROR: unable to fetch market history from CREST
    type_id: 39
    region_id: 10000002
Traceback (most recent call last):
  File "/opt/venvs/prosper-api/lib/python3.5/site-packages/publicAPI/crest_utils.py", line 343, in fetch_market_history
    config=config
  File "/opt/venvs/prosper-api/lib/python3.5/site-packages/publicAPI/crest_utils.py", line 303, in fetch_esi_endpoint
    req.raise_for_status()
  File "/opt/venvs/prosper-api/lib/python3.5/site-packages/requests/models.py", line 909, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://esi.tech.ccp.is/latest/markets/10000002/history/?datasource=tranquility&type_id=39
[ERROR:forecast_utils.py--fetch_extended_history:201]
ERROR: trouble getting data from CREST
    region_id=10000002
    type_id=39
Traceback (most recent call last):
  File "/opt/venvs/prosper-api/lib/python3.5/site-packages/publicAPI/crest_utils.py", line 343, in fetch_market_history
    config=config
  File "/opt/venvs/prosper-api/lib/python3.5/site-packages/publicAPI/crest_utils.py", line 303, in fetch_esi_endpoint
    req.raise_for_status()
  File "/opt/venvs/prosper-api/lib/python3.5/site-packages/requests/models.py", line 909, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 502 Server Error: Bad Gateway for url: https://esi.tech.ccp.is/latest/markets/10000002/history/?datasource=tranquility&type_id=39

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/opt/venvs/prosper-api/lib/python3.5/site-packages/publicAPI/forecast_utils.py", line 194, in fetch_extended_history
    logger=logger
  File "/opt/venvs/prosper-api/lib/python3.5/site-packages/publicAPI/crest_utils.py", line 361, in fetch_market_history
    mode.name
publicAPI.exceptions.CRESTBadMarketData```