Data from Quandl failing -- NASDAQ takeover

For example, when retrieving the Dec 2019 Eurodollar futures contact:

>>> tmp = fe.get('CME/EDZ2019')
ValueError: INVALID symbol string or code variable.

Projects fecon235 and fecon236 have many data dependencies on the Quandl API, including but limited to, futures quotes.

• Bug
• Enhancement
• Upstream disruption

Steps which reproduce behavior

1. Eurodollar futures quotes relies on CME vendor at Quandl.
2. "PAGE NO LONGER EXISTS." at usual vendor page: https://www.quandl.com/data/CME-Chicago-Mercantile-Exchange-Futures-Data/documentation/data-organization
3. Last successful Google crawl on said vendor page was December 1, 2018.
4. Top notification, "Quandl has been acquired by Nasdaq. Learn more" links to a Press Release from Nasdaq: https://business.nasdaq.com/mediacenter/pressreleases/1855930/nasdaq-acquires-quandl-to-advance-the-use-of-alternative-data which includes the following statements:

December 4, 2018, Nasdaq Acquires Quandl

Nasdaq (Nasdaq: NDAQ) announced today it has acquired Quandl, Inc., a leading provider of alternative and core financial data. Quandl from over 350 sources to more than 30,000 active monthly users. The company offers a global database of alternative, financial and public data, including information on capital markets, energy, shipping, healthcare, education, demography, economics and society.

Nasdaq plans to combine Quandl with its existing Analytics Hub business within Global Information Services.

Terms of the deal were not disclosed, but this investment is consistent with both Nasdaq’s strategy to maximize opportunities as a technology and analytics provider to capital markets, as well as its capital deployment and return on investment capital objectives.

Matthew Sheahan
Phone Number: + 1 212 231 5945
Email: [email protected]

5. Search for "CME" on Quandl's search page https://www.quandl.com/search no longer returns any results for futures contracts (bonds, FX, metals, etc.) directly provided by CME.
6. Google search leads to link for "CME Dec 2019 Eurodollar Futures", https://www.quandl.com/CME/EDZ2019 -- but currently displays "PAGE NO LONGER EXISTS." Last successful Google crawl was on November 30, 2018.
7. Disappearance of CME data thus most likely occurred between November 30 and December 7, 2018. Midway, on December 4, Nasdaq announced acquisition of Quandl, a distributor of CME data.

Additional helpful details for bugs

• Problem started recently, but not in older versions

• Problem happens with all files, not only some files

• Problem can be reliably reproduced

• Problem happens randomly

• fecon236 version: 10.8.0

• Python version: 2.7 and 3 series

• Operating system: All

Until advised otherwise, we will assume that the Quandl data will be behind paywalls (Nasdaq's Analytics Hub).

Travis build 63 fail: https://travis-ci.org/MathSci/fecon236/builds/393443445

Example of fecon236 circular dependency

In the fred module, holtwinters.ema() is called to smooth data.
In the holtwinters module, hostess.get() is called to retrieve data.
But the hostess module relies on fred.getfred() in the fred module.

[... continued from fecon235, https://github.com/rsvp/fecon235/issues/9]

Home for our Jupyter notebooks shall remain at fecon235 due to their bulky size. Migration to the python3 kernel should present no problems since fecon235 has been compatible with both python27 and python3.

Version 10 of fecon236 represents the refactoring of only the fecon235 v5.18.0312 library code (not the Jupyter notebooks). It shall maintain compatibility with both python27 and python34.

After 2019-01-01, our official support for python27 will discontinue (following policy at numpy and pandas), however, straddling code may still continue to work.

Version 11 of fecon236 will signal when our Travis builds under python27 fail, and at that point we expect to require at least Python 3.6.

fecon documentation

In-progress

Discussion thread for fecon236 documentation (from #4)

Overview

I have an initial Sphinx documentation implementation in the fecon-docs repository, which in part uses the sphinx-autodoc extension to render the docstrings ("auto-documentation"). There are also small sections which I have built out (such as FRED Codes) for additional context.

The repository is setup to use TravisCI with doctr for CI and CD to Github pages. The travis build lints the documentation source, though I do need to explore options for linting the docstrings themselves (there are a few, each of which have pros and cons).

General Structure

The documentation will be structured as a one-to-one mirror of the modules used in fecon236:

Implementation

Implementation will begin as auto-documentation of each fecon236 module.

In order to complete this process, it is easiest to convert all docstrings to the numpydoc format. An initial implementation of these conversions has been completed (see pull requests #10 through #24)

ATTN mention: @rsvp

flake8 upgrade introduces "W605 invalid escape sequence" error

Travis build 118 on 2018-11-09 has failed -- solely due to fecon236/dst/gaussmix.py -- diagnosis and remedy are discussed herein.

• Bug (Very minor)
• Enhancement

Expected behavior

No SyntaxError was expected.

Observed behavior, build 118.2

$ flake8 --version
3.6.0 (mccabe: 0.6.1, pycodestyle: 2.4.0, pyflakes: 2.0.0) CPython 2.7.15 on Linux
The command "flake8 --version" exited with 0.

$ flake8 fecon236
fecon236/dst/gaussmix.py:66:-1721: W605 invalid escape sequence '\s'
fecon236/dst/gaussmix.py:66:-1494: W605 invalid escape sequence '\s'
fecon236/dst/gaussmix.py:66:-1483: W605 invalid escape sequence '\s'
fecon236/dst/gaussmix.py:66:-1474: W605 invalid escape sequence '\s'
fecon236/dst/gaussmix.py:66:-1417: W605 invalid escape sequence '\s'
fecon236/dst/gaussmix.py:66:-1319: W605 invalid escape sequence '\s'
fecon236/dst/gaussmix.py:66:-1307: W605 invalid escape sequence '\s'
fecon236/dst/gaussmix.py:66:-1293: W605 invalid escape sequence '\s'
fecon236/dst/gaussmix.py:66:-1284: W605 invalid escape sequence '\s'
fecon236/dst/gaussmix.py:66:-934: W605 invalid escape sequence '\k'
fecon236/dst/gaussmix.py:66:-822: W605 invalid escape sequence '\s'
fecon236/dst/gaussmix.py:66:-810: W605 invalid escape sequence '\s'
fecon236/dst/gaussmix.py:66:-796: W605 invalid escape sequence '\s'
fecon236/dst/gaussmix.py:66:-787: W605 invalid escape sequence '\s'
fecon236/dst/gaussmix.py:66:-703: W605 invalid escape sequence '\s'
fecon236/dst/gaussmix.py:66:-661: W605 invalid escape sequence '\s'
fecon236/dst/gaussmix.py:66:-648: W605 invalid escape sequence '\k'
fecon236/dst/gaussmix.py:66:-553: W605 invalid escape sequence '\m'
The command "flake8 fecon236" exited with 1.

Steps which reproduce or enhance behavior

1. PASS: Build 117 https://travis-ci.org/MathSci/fecon236/jobs/449914169

$ flake8 --version
3.5.0 (mccabe: 0.6.1, pycodestyle: 2.3.1, pyflakes: 1.6.0) CPython 2.7.15 on Linux

2. FAIL: Build 118 https://travis-ci.org/MathSci/fecon236/jobs/452950251

$ flake8 --version
3.6.0 (mccabe: 0.6.1, pycodestyle: 2.4.0, pyflakes: 2.0.0) CPython 2.7.15 on Linux

Why would the improvement be useful to most users?

Conformity with latest flake8 lint and pycodestyle guidelines.

Additional helpful details for bugs

What is W605? Prior to Python 3.6, a backslash-character pair that was not a valid escape sequence generated a DeprecationWarning. This has become a SyntaxError as of flake8 version 3.6.0. See https://lintlyci.github.io/Flake8Rules/rules/W605.html

Anti-pattern: regex = '\.png$'

Best practice: regex = r'\.png$'

Thus the best resolution is to use explicit raw string mode.

• Problem started recently, but not in older versions

• Problem happens with all files, not only some files

• Problem can be reliably reproduced

• Problem happens randomly

• fecon236 version: 10.7.1

• pandas version: 0.22

• Python version: 2.7 and 3

• Operating system: All

Our spin-off of fecon235 source code still inherits the upstream problems
outlined in rsvp/fecon235#7

Here at fecon236, it is localized in the stock.py module:
https://github.com/MathSci/fecon236/blob/develop/fecon236/host/stock.py

• check new development at https://github.com/pydata/pandas-datareader/issues
• - write another fecon236 module using reliable data vendor

So far we have relied largely on demonstrative notebooks
at https://github.com/rsvp/fecon235

The source code here at fecon236 is designed to be more
explicit than casual usage, so if you have experience in
setting up a nice documentation framework, please
kindly volunteer.

The current "docs" need a complete redo, one that starts from fresh eyes.
Nobody peeks at the source code unless there is something which
solves their problem or extends their research interest.

As an analogy, Unix-style man pages are really dry,
but the usage examples are most useful in practice.
That's what fecon235 notebooks were intended to do informally,
but as the research topics got more specialized,
formalization of the structure for clarity became necessary,
hence the refactoring into fecon236.

As an example, forecast() used to be called directly,
but now we prefer fe.hw.forecast(). The user may not know
that fe is shorthand for fecon236 (not fecon235), and
that hw is an alias for the holtwinters.py module which
is physically located implicitly under the tsa directory
for time-series analysis.

This is the sort of thing which needs documentation
which should be a skilled blend of unit-based and
integrative explanations.

Let's us know if you would like to join a MathSci team.
Please discuss further at https://gitter.im/MathSci/fecon236