Unfortunately, the three stocks that I want to back test do not exist in the SP100, SP500, or NASDAQ100 universes so I have to ingest the ALL universe which takes forever as I want to go back a few years. Is it in anyway possible that a user can specify the specific symbols that she/he wants to ingest to save time? Sorry, I am not that familiar with zipline so I don't know if this is possible.

Dear shlomikushchi

Do you have a plan to develop IB 1-minute streaming data bundle?

You know, alpaca has already developed pylivetrader that support alpaca data and order execution. It looks like re-invent wheels at zipline-trader.

ib data bundle is really urgent requirements.

Thanks for your great job.

Is support for pandas 1.x in the scope of zipline-trader? Is this up to quantopian/zipline? (quantopian#2243 quantopian#2616)

I started to see what the upgrading looks like
https://github.com/WillForan/zipline-trader/commits/pandas_update

nosetests -e zipline.utils.numpy_utils
...
FAILED (SKIP=6, errors=333, failures=66)

eek

Zipline Trader is currently using Q trading_calendars that is not maintained. Former Q Software Engineer Gerry Manoim started to maintain this callendar under new name exchange_calendars, however there are depreciated some items that zipline is using.

I have opened issue in Gerry's repo's if we could work together. If zipline trader can use Gerry's exchange_calendar or maintain it's own.

Currently Q trading_calendars is getting old and doesn't work the same way under old and new pandas.

When I run the code below I get different results:
pandas 0.22,: true
pandas 1.1.5 : false (No error reported)

import pandas as pd
import trading_calendars
from trading_calendars import TradingCalendar
from datetime import timedelta, time as dtime
pd.__version__
cal: TradingCalendar = trading_calendars.get_calendar('NYSE')
end_date = pd.Timestamp('now', tz='utc').date() - timedelta(days=1)
cal.is_session(end_date)

Quick work around is to add timestamp:
cal.is_session(pd.Timestamp(end_date, tz='utc')

Zipline Q trading_calendars used to have open times one minute later.

Why are open times one minute late?
Due to its historical use in the Zipline backtesting system, trading_calendars will only indicate a market is open upon the completion of the first minute bar in a day. Zipline uses minute bars labeled with the end of the bar, e.g. 9:31AM for 9:30-9:31AM. As an example, on a regular trading day for NYSE:

9:30:00 is treated as closed.
9:30:01 is treated as closed.
9:31:00 is the first time treated as open.
16:00:00 is treated as open
16:00:01 is treated as closed
This may change in the future.

When specifying date earlier than existing bundle, it gives this error:

(zipline-trader) C:\src\lycn\conda-environments\zipline-trader\src\zipline-trader>python zipline\data\bundles\alpaca_api.py
zipline\data\bundles\alpaca_api.py:276: UserWarning: Overwriting bundle with name 'alpaca_api'
def api_to_bundle(interval=['1m']):
zipline\data\bundles\alpaca_api.py:354: UserWarning: Overwriting bundle with name 'alpaca_api'
end_session=end_date

Traceback (most recent call last):
File "zipline\data\bundles\alpaca_api.py", line 362, in
show_progress=True,
File "c:\src\lycn\conda-environments\zipline-trader\src\zipline-trader\zipline\data\bundles\core.py", line 513, in ingest
pth.data_path([name, timestr], environ=environ),
File "zipline\data\bundles\alpaca_api.py", line 306, in ingest
daily_bar_writer.write(daily_data_generator(), assets=assets_to_sids.values(), show_progress=True)
File "c:\src\lycn\conda-environments\zipline-trader\src\zipline-trader\zipline\data\psql_daily_bars.py", line 617, in write
return self._write_internal(it, assets)
File "c:\src\lycn\conda-environments\zipline-trader\src\zipline-trader\zipline\data\psql_daily_bars.py", line 665, in _write_internal
for asset_id, table in iterator:
File "c:\src\lycn\conda-environments\zipline-trader\src\zipline-trader\zipline\data\psql_daily_bars.py", line 658, in iterator
for asset_id, table in iterator:
File "C:\Users\arisl\anaconda3\envs\zipline-trader\lib\site-packages\click_termui_impl.py", line 318, in generator
for rv in self.iter:
File "c:\src\lycn\conda-environments\zipline-trader\src\zipline-trader\zipline\data\psql_daily_bars.py", line 609, in
for sid, df in data
File "c:\src\lycn\conda-environments\zipline-trader\src\zipline-trader\zipline\data\psql_daily_bars.py", line 716, in _write_to_postgres
result = self._format_df_columns_and_index(data, sid)
File "c:\src\lycn\conda-environments\zipline-trader\src\zipline-trader\zipline\data\psql_daily_bars.py", line 731, in _write_to_postgres
result = self._validate_data_consistency_on_edges(sid, data, edge_days, invalid_data_behavior)
File "c:\src\lycn\conda-environments\zipline-trader\src\zipline-trader\zipline\data\psql_daily_bars.py", line 751, in _validate_data_consistency_on_edges
backward_gap = len(self._calendar.sessions_in_range(before_slice.index[-1], first_day))
File "C:\Users\arisl\anaconda3\envs\zipline-trader\lib\site-packages\trading_calendars\trading_calendar.py", line 702, in sessions_in_range
end_session_label
File "C:\Users\arisl\anaconda3\envs\zipline-trader\lib\site-packages\pandas\core\indexes\datetimes.py", line 815, in slice_indexer
return Index.slice_indexer(self, start, end, step, kind=kind)
File "C:\Users\arisl\anaconda3\envs\zipline-trader\lib\site-packages\pandas\core\indexes\base.py", line 4713, in slice_indexer
start_slice, end_slice = self.slice_locs(start, end, step=step, kind=kind)
File "C:\Users\arisl\anaconda3\envs\zipline-trader\lib\site-packages\pandas\core\indexes\base.py", line 4922, in slice_locs
raise ValueError("Both dates must have the same UTC offset")
ValueError: Both dates must have the same UTC offset

Starting zipline 1.6.0, algos with minute level data bundles and data fequency=minute is not working. Getting the below backtrace.
`
Traceback (most recent call last):
File "pandas/_libs/index.pyx", line 471, in pandas._libs.index.DatetimeEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 997, in pandas._libs.hashtable.Int64HashTable.get_item
TypeError: an integer is required

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/anaconda3/envs/new_zipline/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2646, in get_loc
return self._engine.get_loc(key)
File "pandas/_libs/index.pyx", line 433, in pandas._libs.index.DatetimeEngine.get_loc
File "pandas/_libs/index.pyx", line 473, in pandas._libs.index.DatetimeEngine.get_loc
File "pandas/_libs/index.pyx", line 479, in pandas._libs.index.DatetimeEngine._date_check_type
KeyError: 'NaT'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "pandas/_libs/index.pyx", line 471, in pandas._libs.index.DatetimeEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 997, in pandas._libs.hashtable.Int64HashTable.get_item
TypeError: an integer is required

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/anaconda3/envs/new_zipline/lib/python3.6/site-packages/pandas/core/indexes/datetimes.py", line 715, in get_loc
return Index.get_loc(self, key, method, tolerance)
File "/usr/local/anaconda3/envs/new_zipline/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2648, in get_loc
return self._engine.get_loc(self._maybe_cast_indexer(key))
File "pandas/_libs/index.pyx", line 433, in pandas._libs.index.DatetimeEngine.get_loc
File "pandas/_libs/index.pyx", line 473, in pandas._libs.index.DatetimeEngine.get_loc
File "pandas/_libs/index.pyx", line 479, in pandas._libs.index.DatetimeEngine._date_check_type
KeyError: 'NaT'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "pandas/_libs/index.pyx", line 463, in pandas._libs.index.DatetimeEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 997, in pandas._libs.hashtable.Int64HashTable.get_item
File "pandas/_libs/hashtable_class_helper.pxi", line 1004, in pandas._libs.hashtable.Int64HashTable.get_item
KeyError: -9223372036854775808

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/anaconda3/envs/new_zipline/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2646, in get_loc
return self._engine.get_loc(key)
File "pandas/_libs/index.pyx", line 433, in pandas._libs.index.DatetimeEngine.get_loc
File "pandas/_libs/index.pyx", line 465, in pandas._libs.index.DatetimeEngine.get_loc
KeyError: NaT

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "pandas/_libs/index.pyx", line 463, in pandas._libs.index.DatetimeEngine.get_loc
File "pandas/_libs/hashtable_class_helper.pxi", line 997, in pandas._libs.hashtable.Int64HashTable.get_item
File "pandas/_libs/hashtable_class_helper.pxi", line 1004, in pandas._libs.hashtable.Int64HashTable.get_item
KeyError: -9223372036854775808

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/anaconda3/envs/new_zipline/lib/python3.6/site-packages/pandas/core/indexes/datetimes.py", line 728, in get_loc
return Index.get_loc(self, stamp, method, tolerance)
File "/usr/local/anaconda3/envs/new_zipline/lib/python3.6/site-packages/pandas/core/indexes/base.py", line 2648, in get_loc
return self._engine.get_loc(self._maybe_cast_indexer(key))
File "pandas/_libs/index.pyx", line 433, in pandas._libs.index.DatetimeEngine.get_loc
File "pandas/_libs/index.pyx", line 465, in pandas._libs.index.DatetimeEngine.get_loc
KeyError: NaT

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "my_orb_attempt.py", line 248, in
data_frequency='minute'
File "/Users/saurav/PyAlgo/new_zipline/zipline-trader/zipline/utils/run_algo.py", line 489, in run_algorithm
execution_id=execution_id
File "/Users/saurav/PyAlgo/new_zipline/zipline-trader/zipline/utils/run_algo.py", line 207, in _run
adjustment_reader=bundle_data.adjustment_reader,
File "/Users/saurav/PyAlgo/new_zipline/zipline-trader/zipline/data/data_portal.py", line 291, in init
if self._first_trading_day is not None else (None, None)
File "/usr/local/anaconda3/envs/new_zipline/lib/python3.6/site-packages/trading_calendars/trading_calendar.py", line 844, in open_and_close_for_session
self.session_open(session_label),
File "/usr/local/anaconda3/envs/new_zipline/lib/python3.6/site-packages/trading_calendars/trading_calendar.py", line 871, in session_open
'market_open'
File "/usr/local/anaconda3/envs/new_zipline/lib/python3.6/site-packages/pandas/core/indexing.py", line 2178, in getitem
return self.obj._get_value(*key, takeable=self._takeable)
File "/usr/local/anaconda3/envs/new_zipline/lib/python3.6/site-packages/pandas/core/frame.py", line 2919, in _get_value
index = self.index.get_loc(index)
File "/usr/local/anaconda3/envs/new_zipline/lib/python3.6/site-packages/pandas/core/indexes/datetimes.py", line 730, in get_loc
raise KeyError(key)
KeyError: 'NaT'
`

The same algorithm file with same data bundle works fine on another environment where I have zipline 1.5.0 installed. Possibly pandas 1.1.5 is causing some issue but I am not able to root cause it.

Dear Zipline Maintainers,

pip install zipline-trader
ERROR: Could not find a version that satisfies the requirement zipline-trader (from versions: none)
ERROR: No matching distribution found for zipline-trader

Sincerely,

Env: Windows 10, Python 3.6.6, HDF5 installed

I get the same issue with current head, 1.6.0 and 1.5.0 releases.

I've tried many different ways to resolve this but I just can't seem to get past this problem:

(venv_366_new) C:\Users\Andrew\PycharmProjects\investing\zipline-trader>zipline
Traceback (most recent call last):
File "C:\Users\Andrew\PycharmProjects\investing\venv_366_new\Scripts\zipline-script.py", line 33, in
sys.exit(load_entry_point('zipline-trader', 'console_scripts', 'zipline')())
File "c:\users\andrew\pycharmprojects\investing\venv_366_new\lib\site-packages\pkg_resources_init_.py", line 474, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File "c:\users\andrew\pycharmprojects\investing\venv_366_new\lib\site-packages\pkg_resources_init_.py", line 2846, in load_entry_point
return ep.load()
File "c:\users\andrew\pycharmprojects\investing\venv_366_new\lib\site-packages\pkg_resources_init_.py", line 2450, in load
return self.resolve()
File "c:\users\andrew\pycharmprojects\investing\venv_366_new\lib\site-packages\pkg_resources_init_.py", line 2456, in resolve
module = import(self.module_name, fromlist=['name'], level=0)
File "c:\users\andrew\pycharmprojects\investing\zipline-trader\zipline_init_.py", line 29, in
from .utils.run_algo import run_algorithm
File "c:\users\andrew\pycharmprojects\investing\zipline-trader\zipline\utils\run_algo.py", line 24, in
from zipline.data import bundles
File "c:\users\andrew\pycharmprojects\investing\zipline-trader\zipline\data\bundles_init_.py", line 2, in
from . import quandl # noqa
File "c:\users\andrew\pycharmprojects\investing\zipline-trader\zipline\data\bundles\quandl.py", line 16, in
from . import core as bundles
File "c:\users\andrew\pycharmprojects\investing\zipline-trader\zipline\data\bundles\core.py", line 15, in
from ..minute_bars import (
File "c:\users\andrew\pycharmprojects\investing\zipline-trader\zipline\data\minute_bars.py", line 24, in
import h5py
File "c:\users\andrew\pycharmprojects\investing\venv_366_new\lib\site-packages\h5py_init_.py", line 34, in
from . import version
File "c:\users\andrew\pycharmprojects\investing\venv_366_new\lib\site-packages\h5py\version.py", line 17, in
from . import h5 as _h5
File "h5py\h5.pyx", line 1, in init h5py.h5
ImportError: DLL load failed: The specified procedure could not be found.

I've followed all the install instructions and still get this. What's odd is that if I do "import h5py" or "import zipline" in the same virtual env it works.

Any ideas???

Attempting to install this on ubuntu 20.04 LTS with python 3.8.5 (default with ubuntu) following the instructions on the zipline-trader docs page and I get the following error:

pip install -e git://github.com/shlomikushchi/zipline-trader.git#egg=zipline-trader
Obtaining zipline-trader from git+git://github.com/shlomikushchi/zipline-trader.git#egg=zipline-trader
  Cloning git://github.com/shlomikushchi/zipline-trader.git to ./venv/src/zipline-trader
  Running command git clone -q git://github.com/shlomikushchi/zipline-trader.git /home/rod/Projects/zipline/venv/src/zipline-trader
Collecting IbPy2==0.8.0
  Using cached IbPy2-0.8.0.tar.gz (54 kB)
Collecting Logbook>=0.12.5
  Using cached Logbook-1.5.3.tar.gz (85 kB)
Collecting PyYAML==5.3.1
  Using cached PyYAML-5.3.1.tar.gz (269 kB)
Collecting alembic>=0.7.7
  Using cached alembic-1.4.3-py2.py3-none-any.whl (159 kB)
Collecting alpaca-trade-api>=0.51.0
  Using cached alpaca_trade_api-0.51.0-py3-none-any.whl (39 kB)
Collecting bcolz==1.2.1
  Using cached bcolz-1.2.1.tar.gz (1.5 MB)
Processing /home/rod/.cache/pip/wheels/29/60/c9/98f744fb5c7d1ffb38d096318d9e873a08c0f5df07d6487626/Bottleneck-1.3.2-cp38-cp38-linux_x86_64.whl
Collecting click>=4.0.0
  Using cached click-7.1.2-py2.py3-none-any.whl (82 kB)
Processing /home/rod/.cache/pip/wheels/0d/68/bb/926065fb744e7d7cb67334cb1a9c696722abc8303e5dc9a8d0/empyrical-0.5.5-py3-none-any.whl
Collecting h5py==2.10.0
  Using cached h5py-2.10.0-cp38-cp38-manylinux1_x86_64.whl (2.9 MB)
Processing /home/rod/.cache/pip/wheels/45/23/de/5789a92962483fd33cb06674792b9697c1b3766d7c7742830e/intervaltree-3.1.0-py2.py3-none-any.whl
Collecting iso3166>=0.9
  Using cached iso3166-1.0.1-py2.py3-none-any.whl (8.2 kB)
Collecting iso4217>=1.6.20180829
  Using cached iso4217-1.6.20180829-py2.py3-none-any.whl (9.9 kB)
Processing /home/rod/.cache/pip/wheels/c6/91/94/23d462effc91bc844998b2023dc2dfd037a9d0a60034ddab16/lru_dict-1.1.6-cp38-cp38-linux_x86_64.whl
Collecting multipledispatch>=0.6.0
  Using cached multipledispatch-0.6.0-py3-none-any.whl (11 kB)
Collecting networkx<2.0,>=1.9.1
  Using cached networkx-1.11-py2.py3-none-any.whl (1.3 MB)
Collecting numexpr>=2.6.1
  Using cached numexpr-2.7.1-cp38-cp38-manylinux1_x86_64.whl (164 kB)
Collecting numpy>=1.11.3
  Using cached numpy-1.19.4-cp38-cp38-manylinux2010_x86_64.whl (14.5 MB)
Collecting pandas-datareader<0.9.0,>=0.2.1
  Using cached pandas_datareader-0.8.1-py2.py3-none-any.whl (107 kB)
Collecting pandas<=0.22,>=0.18.1
  Using cached pandas-0.22.0.tar.gz (11.3 MB)
  Installing build dependencies ... error
  ERROR: Command errored out with exit status 1:
   command: /home/rod/Projects/zipline/venv/bin/python3 /home/rod/Projects/zipline/venv/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-hkcts2ym/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- wheel setuptools Cython 'numpy==1.9.3; python_version=='"'"'3.5'"'"'' 'numpy==1.12.1; python_version=='"'"'3.6'"'"'' 'numpy==1.13.1; python_version>='"'"'3.7'"'"''
       cwd: None
  Complete output (4265 lines):
  Ignoring numpy: markers 'python_version == "3.5"' don't match your environment
  Ignoring numpy: markers 'python_version == "3.6"' don't match your environment
  Collecting wheel
    Using cached wheel-0.35.1-py2.py3-none-any.whl (33 kB)
  Collecting setuptools
    Using cached setuptools-50.3.2-py3-none-any.whl (785 kB)
  Collecting Cython
    Using cached Cython-0.29.21-cp38-cp38-manylinux1_x86_64.whl (1.9 MB)
  Collecting numpy==1.13.1
    Using cached numpy-1.13.1.zip (5.0 MB)
  Building wheels for collected packages: numpy
    Building wheel for numpy (setup.py): started
    Building wheel for numpy (setup.py): finished with status 'error'
    ERROR: Command errored out with exit status 1:

... after a few more pages of debug info ...

      numpy/random/mtrand/mtrand.c:44867:13: error: ‘PyThreadState’ {aka ‘struct _ts’} has no member named ‘exc_traceback’; did you mean ‘curexc_traceback’?
      44867 |     tstate->exc_traceback = local_tb;
            |             ^~~~~~~~~~~~~
            |             curexc_traceback
      error: Command "x86_64-linux-gnu-gcc -pthread -Wno-unused-result -Wsign-compare -DNDEBUG -g -fwrapv -O2 -Wall -g -fstack-protector-strong -Wformat -Werror=format-security -g -fwrapv -O2 -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE=1 -D_LARGEFILE64_SOURCE=1 -Inumpy/core/include -Ibuild/src.linux-x86_64-3.8/numpy/core/include/numpy -Inumpy/core/src/private -Inumpy/core/src -Inumpy/core -Inumpy/core/src/npymath -Inumpy/core/src/multiarray -Inumpy/core/src/umath -Inumpy/core/src/npysort -I/home/rod/Projects/zipline/venv/include -I/usr/include/python3.8 -Ibuild/src.linux-x86_64-3.8/numpy/core/src/private -Ibuild/src.linux-x86_64-3.8/numpy/core/src/npymath -Ibuild/src.linux-x86_64-3.8/numpy/core/src/private -Ibuild/src.linux-x86_64-3.8/numpy/core/src/npymath -Ibuild/src.linux-x86_64-3.8/numpy/core/src/private -Ibuild/src.linux-x86_64-3.8/numpy/core/src/npymath -c numpy/random/mtrand/mtrand.c -o build/temp.linux-x86_64-3.8/numpy/random/mtrand/mtrand.o -MMD -MF build/temp.linux-x86_64-3.8/numpy/random/mtrand/mtrand.o.d" failed with exit status 1
      ----------------------------------------
  ERROR: Command errored out with exit status 1: /home/rod/Projects/zipline/venv/bin/python3 -u -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'/tmp/pip-install-y97ro9vf/numpy/setup.py'"'"'; __file__='"'"'/tmp/pip-install-y97ro9vf/numpy/setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(__file__);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, __file__, '"'"'exec'"'"'))' install --record /tmp/pip-record-bha06kzm/install-record.txt --single-version-externally-managed --prefix /tmp/pip-build-env-hkcts2ym/overlay --compile --install-headers /home/rod/Projects/zipline/venv/include/site/python3.8/numpy Check the logs for full command output.
  ----------------------------------------
ERROR: Command errored out with exit status 1: /home/rod/Projects/zipline/venv/bin/python3 /home/rod/Projects/zipline/venv/lib/python3.8/site-packages/pip install --ignore-installed --no-user --prefix /tmp/pip-build-env-hkcts2ym/overlay --no-warn-script-location --no-binary :none: --only-binary :none: -i https://pypi.org/simple -- wheel setuptools Cython 'numpy==1.9.3; python_version=='"'"'3.5'"'"'' 'numpy==1.12.1; python_version=='"'"'3.6'"'"'' 'numpy==1.13.1; python_version>='"'"'3.7'"'"'' Check the logs for full command output.
(venv) rod@rod-XPS-13-9360:~/Projects/zipline$ ^C

After building docker and run zipline, it show below error.
pkg_resources.ContextualVersionConflict: (scipy 0.19.1 (/usr/local/lib/python3.6/site-packages), Requirement.parse('scipy>=1.1'), {'statsmodels'})

running pip install scipy==1.1

it works

Getting below error when I try to install it in windows 10 through Anaconda.
I did install latest C++ through Microsoft Visual Studio BuildTool. Not sure if any one of you experienced this or tried installing this in windows?

Thanks,
gv

==========================================================================
Installing collected packages: zipline-trader
Running setup.py develop for zipline-trader
ERROR: Command errored out with exit status 1:
command: 'C:\ProgramData\Anaconda3\envs\Zipline-Trader\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\WINDOWS\system32\src\zipline-trader\setup.py'"'"'; file='"'"'C:\WINDOWS\system32\src\zipline-trader\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' develop --no-deps
cwd: C:\WINDOWS\system32\src\zipline-trader
Complete output (34 lines):
running develop
running egg_info
writing zipline_trader.egg-info\PKG-INFO
writing dependency_links to zipline_trader.egg-info\dependency_links.txt
writing entry points to zipline_trader.egg-info\entry_points.txt
writing requirements to zipline_trader.egg-info\requires.txt
writing top-level names to zipline_trader.egg-info\top_level.txt
reading manifest file 'zipline_trader.egg-info\SOURCES.txt'
reading manifest template 'MANIFEST.in'
writing manifest file 'zipline_trader.egg-info\SOURCES.txt'
running build_ext
skipping 'zipline/assets_assets.c' Cython extension (up-to-date)
skipping 'zipline/assets\continuous_futures.c' Cython extension (up-to-date)
skipping 'zipline/lib\adjustment.c' Cython extension (up-to-date)
skipping 'zipline/lib_factorize.c' Cython extension (up-to-date)
skipping 'zipline/lib_float64window.c' Cython extension (up-to-date)
skipping 'zipline/lib_int64window.c' Cython extension (up-to-date)
skipping 'zipline/lib_int64window.c' Cython extension (up-to-date)
skipping 'zipline/lib_uint8window.c' Cython extension (up-to-date)
skipping 'zipline/lib_labelwindow.c' Cython extension (up-to-date)
skipping 'zipline/lib\rank.c' Cython extension (up-to-date)
skipping 'zipline/data_equities.c' Cython extension (up-to-date)
skipping 'zipline/data_adjustments.c' Cython extension (up-to-date)
skipping 'zipline_protocol.c' Cython extension (up-to-date)
skipping 'zipline/finance_finance_ext.c' Cython extension (up-to-date)
skipping 'zipline/gens\sim_engine.c' Cython extension (up-to-date)
skipping 'zipline/data_minute_bar_internal.c' Cython extension (up-to-date)
skipping 'zipline/data_resample.c' Cython extension (up-to-date)
skipping 'zipline/pipeline/loaders/blaze_core.c' Cython extension (up-to-date)
building 'zipline.assets._assets' extension
C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.28.29333\bin\HostX86\x64\cl.exe /c /nologo /Ox /W3 /GL /DNDEBUG /MD -IC:\ProgramData\Anaconda3\envs\Zipline-Trader\lib\site-packages\numpy\core\include -IC:\ProgramData\Anaconda3\envs\Zipline-Trader\include -IC:\ProgramData\Anaconda3\envs\Zipline-Trader\include "-IC:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.28.29333\include" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\ucrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\shared" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\um" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\winrt" "-IC:\Program Files (x86)\Windows Kits\10\include\10.0.18362.0\cppwinrt" /Tczipline/assets_assets.c /Fobuild\temp.win-amd64-3.6\Release\zipline/assets_assets.obj
_assets.c
c1: fatal error C1083: Cannot open source file: 'zipline/assets_assets.c': No such file or directory
error: command 'C:\Program Files (x86)\Microsoft Visual Studio\2019\BuildTools\VC\Tools\MSVC\14.28.29333\bin\HostX86\x64\cl.exe' failed with exit status 2
----------------------------------------
ERROR: Command errored out with exit status 1: 'C:\ProgramData\Anaconda3\envs\Zipline-Trader\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\WINDOWS\system32\src\zipline-trader\setup.py'"'"'; file='"'"'C:\WINDOWS\system32\src\zipline-trader\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' develop --no-deps Check the logs for full command output.
(Zipline-Trader) PS C:\WINDOWS\system32>

I built a Docker container from the master branch. The build completed with the errors:

statsmodels 0.12.2 has requirement scipy>=1.1, but you'll have scipy 0.19.1 which is incompatible.
aiohttp 3.7.3 has requirement chardet<4.0,>=2.0, but you'll have chardet 4.0.0 which is incompatible.

This makes zipline unusable due to version conflict:

raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (scipy 0.19.1 (/usr/local/lib/python3.6/site-packages), Requirement.parse('scipy>=1.1'), {'statsmodels'})

Is there a quick fix or a workaround I could use? I tried to change version numbers for the built, but that lead to other errors.

Regards,
MS

Encountered when ingesting alpha vantage data with a universe specified in the configuration file.

Please update the link!

Dear Zipline Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

Now that you know a little about me, let me tell you about the issue I am
having:

Description of Issue

• What did you expect to happen?
  I expected to run the algo on Linux but I got an error message
• What happened instead?
  I got the following error:
  ValueError: SQLite file '/home/ubuntu/.zipline/data/alpaca_api/2020-12-07T02;06;17.365878/assets-7.sqlite' doesn't exist.

Here is how you can reproduce this issue on your machine:

Reproduction Steps

1. I used a brand new enviroment but it didn’t resolve the issue.

...
Let me tell everyone what I did with Dan's help:
in my root folder /home/ubuntu: git clone https://github.com/shlomikushchi/zipline-trader.git
conda create -n zipline-trader_py36_v1 python=3.6
conda activate zipline-trader_py36_v1
cd zipline-trader
pip install -e .
change the path for alpaca.yaml in alpaca_api.py to /home/ubuntu/.zipline/alpaca.yaml
add ZIPLINE_ROOT=/home/ubuntu/.zipline my ~/.bashrc file and restart or source ~/.bashrc
I got the following error
alpaca_api.py:259: UserWarning: Overwriting bundle with name 'alpaca_api'
def api_to_bundle(interval=['1m']):
alpaca_api.py:334: UserWarning: Overwriting bundle with name 'alpaca_api'
end_session=end_date
Traceback (most recent call last):
File "/home/ubuntu/anaconda3/envs/zipline-trader_py36_v6/lib/python3.6/site-packages/alpaca_trade_api/rest.py", line 140, in _one_request
resp.raise_for_status()
File "/home/ubuntu/anaconda3/envs/zipline-trader_py36_v6/lib/python3.6/site-packages/requests/models.py", line 943, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 404 Client Error: Not Found for url: https://paper-api.alpaca.markets/v2/assets

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "alpaca_api.py", line 342, in
show_progress=True,
File "/home/ubuntu/zipline-trader/zipline/data/bundles/core.py", line 443, in ingest
pth.data_path([name, timestr], environ=environ),
File "alpaca_api.py", line 284, in ingest
metadata = metadata_df()
File "alpaca_api.py", line 253, in metadata_df
np.empty(len(list_assets()), dtype=metadata_dtype))
File "alpaca_api.py", line 40, in list_assets
ASSETS = [_.symbol for _ in CLIENT.list_assets()]
File "/home/ubuntu/anaconda3/envs/zipline-trader_py36_v6/lib/python3.6/site-packages/alpaca_trade_api/rest.py", line 405, in list_assets
resp = self.get('/assets', params)
File "/home/ubuntu/anaconda3/envs/zipline-trader_py36_v6/lib/python3.6/site-packages/alpaca_trade_api/rest.py", line 156, in get
return self._request('GET', path, data)
File "/home/ubuntu/anaconda3/envs/zipline-trader_py36_v6/lib/python3.6/site-packages/alpaca_trade_api/rest.py", line 119, in _request
return self._one_request(method, url, opts, retry)
File "/home/ubuntu/anaconda3/envs/zipline-trader_py36_v6/lib/python3.6/site-packages/alpaca_trade_api/rest.py", line 148, in _one_request
raise APIError(error, http_error)
alpaca_trade_api.rest.APIError: account not found for f3dbfcc6-839c-4de6-911e-37fcf9bdf180

What steps have you taken to resolve this already?

...

Anything else?

when I do the source ~/.bashrc it exits my environment

and conda activate zipline-trader_py36_v1

python alpaca_api.py returns this error:
Traceback (most recent call last):
File "alpaca_api.py", line 16, in
from zipline.data.bundles import core as bundles
ModuleNotFoundError: No module named 'zipline'

I am surprised that it means that I never installed zipline-trader there

will you please elaborate?
...

Sincerely,
$ whoami

https://github.com/fangchenli/zipline
quantopian/zipline@master...fangchenli:master

    stop testing 2.7 and 3.5

remove py2 compat shim
remove blaze and odo

The documentation says custom asset list is an alternative if universe is selected. But if don't put custom asset list in config, it gives key error:

(zipline-trader) C:\zipline-trader>python zipline\data\bundles\alpaca_api.py
zipline\data\bundles\alpaca_api.py:276: UserWarning: Overwriting bundle with name 'alpaca_api'
def api_to_bundle(interval=['1m']):
zipline\data\bundles\alpaca_api.py:354: UserWarning: Overwriting bundle with name 'alpaca_api'
end_session=end_date
Traceback (most recent call last):
File "zipline\data\bundles\alpaca_api.py", line 362, in
show_progress=True,
File "c:\zipline-trader\zipline\data\bundles\core.py", line 513, in ingest
pth.data_path([name, timestr], environ=environ),
File "zipline\data\bundles\alpaca_api.py", line 290, in ingest
assets_to_sids = asset_to_sid_map(asset_db_writer.asset_finder, list_assets())
File "zipline\data\bundles\alpaca_api.py", line 43, in list_assets
custom_asset_list = conf.custom_asset_list
File "c:\zipline-trader\config\bundle.py", line 47, in custom_asset_list
return self.al["custom_asset_list"]
KeyError: 'custom_asset_list'

An error happened when copy and run the backtest example from https://zipline-trader.readthedocs.io/en/latest/backtest.html. Error details is:

(zipline-trader) C:\src\lycn\zipline-trader-start>python main.py
Traceback (most recent call last):
File "main.py", line 9, in
from zipline.utils.calendars import get_calendar
File "c:\src\lycn\conda-environments\zipline-trader\src\zipline-trader\zipline_init_.py", line 29, in
from .utils.run_algo import run_algorithm
File "c:\src\lycn\conda-environments\zipline-trader\src\zipline-trader\zipline\utils\run_algo.py", line 30, in
from zipline.pipeline.data import USEquityPricing
File "c:\src\lycn\conda-environments\zipline-trader\src\zipline-trader\zipline\pipeline_init_.py", line 8, in
from .graph import ExecutionPlan, TermGraph
File "c:\src\lycn\conda-environments\zipline-trader\src\zipline-trader\zipline\pipeline\graph.py", line 6, in
import networkx as nx
File "C:\Users\arisl\anaconda3\envs\zipline-trader\lib\site-packages\networkx_init_.py", line 61, in
import networkx.generators
File "C:\Users\arisl\anaconda3\envs\zipline-trader\lib\site-packages\networkx\generators_init_.py", line 16, in
from networkx.generators.intersection import *
File "C:\Users\arisl\anaconda3\envs\zipline-trader\lib\site-packages\networkx\generators\intersection.py", line 5, in
from networkx.algorithms import bipartite
File "C:\Users\arisl\anaconda3\envs\zipline-trader\lib\site-packages\networkx\algorithms_init_.py", line 79, in
import networkx.algorithms.tree
File "C:\Users\arisl\anaconda3\envs\zipline-trader\lib\site-packages\networkx\algorithms\tree_init_.py", line 6, in
from .decomposition import *
File "C:\Users\arisl\anaconda3\envs\zipline-trader\lib\site-packages\networkx\algorithms\tree\decomposition.py", line 5, in
from networkx.algorithms import moral, complete_to_chordal_graph, chordal_graph_cliques
ImportError: cannot import name 'complete_to_chordal_graph'

Before I go off and try to reinvent the wheel, is there any "good" way to integrate fundamentals? I use pipeline-live to pull in my iex data for my pylivetrader algos, but I'd like to run some more backtests, and zt looks like my best bet so far. Would there be any way to maybe jimmy pipeline-live to work with zt, or am I looking at starting from scratch if I want fundamentals?

I know you already said anaconda isn't supported, and have already said so in the docs. However, I figured it'd at least be useful for people transitioning to this alternative that have already been using anaconda successfully to see where the project stands on that point.

I know a little python at this point, and don't mind throwing my hand in, but I'd appreciate a point in the right direction if possible.

https://github.com/shlomikushchi/zipline-trader/blob/dbfe73d32072f0a926b1153bad636245e9a359e6/zipline/data/psql_daily_bars.py#L19

Looks like the config package has been included in the 1.60 build at the root of the package but is not available once the package is installed, so I get:

from zipline import run_algorithm
Traceback (most recent call last):
File "C:\Users\Andrew\PycharmProjects\investing\venv_366\lib\site-packages\IPython\core\interactiveshell.py", line 3343, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 1, in
from zipline import run_algorithm
File "C:\Program Files\JetBrains\PyCharm Community Edition 2020.3.2\plugins\python-ce\helpers\pydev_pydev_bundle\pydev_import_hook.py", line 21, in do_import
module = self.system_import(name, *args, **kwargs)
File "C:\Users\Andrew\PycharmProjects\investing\venv_366\lib\site-packages\zipline_init.py", line 29, in
from .utils.run_algo import run_algorithm
File "C:\Program Files\JetBrains\PyCharm Community Edition 2020.3.2\plugins\python-ce\helpers\pydev_pydev_bundle\pydev_import_hook.py", line 21, in do_import
module = self._system_import(name, *args, **kwargs)
File "C:\Users\Andrew\PycharmProjects\investing\venv_366\lib\site-packages\zipline\utils\run_algo.py", line 24, in
from zipline.data import bundles
File "C:\Program Files\JetBrains\PyCharm Community Edition 2020.3.2\plugins\python-ce\helpers\pydev_pydev_bundle\pydev_import_hook.py", line 21, in do_import
module = self.system_import(name, *args, **kwargs)
File "C:\Users\Andrew\PycharmProjects\investing\venv_366\lib\site-packages\zipline\data\bundles_init.py", line 2, in
from . import quandl # noqa
File "C:\Program Files\JetBrains\PyCharm Community Edition 2020.3.2\plugins\python-ce\helpers\pydev_pydev_bundle\pydev_import_hook.py", line 21, in do_import
module = self._system_import(name, *args, **kwargs)
File "C:\Users\Andrew\PycharmProjects\investing\venv_366\lib\site-packages\zipline\data\bundles\quandl.py", line 16, in
from . import core as bundles
File "C:\Program Files\JetBrains\PyCharm Community Edition 2020.3.2\plugins\python-ce\helpers\pydev_pydev_bundle\pydev_import_hook.py", line 21, in do_import
module = self._system_import(name, *args, **kwargs)
File "C:\Users\Andrew\PycharmProjects\investing\venv_366\lib\site-packages\zipline\data\bundles\core.py", line 19, in
from ..psql_daily_bars import PSQLDailyBarReader, PSQLDailyBarWriter
File "C:\Program Files\JetBrains\PyCharm Community Edition 2020.3.2\plugins\python-ce\helpers\pydev_pydev_bundle\pydev_import_hook.py", line 21, in do_import
module = self._system_import(name, *args, **kwargs)
File "C:\Users\Andrew\PycharmProjects\investing\venv_366\lib\site-packages\zipline\data\psql_daily_bars.py", line 19, in
import config.data_backend
File "C:\Program Files\JetBrains\PyCharm Community Edition 2020.3.2\plugins\python-ce\helpers\pydev_pydev_bundle\pydev_import_hook.py", line 21, in do_import
module = self._system_import(name, *args, **kwargs)
ModuleNotFoundError: No module named 'config'

at line 19 in psql_daily_bars.py:

import config.data_backend

ERROR: Failed building wheel for psycopg2 received when installed from latest master.

Workaround is to install psycopg2 package before install from conda forge conda install -c conda-forge psycopg2

I often get this error when running

export ZIPLINE_TRADER_CONFIG=./config.yaml && export ZIPLINE_ROOT=~/.zipline && python zipline/data/bundles/alpha_vantage_api.py

  File "zipline/data/bundles/alpha_vantage_api.py", line 406, in <module>
    show_progress=True,
  File "/Users/rileysteeleparsons/Projects/zipline-trader/zipline/data/bundles/core.py", line 513, in ingest
    pth.data_path([name, timestr], environ=environ),
  File "zipline/data/bundles/alpha_vantage_api.py", line 353, in ingest
    invalid_data_behavior='raise')
  File "/Users/rileysteeleparsons/Projects/zipline-trader/zipline/data/bcolz_daily_bars.py", line 207, in write
    return self._write_internal(it, assets)
  File "/Users/rileysteeleparsons/Projects/zipline-trader/zipline/data/bcolz_daily_bars.py", line 330, in _write_internal
    ).difference(asset_sessions).tolist(),
AssertionError: Got 4850 rows for daily bars table with first day=2001-11-29, last day=2021-03-08, expected 4849 rows.

Hi,

Why choosing this package over zipline-reloaded?
https://github.com/stefan-jansen/zipline-reloaded

Thanks in advance

Hi, since Slack link was dead, is there any other place for the discussion?

Can you post an example of a full backtest with the statistics?

I cant seem to be able to ingest data which didnt come from the alpaca api, which is only availible for us citizens. Is there any way to get the quandl ingestion working? I had to modify the quandl.py by removing the relative imports, but still im not able to ingest any data. I know @shlomikushchi alraidy stated in #142 that there is currently only the alpaca api supported, has thisby any chance changed now or will it ever change? I would submit a PR, but atm im not able to get the quandl ingestion working...

zipline run -f ~/zipline-algos/demo.py --state-file ~/zipline-algos/demo.state --realtime-bar-target ~/zipline-algos/realtime-bars/ --broker ib --broker-uri localhost:7496:1232 --bundle quantopian-quandl --data-frequency minute

Usage: zipline run [OPTIONS]
Try 'zipline run --help' for help.

Error: no such option: --state-file

pip list

Package Version Location

alembic 1.4.3
alpaca-trade-api 0.51.0
argon2-cffi 20.1.0
async-generator 1.10
attrs 20.2.0
backcall 0.2.0
bcolz 0.12.1
bleach 3.2.1
Bottleneck 1.3.2
certifi 2020.6.20
cffi 1.14.3
chardet 3.0.4
click 7.1.2
contextlib2 0.6.0.post1
cycler 0.10.0
cyordereddict 1.0.0
Cython 0.29.21
decorator 4.4.2
defusedxml 0.6.0
empyrical 0.5.5
entrypoints 0.3
h5py 2.10.0
idna 2.10
iexfinance 0.4.3
importlib-metadata 2.0.0
intervaltree 3.1.0
ipykernel 5.3.4
ipython 7.16.1
ipython-genutils 0.2.0
ipywidgets 7.5.1
iso3166 1.0.1
iso4217 1.6.20180829
jedi 0.17.2
Jinja2 2.11.2
jsonschema 3.2.0
jupyter 1.0.0
jupyter-client 6.1.7
jupyter-console 6.2.0
jupyter-core 4.6.3
jupyterlab-pygments 0.1.2
kiwisolver 1.2.0
Logbook 1.5.3
lru-dict 1.1.6
lxml 4.6.1
Mako 1.1.3
MarkupSafe 1.1.1
matplotlib 3.3.2
mistune 0.8.4
multipledispatch 0.6.0
nbclient 0.5.1
nbconvert 6.0.7
nbformat 5.0.8
nest-asyncio 1.4.1
networkx 1.11
notebook 6.1.4
numexpr 2.7.1
numpy 1.19.2
packaging 20.4
pandas 0.22.0
pandas-datareader 0.8.1
pandocfilters 1.4.2
parso 0.7.1
patsy 0.5.1
pexpect 4.8.0
pickleshare 0.7.5
Pillow 8.0.1
pip 18.1
pipeline-live 0.1.9
prometheus-client 0.8.0
prompt-toolkit 3.0.8
ptyprocess 0.6.0
pycparser 2.20
Pygments 2.7.1
pyparsing 2.4.7
pyrsistent 0.17.3
python-dateutil 2.8.1
python-editor 1.0.4
python-interface 1.6.0
pytz 2020.1
pyzmq 19.0.2
qtconsole 4.7.7
QtPy 1.9.0
requests 2.24.0
requests-file 1.5.1
scipy 1.1.0
Send2Trash 1.5.0
setuptools 40.4.3
six 1.15.0
sortedcontainers 2.2.2
SQLAlchemy 1.3.20
statsmodels 0.12.0
TA-Lib 0.4.19
tables 3.6.1
terminado 0.9.1
testpath 0.4.4
toolz 0.11.1
tornado 6.0.4
trading-calendars 2.0.0
traitlets 4.3.3
urllib3 1.25.11
wcwidth 0.2.5
webencodings 0.5.1
websocket-client 0.57.0
websockets 8.1
wheel 0.32.1
widgetsnbextension 3.5.1
zipline 1.3.0
zipline-trader 0+unknown /zipline
zipp 3.3.1

like Blosc/bcolz#398

export DISABLE_BCOLZ_AVX2=true

helps

I tried local use IbPy2, It seems like IbPy2 not work with IB Trader Workstation.

When running back_test example, with this commit up to current commit I get error below.

When I run the same example above with version 1.5.0 there is no problem.

NOTE: After version 1.5.0, there had to be introduced bug related to loading data bundle as timestamp in the error change when bundle ingestion date change. And it is usually one day after bundle start date.

Error msg: KeyError: Timestamp('2012-11-30 05:00:00+0000', tz='UTC')

KeyError                                  Traceback (most recent call last)
pandas/_libs/index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.Int64HashTable.get_item()

KeyError: 1354251600000000000

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)
~/OneDrive/02_Research/examples/zipline_trader/backtesting.py in 
     96                   before_trading_start=before_trading_start,
     97 #                   analyze=analyze,
---> 98                   data_frequency='daily'
     99                   )
    100 fig, axes = plt.subplots(1, 1, figsize=(16, 5), sharex=True)

~/zipline-trader/zipline/utils/run_algo.py in run_algorithm(start, end, initialize, capital_base, handle_data, before_trading_start, analyze, teardown, data_frequency, bundle, bundle_timestamp, trading_calendar, metrics_set, benchmark_returns, default_extension, extensions, strict_extensions, environ, blotter, broker, performance_callback, stop_execution_callback, execution_id, state_filename, realtime_bar_target)
    487         performance_callback=performance_callback,
    488         stop_execution_callback=stop_execution_callback,
--> 489         execution_id=execution_id
    490     )
    491 

~/zipline-trader/zipline/utils/run_algo.py in _run(handle_data, initialize, before_trading_start, analyze, algofile, algotext, defines, data_frequency, capital_base, bundle, bundle_timestamp, start, end, output, trading_calendar, print_algo, metrics_set, local_namespace, environ, blotter, benchmark_spec, broker, state_filename, realtime_bar_target, performance_callback, stop_execution_callback, teardown, execution_id)
    205         equity_minute_reader=bundle_data.equity_minute_bar_reader,
    206         equity_daily_reader=bundle_data.equity_daily_bar_reader,
--> 207         adjustment_reader=bundle_data.adjustment_reader,
    208     )
    209 

~/zipline-trader/zipline/data/data_portal.py in __init__(self, asset_finder, trading_calendar, first_trading_day, equity_daily_reader, equity_minute_reader, future_daily_reader, future_minute_reader, adjustment_reader, last_available_session, last_available_minute, minute_history_prefetch_length, daily_history_prefetch_length)
    289                 self._first_trading_day
    290             )
--> 291             if self._first_trading_day is not None else (None, None)
    292         )
    293 

~/anaconda3/envs/zt151/lib/python3.6/site-packages/trading_calendars/trading_calendar.py in open_and_close_for_session(self, session_label)
    842         """
    843         return (
--> 844             self.session_open(session_label),
    845             self.session_close(session_label)
    846         )

~/anaconda3/envs/zt151/lib/python3.6/site-packages/trading_calendars/trading_calendar.py in session_open(self, session_label)
    869         return self.schedule.at[
    870             session_label,
--> 871             'market_open'
    872         ].tz_localize(UTC)
    873 

~/anaconda3/envs/zt151/lib/python3.6/site-packages/pandas/core/indexing.py in __getitem__(self, key)
   1867 
   1868         key = self._convert_key(key)
-> 1869         return self.obj._get_value(*key, takeable=self._takeable)
   1870 
   1871     def __setitem__(self, key, value):

~/anaconda3/envs/zt151/lib/python3.6/site-packages/pandas/core/frame.py in _get_value(self, index, col, takeable)
   1983 
   1984         try:
-> 1985             return engine.get_value(series._values, index)
   1986         except (TypeError, ValueError):
   1987 

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_value()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_value()

pandas/_libs/index.pyx in pandas._libs.index.DatetimeEngine.get_loc()

KeyError: Timestamp('2012-11-30 05:00:00+0000', tz='UTC')

Dear Zipline Maintainers,

Before I tell you about my issue, let me describe my environment:

Environment

Now that you know a little about me, let me tell you about the issue I am
having:

Description of Issue

• What did you expect to happen? Install zipline-trader
• What happened instead? It cannot install bcolz after doing all of the items mentioned on the installation page

Here is how you can reproduce this issue on your machine:

Reproduction Steps

1. I used a brand new environment

...

What steps have you taken to resolve this already?

...

Anything else?

...

Sincerely,
$ whoami

Dear community,

this is my very first posting on Github, so apologies upfront if I am not following the format you require.

I run a simple backtest (full code below) and would like to set the model's maximum leverage to 1. I read in the official documentation that there is a function set_max_leverage (max_leverage) but I am unsure how and where to integrate it into my code.

May I please ask you for advice?

Thank you very much in advance!

from zipline import run_algorithm
from zipline.api import order_target_percent, symbol, schedule_function,\
    date_rules, time_rules
from datetime import datetime
import pandas as pd
import pytz
import pyfolio as pf

def initialize(context):
    # Which stocks to trade
    dji = [
        "AAPL",
        "AXP",
        "BA",
        "CAT",
        "CSCO",
        "CVX",
        "DIS",
        "DWDP",
        "GS",
        "HD",
        "IBM",
        "INTC",
        "JNJ",
        "JPM",
        "KO",
        "MCD",
        "MMM",
        "MRK",
        "MSFT",
        "NKE",
        "PFE",
        "PG",
        "TRV",
        "UNH",
        "UTX",
        "V",
        "VZ",
        "WBA",
        "WMT",
        "XOM"
    ]
    
    # Make symbol list from tickers
    context.universe = [symbol(s) for s in dji]
    
    # History window
    context.history_window = 20
    
    # Size of our portfolio
    context.stocks_to_hold = 10
    
    # Schedule the daily trading routine for once per month
    schedule_function(handle_data, date_rules.month_start(),\
                      time_rules.market_close())

def month_perf(ts):
    perf = (ts[-1] / ts[0]) - 1
    return perf

def handle_data (context, data):
    # Get history for all the stocks
    hist = data.history (context.universe, "close", context.history_window, "1d")
    
    # This creates a table of percent returns, in order
    perf_table = hist.apply(month_perf).sort_values(ascending = False)
    
    # Make buy list of the top N stocks
    buy_list = perf_table[ : context.stocks_to_hold]
    
    # The rest will not be held
    the_rest = perf_table[context.stocks_to_hold : ]
    
    # Place target buy orders for top N stocks
    for stock, perf in buy_list.iteritems():
        stock_weight = 1 / context.stocks_to_hold
        
        # Place order
        if data.can_trade(stock):
            order_target_percent(stock, stock_weight)
            
    # Make sure we are flat the rest
    for stock, perf in the_rest.iteritems():
        
        # Place order
        if data.can_trade(stock):
            order_target_percent(stock, 0.0)
            
def analyze (context, perf):
    # Use PyFolio to generate a performance report
    returns, positions, transactions = pf.utils.extract_rets_pos_txn_from_zipline(perf)
    pf.create_returns_tear_sheet(returns, benchmark_rets = None)
    
# Set start and end date
start = pd.to_datetime("2003-1-1", utc = True)
end = pd.to_datetime("2017-12-31", utc = True)

# Fire off the backtest
result = run_algorithm(start = start, end = end, initialize = initialize, 
                       analyze = analyze, capital_base = 10000,
                       data_frequency = "daily", bundle = "quandl")  

I try to load historical data from alpaca for custom_asset_list: AAPL, TSLA, GOOG, and changed time period to:

end_date = pd.Timestamp('2005-02-01 0:00', tz='utc')
start_date = end_date - timedelta(days=365)

It gives IndexError: index 0 is out of bounds for axis 0 with size 0 error:

C:\Users\arisl\anaconda3\envs\zipline-trader\python.exe C:/src/lycn/conda-environments/zipline-trader/src/zipline-trader/zipline/data/bundles/alpaca_api.py
C:/src/lycn/conda-environments/zipline-trader/src/zipline-trader/zipline/data/bundles/alpaca_api.py:276: UserWarning: Overwriting bundle with name 'alpaca_api'
def api_to_bundle(interval=['1m']):
C:/src/lycn/conda-environments/zipline-trader/src/zipline-trader/zipline/data/bundles/alpaca_api.py:355: UserWarning: Overwriting bundle with name 'alpaca_api'
end_session=end_date
Traceback (most recent call last):
File "C:/src/lycn/conda-environments/zipline-trader/src/zipline-trader/zipline/data/bundles/alpaca_api.py", line 363, in
show_progress=True,
File "C:\src\lycn\conda-environments\zipline-trader\src\zipline-trader\zipline\data\bundles\core.py", line 513, in ingest
pth.data_path([name, timestr], environ=environ),
File "C:/src/lycn/conda-environments/zipline-trader/src/zipline-trader/zipline/data/bundles/alpaca_api.py", line 306, in ingest
daily_bar_writer.write(daily_data_generator(), assets=assets_to_sids.values(), show_progress=True)
File "C:\src\lycn\conda-environments\zipline-trader\src\zipline-trader\zipline\data\psql_daily_bars.py", line 617, in write
return self._write_internal(it, assets)
File "C:\src\lycn\conda-environments\zipline-trader\src\zipline-trader\zipline\data\psql_daily_bars.py", line 665, in _write_internal
for asset_id, table in iterator:
File "C:\src\lycn\conda-environments\zipline-trader\src\zipline-trader\zipline\data\psql_daily_bars.py", line 658, in iterator
for asset_id, table in iterator:
File "C:\Users\arisl\anaconda3\envs\zipline-trader\lib\site-packages\click_termui_impl.py", line 315, in generator
for rv in self.iter:
File "C:\src\lycn\conda-environments\zipline-trader\src\zipline-trader\zipline\data\psql_daily_bars.py", line 608, in
(sid, self._write_to_postgres(sid, df, invalid_data_behavior))
File "C:/src/lycn/conda-environments/zipline-trader/src/zipline-trader/zipline/data/bundles/alpaca_api.py", line 299, in
return (sid_df for (sid_df, *metadata.iloc[sid_df[0]]) in df_generator(interval='1d',
File "C:/src/lycn/conda-environments/zipline-trader/src/zipline-trader/zipline/data/bundles/alpaca_api.py", line 238, in df_generator
df: pd.DataFrame = get_aggs_from_alpaca(partial, start, end, 'day' if interval == '1d' else 'minute', 1)
File "C:/src/lycn/conda-environments/zipline-trader/src/zipline-trader/zipline/data/bundles/alpaca_api.py", line 203, in get_aggs_from_alpaca
response = _iterate_api_calls()
File "C:/src/lycn/conda-environments/zipline-trader/src/zipline-trader/zipline/data/bundles/alpaca_api.py", line 128, in _iterate_api_calls
if response.index[0] <= (pytz.timezone(NY).localize(
File "C:\Users\arisl\anaconda3\envs\zipline-trader\lib\site-packages\pandas\core\indexes\extension.py", line 209, in getitem
result = self._data[key]
File "C:\Users\arisl\anaconda3\envs\zipline-trader\lib\site-packages\pandas\core\arrays\datetimelike.py", line 514, in getitem
val = getitem(key)
IndexError: index 0 is out of bounds for axis 0 with size 0

How can I raise the API call limit to take advantage of the unlimited API calls?

I wanted to ask for help with the bug.

First ingestion to psql works fine. When I run second time ingestion, I get error located in zipine/data/psql_daily_bars ,py line 744:

before_slice = data[data.index < first_day]

first_day is tz-naive Timestamp('2020-03-05 00:00:00')
data.index is tz-aware DatetimeIndex(['2020-03-05 00:00:00+00:00', '2020-03-06 00:00:00+00:00',...

Note: Bug is related to the part where zipline is reading bars from psql as when I delete psql alpaca_api psql db , data are loaded to database but when data are once in psql the error occures again...

Traceback:

Cannot compare tz-naive and tz-aware datetime-like objects
  File "/home/z/zipline-trader/zipline/data/psql_daily_bars.py", line 744, in _validate_data_consistency_on_edges
    before_slice = data[data.index < first_day]
  File "/home/user/zipline-trader/zipline/data/psql_daily_bars.py", line 731, in _write_to_postgres
    result = self._validate_data_consistency_on_edges(sid, data, edge_days, invalid_data_behavior)
  File "/home/z/zipline-trader/zipline/data/psql_daily_bars.py", line 716, in _write_to_postgres
    result = self._format_df_columns_and_index(data, sid)
  File "/home/z/zipline-trader/zipline/data/psql_daily_bars.py", line 609, in <genexpr>
    for sid, df in data
  File "/home/z/zipline-trader/zipline/data/psql_daily_bars.py", line 658, in iterator
    for asset_id, table in iterator:
  File "/home/z/zipline-trader/zipline/data/psql_daily_bars.py", line 665, in _write_internal
    for asset_id, table in iterator:
  File "/home/z/zipline-trader/zipline/data/psql_daily_bars.py", line 617, in write
    return self._write_internal(it, assets)
  File "/home/z/zipline-trader/zipline/data/bundles/alpaca_api.py", line 306, in ingest
    daily_bar_writer.write(daily_data_generator(), assets=assets_to_sids.values(), show_progress=True)
  File "/home/z/zipline-trader/zipline/data/bundles/core.py", line 513, in ingest
    pth.data_path([name, timestr], environ=environ),
  File "/home/z/zipline-trader/zipline/data/bundles/alpaca_api.py", line 377, in <module>
    show_progress=True,```

/utils/run_algo.py line 201 will be 'NaT'

Can fix this easily with the commented "first_trading_day =
bundle_data.equity_minute_bar_reader.first_trading_day"

May make a pull request.

I run the cli of zipline-trader going to live-trading and got the errors just below:

(ziptrader36) C:\Users\Administrator>zipline run -f Trading_Policy_of_NonePer_venv_zipline_trader.py --state-file ./.zipline\trading-now.state --realtime-bar-target ./.zipline\realtime-bars --broker ib --broker-uri localhost:4002:1636 --bundle custom-csvdir-bundle --data-frequency minute --start 2020-9-1 --no-benchmark --trading-calendar CME --capital-base 69500 --bundle-timestamp 2020-9-1
[2021-04-17 09:27:58.192366] INFO: IB Broker: Connecting: localhost:4002:1636
Server Version: 76
TWS Time at connection:20210417 09:27:57 GMT
[2021-04-17 09:27:58.286117] INFO: IB Broker: [2158] Sec-def data farm connection is OK:secdefil (-1)
[2021-04-17 09:27:58.286117] DEBUG: IB Broker: Execution details completed for request 0
[2021-04-17 09:27:58.489238] INFO: IB Broker: Managed accounts: ['DUX4XXX8X']
[2021-04-17 09:27:58.489238] INFO: IB Broker: Local-Broker Time Skew: 0 days 00:00:00.489238
[2021-04-17 09:27:59.614185] INFO: Blotter Live: Initialized blotter_live
[2021-04-17 09:27:59.739182] INFO: Live Trading: initialization done
Traceback (most recent call last):
File "C:\ProgramData\Miniconda3\envs\ziptrader36\Scripts\zipline-script.py", line 33, in
sys.exit(load_entry_point('zipline-trader', 'console_scripts', 'zipline')())
File "C:\ProgramData\Miniconda3\envs\ziptrader36\lib\site-packages\click\core.py", line 829, in call
return self.main(*args, **kwargs)
File "C:\ProgramData\Miniconda3\envs\ziptrader36\lib\site-packages\click\core.py", line 782, in main
rv = self.invoke(ctx)
File "C:\ProgramData\Miniconda3\envs\ziptrader36\lib\site-packages\click\core.py", line 1259, in invoke
return process_result(sub_ctx.command.invoke(sub_ctx))
File "C:\ProgramData\Miniconda3\envs\ziptrader36\lib\site-packages\click\core.py", line 1066, in invoke
return ctx.invoke(self.callback, **ctx.params)
File "C:\ProgramData\Miniconda3\envs\ziptrader36\lib\site-packages\click\core.py", line 610, in invoke
return callback(*args, **kwargs)
File "c:\users\administrator\zipline-trader\zipline_main.py", line 111, in _
return f(*args, **kwargs)
File "C:\ProgramData\Miniconda3\envs\ziptrader36\lib\site-packages\click\decorators.py", line 21, in new_func
return f(get_current_context(), *args, **kwargs)
File "c:\users\administrator\zipline-trader\zipline_main_.py", line 400, in run
execution_id=None
File "c:\users\administrator\zipline-trader\zipline\utils\run_algo.py", line 271, in _run
'script': algotext,
File "c:\users\administrator\zipline-trader\zipline\algorithm_live.py", line 262, in run
daily_stats = super(self.class, self).run(*args, **kwargs)
File "c:\users\administrator\zipline-trader\zipline\algorithm.py", line 669, in run
for perf in self.get_generator():
File "c:\users\administrator\zipline-trader\zipline\algorithm.py", line 645, in get_generator
return self._create_generator(self.sim_params)
File "c:\users\administrator\zipline-trader\zipline\algorithm_live.py", line 153, in _create_generator
TradingAlgorithm._create_generator(self, self.sim_params)
File "c:\users\administrator\zipline-trader\zipline\algorithm.py", line 618, in _create_generator
metrics_tracker.handle_start_of_simulation(benchmark_source)
File "c:\users\administrator\zipline-trader\zipline\finance\metrics\tracker.py", line 144, in handle_start_of_simulation
benchmark_source,
File "c:\users\administrator\zipline-trader\zipline\finance\metrics\tracker.py", line 127, in hook_implementation
impl(*args, **kwargs)
File "c:\users\administrator\zipline-trader\zipline\finance\metrics\metric.py", line 222, in start_of_simulation
daily_returns_array,
File "zipline\finance_finance_ext.pyx", line 247, in zipline.finance._finance_ext.minute_annual_volatility
File "zipline\finance_finance_ext.pyx", line 277, in zipline.finance._finance_ext.minute_annual_volatility
IndexError: Out of bounds on buffer access (axis 0)

I guess this error refer to pandas version after I found a lots of test .py files inside the zipline-traders\test. (Thanks for your marvellous work!)

On my another pure zipline env, same zipline 1.4.1 installed with its standard requirements of pandas 0.22.0. And my algorithm just above totally success for above custom bundle on back testing of 11 years minute data.

I had met several times of strange error cause by pandas version changed on my env. It was worked again just by changed back to pandas 0.22.0. That normally happened when I installed some other modules occasionally.

I noticed the release note of zipline-trader 1.5.0 said it was already for pandas 1.1.15. And I don't find any test.py of _finance_ext.minute_annual_volatility. Please help ...

Thanks for your help!

urllib3.exceptions.SSLError: [SSL: DECRYPTION_FAILED_OR_BAD_RECORD_MAC] decryption failed or bad record mac (_ssl.c:2305)

I got this error ingesting data. Does this mean I ran out of API calls?

When running Using Alphalens notebook, at get_pricing line, it gives error:

ValueError Traceback (most recent call last)
in
6 asset_list,
7 pipeline_start_date,
----> 8 end_date)
9 prices.head()
c:\src\lycn\conda-environments\zipline-trader\src\zipline-trader\zipline\research\utils.py in get_pricing(data_portal, trading_calendar, assets, start_date, end_date, field)
51
52 global END_DT
---> 53 END_DT = pd.Timestamp(end_date, tz='UTC', freq='C')
54 start_dt = pd.Timestamp(start_date, tz='UTC', freq='C')
55
pandas_libs\tslibs\timestamps.pyx in pandas._libs.tslibs.timestamps.Timestamp.new()
ValueError: Cannot pass a datetime or Timestamp with tzinfo with the tz parameter. Use tz_convert instead.

When loading data universe SP 500 from 2016-01-03 to 2017-01-03, it gives 'Index' object has no attribute 'normalize':

C:\Users\arisl\anaconda3\envs\zipline-trader\python.exe C:/src/lycn/conda-environments/zipline-trader/src/zipline-trader/zipline/data/bundles/alpaca_api.py
C:/src/lycn/conda-environments/zipline-trader/src/zipline-trader/zipline/data/bundles/alpaca_api.py:276: UserWarning: Overwriting bundle with name 'alpaca_api'
def api_to_bundle(interval=['1m']):
C:/src/lycn/conda-environments/zipline-trader/src/zipline-trader/zipline/data/bundles/alpaca_api.py:355: UserWarning: Overwriting bundle with name 'alpaca_api'
end_session=end_date
Traceback (most recent call last):
File "C:/src/lycn/conda-environments/zipline-trader/src/zipline-trader/zipline/data/bundles/alpaca_api.py", line 363, in
show_progress=True,
File "C:\src\lycn\conda-environments\zipline-trader\src\zipline-trader\zipline\data\bundles\core.py", line 513, in ingest
pth.data_path([name, timestr], environ=environ),
File "C:/src/lycn/conda-environments/zipline-trader/src/zipline-trader/zipline/data/bundles/alpaca_api.py", line 306, in ingest
daily_bar_writer.write(daily_data_generator(), assets=assets_to_sids.values(), show_progress=True)
File "C:\src\lycn\conda-environments\zipline-trader\src\zipline-trader\zipline\data\psql_daily_bars.py", line 617, in write
return self._write_internal(it, assets)
File "C:\src\lycn\conda-environments\zipline-trader\src\zipline-trader\zipline\data\psql_daily_bars.py", line 665, in _write_internal
for asset_id, table in iterator:
File "C:\src\lycn\conda-environments\zipline-trader\src\zipline-trader\zipline\data\psql_daily_bars.py", line 658, in iterator
for asset_id, table in iterator:
File "C:\Users\arisl\anaconda3\envs\zipline-trader\lib\site-packages\click_termui_impl.py", line 315, in generator
for rv in self.iter:
File "C:\src\lycn\conda-environments\zipline-trader\src\zipline-trader\zipline\data\psql_daily_bars.py", line 609, in
for sid, df in data
File "C:\src\lycn\conda-environments\zipline-trader\src\zipline-trader\zipline\data\psql_daily_bars.py", line 716, in _write_to_postgres
result = self._format_df_columns_and_index(data, sid)
File "C:\src\lycn\conda-environments\zipline-trader\src\zipline-trader\zipline\data\psql_daily_bars.py", line 716, in _write_to_postgres
result = self._format_df_columns_and_index(data, sid)
File "C:\src\lycn\conda-environments\zipline-trader\src\zipline-trader\zipline\data\psql_daily_bars.py", line 806, in _format_df_columns_and_index
data.index = data.index.normalize()
AttributeError: 'Index' object has no attribute 'normalize'

Hello,

I just installed zipline-trader in python 3.6 env. I see ZT has alpaca api but nothing for IB. Your doc hints to support IB live trading. Is this true or just a plan?

Sincerely,

fimmu

Hello,
i ingest the future data and they are good at datapotal,but the create_continuous_future in volume are strange.They are not always the highest volume in every month,only use few of them.Please help..thank you.

The documentation only supplies instructions for ingesting data with Alpaca's API. I have followed all of the instructions, but I'm faced with a TypeError: an integer is required (got type bytes). Do any of you use zipline with AlphaVantage? Or should I just go to Alpaca?

Code: \src>python zipline-trader/zipline/data/bundles/alpha_vantage_api.py

Error:

File "c:......\src\zipline-trader\zipline\utils\preprocess.py", line 246, in _build_preprocessed_function

new_func.code = CodeType(*map(getitem(args), _code_argorder))

TypeError: an integer is required (got type bytes)

Traceback (most recent call last):
File "alpaca_api.py", line 13, in
import config
ModuleNotFoundError: No module named 'config'

after installing 1.6.0 from pypi and running alpaca ingestion according to video tutorials can't find the config module. Curious where this module comes since it appears it implements alpaca specific logic and the pypi project only talks about generic config management features...

File "zipline/data/bundles/alpaca_api.py", line 307, in ingest
minute_data_generator(), assets=assets_to_sids.values(), show_progress=True)
TypeError: write() got an unexpected keyword argument 'assets'

This is the Error I get when I try to do this:

register(
package_name, # This is the name of the ingest package
# api_to_bundle(interval=['1d', '1m']),
api_to_bundle(interval=['1m']),
# api_to_bundle(interval=['1d']),
calendar_name='NYSE',
start_session=start_date,
end_session=end_date
)

Does the field where I have package name have to be 'alpaca_api'?

Any advice on how to resolve this would be ++appreciated. Thanks!
Andy

Here's my order in the terminal:
(zipline) (base) my-macbookie:zipline-trader andy$ python zipline/data/bundles/alpaca_api.py

Here's the error:
Traceback (most recent call last):
File "zipline/data/bundles/alpaca_api.py", line 11, in
from trading_calendars import TradingCalendar
File "/Users/andy/zipline/lib/python3.7/site-packages/trading_calendars/init.py", line 16, in
from .trading_calendar import TradingCalendar
File "/Users/andy/zipline/lib/python3.7/site-packages/trading_calendars/trading_calendar.py", line 33, in
from .calendar_helpers import (
File "/Users/andy/zipline/lib/python3.7/site-packages/trading_calendars/calendar_helpers.py", line 6, in
NP_NAT = np.array([pd.NaT], dtype=np.int64)[0]
TypeError: int() argument must be a string, a bytes-like object or a number, not 'NaTType'

https://github.com/shlomikushchi/zipline-trader/blob/6b6839efe5fd95d8bc3c4ad8e95dd47b543dffb8/zipline/data/bundles/alpaca_api.py#L23

Hi,

I am still running Zipline 1.41 from the latest quantopian, and had an issue which I am not sure if its been addressed or not in the latest version.

Using Norgate Data to create bundles for both US and Australian (ASX) stocks.

Both run fine with correct trading calendar using handle data, however when using pipeline on the Australian stocks there is no valid pipeline loader if pricing or volume is included (works fine with US stocks):

The error is ValueError: No PipelineLoader registered for column EquityPricing.close::float64

The error is called from ~ line 175 in run_algo.py, I have attempted to change the loader to EquityPricing which appears consistent with the code changes and added some print statements but can't see if this is a flaw in the code or something else is incorrect. Adding domain does not change the issue

Any help appreciated

def choose_loader(column):

    #if column in USEquityPricing.columns:
    # return pipeline_loader
    #raise ValueError(
    # "No PipelineLoader registered for column %s." % column
    #)
    print('Column', column)
    print('Equity Columns',EquityPricing.columns)
    print('Equity Pricing',EquityPricing)
    print('Pipeline Loader',pipeline_loader)
    if column in EquityPricing.columns:
    return pipeline_loader
    raise ValueError(
    "No PipelineLoader registered for column %s." % column

The issue appears to be the as the valid EquityPricing.Columns are below:

Equity Columns frozenset({EquityPricing.close::float64, EquityPricing.low::float64, EquityPricing.high::float64, EquityPricing.volume::float64, EquityPricing.open::float64, EquityPricing.currency::object})

def make_index_pipeline(context):
indexconstituent = NorgateDataIndexConstituent('XJO.au')
dollar_volume = AverageDollarVolume(window_length=30)
high_dollar_volume = (dollar_volume > dollar_min)
return Pipeline(
columns={
'NorgateDataIndexConstituent':indexconstituent},
screen = indexconstituent&high_dollar_volume,
)