Drawdowns currently calculated by $. Keeping them as % would be a more realistic representation of performance, obviously as bigger $ swings with more capital may actually be a lower % drawdown -- but this isn't currently represented correctly.

Will require changing drawdown tests, manually calculating what the drawdown % should be at each of the 10 steps and slightly modifying implementation to do that.

For example

https://github.com/mhallsmoore/qstrader/blob/master/examples/mac_backtest.py

Instead of

def run(config, testing):
    tickers = ["SP500TR"]

we should have

def run(config, testing, tickers):
    ...

@click.command()
@click.option('--config', default=settings.DEFAULT_CONFIG_FILENAME, help='Config filename')
@click.option('--testing/--no-testing', default=False, help='Enable testing mode')
@click.option('--tickers', default='SP500TR', help='Tickers (use comma)')
def main(config, testing, tickers):
    tickers = tickers.split(",")
    config = settings.from_file(config, testing)
    run(config, testing, tickers)

if __name__ == "__main__":
    main()

So it will be possible to test a similar strategy on different tickers without changing code

test_examples.py should be changed accordingly (passing list of tickers to run)

sp500tr_buy_and_hold_backtest.py should be renamed buy_and_hold_backtest.py

Hello,

Home user directory shortcut ~ is not accept as a valid directory in settings.py

Running

$ python qstrader/examples/test_strategy_backtest.py

with this settings file

CSV_DATA_DIR="~/data/random/"
OUTPUT_DIR="~/qstrader/"

raises

FileNotFoundError: [Errno 2] No such file or directory: '~/qstrader/tradelog_2016-06-22'

os.path.expanduser might help https://docs.python.org/3/library/os.path.html#os.path.expanduser

It may be cleaner to create an inherited class for settings
with methods to get CSV_DATA_DIR, OUTPUT_DIR with ~ expanded

Kind regards

This simply requires a try...except block, for all necessary files importing the queue library, as in QSForex.

See https://github.com/mhallsmoore/qstrader/blob/master/qstrader/compliance/compliance.py#L51

self.csv_filename = "tradelog_" + datetime.datetime.today().strftime("%Y-%m-%d") + ".csv"

should be replaced by

today = datetime.datetime.utcnow().date()
self.csv_filename = "tradelog_" + today.strftime("%Y-%m-%d") + ".csv"

Hello,

it will be a good idea to store results into file.
So that we will be able to display again equity / drawdown plots...
and also get a DataFrame showing when trades occur in an interactive way (IPython...) after running a backtest.

Serialization can be done using Pickle.

An other option could be to store results to HDF5 files.

Maybe we should consider to build something on top of PyPET (in order to avoid reinventing a "squared wheel") https://pypet.readthedocs.io/en/latest/manual/introduction.html#getting-started

Kind regards

Backtest (and also live trading of course) should accept several strategies as input (a list of strategies for example).

So we could have for example a PrintStrategy which will for example only responsible of printing ticks, bars, ... (maybe only one out of N with N=1000 for example)

Removing this PrintStrategy from Backtest will help to improve backtest speed.

Hello,

generate_simulated_prices.py shouldn't need any settings by default.
So it will be easier to test sample strategies.

Kind regards

Check unsubscribing a ticker that isn't in the price handler list doesn't raise KeyError
and Subscribe a new ticker, without CSV doesn't raise

in fact problem is that price_handler test assertRaises are incorrects

You should have a look at http://stackoverflow.com/questions/6103825/how-to-properly-use-unit-testings-assertraises-with-nonetype-objects

assertRaises second argument should be just function name... not function call with arguments because in such a case you are calling function it before passing to assertRaises

assertRaises second argument as a "callable"

With Python >= 2.7 you can use a context manager (with)

An other approach is to use a lambda function to pass a "callable" to assertRaises

So test can be written this way:

self.assertRaises(
    KeyError, lambda: self.price_handler.unsubscribe_ticker("PG")
)

or

self.assertRaises(
    KeyError, self.price_handler.unsubscribe_ticker, "PG"
)

Add details into the README.md about Python private_files git_ignore.

Speed has been frustrating me a little bit while backtesting, have had a hunch it was due to the Decimal calculations being used for accuracy's sake. A little research brought up this discussion from the quantopian repo.

I have made a small start here, tests are currently broken though.
https://github.com/ryankennedyio/qstrader/tree/decimal-to-float

Some preliminary testing on my macbook air (1.7GHz dual-core i7, 8GB RAM) show:

SP500 backtest at 7c88d26:
15.6sec
15.2sec
15.8sec

SP500 backtest at 2373c59 (from my branch):
2.88sec
3.03sec
2.90sec

Decimal usage seems to lead to a 5x longer backtest (rule of thumb). This might take even longer for strategies that have lots of price calculations.

I understand the motivation for using Decimal, so I will also update here with the value of the difference between the unit test results when using Decimal and float, to see how major the calculation differences end up being.

@femtotrader - I've just looked at the requirements.txt file after trying to install the latest dependencies (munch, enum34 etc) and we have a line:

yaml>=3.11

On my system this fails with:

Downloading/unpacking yaml>=3.11 (from -r requirements.txt (line 19)) Could not find any downloads that satisfy the requirement yaml>=3.11 (from -r requirements.txt (line 19))

It looks like there is no "yaml" package specifically, but rather we need to use pyyaml: http://stackoverflow.com/questions/14261614/how-do-i-install-the-yaml-package-for-python

If you're happy for me to do so, I'll change this in the requirements.

According discussion #53

Maybe a more general price handler should be done
This price handler could have

• a DataFrame as input (when a backtest for only one ticker is necessary)
• a Panel as input (when a backtest with several tickers is given)

Index will be Time

Depending of columns names it could be either

• a tick price handler (need at least a column Price or Bid/Ask and optionally Volume(s))
• a bar price handler (need at least columns named Open High Low Close)

import pandas as pd
import pandas_datareader.data as web
import datetime
import requests_cache


expire_after = datetime.timedelta(days=3)
session = requests_cache.CachedSession(cache_name='cache', backend='sqlite', expire_after=expire_after)
panel = web.DataReader(["GOOG", "IBM"], "yahoo", session=session)

Panel looks like:

<class 'pandas.core.panel.Panel'>
Dimensions: 6 (items) x 1630 (major_axis) x 2 (minor_axis)
Items axis: Open to Adj Close
Major_axis axis: 2010-01-04 00:00:00 to 2016-06-23 00:00:00
Minor_axis axis: GOOG to IBM

def data_from_dataframe(df):
    for (dt, row) in df.iterrows():
        yield(dt, row)

def data_from_panel(panel):
    for dt, data in panel.transpose(1,0,2).iteritems():
        for (ticker, bar) in data.iteritems():
            yield(dt, ticker, bar)

ticker = "GOOG"
df = panel[:,:,ticker]
for (dt, bar) in data_from_dataframe(df):
    print(dt, ticker, bar)

for (dt, ticker, bar) in data_from_panel(panel):
    print(dt, ticker, bar)

Backtesting a daily bar strategy with about 200 tickers. Involves a monthly rebalance of the portfolio.

Performance starts off at ~5,000 bars per second and fairly quickly degrades down to 2,000 bars after 3 years worth. The degradation happens quickly at first, then levels out once it gets really bad (in terms of time taken, not in terms of computational complexity. In complexity terms, it gets even worse over time, but it's taking slower to get worse -- if that makes sense).

I've done some solid digging and determined that it's because we keep reference to closed out positions in order to calculate portfolio values. This starts to get pretty deep into the core of qstrader, so it's well worth it for us to discuss the options and resolve this correctly.

If I simply do this at the end of portfolio._modify_position(), I can keep a steady 5,000 bars/sec through the entire backtest, regardless of size. That's ideal.

if self.positions[ticker].quantity == 0:
    self.positions.pop(ticker)

Obviously the test goes out the door because we can no longer calculate market value, as we've lost the position.realised_pnl used to calculate portfolio.equity. But the speed is good!!!

To prove this is the issue (and how I found it), here are two profiling results:

Currently

After removing closed positions

I think I ran a different length backtest the second time -- much longer (68s instead of 49s).
Wanted to prove to myself that the performance was now constant as the problem set increased.
Can see that the obvious portfolio.py block is now reasonably consistent with the rest of the system.

An option to explore might be keeping some kind of reference object that should be kept to aggregate the value of closed-out positions. The issue, I think, is pretty obvious when seen; every backtest event causes us to go through and find the realised_pnl for every closed out position. The number of closed positions is ever increasing, leading to the degradation over time.

Might remove
https://github.com/mhallsmoore/qstrader/blob/master/qstrader/compliance/compliance.py#L52

    self.csv_filename = "tradelog_" + datetime.datetime.today().strftime("%Y-%m-%d") + ".csv"
    self.csv_filename = "tradelog_" + today.strftime("%Y-%m-%d") + ".csv"

This was probably because of this morning big merge.

There is an issue with test_strategy_backtest.py

It raises TabError: inconsistent use of tabs and spaces in indentation in test_strategy_backtest.py
see https://raw.githubusercontent.com/mhallsmoore/qstrader/master/examples/test_strategy_backtest.py

    # Use the TestCompliance component
    compliance = TestCompliance();

    # Use a simulated IB Execution Handler
    execution_handler = IBSimulatedExecutionHandler(
        events_queue, price_handler, compliance
    )

ONLY APPLIES TO BACKTESTING
When you unsubscribe from a ticker, although it is removed from the tickers_data and tickers list, it is not removed from the generator. Additionally, if you subscribe to a new ticker mid backtest, the ticker data isnt added to the generator.

so when you hit:

def stream_next_tick:
index, row = next(self.tick_stream) # The generator

You will actually still get the data for the ticker that was removed. This then assigns the removed ticker to ticker = row["ticker"]. Furthermore, when you try and access self.tickers[ticker]["bid"] you get a key error because the key no longer exist in the updated self.ticker dict.

I'm not sure how this can be solved because if you reassign the generator the new data frame that contains the updated ticker list (and data), you reset the generator (correct?).

I'm not sure if i'm explaining this clearly, but if you have any questions on the issue, ill try my best to articulate better.

could you add a read me file that might help a tyro ?

According to discussion #20
a basic benchmarking/profiling tool is necessary

nose-timer can provide this kind of basic results automatically and we can simply have a look at Travis build logs

See for example this log https://travis-ci.org/qstraders/qstrader/jobs/140184530

With Python 3.5

examples.test_examples.TestExamples.test_mac_backtest: 6.9505s
examples.test_examples.TestExamples.test_sp500tr_buy_and_hold_backtest: 5.6768s
qstrader.scripts.test_scripts.TestScripts.test_generate_simulated_prices: 3.6915s
examples.test_examples.TestExamples.test_strategy_backtest: 0.0416s
test_statistics.TestSimpleStatistics.test_calculating_statistics: 0.0290s
test_price_handler.TestPriceHandlerSimpleCase.test_stream_all_ticks: 0.0288s
test_price_handler.TestPriceHandlerSimpleCase.test_get_best_bid_ask: 0.0235s
test_price_handler.TestPriceHandlerSimpleCase.test_subscribe_unsubscribe: 0.0228s
test_position.TestRoundTripPGPosition.test_calculate_round_trip: 0.0008s
test_portfolio.TestAmazonGooglePortfolio.test_calculate_round_trip: 0.0007s
test_portfolio_handler.TestSimpleSignalOrderFillCycleForPortfolioHandler.test_convert_fill_to_portfolio_update_basic_check: 0.0003s
test_position.TestRoundTripXOMPosition.test_calculate_round_trip: 0.0003s
test_portfolio_handler.TestSimpleSignalOrderFillCycleForPortfolioHandler.test_on_signal_basic_check: 0.0002s
test_portfolio_handler.TestSimpleSignalOrderFillCycleForPortfolioHandler.test_place_orders_onto_queue_basic_check: 0.0002s
test_portfolio_handler.TestSimpleSignalOrderFillCycleForPortfolioHandler.test_create_order_from_signal_basic_check: 0.0002s

with Python 2.7

https://travis-ci.org/qstraders/qstrader/jobs/140184526

examples.test_examples.TestExamples.test_mac_backtest: 23.5531s
examples.test_examples.TestExamples.test_sp500tr_buy_and_hold_backtest: 6.4959s
qstrader.scripts.test_scripts.TestScripts.test_generate_simulated_prices: 2.6450s
examples.test_examples.TestExamples.test_strategy_backtest: 0.0447s
test_statistics.TestSimpleStatistics.test_calculating_statistics: 0.0417s
test_price_handler.TestPriceHandlerSimpleCase.test_stream_all_ticks: 0.0324s
test_price_handler.TestPriceHandlerSimpleCase.test_get_best_bid_ask: 0.0251s
test_price_handler.TestPriceHandlerSimpleCase.test_subscribe_unsubscribe: 0.0241s
test_portfolio.TestAmazonGooglePortfolio.test_calculate_round_trip: 0.0147s
test_position.TestRoundTripPGPosition.test_calculate_round_trip: 0.0033s
test_position.TestRoundTripXOMPosition.test_calculate_round_trip: 0.0033s
test_portfolio_handler.TestSimpleSignalOrderFillCycleForPortfolioHandler.test_convert_fill_to_portfolio_update_basic_check: 0.0024s
test_portfolio_handler.TestSimpleSignalOrderFillCycleForPortfolioHandler.test_on_signal_basic_check: 0.0002s
test_portfolio_handler.TestSimpleSignalOrderFillCycleForPortfolioHandler.test_place_orders_onto_queue_basic_check: 0.0002s
test_portfolio_handler.TestSimpleSignalOrderFillCycleForPortfolioHandler.test_create_order_from_signal_basic_check: 0.0002s

We can see here that Python 2.7 is much slower with test_mac_backtest

Same as

mhallsmoore/qsforex#9
mhallsmoore/qsforex#20

It will be easier for packaging to have settings as configuration file
see #40

class named Test... are for unit tests.
it's a bad idea to name them like this because it leads to confusion between example implementation of an abstract class and unit tests.

https://github.com/mhallsmoore/qstrader/search?utf8=%E2%9C%93&q=class+Test&type=Code

TestPositionSizer -> FixedPositionSizer

(default quantity might also be a parameter for __init__)

I also suggest to put abstract class in a base.py and each implementation in a different file.

It could be a good idea to name abstract class Abstract... (this is quite common in Julia... and help a lot to understand code)

Other rename

TestRiskManager -> ExampleRiskManager

TestStrategy -> ExampleStrategy
TestCompliance -> ExampleCompliance or CsvCompliance

I miss probably some others

@femtotrader - Now that the majority of the qstrader source lives under the /qstrader directory, we need to include a default settings.py in order for the strategies to "work out of the box".

I'm not too sure of the changes to the codebase as of yet (I'm still working through them), so I was wondering if you'd be happy adding a Pull Request with a default settings file?

Hi Michael,

Looks like settings.py.example and init.py is missing from the root directory.

Also, import queue statement doesn't work in the execution_handler.py file.

Thanks

As mentioned in: https://www.quantstart.com/articles/Advanced-Trading-Infrastructure-Portfolio-Class

What's missing from this list so far? Perhaps the most important missing piece is any mechanism for calculating trade strategy statistics and viewing the results. This includes performance metrics like Sharpe Ratio and Maximum Drawdown, as well as an equity curve, returns profile and drawdown curve.

Rather than strongly-coupling the results to the PortfolioHandler class, as in the previous QSForex and the event-driven backtester codes, we are going to generate a Results or Statistics class that will calculate and store the necessary performance metrics based on the results of a backtest. We can then use these classes to produce further "client" utilities, such as a web interface or GUI tool, to view the results of a backtest.

Has any work or planning been done on this class? I'll get started if not.

There is currently an equity_file written as part of the Backtest._run_backtest method, but it only tracks portfolio equity over time. Not sure if this is the "final" solution? Should this be tracking more information, and then the Statistics class creates visualisations based on the data in that file? Or should the Statistics class keep more information in-memory?

I'm thinking that a Statistics class might "track" the portfolio (cash, equity value, PnL) as events come through the system. Having these Statistics accessible by other classes would also allow them to be incorporated into strategies, which I think would be quite useful.

To implement this, I think that inside the event loop, we do Statistics.update(). This method will update the equity curve, PnL, cash levels, drawdowns, etc etc etc. I don't like the idea of doing this for every single event that comes through though... ?

Only issue I can see so far is that the unrealised & realised PnL is only calculated when a Position is modified. Might make the equity curves look a bit jumpy for less active strategies ?

Also unsure if Logging should be part of the Statistics class -- I think they would share a lot of functionality.

similar to mhallsmoore/qsforex#42

• PriceHandlerType = Enum("PriceHandlerType", "TICK BAR")
• EventType = Enum("EventType", "TICK BAR SIGNAL ORDER FILL")
• Action = Enum("Action", "BOT", "SLD")

I'm unsure as to what an institutional setting would require and am only familiar with retail discretionary trade logs, but personally this would be most useful for debugging backtests (e.g. what exactly caused a big spike/dip ).

Probably a CSV file including every trade that had taken place would be the most useful way to do this? My alternate method for debugging has been pretty awful, "if timestamp == X: import pdb; pdb.set_trace()" and then poking around current positions.

I'm a bit sick of doing the above when debugging, so will probably work on this before any HDF5 integration. Can anyone include features that might be necessary for their own regulatory environments?

Hi Michael,

Running into this in every way I try to go about it; when running python examples/test_strategy_backtest.py I'll get:

Traceback (most recent call last):
  File "examples/test_strategy_backtest.py", line 3, in <module>
    from qstrader.backtest.backtest import Backtest
ImportError: No module named qstrader.backtest.backtest

This happens after setting up virtualenv and doing
ln -s ~/development/qstrader/ ~/development/qstrader/venv/lib/python2.7/site-packages/qstrader

I also get the same issues when doing the whole thing from scratch inside a Vagrant box specifically provisioned for this, using either virtualenvwrapper or the other one, which is bizarre. Should the setup be the same as it was for qsforex?

As a side note; I'm very keen to contribute a lot. Standard github etiquette applies (feature discussion in issues, feature branches and pull requests from there) ?

Hi,

I'm running into this error that Python 2 doesn't recognise this exception. I guess it's only available in Python 3.

Is qstrader going to be coded for Python 3 only, or should we make it backwards compatible?

What's the best way to handle both python 2 and 3 in this case?

Thanks
Nick

Does anybody know the correct mechanism to include Python version badges in the same vein as Quantopian have done for the Zipline project on their README? https://github.com/quantopian/zipline

Coveralls.io: https://coveralls.io/

This looks good - it's a free to use service for open source repos (a la Travis) that gives us a code coverage percentage, as well as a badge to place next to the "test passing" badge on the README. It should encourage both:

• Development of more unit tests if the percentage is low
• More usage of the software if the percentage is high

What does everyone think? @femtotrader, @ryankennedyio?

I created my simple strategy and tested it with backtesting, as in we have in example. Now I would like to test the same strategy with streaming data.
How and where to plug it.

Can any one give simple class structure to begin with.

I was thinking about possible example strategies that might be good to ship with QSTrader.

Here are some initial thoughts:

• Moving Average Crossover (DONE)
• Some form of Mean-Reversion, either via "technicals" or via a proper cointegration approach
• Machine learning forecast example, e.g. with Scikit-Learn
• Possibly sentiment analysis? Although this is very data dependent

Suggested strategies should be simple, with few free parameters.

(just a reminder)

Travis need to be enable for this repository

Badge link should also be changed in README.md

from

[![Build Status](https://travis-ci.org/qstraders/qstrader.svg?branch=master)](https://travis-ci.org/qstraders/qstrader)

to

[![Build Status](https://travis-ci.org/mhallsmoore/qstrader.svg?branch=master)](https://travis-ci.org/mhallsmoore/qstrader)

So, rather than always executing market orders, I imagine longer term (1min+) strategies will get a lot of benefit from being able to set a limit order a few ticks below market price on instruments [a...m]. As our orders are filled, the risk manager can pull excess limit orders accordingly. I know that manually just buying market all the time is a great way to throw good money, and can't imagine that automated trades would be too different.

As a rough example:
Say we want no more than 10 positions in our portfolio (set by an equal_weight_position_sizer). $100k capital, so position sizer will give $10k to each position.
As we run a strategy, we find 30 instruments that we want to long. We set 30 limit orders 3 ticks down from market. Every time we get a fill, we remove 3 limit orders (ish). At the end we're looking for a fill on 1 stock and we've got maybe 3 limit orders sitting in the bid stack.

All of the above figures are obviously open to being shuffled around. Interesting exercise for those inclined would be optimising these parameters with a self reinforced ML algorithm.

Realise this will open a new can of worms in terms of complexity, so I'd rather have most other aspects of the system nailed before this. It will involve a lot more work in the risk management & position sizing classes.

One risk is that the algorithm might "self-select" stocks that are about to turn against us if the limit order is set too far down. If it's just right, it should be similar in effect to market orders, but it would be easier to overcome brokerage & slippage this way.

Imagine a major difficulty in simulation is that a limit order is only filled a portion of the time when the stock trades at the set limit. Will also need to account for one of my pet hates, partial fills (got a 30% fill yesterday on something and then it gapped up 23% this morning --- hoorah. Edge cases like that will be a pain for an algorithm to deal with).

Any opinions?

Hello,

I noticed this project (qstrader) only very recently
I was looking at qsforex code and I didn't noticed this project.
It seems that a lot of code is common between these 2 projects
Isn't there a way to merge them (or to have a third project) with common parts
of qstrader and qsforex.

Kind regards

Rather simple, but will be quite helpful to only run a backtest through a period we're interested in. Will also help when doing simple in-sample/out-of-sample comparisons.

I believe this should be a function of the backtester rather than price handler, as it's a function specific to backtesting, however this might throw up some performance issues if someone is doing second-resolution backtesting on a large stock universe ?

There is a lot of absolute imports and that's not necessary to do this

Example
https://github.com/mhallsmoore/qstrader/blob/master/qstrader/portfolio_handler/portfolio_handler.py

from qstrader.order.order import SuggestedOrder

could be (using relative imports)

from ..order import SuggestedOrder

When doing a backtest, every FillEvent from the backtester has a timestamp of "now".

Haven't yet devoted any thought to working around this, but it definitely would be a nice thing to know when a trade 'would' have been placed in a backtest.

Getting below error while running the backtest, it started after I did pull all new changes after a month. I am passing a minute bar data.

Any clue what is happening. It used to give result on old code (an month old code.)

Traceback (most recent call last):
  File "lowest_low.py", line 116, in <module>
    main()
  File "/Users/mukeshyadav/.virtualenvs/py3-data/lib/python3.5/site-packages/click/core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "/Users/mukeshyadav/.virtualenvs/py3-data/lib/python3.5/site-packages/click/core.py", line 696, in main
    rv = self.invoke(ctx)
  File "/Users/mukeshyadav/.virtualenvs/py3-data/lib/python3.5/site-packages/click/core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/Users/mukeshyadav/.virtualenvs/py3-data/lib/python3.5/site-packages/click/core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "lowest_low.py", line 112, in main
    run(config, testing, tickers, filename, execute_date)
  File "lowest_low.py", line 83, in run
    results = backtest.simulate_trading(testing=testing)
  File "/Users/mukeshyadav/Projects/stocklysis/server/qstrader/qstrader/trading_session/backtest.py", line 69, in simulate_trading
    results = self.statistics.get_results()
  File "/Users/mukeshyadav/Projects/stocklysis/server/qstrader/qstrader/statistics/simple.py", line 69, in get_results
    statistics["max_drawdown_pct"] = self.calculate_max_drawdown_pct()
  File "/Users/mukeshyadav/Projects/stocklysis/server/qstrader/qstrader/statistics/simple.py", line 105, in calculate_max_drawdown_pct
    top_index = equity_series[:bottom_index].idxmax()
  File "/Users/mukeshyadav/.virtualenvs/py3-data/lib/python3.5/site-packages/pandas/core/series.py", line 1226, in idxmax
    i = nanops.nanargmax(_values_from_object(self), skipna=skipna)
  File "/Users/mukeshyadav/.virtualenvs/py3-data/lib/python3.5/site-packages/pandas/core/nanops.py", line 464, in nanargmax
    result = values.argmax(axis)
ValueError: attempt to get argmax of an empty sequence

So it will be possible to save them (CSV, JSON, YAML, Pickle...)

Moreover it will be possible to use backtest results for unit testing purpose.

(note: I wasn't sure how to mark this with that "Enhancement" tag in github. my apologies up front if this wasn't the way to go about requesting futures trading support...and/or having discussion on this topic)

So I'm looking at using (enhancing) qstrader to live trade and backtest the Futures market. So the first thing that stands out to me is that the qstrader was really geared towards equities and equity pricing, and I would need to first find a way of getting tick and contract sizes of various future contracts. Does anyone have any suggestions on getting this information? Or am I going to have to "hard code" those values into qstrader?

Or is anyone utterly opposed to enhancing qstrader to have futures support? (either in principle or qstrader code support)

I'm likely to be creating a module that will provide simulation of financial time series data in order to test certain statistical machine learning models.

The main reasons for this are:

• The data can be simulated on any timescale (e.g. tick/quote through to monthly)
• Correlated data sources can be generated (e.g. via this method: https://www.quantstart.com/articles/Generating-Correlated-Asset-Paths-in-C-via-Monte-Carlo)
• The data can be simulated with abrupt or gradual changes in simulation parameters to represent "regime change"
• Varying degrees of noise can be added to the data in order to test effectiveness of underlying parameter reconstruction

It should provide a good test-bed for statistical machine learning techniques prior to the usage of real-world financial data, which should help avoid overfitting problems.

Please let me know your thoughts!

Working on this at the moment as a feature, using swigibpy, on my branch here:

https://github.com/ryankennedyio/qstrader/tree/ibgateway

Just gone around setting up a new price_handler for this feature. Will also set up a new execution_handler for this as well.

Will need to update README.md with instructions for swigibpy setup V0.5.0 and IB Gateway setup too.

Believe that live data ("paper traded" to begin with) will require a new price_handler again because IB doesn't let us "stream" historic ticks?

I only have access to IB Demo account for the next ~4 weeks (data is useless in demo acct), so won't do much more than the bare basics.

Not sure if this was intentional (i.e. you wanted user to add final touches), but it seems the examples don't work out of the box. In particular I found:

1. requirements.txt was missing several packages
2. Statistics module is missing "SimpleStatistics" (see test_strategy_backtest.py)

Hello,

in backtest loop portfolio._reset_values() is called twice

One time in
https://github.com/mhallsmoore/qstrader/blob/master/qstrader/backtest/backtest.py#L77

but self.portfolio_handler.update_portfolio_value() call points to

https://github.com/mhallsmoore/qstrader/blob/master/qstrader/portfolio_handler/portfolio_handler.py#L117

where it's also reset.

I suggest to rename methods to a more event oriented approach (like on_tick, on_bar...)
I did a big rewrite like this and it helps a lot to see this kind of double calls.

Kind regard

Hello,

When I run an example I get (now) many errors because of TkAgg backend

$ python examples/buy_and_hold_backtest.py
objc[1826]: Class TKApplication is implemented in both /System/Library/Frameworks/Tk.framework/Versions/8.5/Tk and /Users/femto/anaconda/lib/libtk8.5.dylib. One of the two will be used. Which one is undefined.
objc[1826]: Class TKMenu is implemented in both /System/Library/Frameworks/Tk.framework/Versions/8.5/Tk and /Users/femto/anaconda/lib/libtk8.5.dylib. One of the two will be used. Which one is undefined.
objc[1826]: Class TKContentView is implemented in both /System/Library/Frameworks/Tk.framework/Versions/8.5/Tk and /Users/femto/anaconda/lib/libtk8.5.dylib. One of the two will be used. Which one is undefined.
objc[1826]: Class TKWindow is implemented in both /System/Library/Frameworks/Tk.framework/Versions/8.5/Tk and /Users/femto/anaconda/lib/libtk8.5.dylib. One of the two will be used. Which one is undefined.
Exception in Tkinter callback
Traceback (most recent call last):
  File "/Users/femto/anaconda/lib/python3.5/site-packages/matplotlib/backends/tkagg.py", line 22, in blit
    id(data), colormode, id(bbox_array))
_tkinter.TclError: invalid command name "PyAggImagePhoto"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/femto/anaconda/lib/python3.5/tkinter/__init__.py", line 1550, in __call__
    return self.func(*args)
  File "/Users/femto/anaconda/lib/python3.5/site-packages/matplotlib/backends/backend_tkagg.py", line 283, in resize
    self.show()
  File "/Users/femto/anaconda/lib/python3.5/site-packages/matplotlib/backends/backend_tkagg.py", line 355, in draw
    tkagg.blit(self._tkphoto, self.renderer._renderer, colormode=2)
  File "/Users/femto/anaconda/lib/python3.5/site-packages/matplotlib/backends/tkagg.py", line 30, in blit
    id(data), colormode, id(bbox_array))
_tkinter.TclError
alloc: invalid block: 0x1196c7e90: 10 0
Abort trap: 6

It seems that some other people face this issue
https://www.google.com/search?q=Class+TKApplication+is+implemented+in+both+%2FSystem%2FLibrary%2FFrameworks%2FTk.framework%2FVersions%2F8.5%2FTk&oq=Class+TKApplication+is+implemented+in+both+%2FSystem%2FLibrary%2FFrameworks%2FTk.framework%2FVersions%2F8.5%2FTk&aqs=chrome..69i57.331j0j7&sourceid=chrome&ie=UTF-8#q=Class+TKApplication+is+implemented+in+both+/System/Library/Frameworks/Tk.framework/Versions/8.5/Tk&tbs=qdr:m

I don't know exactly the reason of this... but removing

try:
    matplotlib.use('TkAgg')
except:
    pass

in qstrader/statistics/simple.py

fixed this issue!

So I wonder why should we force backend ?

I think it's a user decision to use one backend or an other one

Here is my current available backends for Matplotlib

$ python -c "import matplotlib; print(matplotlib.rcsetup.all_backends)"
['GTK', 'GTKAgg', 'GTKCairo', 'MacOSX', 'Qt4Agg', 'Qt5Agg', 'TkAgg', 'WX', 'WXAgg', 'CocoaAgg', 'GTK3Cairo', 'GTK3Agg', 'WebAgg', 'nbAgg', 'agg', 'cairo', 'emf', 'gdk', 'pdf', 'pgf', 'ps', 'svg', 'template']

A user can force to use a given backend using a config file named ~/.matplotlib/matplotlibrc
with

backend : Qt4Agg

I'm not sure that IB allows tick-by-tick data in a backtest, and there's currently no provisioning in qstrader for Volume (but that itself should be pretty easy).

I think it would be useful if the system would "roll up" streamed tick data into OHCLV bars of a particular duration before they are analysed by a Strategy. It wouldn't make sense to inundate 1hr+ timeframe strategies with loads and loads of tick-data when they only care about checking things over 5m+

Would this be useful & which part of the system should be responsible for doing this? It would make sense to me that a price_handler should do this, and then the TickEvents that go through the main loop would simply be the OHCLV bars, rather than just Ticks.

$ nosetests -s -v tests/test_price_handler.py

ticker doesn't appear in this message

in price_handler.py - HistoricCSVPriceHandler.get_best_bid_ask

Maybe I'm a little unclear on the vocab, but a fresh position shouldn't give massive unrealised pnls, right? Line 91 in position.py is part calculating unrealised_pnl, and it's subtracting a negative cost_basis. Here's a little example that's mostly plagiarized from the test in the same folder.

from qstrader.position.position import Position

position = Position(
            "SLD", "PG", Decimal('100'), 
            Decimal("77.69"), Decimal("1.00"), 
            Decimal('77.68'), Decimal('77.70')
        )
print(position.cost_basis)
print(position.avg_price)
print(position.total_sld)
print(position.unrealised_pnl) # 15537.00 which is  2*7768.5
print(position.realised_pnl)