In order not to infringe on flat periods when layering random round turns with txnsim, we truncate the duration which overlaps into flat periods. We use a time unit of "seconds" in txnsim to facilitate the building of replicate strategies of higher frequencies than just daily. However, when we compute a truncated duration from the 2nd layer and beyond we get a value in days, and bind it to other trade durations in seconds. txnsim is reading this truncated "days" value as "seconds" though, meaning we add a low value duration to a random start time (to get a random round turn end time), which inevitably results in a round turn with the same day timestamp. Worth noting there are 86400 seconds in a single day.

To fix we will simply use the difftime function for computing the truncated duration, specifying unit="seconds". Currently we are using simple subtraction between 2 timestamps.

Above is an example of a building replicate (using a slight variation of the daily bbands strategy from quantstrat) and the first sampled random start time for the second layer (index 2.1) is 2016-10-24 which should have a duration of 1 day, as the next flat period after that timestamp begins on 2016-10-25. However, we read the 1 as "seconds" meaning when we compute an end time we get the same day, just incremented by 1 second as opposed to 86400.

Below is the result of the proposed fix. The same replicate is being built with a duration of 86400 seconds which will end the round turn in 1 day.

Submitted by: Peter Carl
Assigned to: Peter Carl
R-Forge link

Function to read transactions from a csv file and insert them into a portfolio. Key columns would be:

Portfolio: Could be specified as an argument or read from the file.

Symbol or identifier

Timestamp

Txn.Qty: the quantity, usually in units of contracts, changing hands.
Positive values indicate a 'buy' transaction; negative values
are used to indicate a 'sell.'

Txn.Price: the price at which the transaction was made

Txn.Fees: the sum total of transaction fees associated with the trade

Txn.Value: the notional value of the transaction

Followups:

Date: 2010-01-18 14:09
Sender: Brian Peterson
function addTxns has been added to SVN, and takes a xts object with columns 'Quantity' and 'Price'It is a TODO to extend it to handle fees as well.I'm not sure we need to create a special CSV loader that would only txns<-xts(read.zoo('txnfile.csv))addTxns('port',txns)

Monte Carlo analysis of transactions

Running simulations with similar properties as the backtest or production portfolio may allow the analyst to evaluate the distribution of returns possible with similar trading approaches and evaluate skill versus luck or overfitting.

Cases:

flat.to.flat && replace=FALSE

This is the simplest case, as with equity curve resampling,
we're just rearranging the vector of durations and quanitities.

To create each replicate vector:

• sample(1:nrows(txnsimdf) ,nrow(txnsimdf))
• rearrange our duration and quantity rows by the sample order

It will probably make sense for all these methods to create a little internal
function to do the sampling, and then call replicate() with simplify=FALSE
which should return a listof the replicates which we'll then be able to iterate
over using foreach().

flat.to.flat and replace=TRUE

This is the next simplest case, the risk is that the cumulative duration
of our samples with replacement is longer or shorter than the total duration.

To address this, we'll try to take too many samples and then shorten it to the
correct duration.

To create each replicate vector:

• sample(1:nrows(txnsimdf) ,nrow(txnsimdf)+fudgefactor, replace=TRUE)
• reorganize duration/quantity columns by the sampled row numbers
• take a cumsum of duration
• trim the end to match expected duration fo the entire series

we could choose an arbitrarily large fudge factor, e.g. 50% of nrows, or we
could do something a little more complicated, taking a sample of nrow(txnsimdf),
testing the duration, and taking an additional sample if needed until the
sum(duration) is longer than the duration of

The index for the replicates, in both replace=TRUE, and replace=FALSE, should be
the start date plus the cumsum of the duration at each row. This should create
a monotonically increasing index of the beginning of each trade or flat period.

round turn trade methods which are not flat.to.flat

For any round turn trade methodology which is not measuring round turns as
flat.to.flat, things get more complicated. Fortunately, the complication is
the same for txnsim regardless of the methodology used to pair entry and exit
trades.

The first major complication with any trade that levels into a position is that
sum(txnsimdf$duration) will be longer than the market data. The general
pattern of the solution to this complication is that we sample as usual, to the
a duration equal to the original sum(txnsimdf$duration), and then overlap any
overage onto the first set of samples to get leveled trades.

The next major complication would be on max position. For this first rendition,
I think we should focus on the quantstrat-compatible Poslimit slot in the
blotter portfolio object. It can be found here:

portf$symbols[[symbol]]$PosLimit

and has columns:

"MaxPos", "LongLevels", "MinPos", "ShortLevels"

The general pattern of the solution to the maxpos problem is that we should
check the cumsum of the position implied by the replicate transactions, and
reduce or eliminate trades that would violate the max position limits. (this is done with commit 8cb479e, see #63 )

if replace=FALSE

If replace=FALSE, we start the same way as for flat.to.flat:

• sample(1:nrows(txnsimdf) ,nrow(txnsimdf))
• rearrange our duration and quantity rows by the sample order

Note that the duration/quantity tuples will have total duration longer than
mktdata, as described above. We'll deal with this when generating transactions.

if replace=TRUE

If replace=TRUE, we again start the same way as for flat.to.flat:

• sample(1:nrows(txnsimdf) ,nrow(txnsimdf)+fudgefactor, replace=TRUE)
• reorganize duration/quantity columns by the sampled row numbers

In all four scenarios described here, it seems as though we can have the same
return from the function that generates replicates. We should return a list of
n replicates which contains the duration/quantity tuples

Generating transactions

round turn trade method is flat.to.flat

At this point, replace=TRUE and replace=FALSE are immaterial. The object has a
total duration equal to the market data, and all we need to do is create the
addTxns object and apply it.

round turn trade methods which are not flat.to.flat

• add column of cumsum of duration
• split the rearranged subset at the length of the market data, trim duration of the last row if required
• set the index of the initial grouping as start date plus the cumsum of the duration at each row
• sample from index(mktdata) a number of times equal to the remaining rows
• insert the remaining rows at the sampled mktdata index dates (this gives us overlapping trades)

addTxns object:

At this point, we should have an object containing a single index that fits
within the mktdata, of the same [start, duration, qty] form as the txnsimdf
object.

Generating data for addTxns will proceed in the same manner for all input data
at this point.

• create an empty xts object for use with addTxns()
• generate rows for entry trades at the index timestamp
• generate exit trades (reverse the quantity) at start+duration

It is important that this step be done as an xts object, because xts will keep
everything ordered correctly.

Applying Transactions:

• initPortf with the name of the original portfolio + replicate number + n
• call addTxns
• call updatePortf
• return the portfolio object to a custom .combine function (see paramset and walk.forward code for examples)

@jaymon0703 and @braverock initially worked on and discussed this issue here

Found a potential edge case where if a transaction results in a position going through zero and the resulting open trade is part of the final open position at the end of the strategy, then we do not build the trades$Start vector appropriately...adding both time stamps as opposed to only the one which resulted in the open trade.

Solution (ln 205) is to reference only the last matching timestamp from which(index(incrPos) == testdf$start_ts[first(which(is.na(testdf$end_ts)))]) which is the index of the opening portion of the relevant transaction taking the position through zero.

           # add extra 'trade start' if there's an open trade, so 'includeOpenTrade' logic will work
           if(any(is.na(testdf$end_ts))){
             trades$Start <- c(trades$Start,last(which(index(incrPos) == testdf$start_ts[first(which(is.na(testdf$end_ts)))])))
           }

Submitted by: Nicholas Ashley
Assigned to: Brian Peterson
R-Forge link

When running own script and the demo (Turtles and Longtrend) getting:

Error in if (nrow(Txns) == 1) { : argument is of length zero

Appears to be occurring when calling updatePortf on the first loop, and looking at the source, the only place I found this type of call was in the updatePosPL.

My installation:
TradeAnalytics is installed from source.
Runing R version 2.11.1 (2010-05-31)
[R.app GUI 1.34 (5589) i386-apple-darwin9.8.0]

The same scripts were working recently on my windows installation, which is also running the latest TradeAnalytics. I do not have further details of that installation to hand.

Followups:

Date: 2010-09-20 12:31
Sender: Brian Peterson
RESOLVED. FIXED.A recent change to subset in xts broke this functionality. Current versions of xts and blotter have resolved the problem.Thanks for the report.Regards, - Brian

Submitted by: Mark Breman
Assigned to: Peter Carl
R-Forge link

When I try to run the example from the blotter documentation I get warnings and a error on addTxn():

library(blotter)
Loading required package: xts
Loading required package: zoo
Loading required package: quantmod
Loading required package: Defaults
Loading required package: TTR
Loading required package: FinancialInstrument
symbols = c('IBM','F','MMM')
require(quantmod)
getSymbols(symbols, from='2007-01-01', to='2007-01-31', index.class='POSIXct')
[1] 'IBM' 'F' 'MMM'

Initialize a portfolio object 'p'

print('Creating portfolio 'p'...')
[1] 'Creating portfolio 'p'...'
p = initPortf(symbols=symbols)

Trades must be made in date order.

print('Adding trades to 'p'...')
[1] 'Adding trades to 'p'...'

Make a couple of trades in IBM

p = addTxn(p, 'IBM', '2007-01-03', 50, 96.5, -0.05_50)
[1] '2007-01-03 IBM 50 @ 96.5'
Warning messages:
1: In getInstrument(Symbol) :
Instrument IBM not found, please create it first.
2: In addTxn(p, 'IBM', '2007-01-03', 50, 96.5, -0.05 * 50) :
Instrument IBM not found, using contract multiplier of 1
p = addTxn(p, 'IBM', '2007-01-04', 50, 97.1, -0.05_50)
Error in get(paste('portfolio', pname, sep = '.'), envir = .blotter) :
object 'portfolio.list(txn = c(0, 50, 0, 96.5, 0, 0, 0, 4825, 0, 96.5, 0, 50, 0, 96.5, 0, 0, 0, 1), posPL = c(0, 1, 1, 0, 0, 0, 0, 0, 0))' not found

I am using the latest build from R-Forge and the following versions:

sessionInfo()
R version 2.10.1 (2009-12-14)
i386-pc-mingw32

locale:
[1] LC_COLLATE=Dutch_Netherlands.1252 LC_CTYPE=Dutch_Netherlands.1252 LC_MONETARY=Dutch_Netherlands.1252
[4] LC_NUMERIC=C LC_TIME=Dutch_Netherlands.1252

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] blotter_0.4 FinancialInstrument_0.0.2 quantmod_0.3-14 TTR_0.20-1 Defaults_1.1-1
[6] xts_0.7-1 zoo_1.6-2

loaded via a namespace (and not attached):
[1] grid_2.10.1 lattice_0.18-3 tools_2.10.1

I looked in the code for getInstrument() and it looks like it's trying to fetch the symbol's from the .instrument environment. There are no symbols there because the example fetches the xts timeseries for the symbols in the 'base' environment (using getSymbols() from quantmod).

I'm not sure if my analysis is correct and if so, i'm not sure what is correct environment for the symbols (Various tests should also fail I think).

Regards,

-Mark-

Followups:

Date: 2010-03-29 03:49
Sender: Peter Carl
I've updated the code to work with the syntax and argument changes we've made recently. Let me know if you see any other issues. Thanks again...

Date: 2010-03-02 13:54
Sender: Peter Carl
Thanks for submitting this, Mark.

Transactions should always be added to the portfolio in time-order. Strange behavior can occur if they're added out of order. But the blotter 'txn' table is always a POSIXct index, so the comparison fails if the user supplies data with a Date index.

require(blotter)
currency("USD")
initPortf("test", symbols="AAPL", currency="USD")

addTxn("test", "AAPL", as.Date("2017-01-01"),  1, 10)
# [1] "2017-01-01 00:00:00 AAPL 1 @ 10"
# Warning messages:
# 1: In addTxn("test", "AAPL", as.Date("2017-01-01"), 1, 10) :
#   Incompatible methods ("Ops.Date", "Ops.POSIXt") for "<"
# 2: In getInstrument(Symbol) :
#   instrument AAPL not found, please create it first.
# 3: In addTxn("test", "AAPL", as.Date("2017-01-01"), 1, 10) :
#   Instrument AAPL  not found, using contract multiplier of 1
addTxn("test", "AAPL", as.Date("2017-01-02"), -1, 10)
# Error in addTxn("test", "AAPL", as.Date("2017-01-02"), -1, 10) : 
#   Transactions must be added in order. TxnDate (2017-01-02) is before last transaction in portfolio (2016-# 12-31 18:00:00) for AAPL
# In addition: Warning message:
# In addTxn("test", "AAPL", as.Date("2017-01-02"), -1, 10) :
#   Incompatible methods ("Ops.Date", "Ops.POSIXt") for "<"

It would be convenient and an easy fix to add an argument:

legend.position='bottomright'

to chart.ME and then change the last statement in the function to:

'''legend(x = legend.position, ...)

to allow the user to move the legend. Thanks.

Submitted by: Joshua Ulrich
Assigned to: Joshua Ulrich
R-Forge link

When a trade ends at midnight, the timespan will be '%Y-%m-%d %H:%M:%S/%Y-%m-%d'. The lack of '%H:%M:%S' on the end of the range causes the entire day to be selected. This may be considered a bug in xts subsetting, but there's no need to use a timespan in this function, since the trade start and end could be kept as the corresponding row indices of the PosPL table. Dirk Eddelbuettel has reported other issues caused by the digits.secs option.

Followups:

Date: 2013-10-19 17:26
Sender: Joshua Ulrich
Fixed in r1543.

Testing txnsim() on RSI highlighted the bug, which calls the subsample() function on flat period durations when there aren't any.

For the solution in ln 417,

flatdf <- subsample(svector = flatrows, targetdur = flatdur)

becomes

      if(flatdur > 0){
        flatdf  <- subsample(svector = flatrows, targetdur = flatdur)
      } else {
        flatdf <- NULL
      }

When layering trade durations onto the first layer in txnsim, we check to see if the sampled layering trade duration overlaps into another trade duration. If this happens, we check to see if the trade duration it overlaps is a similar side. If so, move onto the next layering long or short trade. If we instead overlap into a flat duration or an opposing side duration, we need to truncate.

Currently, we do not increment correctly down the list of first layer trades when looping >=2 times through the While loop, instead hard-coding an increment of 1 to the fixed first layer start timestamp. We need to add a counter to the While loop to increment appropriately for 2 or more loops.

Also, our While loop condition specifies only a > operator, but we need to account for the possibility that our first layer trade duration and our 2nd and subsequent layer trade durations could end on the same timestamp, and update the longdf and shortdf data frames accordingly.

Submitted by: Joshua Ulrich
Assigned to: Nobody
R-Forge link

These functions return the actual account and portfolio environments, which means changes to the returned object directly affect the .blotter environment. Instead, these functions should return lists, by wrapping the environment in an as.list() call.

The current ability of these functions should be encapsulated in new functions (.getAccount and .getPortfolio) and not exported.

Submitted by: Mark Breman
Assigned to: Peter Carl
R-Forge link

If I call updateAcct() with a startdate which is equal to the initdate of the account (and portfolio), it produces a double record in the $TOTAL section of the account.

Reproduction:

initDate='2010-01-01'
currency('USD')
stock(primary_id='PLW', currency='USD', multiplier=1)
getSymbols('PLW', from=initDate)
[1] 'PLW'

portfolio='faber'
account='IB'

initPortf(name=portfolio, symbols='PLW', initDate=initDate)
[1] 'faber'
initAcct(name=account, portfolios=portfolio, initDate=initDate, initEq=50000)
[1] 'IB'

verbose=TRUE

addTxn(Portfolio=portfolio, Symbol='PLW', TxnDate='2010-02-04', TxnQty=12, TxnPrice=27.12, TxnFees=1.0)
[1] '2010-02-04 PLW 12 @ 27.12'
addTxn(Portfolio=portfolio, Symbol='PLW', TxnDate='2010-02-05', TxnQty=24, TxnPrice=27.12, TxnFees=0.0)
[1] '2010-02-05 PLW 24 @ 27.12'

updatePortf(Portfolio=portfolio, Dates='2010::')
[1] 'faber'

CurrentSpan='2010-01-05::2010-03-01'

updateAcct(name=account, Dates='2010-01-01:2010-03-01')
[1] 'IB'
getAccount(Account=account)
$TOTAL
Additions Withdrawals Txn.Fees Realized.PL Unrealized.PL Int.Income Trading.PL Advisory.Fees Net.Performance End.Eq
2010-01-01 0 0 0 0 0 0 0 0 0 50000
2010-01-01 0 0 0 0 0 0 0 0 0 0

$portfolio.faber
Long.Value Short.Value Net.Value Gross.Value Txn.Fees Realized.PL Unrealized.PL Trading.PL
2010-01-01 0 0 0 0 0 0 0 0

attr(,'class')
[1] 'portfolio_account' 'account'

Look at the records in the $TOTAL section of the account.

I suspect hat this has to do with the fact that for the startdate of account and/or portfolio there is no pricedata available, but I'm not sure.

Regards,

-Mark-

Followups:

Date: 2010-10-07 16:55
Sender: Brian Peterson
Closed. Resolved. Fixed.r408 applies the appropriate trimming to Account as well, so both Portfolio and Account will not have duplicate rows if called repeatedly over the same Dates range.

Date: 2010-08-17 19:25
Sender: Brian Peterson
in updatePosPL v 374, I eliminated this problem through vectorization, and throwing out all the rows after the first Date in the date span. I think we will do something similar for Acct, allowing the user to call it as often as they want.

Date: 2010-03-02 14:02
Sender: Peter Carl
Thanks for submitting this Mark - I'll take a look.

Submitted by: Mark Breman
Assigned to: Brian Peterson
R-Forge link

After running the following scenario and calling updateAcct() I get warnings (NA's introduced by coercion) and end up with double starting rows in the portfolio$myportf section of the account:

rm(list=ls(envir=.blotter),envir=.blotter)
rm('portfolio.myportfolio', 'account.myaccount', pos=.blotter)
Warning messages:
1: In rm('portfolio.myportfolio', 'account.myaccount', pos = .blotter) :
object 'portfolio.myportfolio' not found
2: In rm('portfolio.myportfolio', 'account.myaccount', pos = .blotter) :
object 'account.myaccount' not found

Sys.setenv(TZ='GMT')

prices = c(23.07, 23.05, 23.05, 23.09, 23.28, 23.36)
MYSTOCK = xts(prices, as.POSIXct(strptime(paste('2009-01-', seq(1:length(prices)), sep=''),'%Y-%m-%d')))
colnames(MYSTOCK) = 'Close'

initDate='2009-01-01'
currency('USD')
stock(primary_id='MYSTOCK', currency='USD', multiplier=1)

portfolio='myportf'
account='myacct'

initPortf(name=portfolio, symbols='MYSTOCK', initDate=initDate)
[1] 'myportf'
initAcct(name=account, portfolios=portfolio, initDate=initDate, initEq=1000)
[1] 'myacct'

verbose=TRUE

addTxn(Portfolio=portfolio, Symbol='MYSTOCK', TxnDate='2009-01-02', TxnQty=36, TxnPrice=23.05, TxnFees=-1.0)
[1] '2009-01-02 MYSTOCK 36 @ 23.05'
addTxn(Portfolio=portfolio, Symbol='MYSTOCK', TxnDate='2009-01-04', TxnQty=-36, TxnPrice=23.09, TxnFees=-1.0)
[1] '2009-01-04 MYSTOCK -36 @ 23.09'

updatePortf(Portfolio=portfolio, Dates='2009::')
[1] 'myportf'
getPortfolio(portfolio)
$MYSTOCK
$MYSTOCK$txn
Txn.Qty Txn.Price Txn.Fees Txn.Value Txn.Avg.Cost Pos.Qty Pos.Avg.Cost Realized.PL Con.Mult
2009-01-01 0 0.00 0 0.00 0.00000 0 0.00000 0.00 0
2009-01-02 36 23.05 -1 830.80 23.07778 36 23.07778 0.00 1
2009-01-04 -36 23.09 -1 -830.24 23.06222 0 0.00000 -0.56 1

$MYSTOCK$posPL
Pos.Qty Con.Mult Ccy.Mult Pos.Value Txn.Value Txn.Fees Realized.PL Unrealized.PL Trading.PL
2009-01-01 0 1 1 0.0 0.00 0 0.00 0 0.00
2009-01-02 36 1 1 829.8 830.80 -1 0.00 -1 -1.00
2009-01-03 36 1 1 829.8 0.00 0 0.00 0 0.00
2009-01-04 0 1 1 0.0 -830.24 -1 -0.56 1 0.44
2009-01-05 0 1 1 0.0 0.00 0 0.00 0 0.00
2009-01-06 0 1 1 0.0 0.00 0 0.00 0 0.00

attr(,'class')
[1] 'blotter_portfolio' 'portfolio'

2-1.44
[1] 0.56
36*0.04
[1] 1.44
MYSTOCK
Close
2009-01-01 23.07
2009-01-02 23.05
2009-01-03 23.05
2009-01-04 23.09
2009-01-05 23.28
2009-01-06 23.36
getAccount(account)
$TOTAL
Additions Withdrawals Txn.Fees Realized.PL Unrealized.PL Int.Income Trading.PL Advisory.Fees Net.Performance End.Eq
2009-01-01 0 0 0 0 0 0 0 0 0 1000

$portfolio.myportf
Long.Value Short.Value Net.Value Gross.Value Txn.Fees Realized.PL Unrealized.PL Trading.PL
2009-01-01 0 0 0 0 0 0 0 0

attr(,'class')
[1] 'portfolio_account' 'account'

updateAcct(name=account, Dates='2009::')
[1] 'myacct'
There were 12 warnings (use warnings() to see them)
warnings()
Warning messages:
1: In as_numeric(YYYY) : NAs introduced by coercion
2: In as_numeric(YYYY) : NAs introduced by coercion
3: In as_numeric(YYYY) : NAs introduced by coercion
4: In as_numeric(YYYY) : NAs introduced by coercion
5: In as_numeric(YYYY) : NAs introduced by coercion
6: In as_numeric(YYYY) : NAs introduced by coercion
7: In as_numeric(YYYY) : NAs introduced by coercion
8: In as_numeric(YYYY) : NAs introduced by coercion
9: In as_numeric(YYYY) : NAs introduced by coercion
10: In as_numeric(YYYY) : NAs introduced by coercion
11: In as_numeric(YYYY) : NAs introduced by coercion
12: In as_numeric(YYYY) : NAs introduced by coercion
getPortfolio(portfolio)
$MYSTOCK
$MYSTOCK$txn
Txn.Qty Txn.Price Txn.Fees Txn.Value Txn.Avg.Cost Pos.Qty Pos.Avg.Cost Realized.PL Con.Mult
2009-01-01 0 0.00 0 0.00 0.00000 0 0.00000 0.00 0
2009-01-02 36 23.05 -1 830.80 23.07778 36 23.07778 0.00 1
2009-01-04 -36 23.09 -1 -830.24 23.06222 0 0.00000 -0.56 1

$MYSTOCK$posPL
Pos.Qty Con.Mult Ccy.Mult Pos.Value Txn.Value Txn.Fees Realized.PL Unrealized.PL Trading.PL
2009-01-01 0 1 1 0.0 0.00 0 0.00 0 0.00
2009-01-02 36 1 1 829.8 830.80 -1 0.00 -1 -1.00
2009-01-03 36 1 1 829.8 0.00 0 0.00 0 0.00
2009-01-04 0 1 1 0.0 -830.24 -1 -0.56 1 0.44
2009-01-05 0 1 1 0.0 0.00 0 0.00 0 0.00
2009-01-06 0 1 1 0.0 0.00 0 0.00 0 0.00

attr(,'class')
[1] 'blotter_portfolio' 'portfolio'

getAccount(account)
$TOTAL
Additions Withdrawals Txn.Fees Realized.PL Unrealized.PL Int.Income Trading.PL Advisory.Fees Net.Performance End.Eq
2009-01-01 0 0 0 0.00 0 0 0.00 0 0 1000
2009-01-01 0 0 0 0.00 0 0 0.00 0 0 0
2009-01-01 0 0 0 0.00 0 0 0.00 0 0 0
2009-01-02 0 0 -1 0.00 -1 0 -1.00 0 0 0
2009-01-03 0 0 0 0.00 0 0 0.00 0 0 0
2009-01-04 0 0 -1 -0.56 1 0 0.44 0 0 0
2009-01-05 0 0 0 0.00 0 0 0.00 0 0 0
2009-01-06 0 0 0 0.00 0 0 0.00 0 0 0

$portfolio.myportf
Long.Value Short.Value Net.Value Gross.Value Txn.Fees Realized.PL Unrealized.PL Trading.PL
2009-01-01 0.0 0 0.0 0.0 0 0.00 0 0.00 <<=============== FIRST ROW
2009-01-01 0.0 0 0.0 0.0 0 0.00 0 0.00 <<=============== DOUBLE ROW!!
2009-01-02 829.8 0 829.8 829.8 -1 0.00 -1 -1.00
2009-01-03 829.8 0 829.8 829.8 0 0.00 0 0.00
2009-01-04 0.0 0 0.0 0.0 -1 -0.56 1 0.44
2009-01-05 0.0 0 0.0 0.0 0 0.00 0 0.00
2009-01-06 0.0 0 0.0 0.0 0 0.00 0 0.00

attr(,'class')
[1] 'portfolio_account' 'account'

Also note the triple starting row in the $TOTAL section (already reported as another bug with ID: 831)

Followups:

Date: 2010-10-07 16:57
Sender: Brian Peterson
Resolved. Closed. Fixed. Account will not include double rows anymore. - Brian

Date: 2010-08-17 21:01
Sender: Brian Peterson
This has been fixed in SVN rev 374 for updatePortf.UpdateAcct still to come.

Date: 2010-03-04 03:40
Sender: Peter Carl
I've picked Brian's comments about his proposed solution out of the comments below and added them to Feature Request 289.Although I think we should fix the problem (at some point soon), I'll propose that we insert a stop() for now.Does that seem like a reasonable temporary approach?

Date: 2010-03-03 18:36
Sender: Mark Breman
Brian,Just a remark about your message regarding the initDate should be a date prior to the first close price given.If I retry my test scenario with an initDate set to a date prior to the first price date, the problems I reported persist:> rm(list=ls())> rm(list=ls(envir=.blotter),envir=.blotter)> rm('portfolio.myportfolio', 'account.myaccount', pos=.blotter)Warning messages:1: In rm('portfolio.myportfolio', 'account.myaccount', pos = .blotter) : object 'portfolio.myportfolio' not found2: In rm('portfolio.myportfolio', 'account.myaccount', pos = .blotter) : object 'account.myaccount' not found> > Sys.setenv(TZ='GMT')> > prices = c(23.07, 23.05, 23.05, 23.09, 23.28, 23.36)> MYSTOCK = xts(prices, as.POSIXct(strptime(paste('2009-01-', seq(1:length(prices)), sep=''),'%Y-%m-%d')))> colnames(MYSTOCK) = 'Close'> > initDate='2008-12-31'> currency('USD')> stock(primary_id='MYSTOCK', currency='USD', multiplier=1)> > portfolio='myportf'> account='myacct'> > initPortf(name=portfolio, symbols='MYSTOCK', initDate=initDate)[1] 'myportf'> initAcct(name=account, portfolios=portfolio, initDate=initDate, initEq=1000)[1] 'myacct'> > verbose=TRUE> > addTxn(Portfolio=portfolio, Symbol='MYSTOCK', TxnDate='2009-01-02', TxnQty=36, TxnPrice=23.05, TxnFees=-1.0)[1] '2009-01-02 MYSTOCK 36 @ 23.05'> addTxn(Portfolio=portfolio, Symbol='MYSTOCK', TxnDate='2009-01-04', TxnQty=-36, TxnPrice=23.09, TxnFees=-1.0)[1] '2009-01-04 MYSTOCK -36 @ 23.09'> > updatePortf(Portfolio=portfolio, Dates='2009::')[1] 'myportf'> > #CurrentSpan='2010-01-05::2010-03-01'> #updateAcct(name=account, Dates='2009-01-01:2009-01-06')> updateAcct(name=account, Dates='2009::')[1] 'myacct'There were 12 warnings (use warnings() to see them)> > #updateEndEq(Account=account)> > #getTxns(Portfolio=portfolio, Symbol='PLW', Date='2010::')> > getPortfolio(Portfolio=portfolio)$MYSTOCK$MYSTOCK$txn Txn.Qty Txn.Price Txn.Fees Txn.Value Txn.Avg.Cost Pos.Qty Pos.Avg.Cost Realized.PL Con.Mult2008-12-31 0 0.00 0 0.00 0.00000 0 0.00000 0.00 02009-01-02 36 23.05 -1 830.80 23.07778 36 23.07778 0.00 12009-01-04 -36 23.09 -1 -830.24 23.06222 0 0.00000 -0.56 1$MYSTOCK$posPL Pos.Qty Con.Mult Ccy.Mult Pos.Value Txn.Value Txn.Fees Realized.PL Unrealized.PL Trading.PL2008-12-31 0 1 1 0.0 0.00 0 0.00 0 0.002009-01-02 36 1 1 829.8 830.80 -1 0.00 -1 -1.002009-01-03 36 1 1 829.8 0.00 0 0.00 0 0.002009-01-04 0 1 1 0.0 -830.24 -1 -0.56 1 0.442009-01-05 0 1 1 0.0 0.00 0 0.00 0 0.002009-01-06 0 1 1 0.0 0.00 0 0.00 0 0.00attr(,'class')[1] 'blotter_portfolio' 'portfolio' > getAccount(Account=account)$TOTAL Additions Withdrawals Txn.Fees Realized.PL Unrealized.PL Int.Income Trading.PL Advisory.Fees Net.Performance End.Eq2008-12-31 0 0 0 0.00 0 0 0.00 0 0 10002008-12-31 0 0 0 0.00 0 0 0.00 0 0 02008-12-31 0 0 0 0.00 0 0 0.00 0 0 02009-01-02 0 0 -1 0.00 -1 0 -1.00 0 0 02009-01-03 0 0 0 0.00 0 0 0.00 0 0 02009-01-04 0 0 -1 -0.56 1 0 0.44 0 0 02009-01-05 0 0 0 0.00 0 0 0.00 0 0 02009-01-06 0 0 0 0.00 0 0 0.00 0 0 0$portfolio.myportf Long.Value Short.Value Net.Value Gross.Value Txn.Fees Realized.PL Unrealized.PL Trading.PL2008-12-31 0.0 0 0.0 0.0 0 0.00 0 0.002008-12-31 0.0 0 0.0 0.0 0 0.00 0 0.002009-01-02 829.8 0 829.8 829.8 -1 0.00 -1 -1.002009-01-03 829.8 0 829.8 829.8 0 0.00 0 0.002009-01-04 0.0 0 0.0 0.0 -1 -0.56 1 0.442009-01-05 0.0 0 0.0 0.0 0 0.00 0 0.002009-01-06 0.0 0 0.0 0.0 0 0.00 0 0.00attr(,'class')[1] 'portfolio_account' 'account' > Regarding your remarks about handling out-of-order transactions, multiple calls to update* functions etc. I think that if you do accept the parameters of a user you should always produce a correct result or die with an (clear) error. Recalculation of the entire portfolio/account should be possible at all times I think.Is it not possible to sort the user provided transactions according to date before storing them in the blotter administration? I'm not sure about this at all and have to think about it a bit.Regards,-Mark-

Date: 2010-03-03 15:30
Sender: Brian Peterson
The issue in the transactions is that Pos.Qty Pos.Avg.Cost Realized.PLare all calculated, and path dependent.So if things come in out of order, everything later is wrong.I agree that if we decide to throw a stop() error it should be with a clear error. The question is whether we should put in the stop() error for now, or try to fix the problem.Clearly the initial assumption was overly restrictive. Now we need to deal with it. - Brian

Date: 2010-03-03 15:24
Sender: Mark Breman
Brian,Just a remark about your message regarding the initDate should be a date prior to the first close price given.If I retry my test scenario with an initDate set to a date prior to the first price date, the problems I reported persist:> rm(list=ls())> rm(list=ls(envir=.blotter),envir=.blotter)> rm('portfolio.myportfolio', 'account.myaccount', pos=.blotter)Warning messages:1: In rm('portfolio.myportfolio', 'account.myaccount', pos = .blotter) : object 'portfolio.myportfolio' not found2: In rm('portfolio.myportfolio', 'account.myaccount', pos = .blotter) : object 'account.myaccount' not found> > Sys.setenv(TZ='GMT')> > prices = c(23.07, 23.05, 23.05, 23.09, 23.28, 23.36)> MYSTOCK = xts(prices, as.POSIXct(strptime(paste('2009-01-', seq(1:length(prices)), sep=''),'%Y-%m-%d')))> colnames(MYSTOCK) = 'Close'> > initDate='2008-12-31'> currency('USD')> stock(primary_id='MYSTOCK', currency='USD', multiplier=1)> > portfolio='myportf'> account='myacct'> > initPortf(name=portfolio, symbols='MYSTOCK', initDate=initDate)[1] 'myportf'> initAcct(name=account, portfolios=portfolio, initDate=initDate, initEq=1000)[1] 'myacct'> > verbose=TRUE> > addTxn(Portfolio=portfolio, Symbol='MYSTOCK', TxnDate='2009-01-02', TxnQty=36, TxnPrice=23.05, TxnFees=-1.0)[1] '2009-01-02 MYSTOCK 36 @ 23.05'> addTxn(Portfolio=portfolio, Symbol='MYSTOCK', TxnDate='2009-01-04', TxnQty=-36, TxnPrice=23.09, TxnFees=-1.0)[1] '2009-01-04 MYSTOCK -36 @ 23.09'> > updatePortf(Portfolio=portfolio, Dates='2009::')[1] 'myportf'> > #CurrentSpan='2010-01-05::2010-03-01'> #updateAcct(name=account, Dates='2009-01-01:2009-01-06')> updateAcct(name=account, Dates='2009::')[1] 'myacct'There were 12 warnings (use warnings() to see them)> > #updateEndEq(Account=account)> > #getTxns(Portfolio=portfolio, Symbol='PLW', Date='2010::')> > getPortfolio(Portfolio=portfolio)$MYSTOCK$MYSTOCK$txn Txn.Qty Txn.Price Txn.Fees Txn.Value Txn.Avg.Cost Pos.Qty Pos.Avg.Cost Realized.PL Con.Mult2008-12-31 0 0.00 0 0.00 0.00000 0 0.00000 0.00 02009-01-02 36 23.05 -1 830.80 23.07778 36 23.07778 0.00 12009-01-04 -36 23.09 -1 -830.24 23.06222 0 0.00000 -0.56 1$MYSTOCK$posPL Pos.Qty Con.Mult Ccy.Mult Pos.Value Txn.Value Txn.Fees Realized.PL Unrealized.PL Trading.PL2008-12-31 0 1 1 0.0 0.00 0 0.00 0 0.002009-01-02 36 1 1 829.8 830.80 -1 0.00 -1 -1.002009-01-03 36 1 1 829.8 0.00 0 0.00 0 0.002009-01-04 0 1 1 0.0 -830.24 -1 -0.56 1 0.442009-01-05 0 1 1 0.0 0.00 0 0.00 0 0.002009-01-06 0 1 1 0.0 0.00 0 0.00 0 0.00attr(,'class')[1] 'blotter_portfolio' 'portfolio' > getAccount(Account=account)$TOTAL Additions Withdrawals Txn.Fees Realized.PL Unrealized.PL Int.Income Trading.PL Advisory.Fees Net.Performance End.Eq2008-12-31 0 0 0 0.00 0 0 0.00 0 0 10002008-12-31 0 0 0 0.00 0 0 0.00 0 0 02008-12-31 0 0 0 0.00 0 0 0.00 0 0 02009-01-02 0 0 -1 0.00 -1 0 -1.00 0 0 02009-01-03 0 0 0 0.00 0 0 0.00 0 0 02009-01-04 0 0 -1 -0.56 1 0 0.44 0 0 02009-01-05 0 0 0 0.00 0 0 0.00 0 0 02009-01-06 0 0 0 0.00 0 0 0.00 0 0 0$portfolio.myportf Long.Value Short.Value Net.Value Gross.Value Txn.Fees Realized.PL Unrealized.PL Trading.PL2008-12-31 0.0 0 0.0 0.0 0 0.00 0 0.002008-12-31 0.0 0 0.0 0.0 0 0.00 0 0.002009-01-02 829.8 0 829.8 829.8 -1 0.00 -1 -1.002009-01-03 829.8 0 829.8 829.8 0 0.00 0 0.002009-01-04 0.0 0 0.0 0.0 -1 -0.56 1 0.442009-01-05 0.0 0 0.0 0.0 0 0.00 0 0.002009-01-06 0.0 0 0.0 0.0 0 0.00 0 0.00attr(,'class')[1] 'portfolio_account' 'account' > Regarding your remarks about handling out-of-order transactions, multiple calls to update* functions etc. I think that if you do accept the parameters of a user you should always produce a correct result or die with an (clear) error. Recalculation of the entire portfolio/account should be possible at all times I think.Is it not possible to sort the user provided transactions according to date before storing them in the blotter administration? I'm not sure about this at all and have to think about it a bit.Regards,-Mark-

Date: 2010-03-03 14:32
Sender: Brian Peterson
one of the design assumptions we made to get blotter off the ground was that we weren't going to support things happening 'out of order'As a 'trade blotter', the design assumption went, the user will have control over where and how it is run, and will run things in linear time without overlaps.The documentation supports this assumption, stating that initDate should be a date prior to the first close price given.Now, all that aside, users will run update* more than once on the same date. What do we do about it?Today, we 'rbind' the new calculations to the time series, which can result in double and triple rows, and definitely results in incorrect calculations.I think we have a few options. - die with an error if you've already calc'd over those timestamps- throw out any rows after currentDate and recalc- insert rows on currentDate by index, overwriting previously calculated rows- apply adjustments to bring all future rows into complianceIn an accounting system, the last option (adjustments) would be correct. 'blotter' is not a portfolio accounting stystem, so I think we can toss that option for portfolio and account updates. However, I think we do need to apply adjustments, of a sort, to transactions. I'll come to that in a bit.If we wanted to hold to our design assumption, we'd pick the first option, and insert a stop() error and make the user sort it out. I don't think anyone would prefer this option.That leaves the second and third options.For updatePortf and LupdateAcct, I think we should go with the second option. eave any rows prior to currentDate, throw out any later rows, and calc over the Dates range sent to the update* function. This should be minimal code to add, and will work.The third option, to insert rows directly, is very fragile if the index of updates changes, so I'd rather not go down that path, though it is also only a small code change.For transactions, I don't think the solution is nearly so clear. If a transaction comes in out of order, I'd be inclined for now to try to trap that (if txnDate is before last() txnDate) abd throw a stop error. If we want to handle them, then I think we'd need to retrieve all transactions after the current txnDate (using getTxns with a date range), strip the calculated values, leaving only the reported values (time, qty, price, fees), and remove those rows from the table. Then you'd insert the new transaction as normal, and insert the other out of order transactions using addTxns(). You'd also need to clear out or recalc any portfolio or account updates after txnDate, since they're now wrong.We need to be very careful in balancing robustness against speed. The more we recalc, the longer it takes. all of the add* functions are currently quite fast, but the update* functions are slow (painfully so in some cases). Supporting out or order and duplicate updates may exacerbate these performance issues. - Brian

R CMD check now checks to ensure functions in compiled code are registered. How to do this is described in Writing R Extensions Section 5.4.

Using a vectorised approach for the construction of the replicate transactions in txnsim.txns() we can reduce the memory required to simulate 'n' replicates. The improvement in performance will be orders of magnitude, making the simulation of 1k replicates possible in the single digit minute time space (currently ranging from 20 mins to 15 hours depending on hardware) and making feasible simulations for a greater number of replicates.

In the example below I get a 6.9% portfolio return and a slightly negative account return and the problem is with the account return. I took a look at the function and found one way to fix it, but I might be way off base, please let me know.

In AcctReturns line 102 adds the daily P&L to initEq...

V = initEq + reclass(rowSums(table), table)                  # Account values

... but it seems we need the cumulative P&L, as such:

V = initEq + reclass(cumsum(rowSums(table)), table)                  # Account values

Also, line 107 calculates the returns as:

  returns =  V  / (lag(V) + CF) - 1

But the return on the first day is NA, so we need to fill it in, as such:

  returns =  V  / (lag(V) + CF) - 1
  returns[1] = V[1] / initEq - 1

This produces an account return which matches closely to the portfolio return. I am really interested in hearing feedback on this patch.

Start Example Code

require(blotter)
require(FinancialInstrument)
require(quantmod)
rm(.blotter)
if(!exists(".instrument")) .instrument <<- new.env()
if(!exists(".blotter")) .blotter <<- new.env()

currency("USD")
symbols = c("IBM")
for(symbol in symbols){ # establish tradable instruments
stock(symbol, currency="USD", multiplier=1)
}

getSymbols(symbols, from='2017-01-01', to='2017-01-31', src='yahoo', index.class=c("POSIXt","POSIXct"))

initPortf("p", symbols=symbols, currency="USD")

addTxn(Portfolio = "p", Symbol = "IBM", TxnDate = '2017-01-03', TxnQty = 1, TxnPrice = 166, TxnFees = 00)
addTxn("p", "IBM", '2017-01-27', -1, 177.30, TxnFees = 0)

updatePortf(Portfolio="p",Dates='2017-01')

initAcct(name="a", portfolios="p", initEq=166, currency="USD")
updateAcct("a",'2017-01')
updateEndEq("a",'2017-01')

portfolio_rets <- PortfReturns(Account="a", Dates="2017", Portfolios="p")
Return.cumulative(portfolio_rets)

account_rets <- AcctReturns(Account="a", Dates="2017", Portfolios="p")
Return.cumulative(account_rets)

p = getPortfolio("p")
a = getAccount("a")

account_test <- CalculateReturns(a$summary$End.Eq)
cbind(portfolio_rets, account_rets, account_test)

The NFA 13-column report is described here:

https://www.nfa.futures.org/NFA-compliance/publication-library/disclosure-document-guide.pdf

blotter currently supports most of the features of the 13-column report.

It would be desirable to add pro-forma fees to a blotter account on a prospective basis after the fact, so that the effects of different fee structures could be examined.

Submitted by: Christian Neuabuer
Assigned to: Nobody
R-Forge link

Hi,

I am planning to user blotter with currencies. However there are the following errors in the update functions.

updatePortf('portf', Dates = CurrentDate)
Error in NextMethod(.Generic) :
number of items to replace is not a multiple of replacement length
In addition: Warning message:
In if (is.na(CcyMult) & !is.na(FXrate)) { :
the condition has length > 1 and only the first element will be used

updateAcct('accnt', Dates = CurrentDate)
Error in xts(rep(0, obsLength), order.by = obsDates) :
order.by requires an appropriate time-based object

updateEndEq('accnt', Dates = CurrentDate)
Error in if (!all(i <= 0)) stop('only zeros may be mixed with negative subscripts') :
missing value where TRUE/FALSE needed
In addition: Warning message:
In max(i) : no non-missing arguments to max; returning -Inf

I have reduced the problem to a stand alone script attached.

Any help is appreciated.

Regards
Chris

Followups:

Date: 2011-08-19 13:39
Sender: Brian Peterson
I'm closing this because I don't have this problem on multi-currency portfolios, and we don't have a reproducible example.Marking Closed Invalid.If you want to produce a working reproducible example, we'll fix any issues that exist.regards, - Brian

Date: 2011-03-11 13:09
Sender: Christian Neuabuer
Dear Brian,you are completely right. I tried to minimalize the bug report code, so I stripped out all the addTxn(...) stuff. What I want to do is to simulate a strategy that operates on baskets of (cointegrated) currencies. I plan to use blotter to simplify the transaction stuff in order to do some backtesting of my strategy.I hope this clarifies a little bit.RegardsChris

Date: 2011-03-11 12:39
Sender: Brian Peterson
I'm confused about what you're trying to accomplish.You don't have any transactions, thus no position, so there's nothing to update.I'll look into the section of updatePosPL that does the currency multiplier, but a little more context would be appreciated.Regards, - Brian

Date: 2011-03-11 08:29
Sender: Christian Neuabuer
my installaltionR version 2.12.2 (2011-02-25)Copyright (C) 2011 The R Foundation for Statistical ComputingISBN 3-900051-07-0Platform: x86_64-apple-darwin9.8.0/x86_64 (64-bit)RegardsChris

Submitted by: Ben McCann
Assigned to: Joshua Ulrich
R-Forge link

This example:
http://blog.fosstrading.com/2009/11/tactical-asset-allocation-using-blotter.html

Fails on Line 52:
tmpPL$Pos.Qty <- na.locf(tmpPL$Pos.Qty)

With error:
Error in complete.cases(object) :
no input has determined the number of cases
Calls: updatePortf ... na.locf.default -> na.trim -> na.trim.default -> complete.cases
In addition: Warning messages:
1: In max(i) : no non-missing arguments to max; returning -Inf
2: In is.na(x) : is.na() applied to non-(list or vector) of type 'NULL'

Followups:

Date: 2010-09-20 12:28
Sender: Brian Peterson
The FOSSTrading example works again with current blotter/xts code.RESOLVED. FIXED.Regards, - Brian

Submitted by: Phong Nguyen
Assigned to: Nobody
R-Forge link

Hello,

I use blotter to track/simulate portfolio to replay data from an actual transactions but ran into a problem that I'm not sure if it's a subtle bug with regards to timezone or something else, as only 2/163 transactions cause issue. Details of problematic transactions are below:

• on 2014-03-28, I bought 530 FPT
• on 2014-05-09, this has 4:1 stock split, which is simulated by adding to the portfolio 132 (530/4, rounding to nearest integer) with price of 0. The total position in portfolio should be 662 (shown correctly from getPos on 2014-05-27)
• on 2014-03-28, I bought another 1250 FPT, the total should be 1912, but getPos shows only 1780 (=530+1250).

Any ideas/help for work-around is highly appreciated. Many thanks in advance.

[1] '2014-03-28 00:00:00 FPT 530 @ 69.5'
[1] '2014-05-28 00:00:00 FPT 1250 @ 46.1'
[1] '2014-05-29 00:00:00 FPT -530 @ 47.1'
[1] '2014-06-12 00:00:00 FPT -1250 @ 47.3'
[1] '2014-07-03 00:00:00 FPT -130 @ 48.2'
[1] '2014-05-09 00:00:00 FPT 132 @ 0.000'
[1] '2014-05-09 FPT Dividend 1.425 on 662 shares: 943.35'

getPos(Portfolio=portfolio.VNDS,Symbol='FPT',Date=as.Date('2014-05-27'))
Pos.Qty Pos.Avg.Cost
2014-05-12 662 55.6419939577039
getPos(Portfolio=portfolio.VNDS,Symbol='FPT',Date=as.Date('2014-05-28'))
Pos.Qty Pos.Avg.Cost
2014-05-28 07:00:00 1780 53.0674157303371
getPos(Portfolio=portfolio.VNDS,Symbol='FPT',Date=as.Date('2014-05-29'))
Pos.Qty Pos.Avg.Cost
2014-05-29 07:00:00 1250 53.0674157303371
getPos(Portfolio=portfolio.VNDS,Symbol='FPT',Date=as.Date('2014-05-30'))
Pos.Qty Pos.Avg.Cost
2014-05-29 07:00:00 1250 53.0674157303371

sessionInfo()
R version 3.0.3 (2014-03-06)
Platform: x86_64-w64-mingw32/x64 (64-bit)

locale:
[1] LC_COLLATE=English_Australia.1252 LC_CTYPE=English_Australia.1252
[3] LC_MONETARY=English_Australia.1252 LC_NUMERIC=C
[5] LC_TIME=English_Australia.1252

attached base packages:
[1] grDevices datasets parallel graphics utils stats methods
[8] base

other attached packages:
[1] XML_3.98-1.1 vTrading_0.2.496
[3] scales_0.2.4 ggplot2_1.0.0
[5] reshape2_1.4 png_0.1-7
[7] plyr_1.8.1 timeDate_3010.98
[9] logging_0.7-103 quantstrat_0.8.2
[11] foreach_1.4.2 gdata_2.13.3
[13] blotter_0.8.18 PerformanceAnalytics_1.1.0
[15] FinancialInstrument_1.1 quantmod_0.4-0
[17] TTR_0.22-0 xts_0.9-7
[19] zoo_1.7-11 Defaults_1.1-1

loaded via a namespace (and not attached):
[1] codetools_0.2-8 colorspace_1.2-4 digest_0.6.4 grid_3.0.3
[5] gtable_0.1.2 gtools_3.4.1 iterators_1.0.7 labeling_0.2
[9] lattice_0.20-29 MASS_7.3-33 munsell_0.4.2 proto_0.3-10
[13] Rcpp_0.11.2 stringr_0.6.2 tools_3.0.3

P/S: I tried to submit to forum but it doesn't seem to get registered

Based on the example code at the end, my additions, withdrawals and interest are not being added to the account summary. One fix would be to modify line 94 of updateAcct.R from

obsDates = index(table)

to

obsDates = as.Date(index(table))

I thought this was more of a hack than a fix, but then I saw this post where a user says "If you supply a POSIXct time xts tries to make an exact match, which it doesn't find in this case." and obsDates and TxnDate are both POSIXct, so it may be good enough to change obsDates to non-time based so xts will match it.

require(blotter)
require(FinancialInstrument)
require(quantmod)
rm(.blotter)
if(!exists(".instrument")) .instrument <<- new.env()
if(!exists(".blotter")) .blotter <<- new.env()

currency("USD")
symbols = c("IBM")
for(symbol in symbols){ # establish tradable instruments
  stock(symbol, currency="USD", multiplier=1)
}

getSymbols(symbols, from='2017-01-01', to='2017-01-10', src='yahoo', index.class=c("POSIXt","POSIXct"))

initPortf("p", symbols=symbols, currency="USD")

addTxn(Portfolio = "p", Symbol = "IBM", TxnDate = '2017-01-03', TxnQty = 1, TxnPrice = 166, TxnFees = 00)
addTxn("p", "IBM", '2017-01-10', -1, 170, TxnFees = 0)

updatePortf(Portfolio="p",Dates='2017-01')

initAcct(name="a", portfolios="p", initEq=166, currency="USD")

addAcctTxn(Account="a", TxnDate='2017-01-05', TxnType='Additions', Amount=1000)
addAcctTxn(Account="a", TxnDate='2017-01-06', TxnType='Withdrawals', Amount=1000)
addAcctTxn(Account="a", TxnDate='2017-01-09', TxnType='Interest', Amount=10)

updateAcct("a")
updateEndEq("a")

a <- getAccount("a")
a$summary
index(a$Additions)
a$Interest

Inside txnstruct() we currently use !is.null(maxlongpos) and !is.null(maxshortpos) to check whether maxlongpos or maxshortpos exists. When one of these variables does not exist, the function will error out as the argument is missing.

Instead using try() would work.

longcheck <- try(missing(maxlongpos), silent = TRUE)
shortcheck <- try(missing(maxshortpos), silent = TRUE)
if (class(longcheck) != "try-error") attr(txnsimdf,"maxlongpos") <- maxlongpos
if (class(shortcheck) != "try-error") attr(txnsimdf,"maxshortpos") <- maxshortpos

Note joran's reply to this SO question. Note also @gsee reply.

Currently in txnsim() we do not limit the quantity from layering long and short trades when building out the replicate strategies.

To remedy this, we can create a new stylized fact, namely; 'maxlongpos' for the maximum long position observed in the original strategy, and maxshortpos for the maximum absolute short position observed in the original strategy. Then keep track of the cumsum of the layered quantities as we layer, making sure we only add to the long and short layer if the cumsum is within the max long or short positions observed in the original strategy, respectively.

Submitted by: Mark Breman
Assigned to: Peter Carl
R-Forge link

If I run the amzn_test.R demo it looks like it's working fine although I get warnings about my TZ being different than the TZ from the data:

require(blotter)

Remove portfolio and account data if run previously

try(rm('portfolio.amzn_port','account.amzn_acct',pos=.blotter))

load the example data

data('amzn')
currency('USD')
stock('amzn',currency='USD',multiplier=1)

Initialize the Portfolio

initPortf('amzn_port',symbols='amzn',initDate='2010-01-14')
[1] 'amzn_port'
initAcct('amzn_acct',portfolios='amzn_port',initDate='2010-01-14', initEq=10000)
[1] 'amzn_acct'

look at the transactions data

amzn.trades
Price Quantity
2010-01-14 15:18:50.00000 127.49 -400
2010-01-14 15:18:53.00000 127.49 400
2010-01-14 15:21:50.00000 127.26 -300
2010-01-14 15:21:53.00000 127.26 300
2010-01-14 15:23:07.00000 127.17 100
2010-01-14 15:23:10.00000 127.19 -100
2010-01-14 15:37:56.00000 127.12 -400
2010-01-14 15:37:59.00000 127.12 400
2010-01-14 15:38:32.00001 127.23 -500
2010-01-14 15:38:35.00000 127.23 500
2010-01-14 15:38:37.00001 127.27 300
2010-01-14 15:38:40.00000 127.30 -300
2010-01-14 15:46:08.00001 126.95 -1300
2010-01-14 15:46:11.00000 126.95 1300
Warning message:
In check.TZ(x) :
timezone of object () is different than current timezone (GMT).

Add the transactions to the portfolio

if you wanted to avoid the contract multiplier warning, you would add an instrument first

addTxns('amzn_port','amzn',TxnData=amzn.trades,verbose=TRUE)
Txn.Qty Txn.Price Txn.Fees Txn.Value Txn.Avg.Cost Pos.Qty Pos.Avg.Cost Realized.PL Con.Mult
2010-01-14 15:18:50.00000 -400 127.49 0 -50996 127.49 -400 127.49 0 1
2010-01-14 15:18:53.00000 400 127.49 0 50996 127.49 0 0.00 0 1
2010-01-14 15:21:50.00000 -300 127.26 0 -38178 127.26 -300 127.26 0 1
2010-01-14 15:21:53.00000 300 127.26 0 38178 127.26 0 0.00 0 1
2010-01-14 15:23:07.00000 100 127.17 0 12717 127.17 100 127.17 0 1
2010-01-14 15:23:10.00000 -100 127.19 0 -12719 127.19 0 0.00 2 1
2010-01-14 15:37:56.00000 -400 127.12 0 -50848 127.12 -400 127.12 0 1
2010-01-14 15:37:59.00000 400 127.12 0 50848 127.12 0 0.00 0 1
2010-01-14 15:38:32.00001 -500 127.23 0 -63615 127.23 -500 127.23 0 1
2010-01-14 15:38:35.00000 500 127.23 0 63615 127.23 0 0.00 0 1
2010-01-14 15:38:37.00001 300 127.27 0 38181 127.27 300 127.27 0 1
2010-01-14 15:38:40.00000 -300 127.30 0 -38190 127.30 0 0.00 9 1
2010-01-14 15:46:08.00001 -1300 126.95 0 -165035 126.95 -1300 126.95 0 1
2010-01-14 15:46:11.00000 1300 126.95 0 165035 126.95 0 0.00 0 1

update the portfolio stats

updatePortf('amzn_port',Dates='2010-01-14')
[1] 'amzn_port'

update the account P&L

updateAcct('amzn_acct',Dates='2010-01-14')
[1] 'amzn_acct'
There were 12 warnings (use warnings() to see them)

and look at it

chart.Posn('amzn_port','amzn',Dates='2010-01-14',theme='white')

portfolio = getPortfolio('amzn_port')
account = getAccount('amzn_acct')
warnings()
Warning messages:
1: In as_numeric(YYYY) : NAs introduced by coercion
2: In as_numeric(YYYY) : NAs introduced by coercion
3: In as_numeric(YYYY) : NAs introduced by coercion
4: In as_numeric(YYYY) : NAs introduced by coercion
5: In as_numeric(YYYY) : NAs introduced by coercion
6: In as_numeric(YYYY) : NAs introduced by coercion
7: In as_numeric(YYYY) : NAs introduced by coercion
8: In as_numeric(YYYY) : NAs introduced by coercion
9: In as_numeric(YYYY) : NAs introduced by coercion
10: In as_numeric(YYYY) : NAs introduced by coercion
11: In as_numeric(YYYY) : NAs introduced by coercion
12: In as_numeric(YYYY) : NAs introduced by coercion

But then if I try to update the end equity in the account after this it does not update the End.Eq column in $TOTAL section of the account:

updateEndEq('amzn_acct', Dates='2010-01-14')
[1] 'amzn_acct'
getEndEq('amzn_acct', Date='2010-01-14')
[1] 0

I am not sure if this has to do with the different TZ settings (my TZ is 'GMT').

Followups:

Date: 2011-02-11 17:58
Sender: Brian Peterson
Mark,A patch by Grigory Bykov here:https://r-forge.r-project.org/tracker/index.php?func=detail&aid=1271&group_id=316&atid=1271(patch 1271)has been applied in SVN r556Thanks again for the report.

Date: 2010-10-07 16:59
Sender: Brian Peterson
The vectorized/multi-currency version of updateAcct resolves this problem.Closed. Fixed. - Brian

Date: 2010-03-04 03:54
Sender: Peter Carl
Mark,Thanks for pointing this out. updateEndEq needs to be updated to handle irregular time series (as in updateAcct).

Submitted by: Joshua Ulrich
Assigned to: Joshua Ulrich
R-Forge link

This behavior was introduced in updateAcct.R in revisions between 1481-1495. The discussion on R-SIG-Finance includes a reproducible example.

Followups:

Date: 2014-12-01 16:03
Sender: Joshua Ulrich
Fixed in r1658 (with the solution suggested in my previous message).

Date: 2014-11-12 20:22
Sender: Joshua Ulrich
I'm leaning toward agreeing with Alexios' comment that paste('::',obsDates, sep='') should be paste(obsDates, sep=''), since we know that obsDates is only one observation when on='none'...

Submitted by: Alex Lee
Assigned to: Nobody
R-Forge link

At line 42-44 in updatePosPL.R

if(.parseISO8601(Dates)$first.time < as.POSIXct(first(index(prices))) || is.na(.parseISO8601(Dates)$first.time)){
    Dates<-index(prices[paste('/',.parseISO8601(Dates)$last.time,sep='')])
}

If .parseISO8601(Dates)$last.time returns NA, the [.xts will be called with '/NA', the execution stops.

I change this line to:

last.time <- .parseISO8601(Dates)$last.time
if (is.na(last.time)) last.time=NULL 

This should be correct.

Submitted by: Jonathan Owen
Assigned to: Joshua Ulrich
R-Forge link

Looking at the code for addTxn in blotter, I believe the logic for determining if TxnFees is a character or function (lines 100 - 108) should come before the logic for splitting transactions crossing 0 (lines 68-76). The transaction splitting logic splits TxnFees by TxnQty, which generates an error if TxnFees is a character or function. After the move, the splitting logic should also use the txnfees variable instead of TxnFees.

While making this change, I'd also propose changing the TxnFees function call to also pass the Symbol variable. A more general solution could be to pass all addTxn parameters to the function call, allowing even more flexibility.

if (is.function(TxnFees)) {
fcall <- match.call(expand.dots = FALSE)
fcall[[1]] = TxnFees
txnfees <- eval(fcall, parent.frame())
} else {
txnfees<- as.numeric(TxnFees)
}

Proposed updates are attached.

Followups:

Date: 2015-02-12 12:39
Sender: Joshua Ulrich
Fixed in r1681. The other proposed changes were not included in the commit. They should be a separate feature request. Also, the TxnFees call already passes Symbol (with TxnQty and TxnPrice), so I don't understand that portion of the request.

Algorithmic Transaction Cost Analysis

In Chapter 3 of "The Science of Algorithmic Trading and Portfolio Management," Robert Kissell, Ph.D covers a few models that may be worth implementing in blotter especially considering the extent to which blotter is used in the industry for post-trade analysis. Models eligible for inclusion are:

1. Implementation Shortfall

(first draft is complete and merged into 54_algotca)

Implementation Shortfall is a measure that represents the total cost of executing the investment idea. Implementation Shortfall is calculated as the difference between the paper return of a portfolio where all shares are assumed to have transacted at the manager's decision price and the actual return of the portfolio using actual transaction prices and shares executed.

is the total number of shares to trade. This should be made available as a parameter, with a reasonable default of simply what was executed, which assumes entire share allocation was transacted and the opportunity cost (below) will be zero.

is the manager's decision price. We should make this available as a param value to be specified by the user, with a reasonable default such as the Arrival Price.

is the price at the end of period n. This should be made available as a parameter as well (since it is the mid-price at the end of the trading horizon), with a suitable default perhaps being the price of the last executed transaction.

Actual portfolio return is the difference between the actual ending portfolio value and the value that was required to acquire the portfolio minus all fees.

represents the total number of shares in the portfolio

is the ending portfolio value

is the price paid to acquire the portfolio

Implementation Shortfall can be implemented in 4 ways:

1. Assuming Complete Execution, in which case trading horizon end price for the opportunity cost (relevant for the Paper Return) is irrelevant, and Paper Return will be invalid.

2. Using an Opportunity Cost (Perold 1988) component, where not all shares originally allocated for trading are finally executed. Opportunity Cost is the cost of not executing a portion of the originally allocated shares (S) for execution. This could be due to limit price constraints or a lack of liquidity. The formulation for Opportunity Cost is below:

The Implementation Shortfall formulation of Perold (1988) can be written as:

3. Expanded Implementation Shortfall (Wayne Wagner)

Wayne Wagner's implementation categorizes costs into delay, trading and opportunity related costs. Assuming is the decision price, is the price when trading begins (ideally mid-point of bid-ask spread on this timestamp, when first child order enters the market...aka Arrival Price) and is the price at the end of trading. Then the Expanded IS can be written as:

If you substitute this price into Perold's IS, then IS can be written as:

This formula can be re-written into their separate delay, trading and opportunity cost related components as follow:

where each term (excluding fees) reflects the delay, trading and opportunity cost components respectively.

4. Market Activity IS, which assumes the analyst is unaware of the manager's decision price. This should be our default method, and could be validated against the existence of a parameter for the decision price. For this method, we cut the first term off the Expanded IS implementation in order to assess only market activity IS:

Taking all the above into account, we should at a minimum have different methods for implementing Implementation Shortfall, with a parameter value for method=c("Complete", "Perold", "Wagner", "Market").

2. Trading Cost/Arrival Cost

See Step 1 in model # 9!

3. Trading Price Performance

Identical to Trading/Arrival Cost except underperformance against benchmark (ie. Arrival Price) is a negative value and outperformance is a positive value.

4. Benchmark Price Performance

Benchmark prices could be anything from Open (for arrival price) to Close (for index funds or otherwise using EOD prices for valuation), or Next Day Open/Close to ascertain permanent v temporary market impact. "Benchmark Price Performance Measures are the simplest of the TCA performance evaluation techniques."

5. VWAP Benchmark

Positive value indicates better performance, negative value indicates underperformance.

3 drawbacks: 1. The larger the order the closer the execution will be to VWAP, 2. where large block trades occur these could skew the results especially where executing algo is not able to participate as is normally the case for block trades, 3. not a useful comparison across stocks or different days for the same stock.

Execution quality as measured against VWAP should be considered in terms of the stock-specific spread. A 10bp underperformance for a stock with an average 20bp spread is worse than a 30bp underperformance for a stock with a 100bp average spread. Jeffrey Mazar and @braverock's obmodeling package may come in handy here.

3 Common variations (of which only the 3rd may need elaborating) include 1. Full-day VWAP, 2. Interval VWAP and 3. VWAP to EOD - typically used to compare performance of an execution which finished early, assuming broker had free reign to determine when to finish the order.

6. Participation Weighted Price (PWP) Benchmark

A variation of the VWAP benchmark in which the objective is to compare performance of the average execution with a likely realize price had the trader participated with a specific percentage of volume during the duration of the order.

Using an example, Kissell explains the PWP Benchmark assuming 100k shares are transacted, and specified participation rate is 20%, then PWP-20% benchmark price is computed as the volume weighted average price for the first 500k shares (100k/20%) that traded from the time the order arrived.

PWP Price = volume weighted price of the first PWP shares starting at the arrival time t_0

This benchmark suffers the same limitations as for VWAP in that the metric is not comparable between stocks or across days for the same stock. Also, the benchmark is subject to manipulation thanks to the potentially slowly dissipating temporary impact of an order, keeping the price inflated for larger buy orders or lower for larger sell orders to the benefit of the benchmark.

7. Relative Performance Measure

A percentile ranking of trading activity, this measure is preferable to the VWAP benchmark as it can be used to compare performance across stocks, days and different volatilities. The RPM measure can be mapped to a score:

8. Pre-Trade Benchmark

This benchmark is used to evaluate performance based on what was expected to occur. Of course any difference between what did occur and what was expected to occur could be beyond the control of the trader (e.g. price momentum, changing liquidity conditions etc). These are accounted for with the below z-score and market-adjusted cost analyses.

Pre-Trade Difference = Estimated Arrival Cost - Actual Arrival Cost

9. Index-Adjusted Performance Metric

Step 1: Compute Arrival Cost

Step 2: Compute Index Cost

Step 3: Compute Index Adjusted Cost

10. Z-Score Evaluation Metric

11. Market Cost Adjusted Z-Score

In addition to modelling TCA Kissell also covers models for comparing performance across algorithms by determining whether or not they are equivalent using non-parametric tests, including:

Paired Samples

• Sign Test
• Wilcoxon Signed Rank Test

Independent Samples

• Median Test
• Mann-Whitney U Test

Distribution Analysis

• Chi-Square
• Kolmogorov-Smirnov goodness of fit

There will almost certainly be opportunity to add Market Impact models to blotter but these will be opened as separate issues. We will add notes for Market Impact models in this issue, since there is a branch prefixed with the number of this issue (#54) which we would like to use for the duration of the GSoC 2019 project which addresses this issue.

Kissell ends off chapter 3 with the following statement which sums up the intention of this issue very nicely: "Transaction cost analysis remains an essential ingredient to achieve best execution. When administered properly, improved stock selection and reduced costs have proven to boost portfolio performance. As such, advancement of TCA models is an essential catalyst to further develop the algorithmic trading and market efficiency space."

Market Impact Models

Market impact is defined as the price impact on a financial instrument as the result of an order or a trade. The impact is broadly categorized into 2 components: Temporary Price Impact and Permanent Price Impact. Temporary impact is a function of order urgency. In order to trade a relatively larger amount of an instrument the trader will have to increase/decrease price for buy/sell orders. This a fundamental implication of Demand-Supply equilibrium. Permanent price impact would transpire where the act of a participant trading a particular stock in a particular direction leaks information to the market. The most obvious example of permanent price impact is with company announcements of earnings, or M&A announcements. For example, when AB InBev announced they were making a bid for SAB Miller the price in ZAR implied an almost 20% premium to the prevailing market price and the market adjusted immediately, accounting for the probability that the acquisition still needed regulatory approval. In these instances the information that led to the permanent price impact was an official announcement to the market by the company concerned. In the case of "market impact models" and permanent price impact the act of disclosing order volume on the order book by an informed trader could result in permanent price impact if the signal is picked up by other participants.

Modeling market impact is heavily biased by assumptions due to the fact that one's impact on the market cannot be observed in isolation to the impact without one's participation in the market. This is otherwise referred to as the Heisenberg uncertainty principle of Finance.

According to Kissell, there are 10 essential properties of a market impact model:

1. Impact costs increase with size. Relatively larger orders with more likely leak information, or indicate increasing demand or supply, resulting the shift in demand or supply curves.
2. Impact costs increase with volatility. Volatility serves as a proxy for price elasticity.
3. Impact costs depend on trading strategy. More aggressive strategies will incur larger impact due to immediacy of trading at a larger relative percentage of volume (POV). Trading at a slower rate will have less impact, but more market risk. This is the "trader's dilemma" which is managing the trade-off between trading as urgently as possible and incurring the resulting impact cost, or slowing down the trade rate to reduce the impact costs but increasing the likelihood that future trading will occur at a worse price than what is currently available.
4. Impact costs are dependent on market conditions and trading patterns. Stocks with larger average daily volumes (ADV) will incur relatively lower impact costs for the same volume than less liquid stocks.
5. Impact costs have a temporary and permanent component.
6. Impact costs are inversely dependent on market cap. This is a generalization but can be a proxy for liquidity and volatility. In other works stocks with bigger market caps will be larger index constituents and therefore have more active participants buying and selling interests in the company which should reduce the relative price volatility.
7. Trading costs increase with spreads. This is intuitive if you consider the cost for trading immediately is crossing a bid-ask spread. The larger the spread the larger the impact cost. Generally less liquid stocks will have larger spreads.

Kissell mentions other factors but these are most likely covered above already in some shape or form.

Kissell elaborates on 2 types of models in Chapter 4 - Market Impact Models. 1. Almgren and Chriss, 2. I-Star. The AC model is a path-dependent approach in that it makes cost estimates based on the actual sequence of trades. Kissell refers to this as a bottom-up approach. The I* model by Kissell & Malamut is a top-down approach. Kissell mentions calculating the cost of the entire order first and the allocating to trade periods based on the actual trade schedule. TODO: understand whether the I model can be used with public data alone, and as a pre-trade benchmark.*

We will cover summaries of both models below:

Almgren and Chriss

Submitted by: Mark Breman
Assigned to: Garrett See
R-Forge link

Hi,

I think the addDiv implementation is not correct in all cases as it has only one date param and no quantity param.

Here is a scenario where it's producing the wrong result:

############################ start script

try(rm('account.IB','portfolio.faber',pos=.blotter),silent=TRUE)
try(rm('PLW','USD',pos=.instrument),silent=TRUE)

initDate='2010-01-01'
currency('USD')
stock(primary_id='PLW', currency='USD', multiplier=1)
getSymbols('PLW', from=initDate)

portfolio='faber'
account='IB'

initPortf(name=portfolio, symbols='PLW', initDate=initDate)
initAcct(name=account, portfolios=portfolio, initDate=initDate, initEq=50000)

verbose=TRUE

addTxn(Portfolio=portfolio, Symbol='PLW', TxnDate='2010-02-04', TxnQty=12, TxnPrice=27.12, TxnFees=-1.0)
addTxn(Portfolio=portfolio, Symbol='PLW', TxnDate='2010-02-24', TxnQty=24, TxnPrice=27.12, TxnFees=0.0)

actually the cumdate='2010-02-11'

blotter:::addDiv(Portfolio=portfolio, Symbol='PLW', TxnDate='2010-02-26', DivPerShare=.0969)
addTxn(Portfolio=portfolio, Symbol='PLW', TxnDate='2010-05-20', TxnQty=-36, TxnPrice=28.14, TxnFees=-1.0)

updatePortf(Portfolio=portfolio, Dates='2010::')
updateAcct(name=account, Dates='2010::')
updateEndEq(Account=account)

getTxns(Portfolio=portfolio, Symbol='PLW', Date='2010::')

############################## end script

In the Net.Txn.Realized.PL results the #shares used is 36 (0.0969*36 = 3.4884), while the #shares should have been 12 because that was the amount of shares I held at the date before the ex-dividend date (I call this the cum-dividend date).

For cash dividends I think there should be two date params to addDiv: cumdate and paydate (or altenativly a #shares and a paydate).

Kind regards,

-Mark-

Followups:

Date: 2011-08-19 14:42
Sender: Brian Peterson
Garrett,Can you take a look at this? I know you've poked around at addDiv - Brian

Date: 2010-10-11 13:28
Sender: Mark Breman
You could also consider generating two transactions if someone adds a cash dividend:1) transaction for determining position held (period preceding exDivDate)2) payout of dividend transactionThe time in between those two transactions you have a sort-of 'unrealized dividend'.This way you can do with just one date column in the slot, and things can be reconstructed later (bit more difficult though). -Mark-

Date: 2010-10-11 13:14
Sender: Mark Breman
I think to use the position at the end of the exDivDate is not correct. Better would be to use the position held preceding the exDivDate, see http://en.wikipedia.org/wiki/Ex-dividend_date'We might need to add another column in addition to the Txn.Typecolumn to hold extra info to be able to reconstruct things later.'This is another reason why I declared a different slot for corporate actions in the pnl-package.The column headers for transactions (typically buy/sell transactions) just don't match with what's required for corporate actions, although they do have some headers in common. It's also confusing if you look at the transactions slot and the buy/sell transactions are mixed with dividends and possibly more corporate actions in the future.Kind regards,-Mark-

Date: 2010-10-11 12:34
Sender: Brian Peterson
My inclination for this is to have exDivDate and payDate. We'll use the position at the end of the exDivDate to calculate number of shares to pay the dividend on.We might need to add another column in addition to the Txn.Type column to hold extra info to be able to reconstruct things later.Thoughts? - Brian

Submitted by: Mark Breman
Assigned to: Brian Peterson
R-Forge link

Hi,

I'm trying to run my very basic blotter test with the latest build from R-Forge and updatePortf() throws me an 'subscript out of bounds exception':

try(rm('account.IB','portfolio.faber',pos=.blotter),silent=TRUE)
try(rm('PLW','USD',pos=.instrument),silent=TRUE)

initDate='2010-01-01'

currency('USD')
stock(primary_id='PLW', currency='USD', multiplier=1)
getSymbols('PLW', from=initDate)
[1] 'PLW'

portfolio='faber'
account='IB'

initPortf(name=portfolio, symbols='PLW', initDate=initDate)
[1] 'faber'
initAcct(name=account, portfolios=portfolio, initDate=initDate, initEq=50000)
[1] 'IB'

verbose=TRUE

addTxn(Portfolio=portfolio, Symbol='PLW', TxnDate='2010-02-04', TxnQty=12, TxnPrice=27.12, TxnFees=1.0)
[1] '2010-02-04 PLW 12 @ 27.12'

addTxn(Portfolio=portfolio, Symbol='PLW', TxnDate='2010-02-05', TxnQty=24, TxnPrice=27.12, TxnFees=0.0)
[1] '2010-02-05 PLW 24 @ 27.12'

addTxn(Portfolio=portfolio, Symbol='PLW', TxnDate='2010-02-24', TxnQty=-36, TxnPrice=26.84, TxnFees=1.0)
[1] '2010-02-24 PLW -36 @ 26.84'

updatePortf(Portfolio=portfolio, Dates='2010::')
Error in [.xts(x, ((NROW(x) - n + 1):NROW(x))) :
subscript out of bounds

I tried with different Dates values but it looks like everything I try throws an error.

I also tried one of the tests in the blotter package (test.txnFees() in runitUpdatePortf.R) but it fails with the same error on updatePortf():

test.txnFees()
[1] '2007-01-04 IBM 100 @ 96.5'
Error in [.xts(x, ((NROW(x) - n + 1):NROW(x))) :
subscript out of bounds

Here is my sessionInfo():

sessionInfo()
R version 2.11.1 (2010-05-31)
i486-pc-linux-gnu

locale:
[1] LC_CTYPE=en_US.utf8 LC_NUMERIC=C LC_TIME=en_US.utf8 LC_COLLATE=en_US.utf8 LC_MONETARY=C
[6] LC_MESSAGES=en_US.utf8 LC_PAPER=en_US.utf8 LC_NAME=C LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.utf8 LC_IDENTIFICATION=C

attached base packages:
[1] stats graphics grDevices utils datasets methods base

other attached packages:
[1] blotter_0.7 FinancialInstrument_0.0.2 quantmod_0.3-14 TTR_0.20-2 Defaults_1.1-1
[6] PerformanceAnalytics_1.0.3 pnl_0.1 RUnit_0.4.25 xts_0.7-5 zoo_1.6-4

loaded via a namespace (and not attached):
[1] grid_2.11.1 lattice_0.19-11 tools_2.11.1

Kind regards,

-Mark-

Followups:

Date: 2010-10-07 15:14
Sender: Brian Peterson
Thanks, Closing. FWIW, I don't consider this 'bothering' me or a waste of my time. Bugs are bugs, and need to be fixed, so reports that are reproducible are always welcome.Regards, - Brian

Date: 2010-10-07 13:29
Sender: Mark Breman
Hi Brian,After installing the latest xts from R-Forge the problem is gone (although the version of xts is still 0.7-6).Sorry for bothering you.-Mark-

Date: 2010-10-07 12:43
Sender: Brian Peterson
I am unable to replicate your problem.Your example, now attached to this bug report as a .R file here, 'works for me', The 'amzn_test', 'longtrend' and 'turtles' demos on blotter all produce the expected results, and the quantstrat demos all work...The RUnit tests mostly work too. The one that checks Account fails, as there is not a $TOTAL slot in account anymore (it has been renamed to $summary to match the portfolio $summary slot). I'll update that test today.The only obvious difference in your sessionInfo(0 is the xts version. Perhaps try updating xts? blotter, quantstrat, etc. rather routinely flush out edge case bugs in xts, we seem to be pusing it harder than most users, especially on tick data. There have been some recent changes to subsetting.Regards, - Brian

Date: 2010-10-07 11:39
Sender: Brian Peterson
I'll try to replicate this this morning. - Brian

The chart.ME function plots the MAE/MFE per symbol. In some cases, it is useful to analyze the MAE and MFE statistics at an aggregated portfolio level. I propose an option to chart the MAE/MFE for all symbols in a given portfolio, e.g. Symbol = "all".

I see a few options for achieving this:

1. Loop over the symbols in the portfolio and rbind the output of perTradeStats.
2. Modify perTradeStats to accept an argument Symbol = "all" to return the trade statistics aggregated at the portfolio level.

Submitted by: Mark Breman
Assigned to: Peter Carl
R-Forge link

If I run the following test scenario I think I get some incorrect values in the Trading.PL and Unrealized.PL columns after calling updatePortf():

rm(list=ls(envir=.blotter),envir=.blotter)
rm('portfolio.myportfolio', 'account.myaccount', pos=.blotter)
Warning messages:
1: In rm('portfolio.myportfolio', 'account.myaccount', pos = .blotter) :
object 'portfolio.myportfolio' not found
2: In rm('portfolio.myportfolio', 'account.myaccount', pos = .blotter) :
object 'account.myaccount' not found

Sys.setenv(TZ='GMT')

prices = c(23.07, 23.05, 23.05, 23.09, 23.28, 23.36)
MYSTOCK = xts(prices, as.POSIXct(strptime(paste('2009-01-', seq(1:length(prices)), sep=''),'%Y-%m-%d')))
colnames(MYSTOCK) = 'Close'

initDate='2009-01-01'
currency('USD')
stock(primary_id='MYSTOCK', currency='USD', multiplier=1)

portfolio='myportf'
account='myacct'

initPortf(name=portfolio, symbols='MYSTOCK', initDate=initDate)
[1] 'myportf'
initAcct(name=account, portfolios=portfolio, initDate=initDate, initEq=1000)
[1] 'myacct'

verbose=TRUE

addTxn(Portfolio=portfolio, Symbol='MYSTOCK', TxnDate='2009-01-02', TxnQty=36, TxnPrice=23.05, TxnFees=-1.0)
[1] '2009-01-02 MYSTOCK 36 @ 23.05'
addTxn(Portfolio=portfolio, Symbol='MYSTOCK', TxnDate='2009-01-04', TxnQty=-36, TxnPrice=23.09, TxnFees=-1.0)
[1] '2009-01-04 MYSTOCK -36 @ 23.09'

updatePortf(Portfolio=portfolio, Dates='2009::')
[1] 'myportf'
getPortfolio(portfolio)
$MYSTOCK
$MYSTOCK$txn
Txn.Qty Txn.Price Txn.Fees Txn.Value Txn.Avg.Cost Pos.Qty Pos.Avg.Cost Realized.PL Con.Mult
2009-01-01 0 0.00 0 0.00 0.00000 0 0.00000 0.00 0
2009-01-02 36 23.05 -1 830.80 23.07778 36 23.07778 0.00 1
2009-01-04 -36 23.09 -1 -830.24 23.06222 0 0.00000 -0.56 1

$MYSTOCK$posPL
Pos.Qty Con.Mult Ccy.Mult Pos.Value Txn.Value Txn.Fees Realized.PL Unrealized.PL Trading.PL
2009-01-01 0 1 1 0.0 0.00 0 0.00 0 0.00
2009-01-02 36 1 1 829.8 830.80 -1 0.00 -1 -1.00
2009-01-03 36 1 1 829.8 0.00 0 0.00 0 0.00
2009-01-04 0 1 1 0.0 -830.24 -1 -0.56 1 0.44
2009-01-05 0 1 1 0.0 0.00 0 0.00 0 0.00
2009-01-06 0 1 1 0.0 0.00 0 0.00 0 0.00

attr(,'class')
[1] 'blotter_portfolio' 'portfolio'

2-1.44
[1] 0.56
36*0.04
[1] 1.44
MYSTOCK
Close
2009-01-01 23.07
2009-01-02 23.05
2009-01-03 23.05
2009-01-04 23.09
2009-01-05 23.28
2009-01-06 23.36

Look at the columns for 2009-01-04 in the MYSTOCK$posPL section for myportf: Unrealized.PL is 1 and Trading.PL is 0.44. I expected 1.44 in Trading.PL ((23.09_36)-(23.05_36)), and 0.44 in Unrealized.PL (i.e. ((23.09_36)-(23.05_36) -1.0 for transaction fees). Is this correct?

Followups:

Date: 2010-03-27 04:22
Sender: Mark Breman
Hi Peter,Now I must admit that I am confused. I thought I had a clear understanding of how blotter defined the different terms but I still see values in the result of the test case that do not match with this understanding (i.e. Realized.PL on 2009-01-04 of the posPL table should be -0.56 instead of 1.08 and Unrealized.PL should be a total (including fees) cumulative summation for an open position)Is there a list somewhere with definitions/formulas for these terms? and if not, would it be a good idea if we made such a list to clarify things? Such a list would also be very nice for the documentation I think. If I buy 36@

Date: 2010-03-27 00:11
Sender: Peter Carl
Mark,I've made some changes to the structure of the txn and posPL table that I think will make the calculations easier to understand. I've also simplified how the transaction fees are applied.> Sys.setenv(TZ='GMT')>> prices = c(23.07, 23.05, 23.06, 23.09, 23.28, 23.36)> MYSTOCK = xts(prices, as.POSIXct(strptime(paste('2009-01-', seq(1:length(prices)), sep=''),'%Y-%m-%d')))> colnames(MYSTOCK) = 'Close'>> initDate='2009-01-01'> currency('USD')> stock(primary_id='MYSTOCK', currency='USD', multiplier=1)>> portfolio='myportf'> account='myacct'>> initPortf(name=portfolio, symbols='MYSTOCK', initDate=initDate)[1] 'myportf'> initAcct(name=account, portfolios=portfolio, initDate=initDate, initEq=1000)[1] 'myacct'>> verbose=TRUE>> addTxn(Portfolio=portfolio, Symbol='MYSTOCK', TxnDate='2009-01-02', TxnQty=36, TxnPrice=23.05, TxnFees=-1.0)[1] '2009-01-02 MYSTOCK 36 @ 23.05'> addTxn(Portfolio=portfolio, Symbol='MYSTOCK', TxnDate='2009-01-04', TxnQty=-36, TxnPrice=23.09, TxnFees=-1.0)[1] '2009-01-04 MYSTOCK -36 @ 23.09'>> updatePortf(Portfolio=portfolio, Dates='2009::')[1] 'myportf'> getPortfolio(Portfolio=portfolio)$MYSTOCK$MYSTOCK$txn Txn.Qty Txn.Price Txn.Value Txn.Avg.Cost Pos.Qty Pos.Avg.Cost2009-01-01 0 0.00 0.00 0.00 0 0.002009-01-02 36 23.05 829.80 23.05 36 23.052009-01-04 -36 23.09 -831.24 23.09 0 0.00 Gross.Txn.Realized.PL Txn.Fees Net.Txn.Realized.PL Con.Mult2009-01-01 0.00 0 0.00 02009-01-02 0.00 -1 -1.00 12009-01-04 1.44 -1 0.44 1$MYSTOCK$posPL Pos.Qty Con.Mult Ccy.Mult Pos.Value Txn.Value Realized.PL2009-01-01 0 1 1 0.00 0.00 0.002009-01-02 36 1 1 829.80 829.80 0.002009-01-03 36 1 1 830.16 0.00 0.002009-01-04 0 1 1 0.00 -831.24 1.082009-01-05 0 1 1 0.00 0.00 0.002009-01-06 0 1 1 0.00 0.00 0.00 Unrealized.PL Gross.Trading.PL Txn.Fees Net.Trading.PL2009-01-01 0.00 0.00 0 0.002009-01-02 0.00 0.00 -1 -1.002009-01-03 0.36 0.36 0 0.362009-01-04 0.00 1.08 -1 0.082009-01-05 0.00 0.00 0 0.002009-01-06 0.00 0.00 0 0.00attr(,'class')[1] 'blotter_portfolio' 'portfolio'This new structure has also been applied to the Account level in updateAccount.The Txn.Realized.PL in the txn table is the aggregate P&L for the transaction. In the posPL table, that P&L is distributed through time. You can see that the cumulative Net.Trading.PL is equal to the sum of the Txn.Realized.PL in the txn table.I hope that's clear, but don't hesitate to ask if you have any questions.pcc

Date: 2010-03-26 13:34
Sender: Brian Peterson
We've made some changes in things in svn(2010-03-25), and all the totals/sums are now correct, but it seems like there's still some work to do to get to a 13-column-like report.Testing and feedback would be appreciated.Regards, - Brian

Date: 2010-03-25 08:59
Sender: Mark Breman
Hi Peter,I have build blotter with the latest sources and rerun the test. The results are better, but now the Unrealized.PL column in the PosPL section shows incorrect values for dates 2009-01-02 and 2009-01-03:> Sys.setenv(TZ='GMT')> > prices = c(23.07, 23.05, 23.06, 23.09, 23.28, 23.36)> MYSTOCK = xts(prices, as.POSIXct(strptime(paste('2009-01-', seq(1:length(prices)), sep=''),'%Y-%m-%d')))> colnames(MYSTOCK) = 'Close'> > initDate='2009-01-01'> currency('USD')> stock(primary_id='MYSTOCK', currency='USD', multiplier=1)> > portfolio='myportf'> account='myacct'> > initPortf(name=portfolio, symbols='MYSTOCK', initDate=initDate)[1] 'myportf'> initAcct(name=account, portfolios=portfolio, initDate=initDate, initEq=1000)[1] 'myacct'> > verbose=TRUE> > addTxn(Portfolio=portfolio, Symbol='MYSTOCK', TxnDate='2009-01-02', TxnQty=36, TxnPrice=23.05, TxnFees=-1.0)[1] '2009-01-02 MYSTOCK 36 @ 23.05'> #addDiv(Portfolio=portfolio, Symbol='MYSTOCK', PayDate='2009-01-03', Amount=0.0937)> addTxn(Portfolio=portfolio, Symbol='MYSTOCK', TxnDate='2009-01-04', TxnQty=-36, TxnPrice=23.09, TxnFees=-1.0)[1] '2009-01-04 MYSTOCK -36 @ 23.09'> > #addTxn(Portfolio=portfolio, Symbol='MYSTOCK', TxnDate='2009-01-05', TxnQty=36, TxnPrice=23.28, TxnFees=-1.0)> #addDiv(Portfolio=portfolio, Symbol='MYSTOCK', PayDate='2009-01-04', Amount=0.0937)> #addTxn(Portfolio=portfolio, Symbol='MYSTOCK', TxnDate='2009-01-06', TxnQty=-36, TxnPrice=23.36, TxnFees=-1.0)> > updatePortf(Portfolio=portfolio, Dates='2009::')[1] 'myportf'> getPortfolio(portfolio)$MYSTOCK$MYSTOCK$txn Txn.Type Txn.Qty Txn.Price Txn.Fees Txn.Value Txn.Avg.Cost Pos.Qty Pos.Avg.Cost Realized.PL Con.Mult2009-01-01 0 0 0.00 0 0.00 0.00000 0 0.00000 0.00 02009-01-02 0 36 23.05 -1 830.80 23.07778 36 23.07778 0.00 12009-01-04 0 -36 23.09 -1 -830.24 23.06222 0 0.00000 -0.56 1$MYSTOCK$posPL Pos.Qty Con.Mult Ccy.Mult Pos.Value Txn.Value Txn.Fees Realized.PL Unrealized.PL Trading.PL2009-01-01 0 1 1 0.00 0.00 0 0.00 0.0000 0.002009-01-02 36 1 1 829.80 830.80 -1 0.00 806.7222 -1.002009-01-03 36 1 1 830.16 0.00 0 0.00 807.0822 0.362009-01-04 0 1 1 0.00 -830.24 -1 -0.56 0.0000 0.082009-01-05 0 1 1 0.00 0.00 0 0.00 0.0000 0.002009-01-06 0 1 1 0.00 0.00 0 0.00 0.0000 0.00I also think the Trading.PL column has incorrect values now. I see Trading.PL as 'daily P&L on a position resulting from price changes of the asset but WITHOUT fees'. If this is the correct definition than the values in the column are wrong.I have attached the test case (see updatePortf1.R).HTH,-Mark-

Date: 2010-03-25 04:35
Sender: Peter Carl
Mark, Take a look at Brian's recent commits. I haven't had a chance to test them with your case yet, but let me know if problems persist. pcc

Date: 2010-03-04 05:07
Sender: Peter Carl
I've narrowed this down to an error in how transaction fees are being applied.> addTxn(Portfolio=portfolio, Symbol='MYSTOCK', TxnDate='2009-01-02', TxnQty=36, TxnPrice=23.05,+ TxnFees=0)[1] '2009-01-02 MYSTOCK 36 @ 23.05'> addTxn(Portfolio=portfolio, Symbol='MYSTOCK', TxnDate='2009-01-04', TxnQty=-36, TxnPrice=23.09, TxnFees=0)[1] '2009-01-04 MYSTOCK -36 @ 23.09'> updatePortf(Portfolio=portfolio, Dates='2009::')[1] 'myportf'> getPortfolio(portfolio)$MYSTOCK$MYSTOCK$txn Txn.Qty Txn.Price Txn.Fees Txn.Value Txn.Avg.Cost Pos.Qty2009-01-01 0 0.00 0 0.00 0.00 02009-01-02 36 23.05 0 829.80 23.05 362009-01-04 -36 23.09 0 -831.24 23.09 0 Pos.Avg.Cost Realized.PL Con.Mult2009-01-01 0.00 0.00 02009-01-02 23.05 0.00 12009-01-04 0.00 1.44 1$MYSTOCK$posPL Pos.Qty Con.Mult Ccy.Mult Pos.Value Txn.Value Txn.Fees Realized.PL2009-01-01 0 1 1 0.0 0.00 0 0.002009-01-02 36 1 1 829.8 829.80 0 0.002009-01-03 36 1 1 829.8 0.00 0 0.002009-01-04 0 1 1 0.0 -831.24 0 1.442009-01-05 0 1 1 0.0 0.00 0 0.002009-01-06 0 1 1 0.0 0.00 0 0.00 Unrealized.PL Trading.PL2009-01-01 0.000000e+00 0.002009-01-02 0.000000e+00 0.002009-01-03 0.000000e+00 0.002009-01-04 -2.842171e-14 1.442009-01-05 0.000000e+00 0.002009-01-06 0.000000e+00 0.00attr(,'class')[1] 'blotter_portfolio' 'portfolio'I'll work on this more tomorrow if I can.

Date: 2010-03-04 04:07
Sender: Peter Carl
Thanks for the very good test case. I've confirmed the bug: you shouldn't have an unrealized gain on 01-04 since you don't have a position.

Submitted by: Mark Breman
Assigned to: Nobody
R-Forge link

It was last seen in rev: 308, but seems to be gone now.

Is it dropped for something else?

Thank you,

-Mark-

Followups:

Date: 2010-10-08 11:46
Sender: Brian Peterson
moving to feature requests. I'll try to make some progress on this today along with other things. - Brian

Date: 2010-10-08 06:10
Sender: Mark Breman
Maybe corporate actions resulting from M&A needs to be supported in the future, but I'm not sure.What happens for instance if a company listing is ended due to an acquisition?All open positions have to be closed at the takeover price?Regards,-Mark-

Date: 2010-10-07 19:58
Sender: Brian Peterson
The method you suggest could certainly work. It would be easy enough to put a text->numeric lookup into some internal function in blotter that would keep the numeric codes.So, just to think through some other types:cash additions and withdrawals go in at the account level, no transaction needed.interest paid probably belongs at the account level as well, for simplicitysplits don't realize any P&L, they are changes to positions (and multipliers as well with most derivatives). the simplest way to implement them is to simply adjust the position, leaving the position average cost the sameWith options, we need ASSIGN and EXERCISE transactions, which have some fun characteristics that I'd have to go dig out of my design notes from the last portfolio accounting system we did.Were there any other types you wanted to think about?Regards, - Brian

Date: 2010-10-07 17:50
Sender: Mark Breman
Ok I will do some testing with it tomorrow.Why can't Txn.Type not be numeric (like buy/sell = 0, cash dividend = 1 etc.)? What I did in pnl is create an entire new slot for corporate actions ($corp) as I anticipated supporting a lot more types of corporate actions in the future (stock dividends, stock splits etc.) and mixing it all in one slot would clutter the code/data beyond comprehension.Regards,-Mark-

Date: 2010-10-07 17:33
Sender: Brian Peterson
It's not exposed in the NAMESPACE, but it's still there.You can access it with blotter:::addDiv()Honestly, it needs more testing, and we should probably add a Txn.Type column to the $txn slot. I'm a little befuddled about how to do that 'safely', as we currently have an all-numeric data store, and adding Txn.Type would make it a character storage mechanism, which I'm afraid would cause bugs in other places.Let us know if you have any feedback or need modifications to this function, as it isn't publicly exposed, it should be easy to change without breaking working code for other people.Regards, - Brian

updateAcct("amzn_acct",Dates="2010-01-14")
Error in match.arg(on, c("years", "quarters", "months", "weeks", "days", :
'arg' should be one of “years”, “quarters”, “months”, “weeks”, “days”, “hours”, “minutes”, “seconds”, “milliseconds”, “microseconds”, “ms”, “us”

I believe the fix is to remove the Dates argument from both updateAcct and updatePortf.

txnsim - When buiding the transactions data frame from our sampled durations, we correctly apply an adjustment to our end date where the specific end date is not in our market data object. Typically, this will be when a sampled end timestamp falls on a weekend or holiday. We need to apply a similar adjustment for our sampled start timestamps.

Submitted by: Mark Breman
Assigned to: Peter Carl
R-Forge link

The updateAcct() function throws a error when called without the Dates parameter.

To reproduce:

initDate='2010-01-01'
currency('USD')
stock(primary_id='PLW', currency='USD', multiplier=1)
getSymbols('PLW', from='2010-01-01')
[1] 'PLW'

portfolio='faber'
account='IB'

initPortf(name=portfolio, symbols='PLW', initDate=initDate)
[1] 'faber'
initAcct(name=account, portfolios=portfolio, initDate=initDate, initEq=50000)
[1] 'IB'

verbose=TRUE

addTxn(Portfolio=portfolio, Symbol='PLW', TxnDate='2010-02-04', TxnQty=12, TxnPrice=27.12, TxnFees=1.0)
[1] '2010-02-04 PLW 12 @ 27.12'
addTxn(Portfolio=portfolio, Symbol='PLW', TxnDate='2010-02-05', TxnQty=24, TxnPrice=27.12, TxnFees=0.0)
[1] '2010-02-05 PLW 24 @ 27.12'

updatePortf(Portfolio=portfolio, Dates='2010::')
[1] 'faber'
updateAcct(name=account)
Error: object 'CurrentSpan' not found

I think the error is caused by the CurrentSpan variable which is defined inside the for-loop, but is also tested just outside the loop:

.....
for (d in 1:length(Dates)) {
CurrentSpan=NULL
for (pname in Portfolios) {
Portfolio = getPortfolio(pname)
CurrentDate = Dates[d]
PrevDate = time(Portfolio[[1]]$posPL[Portfolio[[1]]$posPL[CurrentDate,
which.i = TRUE] - 1])
if (length(PrevDate) == 0)
next()
PrevDateWidth = xts:::.parseISO8601(PrevDate)
PrevDateLast = PrevDateWidth$last.time
CurrentSpan = paste(PrevDateLast, CurrentDate, sep = '::')
rows = calcPortfSummary(Portfolio, CurrentSpan)
rows = na.omit(rows)
Account[[pname]] = rbind(Account[[pname]], rows)
}
if (is.null(CurrentSpan)) <<========= tested but not defined
next()
....

Regards,

-Mark-

Followups:

Date: 2010-10-07 12:44
Sender: Brian Peterson
Closed. Resolved. Fixed.

Date: 2010-08-17 19:24
Sender: Brian Peterson
Resolved, fixed in svn version 374vectorized version of updatePosPL doesn't need to use the loop anymore

Date: 2010-08-16 08:02
Sender: Ben McCann
BTW, replacing CurrentSpan with CurrentDate will fix 3 bugs: 830, 831, 839.

Date: 2010-08-16 07:59
Sender: Ben McCann
Actually, setting 'CurrentSpan = NULL' is probably not what we want to do because I just tested it out and it creates a ton of duplicate rows in the account. E.g. using CurrentSpan when happens is it will update the account for Jun 22-Jun 23, then Jun 23-Jun 24, etc. so there's a bunch of double counting. Better would be to just replace all occurrences of CurrentSpan with CurrentDate. Then in the previous example you would only update for Jun 22, Jun 23, etc. and won't produce the duplicate rows.

Date: 2010-08-16 07:51
Sender: Ben McCann
Nevermind my previous comment, Dates is being initialized correctly.Just need to add 'CurrentSpan = NULL' on the line right before 'for(pname in Portfolios){' and this issue will be fixed.

Date: 2010-08-06 07:27
Sender: Ben McCann
There are a couple problems here. The main problem is that when Dates is initialized it is set to a single date: if(is.null(Dates)) #[[1]] here is the first instrument in the portfolio Dates = time(Portfolio[[1]]$posPL )Then we skip the first for loop iteration because there is no previous date so CurrentSpan never gets defined:if (length(PrevDate)==0) next()And then when we check to see if it is null we fail because it was never defined:if(is.null(CurrentSpan)) next()It believe there are two fixes needed. The first is that I do not think Dates is being initialized properly. The second is that we should initialize CurrentSpan to NULL outside of the for loop so that it will not fail the is.null check.

Date: 2010-08-02 02:05
Sender: Ben McCann
I encountered this issue as well.

Date: 2010-03-02 13:51
Sender: Peter Carl
Thanks for the posting, Mark.

A useful feature in the Monte Carlo style function txnsim() is the computation of empirical p-values from the output. The output from simulations of portfolio PL (ie. mcsim) tends to tell us less information about standard sample statistics than metrics like Max Drawdown for instance, and viewing this p-value from the output of mcsim() could be useful to the analyst.

See, North, Curtis, and Sham mention in "A Note on the Calculation of Empirical P Values from Monte Carlo Procedures" - https://www.ncbi.nlm.nih.gov/pmc/articles/PMC379178/

perTradeStats correctly segments the start, end, and quantity of round turn trades using three mechanisms: flat.to.flat, flat.to.reduced, and increased.to.reduced (also available as acfifo).

There are problems with some of the statistics for flat.to.reduced, and increased.to.reduced.

Net.Trading.PL

Net.Trading.PL is calculated using
trade[,"Pos.Value"]-Pos.Cost.Basis

from the PosPL slot of the portfolio for the time of the end of the round turn trade.

I believe this is correct for flat.to.flat, but incorrect for the other round turn trade segmentation mechanisms.

The correct method for flat.to.reduced, and increased.to.reduced should use the Period.Realized.PL column.

In the simplest case, Period.Realized.PL will be for a single transaction, and that transaction size will match the transaction size of the round turn trade. In this case, the entire Period.Realized.PL may be applied to the round turn trade.

If the quantity of the round turn trade is smaller than the quantity of the closing transaction, then Period.Realized.PL will need to be pro-rated to reflect the round turn trade quantity.

It should be noted that Period.Realized.PL could be used for flat.to.flat as well, but it would then need to be the cumsum of all Period.Realized.PL's during the course of the flat.to.flat round turn. It may make sense to use Period.Realized.PL for all methods, just for consistency.

MAE and MFE

MAE and MFE are currently specified as the min or max of:

Pos.PL <- trade[,"Pos.Value"]-Pos.Cost.Basis

during the duration of the round turn trade. These (and their percent and tick equivalents) need to be pro-rated for position.

Submitted by: Xin Xiao
Assigned to: Brian Peterson
R-Forge link

now i can change portfolio by adding new trade.

i want a function that accept target position and figure out the correspondent trades, and then change the internal data structure, according to the difference between current position and target position.

in this way, i can use the result of a portfolio optimizer directly.

and how to take part in the development of blotter, since i find the package is very useful.

thanks

Followups:

Date: 2013-09-18 22:13
Sender: Rashid Rasul
Has any more progress been made on this? I am also looking for similar functionality, i.e. to calculate target positions and create transactions to move from the current position to the target positions.I'm writing this now when I came across this note. I will attach once completed, but it is a little more complicated than I expected as the positions are in different currencies and I need to work out the correct GBP equivalent to go long/short in a basket of other pairs and then calculate P&L and carry in GBP! Regards, - Rashid

Date: 2010-03-29 11:25
Sender: Brian Peterson
Thanks for your note. I agree that this would be useful functionality to have available for a portfolio manager. One way to participate in the development of blotter would be to write the function you are requesting, and attach the file to this feature request.I think that the function is relatively straightforward.First, one would getPos() for a given time, and then calculate the difference between current and desired position. If the position crosses through zero, you would want to split the transaction into two transactions, one to get to zero, and one to go to the new position.These could then be added via addTxn() or addTxns()I do wonder whether this functionality is better modeled as orders in a trading/strategy system than directly modeled as transactions, but any implementation will be a good starting point. Perhaps there will be one function that will take weights and generate target changes in position based on current positions, and another that generates the transactions from that.As for broader involvement, the best thing to do is just get involved. write test cases, report issues, write code to enhance functionality and add it as feature requests, etc. We actively desire participation of other R finance community users and developers.Regards, - Brian

Submitted by: Mark Breman
Assigned to: Nobody
R-Forge link

I get very strange errors on addTxn() when I create a portfolio with the name 'myportfolio':

Sys.setenv(TZ='GMT')

prices = c(23.07, 23.05, 23.05, 23.09, 23.28, 23.36)
MYSTOCK = xts(prices, as.POSIXct(strptime(paste('2009-01-', seq(1:length(prices)), sep=''),'%Y-%m-%d')))
colnames(MYSTOCK) = 'Adjusted'

initDate='2009-01-01'
currency('USD')
stock(primary_id='MYSTOCK', currency='USD', multiplier=1)

portfolio='myportfolio'
account='myacct'

initPortf(name=portfolio, symbols='MYSTOCK', initDate=initDate)
[1] 'myportfolio'
initAcct(name=account, portfolios=portfolio, initDate=initDate, initEq=1000)
[1] 'myacct'

verbose=TRUE

addTxn(Portfolio=portfolio, Symbol='MYSTOCK', TxnDate='2009-01-02', TxnQty=36, TxnPrice=23.05, TxnFees=1.0)
Error in get(pname, envir = .blotter) : object 'myportfolio' not found
Error in getPortfolio(Portfolio) :
Portfolio myportfolio not found, use initPortf() to create a new portfolio
ls()
[1] 'account' 'initDate' 'MYSTOCK' 'portfolio' 'prices' 'verbose'
ls(.blotter)
[1] 'account.myacct' 'portfolio.myportfolio'

The portfolio is created (see above) but when I call addTxn() it gives me an error?!

If I call the portfolio 'myportf' instead of 'myportfolio' addTxn() works fine:

rm('portfolio.myportfolio', 'account.myaccount', pos=.blotter)
Warning messages:
1: In rm('portfolio.myportfolio', 'account.myaccount', pos = .blotter) :
object 'portfolio.myportfolio' not found
2: In rm('portfolio.myportfolio', 'account.myaccount', pos = .blotter) :
object 'account.myaccount' not found

Sys.setenv(TZ='GMT')

prices = c(23.07, 23.05, 23.05, 23.09, 23.28, 23.36)
MYSTOCK = xts(prices, as.POSIXct(strptime(paste('2009-01-', seq(1:length(prices)), sep=''),'%Y-%m-%d')))
colnames(MYSTOCK) = 'Adjusted'

initDate='2009-01-01'
currency('USD')
stock(primary_id='MYSTOCK', currency='USD', multiplier=1)

portfolio='myportf'
account='myacct'

initPortf(name=portfolio, symbols='MYSTOCK', initDate=initDate)
[1] 'myportf'
initAcct(name=account, portfolios=portfolio, initDate=initDate, initEq=1000)
[1] 'myacct'

verbose=TRUE

addTxn(Portfolio=portfolio, Symbol='MYSTOCK', TxnDate='2009-01-02', TxnQty=36, TxnPrice=23.05, TxnFees=1.0)
[1] '2009-01-02 MYSTOCK 36 @ 23.05'

Followups:

Date: 2010-03-03 15:23
Sender: Brian Peterson
RESOLVED FIXEDWe store portfolio and account names with 'portfolio.' and 'account.'as prefixes.I was previously searching for the string 'portfolio' or 'account' in the passed name to try to figure out how to get the data out of the environment.I'm now searching for:'portfolio.'and'account.'which should allow 'portfolio' and 'account' to be used by the user in user-supplied names.There are still edge cases, but the incidence should go down dramatically.Regards, - Brian

Ideally, we would like to be able to stack the MAE/MFE series of gains/losses and have that reconcile to the total equity curve of the portfolio.

• for flat.to.flat, MAE/MFE correspond directly to the equity curve, so this is not an issue there.

• for increased.to.reduced/acfifo/flat.to.reduced, MAE/MFE are not currently coherent/additive to the portfolio equity curve

Transaction-Based Reconciliation

Doing this from transactions would require deeper changes to blotter

1. we'd need to store the market price series used to mark the portfolio with updatePortf

2. we'd need to generate MAE/MFE from the transactions, not from the posPL 'equity curve' data (for increased.to.reduced and flat.to.reduced)

One potential advantage of constructing MAE/MFE from a combinationof transactions and market data is that we could choose whether to do things in a portfolio average cost manner, as is done throughout blotter today, or todeviate from that and use the cost of the transaction only for the purposes of constructing MAE/MFE. Arguably supporting per-transaction cost basis makes sense in this context.

Equity Curve Based Reconciliation

Making MAE/MFE reconcile with the portfolio equity curve could also be done for increased.to.reduced/acfifo can be done from the equity curve with a time varying pro-rata.

1. Construct a time varying pro-rata based on the ratio of trade size used for the round turn Closing.Txn.Qty to Pos.Qty

prorata <- trades$Closing.Txn.Qty[i]/Pos.Qty

2. per period gross P&L is multiplied by the prorata

Cum.PL <- (cumsum(trade[,'Period.Realized.PL'] + trade[,'Period.Unrealized.PL']) + trade[,'Txn.Fees'])*prorata

Note that this isn't exactly right for Txn.Fees as this round turn should only participate in the fees for its opening and closing trades like so:

fees <- as.numeric(trade[1,'Txn.Fees'] * prorata) + as.numeric(trade[n,'Txn.Fees'])

A time varying pro-rata could be done from data we already have, without changing the trade subsetting, and without adding the prices used to mark the book to the portfolio object.

Some decisions would need to be made for either a transaction based approach or a equity curve based approach for flat.to.reduced round turn definition. Since by definition all round turns start at flat, we would need to decide how to pro-rate the early parts of a round turn before reaching maximum position, as well as how to treat things as the overall position got smaller (and potentially larger again). increased.to.reduced/acfifo are far more intuitive in this regard.

Submitted by: Mark Breman
Assigned to: Brian Peterson
R-Forge link

Hi,

I installed the latest build of blotter and it looks like the calcRealizedPL() is missing:

########################## start script

try(rm('account.IB','portfolio.faber',pos=.blotter),silent=TRUE)
try(rm('PLW','USD',pos=.instrument),silent=TRUE)

initDate='2010-01-01'
currency('USD')
stock(primary_id='PLW', currency='USD', multiplier=1)
getSymbols('PLW', from=initDate)

portfolio='faber'
account='IB'

initPortf(name=portfolio, symbols='PLW', initDate=initDate)
initAcct(name=account, portfolios=portfolio, initDate=initDate, initEq=50000)

verbose=TRUE

addTxn(Portfolio=portfolio, Symbol='PLW', TxnDate='2010-02-04', TxnQty=12, TxnPrice=27.12, TxnFees=-1.0)

############################# end script

If I run the script I get:

...
addTxn(Portfolio=portfolio, Symbol='PLW', TxnDate='2010-02-04', TxnQty=12, TxnPrice=27.12, TxnFees=-1.0)
Error in addTxn(Portfolio = portfolio, Symbol = 'PLW', TxnDate = '2010-02-04', :
could not find function 'calcRealizedPL'

Regards,

-Mark-

Followups:

Date: 2010-10-11 12:26
Sender: Brian Peterson
Closing this. It was a miss on my part between r411 and r412-413. R-Forge built r411 before I noticed the mistake.Regards, - Brian

Date: 2010-10-11 08:42
Sender: Brian Peterson
This should have been resolved in r412, Sorry.

Submitted by: Mark Breman
Assigned to: Peter Carl
R-Forge link

There is no windows install package available for blotter 0.3.

Installation by repacking the source package as .zip fails under R for Windows 2.10.0 fails:

install.packages('blotter.zip',repos=NULL)
library(blotter)
Error in library(blotter) : 'blotter' is not a valid installed package

Followups:

Date: 2009-11-30 16:57
Sender: Peter Carl
Ah, good. Thanks very much for following up. I'll mark this as Fixed then. pcc

Date: 2009-11-30 15:29
Sender: Mark Breman
Looks like you fixed the problem Peter Carl.Thank you.

Date: 2009-11-24 04:13
Sender: Peter Carl
Tests were failing perhaps because of changes in 2.10, but also because they weren't updated after making changes to initPortf. I've checked in changes and R CMD check now passes (although with warnings). Once the build process runs again, recheck to see if the packaging works.Installing from source should still work.

Submitted by: Wolfgang Wu
Assigned to: Peter Carl
R-Forge link

When you have two stocks in different currencies then updateAcct is not calculating the EndEquity and other columns in the $TOTAL and $portfolio.* tables correctly.

The following example is buying one security and selling another in different currencies. At the next day only the currency has changed and the EndEquity should be different now. But it isn't. I think the problem is that for the calculation in the function getBySymbol.R it uses the line:

tmp_col = Portfolio[[symbol]]$posPL[Dates,Attribute,drop=FALSE]

Shouldn't it better do this?

tmp_col = Portfolio[[symbol]]$posPL.USD[Dates,Attribute,drop=FALSE]

The example code to run is here:

Example

library(quantmod);
library(PerformanceAnalytics);
library(FinancialInstrument)
library(blotter)

currency('USD', multiplier=1);
currency('JPY', multiplier=1);

stock('StockA', currency='USD',multiplier=1)
stock('StockB', currency='JPY',multiplier=1)

getSymbols(Symbols=c('EUR/JPY') ,src='oanda', from='2008-12-28', to='2009-01-01');
USDJPY <- EURJPY;
JPYUSD <- 1/USDJPY;
colnames(USDJPY) <- 'Close'
colnames(JPYUSD) <- 'Close'

Set the stocks to 100

StockA <- (USDJPY_0+100); #worth 100 USD
StockB <- (USDJPY_0+100); #worth 100 JPY
StartDate <- time(first(USDJPY));

initPortf(name='fPortfolio', symbols = c('StockA', 'StockB'), initPosQty = 0, initDate=StartDate);
initAcct(name='fAccount',portfolios=c('fPortfolio'), initDate=StartDate, initEq=1000);
updatePortf(Portfolio='fPortfolio', Dates = StartDate);

Buy 1 StockA and Sell 127.461 StockB which should have the same value on day 1

currentDate = time(StockA[2]);
addTxn('fPortfolio', Symbol='StockA', TxnDate=currentDate, TxnPrice=as.numeric(StockA[2]), TxnQty=1, TxnFees=0, verbose=TRUE);
addTxn('fPortfolio', Symbol='StockB', TxnDate=currentDate, TxnPrice=as.numeric(StockB[2]), TxnQty=(-1*as.numeric(USDJPY[2])), TxnFees=0, verbose=TRUE);
updatePortf(Portfolio='fPortfolio', Dates = currentDate);
updateAcct('fAccount', currentDate);
updateEndEq(Account='fAccount', Dates = currentDate);

Now value the portfolio on the next day

currentDate = time(StockA[3]);
updatePortf(Portfolio='fPortfolio', Dates = currentDate);
updateAcct('fAccount', currentDate);
updateEndEq(Account='fAccount', Dates = currentDate);
getEndEq(Account='fAccount', Date=currentDate)

fPortfolio = getPortfolio('fPortfolio');
fAccount = getAccount('fAccount');

Followups:

Date: 2010-10-07 16:56
Sender: Brian Peterson
Closed. Fixed.Account is now completely multi-currency aware.Regards, - Brian

Submitted by: Mark Breman
Assigned to: Brian Peterson
R-Forge link

Hi,

Ok, here is my test case again:

################ Start test script

try(rm('account.IB','portfolio.faber',pos=.blotter),silent=TRUE)
try(rm('PLW','USD',pos=.instrument),silent=TRUE)

initDate='2010-01-01'
currency('USD')
stock(primary_id='PLW', currency='USD', multiplier=1)
getSymbols('PLW', from=initDate)

portfolio='faber'
account='IB'

initPortf(name=portfolio, symbols='PLW', initDate=initDate)
initAcct(name=account, portfolios=portfolio, initDate=initDate, initEq=50000)

verbose=TRUE

addTxn(Portfolio=portfolio, Symbol='PLW', TxnDate='2010-02-04', TxnQty=12, TxnPrice=27.12, TxnFees=1.0)
addTxn(Portfolio=portfolio, Symbol='PLW', TxnDate='2010-02-05', TxnQty=24, TxnPrice=27.12, TxnFees=0.0)
addTxn(Portfolio=portfolio, Symbol='PLW', TxnDate='2010-05-20', TxnQty=-36, TxnPrice=28.14, TxnFees=1.0)

updatePortf(Portfolio=portfolio, Dates='2010::')
updateAcct(name=account, Dates='2010::')
updateEndEq(Account=account)

############## End test script

In this script I buy 36 stocks at 27.12 and sell them some time later at 28.14. I pay a total of 2.0 in commission.
After running the script I would expect somewhere in the portfolio or the transaction overviews to find a Net. value of 34.72 as:

((28.14-27.12)*36)-2
[1] 34.72

But I only find 37.72 in the Net.Realized.PL and Period.Realized.PL columns.
Am I missing something here or are the results not correct?

Kind regards,

-Mark-

Followups:

Date: 2010-10-07 17:12
Sender: Brian Peterson
Mark,No, your example is fine with the TxnFees put in as negative numbers.Keep submitting questions/problems. It is incredibly helpful to all of us to have more eyes on things, making sure that we're doing the right thing. I hope that we are close at this point to blotter being 'stable', so that we can focus almost all of our energy in quantstrat, and building strategies.Closing this bug report.Regards, - Brian

Date: 2010-10-07 17:07
Sender: Mark Breman
Hi Brian,Ahrg, yes now I see it! Now I also understand the mystirious 1.0 in the Period.Realized.PL column. Ofcourse when it's -1.0 it's correct, as the commission is paid (realized) inmediatly.Thanks for pointing it out to me.'Let me know if I need to spend some more time on your example to 'fix' it'Does this mean you already spotted more errors in my example? If so please let me know because I think it should be correct by now.Thanks for your patience with me.-Mark-

Date: 2010-10-07 15:12
Sender: Brian Peterson
Per the documentation, TxnFees should usually be represented as negative numbers. Positive fees are rebates.If I reverse the sign of your TxnFees, I see:getTxns('faber',Symbol='PLW') Txn.Qty Txn.Price Txn.Fees Txn.Value Txn.Avg.Cost Net.Txn.Realized.PL2010-01-01 0 0.00 0 0.00 0.00 0.002010-02-04 12 27.12 -1 325.44 27.12 -1.002010-02-05 24 27.12 0 650.88 27.12 0.002010-02-24 -36 28.14 -1 -1013.04 28.14 35.72Which sums to the 34.72 you expect.Let me know if I need to spend some more time on your example to 'fix' it. - Brian

Submitted by: Xie Wenfei
Assigned to: Nobody
R-Forge link

I am using latest versions of all libraries.
I met a problem. I should be a bug.
Net.Trading.PL should be calcuated by (2021.285 - 2024.951) * 8468 = -31043.688.
But in perTradeStats Net.Trading.PL result 112488.912

tail(book$holyGrail1$SINAPLATEINFO)
Order.Qty Order.Price Order.Type Order.Side Order.Threshold Order.Status Order.StatusTime Prefer Order.Set Txn.Fees
2014-05-21 '8468' '2024.951' 'market' 'long' NA 'closed' '2014-05-22 00:00:00' 'Close' NA '0'
2014-05-22 'all' '2021.285' 'market' 'long' NA 'closed' '2014-05-23 00:00:00' '' NA '0'

tail(perTradeStats(q.strategy, includeOpenTrade = FALSE))
Start End Init.Pos Max.Pos Num.Txns Max.Notional.Cost Net.Trading.PL MAE MFE Pct.Net.Trading.PL
41 2014-05-22 2014-05-23 8468 8468 2 17116241 112488.912 0.00 112488.912 0.0065720569

My R code is attached:

add.signal(q.strategy, name = 'sigCrossover', arguments = list(columns = c('Close',
'TrendMA'), relationship = 'gte'), label = 'Cl.gt.TrendMA')
add.signal(q.strategy, name = 'sigCrossover', arguments = list(columns = c('Close',
'TrendMA'), relationship = 'lt'), label = 'Cl.lt.TrendMA')

add.signal(q.strategy,name='sigThreshold',
arguments = list(column='WinLosePredict',
relationship='eq',
threshold=1,
cross=FALSE),
label='WinLosePredict.LongWin')

add.rule(q.strategy, name = 'ruleSignal', arguments = list(sigcol = 'Cl.lt.TrendMA',
sigval = TRUE, orderqty = 'all', ordertype = 'market', orderside = 'long', osFUN = 'osMaxPos',
pricemethod = 'market', replace=TRUE), type = 'exit', path.dep = TRUE, label = 'ExitLONG')

add.rule(q.strategy, name = 'ruleSignal', arguments = list(sigcol = 'WinLosePredict.LongWin',
sigval = TRUE, orderqty = pos, ordertype = 'market', prefer='Close',orderside = 'long', osFUN = 'osMaxPos',
pricemethod = 'market', replace=FALSE), type = 'enter', path.dep = TRUE, label = 'EnterLONG')

out <- applyStrategy(strategy = q.strategy, portfolios = q.strategy)

updatePortf(q.strategy)

updateAcct(q.strategy)

updateEndEq(q.strategy)

Followups:

Date: 2015-05-21 23:48
Sender: Xie Wenfei
Dear Brian, Thanks so much for your quick response. I really appreciate your brilliant work in these R projects. Well, I will clarify it with demo example. I commented 'long' add.rule lines and only left 'short' add.rule lines in luxor.1.strategy.basic. You can run luxor.1.strategy.basic as attached. From those diagrams generated, we can find that Issue 1:In perTradeStatsNet.Trading.PL -390 -250 170 -650 40I suppose we can calculate perTradeStats from getOrderBook. How Net.Trading.PL is calculated from getOrderBook? Could you explain how -250 is calculated for example?Issue 2: In the diagrams of both 'chart.Posn(portfolio.st, GBPUSD)' and 'charts.PerformanceSummary(ret1$total, geometric=FALSE, wealth.index=TRUE)', we can see that Cumulative Return and Drawback Daily return is so poor. I doubt the tradeStats calculation of short trades. Finally, we really appreciate your team for these excellent projects to explore market secret. Looking forward to your reply.

Date: 2015-05-21 09:13
Sender: Brian Peterson
Your example is not reproducible.At the very least, pleas attach the data which generates what you believe to be a problem.Better yet, produce an actual minimal reproducible example.There are a number of ways to do this...Probably the simplest aside from attaching the data used to generate your test would be to call extractTxns on the portfolio which you believe contains an error, and extract that along with an explanation of where you think the error occurs.Even better would be a small crafted set of trades which replicates what you believe to be the error.Otherwise, I can't evaluate your hand waving, and can't help you.

Submitted by: Mark Breman
Assigned to: Brian Peterson
R-Forge link

Are the tests for blotter not running on the build system?

The tests look very out of date. For instance addTxn.R and txnFee.R.

Are the tests not updated after modifications to the blotter code?

Regards,

-Mark-

Followups:

Date: 2010-08-17 21:02
Sender: Brian Peterson
Ben McCann updated all the tests to use RUnit. All pass now. CLOSED FIXED.

Date: 2010-08-06 01:07
Sender: Ben McCann
I provided a rewrite of the tests, so this should be able to be closed after the patch is committed:https://r-forge.r-project.org/tracker/index.php?func=detail&aid=1019&group_id=316&atid=1271

Date: 2010-03-02 16:26
Sender: Peter Carl
Thanks, Mark - I appreciate the pointer... I've flagged this as a bug and it certainly deserves the cleanup you pointed out originally.

Date: 2010-03-02 15:54
Sender: Mark Breman
Hi Peter Carl,You are right that the first call to addTxn() results in just warnings, but a subsequent call to addTxn() results in error(s):> rm('portfolio.default', pos=.blotter)> Sys.setenv(TZ='America/Chicago') # as the data set got save with this TZ> options('width'=78) # to tie down the print() statement width> verbose <- FALSE> data(IBM) # data included in package> symbols <- c('IBM')> > suppressMessages(library(blotter)) # to suppress the TZ noise from xts> suppressMessages(library(quantmod))> # Initialize a portfolio object 'p'> # Creating portfolio:> p = initPortf(symbols=symbols)> > ## Trades must be made in date order.> # Make a couple of trades in IBM> p = addTxn(p, 'IBM', '2007-01-03', 50, 96.5, 0.05_50)[1] '2007-01-03 IBM 50 @ 96.5'Warning messages:1: In getInstrument(Symbol) : Instrument IBM not found, please create it first.2: In addTxn(p, 'IBM', '2007-01-03', 50, 96.5, 0.05 * 50) : Instrument IBM not found, using contract multiplier of 1> p = addTxn(p, 'IBM', '2007-01-04', -50, 97.1, 0.05_50)Error in get(paste('portfolio', pname, sep = '.'), envir = .blotter) : object 'portfolio.list(txn = c(0, 50, 0, 96.5, 0, 0, 0, 4825, 0, 96.5, 0, 50, 0, 96.5, 0, 0, 0, 1), posPL = c(0, 1, 1, 0, 0, 0, 0, 0, 0))' not found>HTH,-Mark-

Date: 2010-03-02 15:42
Sender: Peter Carl
Thanks for the clarification. What you're seeing are warnings rather than errors - we don't strictly require the instruments to be established, but we make some important assumptions if they aren't. So those warnings are to make sure that the user understands that the instrument wasn't found and as a result we're assuming it has a contract multiplier of 1. But it doesn't throw an error and it processes the trade normally. The risk of not establishing the instrument would be that the P&L and other calculations would be incorrect (the default assumption happens to work fine for stocks).I'll ask Brian to review the test suite to see what we need to clean up.

Date: 2010-03-02 14:45
Sender: Mark Breman
For instance if I try to run the addTxn() test in the tests directory of blotter this is what happens:> Sys.setenv(TZ='America/Chicago') # as the data set got save with this TZ> options('width'=78) # to tie down the print() statement width> verbose <- FALSE> data(IBM) # data included in package> symbols <- c('IBM')> > suppressMessages(library(blotter)) # to suppress the TZ noise from xts> suppressMessages(library(quantmod))> # Initialize a portfolio object 'p'> # Creating portfolio:> p = initPortf(symbols=symbols)> > ## Trades must be made in date order.> # Make a couple of trades in IBM> p = addTxn(p, 'IBM', '2007-01-03', 50, 96.5, 0.05*50)[1] '2007-01-03 IBM 50 @ 96.5'Warning messages:1: In getInstrument(Symbol) : Instrument IBM not found, please create it first.2: In addTxn(p, 'IBM', '2007-01-03', 50, 96.5, 0.05 * 50) : Instrument IBM not found, using contract multiplier of 1> That the test fails with the shown error seems logical to me as blotter requires the creation of an instrument before using it in a transaction, which is not done in the test right?Regards,-Mark-

Date: 2010-03-02 13:56
Sender: Peter Carl
Thanks for posting this Mark. We'll take a look, but would you mind adding a bit more detail so that we can address what you are specifically seeing?

perTradeStats computes the trade statistics for a given symbol. We should also provide an option for trade statistics aggregated at the portfolio level (related to #42). One way is to modify perTradeStats to loop over all the symbols in the portfolio.

portf <- .getPortfolio(Portfolio)
for(Symbol in names(portf$symbols){
    # do trade stats stuff
}

Another option is to do this with a separate function such as perTradeStats.portfolio. Then perTradeStats could call perTradeStats.portfolio if Symbol = "all" is passed in as an argument.

For example, something like

perTradeStats.portfolio <- function(Portfolio, Symbol, includeOpenTrade = TRUE, tradeDef = "flat.to.flat", ...){
  symbols <- names(.getPortfolio(Portfolio)$symbols)
  pts.list <- lapply(symbols, function(x) perTradeStats(Portfolio, Symbol = x, includeOpenTrade = includeOpenTrade, tradeDef = tradeDef, ... = ...))
  # either return as a list or rbind and return a single data.frame (e.g. do.call(rbind, pts.list))
  return(pts.list)
}

The R-Index by Schimack (2014) intends to penalize QRPs (Questionable Research Practices) and is a "doping test for science." We should be able to use the output from txnsim() to come up with our own version of the R-Index and in so doing add to the list of diagnostic tools available in blotter/quantstrat for determining luck vs skill or overfitting.

http://www.r-index.org/uploads/3/5/6/7/3567479/introduction_to_the_r-index__14-12-01.pdf

where

and

Statistical Power (or Success Rate) in Schimmack is defined as the LR probability of finding a significant result. From a large enough 'n' in txnsim() we could rely on the pvalues output of our strategy within the sampled distribution.

For Effect Size (or Inflation) we could rely on the effsize R package. This will require more research.

Below are more references that could prove useful, from the blog replicationindex.wordpress.com

• https://replicationindex.wordpress.com/2016/01/31/a-revised-introduction-to-the-r-index/
• https://replicationindex.wordpress.com/2015/04/01/meta-analysis-of-observed-power-comparison-of-estimation-methods/
• https://replicationindex.wordpress.com/2015/03/24/an-introduction-to-observed-power-based-on-yuan-and-maxwell-2005/
• https://replicationindex.wordpress.com/2016/01/14/on-the-definition-of-statistical-power/
• https://stackoverflow.blog/2017/10/17/power-calculations-p-values-ab-testing-stack-overflow/
• http://rpsychologist.com/d3/NHST/
• http://meera.snre.umich.edu/power-analysis-statistical-significance-effect-size