Statsmodel has two-sided HP filter implemented: http://www.statsmodels.org/devel/generated/statsmodels.tsa.filters.hp_filter.hpfilter.html

Implement method for this. One-sided filter comes via applying filter to expanding sample.

Update 20190521
Possible fix in SumVariables(): added 5 lines that, when no entities selected, get entities for which all given variables exists. Seems to work.

This fix needs to be applied to other methods as well that have the same problem! Similar can be used for others if we construct a separate loop for variables at start. This entity selection needs to be wrapped as separate static method and apply it to all.

Init post
Default entity selection
if entities == None:
entities = list(np.unique(self.df.columns.get_level_values(1).values))
is not working (at least) for following methods:

SumVariables()
DetrendVariables()
MRADecomposition()
SumVariables()
ReduceVariableDimension()
Example case: MRADecomposition() is performed on entity DEU:

variables = ['Credit_def_ld1','ResidentialPrices_ld1','StockPrices_def_ld1']
do.MRADecomposition(variables,entities=['DEU'],levels = 6,expanding=False)

Later we try to perform SumVariables() on default entities:
variables = {'L45':['StockPrices_def_ld1_wl4','StockPrices_def_ld1_wl5','Credit_def_ld1_wl4','Credit_def_ld1_wl5','ResidentialPrices_ld1_wl4','ResidentialPrices_ld1_wl5']}
do.SumVariables(variables)

For entities that were not included in MRADecomposition, SumVariables() results in variable with values 0 throughtout, although it should throw an error.

In ExpandingSampleCalc(), in the first counter loop line

resultframefull = pd.merge(resultframefull, crtresultframe, left_index = True, right_index = True, how = 'left')

causes a warning as columns in resultframefull are not a multi-index for some weird reason. Functionality seems to be correct despite the warning.

Before merge resultframefull is an empty frame with just index but no columns. After the merge we currently force the columns to multi-index so in rest of loops warning disappears, but this should be somehow corrected such that there will be no warning in the first loop either.

Use this example to reproduce: how to get multi-index column to gg when there is no data in it?
df = do.ReturnDf()
gg = pd.DataFrame(index=df.index)

Add similar checks as in SumVariables to other methods as well.
Related to #1.

It is better to throw an explicit error when something is not working as user inputs it.

It seems that endogenous dummy variables are treated as integers. However, in vulnerability horizon treatment we need to introduce NaNs and this causes these columns to become floats. We could probably use integers with the new nullable integer type (https://pandas.pydata.org/pandas-docs/stable/user_guide/integer_na.html) but I guess in this context it is enough to use just floats. This, however, should be properly defined; that is, all numerical column on MTSDataModel should be defined as floats.

It does not really make sense that SumVariables() differs from other methods in how it takes in variables. The idea was to let the dict key to designate the name of the resulting new variable, and the variables itself are passed in as list in dict value element. To match with others, it makes more sense to pass in two lists variables and entities, and a mandatory 3rd input variable to designate the name, similar to ReduceVariableDimension.

What this does is perhibit summing multiple variables combindations at one go, but the others work in this way as well.

Once this is done, do #1

Need to check how well HP filtering works!

Data model needs method to reduce sample to given vulnerability horizon.

Questions:

• where to put this methods? Under class MTSModel? Do not really belong to that particular class, predictive modelling part are not just data handling anymore.