jankrepl / deepdow Goto Github PK
View Code? Open in Web Editor NEWPortfolio optimization with deep learning.
Home Page: https://deepdow.readthedocs.io
License: Apache License 2.0
Portfolio optimization with deep learning.
Home Page: https://deepdow.readthedocs.io
License: Apache License 2.0
And make sure or operations inside propagate it. Problems with torch.eye inside of run
Would be cool to create an allocator that just learns a single weight per each asset. To make sure all the weights sum up to one can only consider positive weights and then divide them by the sum.
For determinstic benchmarks metrics can be just copied from previous step rather than recomputed
Apart from generating iid sequences one can do a lot of different things. Just need to pay attention to using too many external dependencies.
Additionally Markowitz benchmarks
Unfortunately installing Zipline is a nightmare, only supports Python 3.5.
So maybe investigate open-source backtesters
These days I have been using your deepdow package to do some experiments about portfolio optimization. Thanks for your great work!
But I have a problem here. I want to add the turnover rate constraint into the optimization. To achieve that, every round the network run over the training optimization, I have to keep the weights that has been calculated so that in the next round I can assure the new weight calculated won’t be too far from the previous one.
So I want to ask you that if there is some way I can save the weight each time the network calculated during the training process?
raw_to_Xy appears to handle regular gaps in data (e.g. weekend days) but cannot handle irregular gaps such as holidays.
When fed trading data similar to the example at https://deepdow.readthedocs.io/en/latest/source/data_loading.html but covering an entire trading year it get out of sync on every holiday. E.g. a Monday that would typically trade but does not on a holiday such as Jan 20, 2020.
The result is that the assertion assert timestamps[0] == raw_df.index[lookback]
fails.
This, and likely other data formatting issues, causes an error when executing history = run.launch(30)
which is RuntimeError: mat1 and mat2 shapes cannot be multiplied
Some ideas below
visualize
model would use some helper function that inputs network
and dataloader
and returns DataFrame
weight_image
Currently, deepdow.utils.raw_to_Xy
does a lot of magic inside and outputs only the bare minimum for training:
X
timestamps
y
asset_names
indicators
Would be nice to have some debug mode that returns more.
Currently one cannot just do -SomeLoss()
. Of course it could be hacked by doing (-1) * SomeLoss()
. We want to implement the first syntax via __sub__
.
Complete preprocessor
Implement some dunders...
Would be nice to show, how deepdow
is able to directly learn or have a network predictor of any input variables of the deepdow.layers.NumericalMarkowitz
It might be a good idea to use real data (i.e. yfinance
), however one needs to be careful about the example running too long (both CI and readthedocs need to run it)
I'm able to get the out-of-the box examples to execute successfully (getting_started and iid) when using the generated data but when using differenty toy datasets I get an AssertionError in the cvxcpy module.
/opt/conda/lib/python3.6/site-packages/cvxpy/cvxcore/python/canonInterface.py in nonzero_csc_matrix(A)
162 # this function returns (rows, cols) corresponding to nonzero entries in
163 # A; an entry that is explicitly set to zero is treated as nonzero
--> 164 assert not np.isnan(A.data).any()
165
166 # scipy drops rows, cols with explicit zeros; use nan as a sentinel
AssertionError:
Steps to reproduce:
from deepdow.nn import BachelierNet
n_channels = X.shape[1]
lookback = X.shape[2]
n_assets = X.shape[3]
max_weight = 0.5
hidden_size = 32
network = BachelierNet(n_channels, n_assets, hidden_size=hidden_size, max_weight=max_weight)
print(network)
Same error occurs even if reducing channels to 1, increasing number of samples, keeping lookback, gap, horizion small (5,0,1).
Using torch.distributions
Probably not possible since we would encounter zero gradients
In the SoftmaxAllocator
the nonnegativity constraint w >= 0
is missing.
4 options:
Currently, not clear how to contribute to the project.
It would be nice to have a benchmark that is just some predefined portfolio. One would construct it by passing all the weights.
Design new network with a fixed sized size - major speedups probably
Show that sliding the window by one does not change the allocation much because of the underlying 1D convolutions.
The same idea as SoftmaxAllocator
but with additional quite useful features for portfolio optimization
Inspired by https://locuslab.github.io/2019-10-28-cvxpylayers/ however implementing via cvxpylayers
(as written in the blog) is not the most efficient way to do it.
Currently, we implement one "Explainable" algorithm in deepdow.explain.gradient_wrt_input
. The problem is that we do not restrict the values the input can have. One solution would be to implement some projection/clipping logic that takes place after each optimizer step and thus forces the values to be in a given range.
Would be nice for the network to do translations along the time dimensions (possibly independently for each assets) https://pytorch.org/docs/stable/nn.functional.html#grid-sample
Scaling potentially also useful
Papers
Can be used for mlflow logging
With the convex optimization it could happen that the solver does not find solution and then it results in weights not summing up to one (sometimes drastically different).
Possible solutions
Currently all metrics input and output torch.tensors...limiting!?
Or at least give a user an option to do it
Rather than reinventing the wheel one could just use torchvision
transforms https://pytorch.org/docs/stable/torchvision/transforms.html
Compose
(already recreated in deepdow
)RandomApply
- apply all with some probabilityRandomChoice
- apply exactly one but at randomRandomOrder
- apply all but in random order1Dwarping
- Affine would be a special case, one could in theory have any increasing function (derivative > 0)RandomAffine
- scaling and translation along the y axis (lookback) could be a brilliant augmentation for deepdow
tensorsRandomHorizontalFlip
- flipping the time flow, probably super confusing if one wants to pic up mean reversionNormalize
- a must together with some helper function that computes means, stds in the training set. However, it still assumes that the time series is stationary.RandomErasing
- (similar to the current Dropout
however it is contiguous regions)Additionally, torchvision might be also helpful in other tasks (see #39)
The clear downside is introducing yet another dependency. Additionally, one might argue that it is better to go all the way and use imgaug
, albumentation
,...
Other nonvision augmentations:
Assert Python version via python_requires
. Should correspond to what is tested (.travis.yml
)
Currently, there are a lot of typos, poorly written or unfinished sentences, etc...
Write a torch.utils.data.Dataset
subclass such that it randomly subsamples both samples and assets. Seems like cvxpylayers does not scale well
A declarative, efficient, and flexible JavaScript library for building user interfaces.
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
An Open Source Machine Learning Framework for Everyone
The Web framework for perfectionists with deadlines.
A PHP framework for web artisans
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
Some thing interesting about web. New door for the world.
A server is a program made to process requests and deliver data to clients.
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Some thing interesting about visualization, use data art
Some thing interesting about game, make everyone happy.
We are working to build community through open source technology. NB: members must have two-factor auth.
Open source projects and samples from Microsoft.
Google ❤️ Open Source for everyone.
Alibaba Open Source for everyone
Data-Driven Documents codes.
China tencent open source team.