samre12 / deep-trading-agent Goto Github PK

Deep Reinforcement Learning based Trading Agent for Bitcoin

License: MIT License

Python 98.43% Shell 1.57%

deep-reinforcement-learning deep-q-network bitcoin deep-sense trading artificial-intelligence agent recurrent-neural-networks

deep-trading-agent's Introduction

Deep Trading Agent

Deep Reinforcement Learning based Trading Agent for Bitcoin using DeepSense Network for Q function approximation.

For complete details of the dataset, preprocessing, network architecture and implementation, refer to the Wiki of this repository.

Requirements

Python 2.7
Tensorflow
Pandas (for pre-processing Bitcoin Price Series)
tqdm (for displaying progress of training)

To setup a ubuntu virtual machine with all the dependencies to run the code, refer to assets/vm.

Run with Docker

Pull the prebuilt docker image directly from docker hub and run it as

docker pull samre12/deep-trading-agent:latest
docker run -p 6006:6006 -it samre12/deep-trading-agent:latest

Build the docker image locally by executing the command and the run the image as

docker build -t deep-trading-agent .
docker run -p 6006:6006 -it deep-trading-agent

This will setup the repository for training the agent and

mount the current directory into /deep-trading-agent in the container
during image build, the latest transactions history from the exchange is pulled and sampled to create per-minute scale dataset of Bitcoin prices. This dataset is placed at /deep-trading-agent/data/btc.csv
to initiate training of the agent, specify suitable parameters in a config file (an example config file is provided at /deep-trading-agent/code/config/config.cfg) and run the code using /deep-trading-agent/code/main.py
training supports logging and monitoring through Tensorboard
vim and screen are installed in the container to edit the configuration files and run tensorboard
bind port 6006 of container to 6006 of host machine to monitor training using Tensorboard

Support

Please give a ⭐ to this repository to support the project 😄.

ToDo

Docker Support

Add Docker support for a fast and easy start with the project

Improve Model performance

Extract highest and lowest prices and the volume of Bitcoin traded within a given time interval in the Preprocessor
Use closing, highest, lowest prices and the volume traded as input channels to the model (remove features calculated just using closing prices)
Normalize the price tensors using the price of the previous time step
For the complete state representation, input the remaining number of trades to the model
Use separate diff price blocks to calculate the unrealized PnL
Use exponentially decayed weighted unrealized PnL as a reward function to incorporate current state of investment and stabilize the learning of the agent

Trading Model

is inspired by Deep Q-Trading where they solve a simplified trading problem for a single asset.
For each trading unit, only one of the three actions: neutral(1), long(2) and short(3) are allowed and a reward is obtained depending upon the current position of agent. Deep Q-Learning agent is trained to maximize the total accumulated rewards.
Current Deep Q-Trading model is modified by using the Deep Sense architecture for Q function approximation.

Dataset

Per minute Bitcoin series is obtained by modifying the procedure mentioned in this repository. Transactions in the Coinbase exchange are sampled to generate the Bitcoin price series.
Refer to assets/dataset to download the dataset.

Preprocessing

Basic Preprocessing
Completely ignore missing values and remove them from the dataset and accumulate blocks of continuous values using the timestamps of the prices.
All the accumulated blocks with number of timestamps lesser than the combined history length of the state and horizon of the agent are then filtered out since they cannot be used for training of the agent.
In the current implementation, past 3 hours (180 minutes) of per minute Bitcoin prices are used to generate the representation of the current state of the agent.
With the existing dataset (at the time of writing), following are the logs generated while preprocessing the dataset:

INFO:root:Number of blocks of continuous prices found are 58863
INFO:root:Number of usable blocks obtained from the dataset are 887
INFO:root:Number of distinct episodes for the current configuration are 558471

Advanced Preprocessing
Process missing values and concatenate smaller blocks to increase the sizes of continuous price blocks.
Standard technique in literature to fill the missing values in a way that does not much affect the performance of the model is using exponential filling with no decay.
(To be implemented)

Implementation

Tensorflow "1.1.0" version is used for the implementation of the Deep Sense network.

Deep Sense

Implementation is adapted from this Github repository with a few simplifications in the network architecture to incorporate learning over a single time series of the Bitcoin data.

Deep Q Trading

Implementation and preprocessing is inspired from this Medium post. The actual implementation of the Deep Q Network is adapted from DQN-tensorflow.

deep-trading-agent's People

Contributors

Stargazers

Watchers

Forkers

easyrider gcwk shubhampachori12110095 josealb phejimlin canerdogan mohala562 trungtv suqi mcgnus txizzle htq310542 caserzer laranea rohansaphal97 zebratiger4 thesoulbender marlesson amin1softco meelement parrondo aiot-tech bohaohan luciferkonn starsmall-xiaoqq peltierchip ianmadlenya researchase sealter philipshurpik donghaima matrix10 fbracun snsie vitiusconsulting aparnadhinakaran xavier66 ronghuazhao joseph-zhong thegu noke8868 qianchaoliu falconzyx leedaga linpingchuan amoliu chenyang1999 xcbat allensmile andrewgeorge whnj badman1111 blockchaincapital siylipson xlz209 baifengbai sunjiajia-19930606 bhysyq datafu youngjunseo hocgabri chinshou testmana2 openself gavinzjchao stefaj lyonleelpl kgkesoulis yonghe1979 hbcbh1999 wesseldr paracats mcskyn charleswwang jiangilhui0330 mave5 imserkan arcaine qomposer ramananm yisun518 fomhko sahasas lightvault jbdatascience loponly github59987 gameweb mathematixy wayne980 edwarts tony32769 jingfei05 darkstar4401 maba18 wknight1 afshar1 kahthong elulue linglonggh

deep-trading-agent's Issues

ModuleNotFoundError: No module named 'ConfigParser'

Hi! Tried to run your agent but found an error - "ModuleNotFoundError: No module named 'ConfigParser'" - seems that this file was not committed
ah, okay - it's python 2.7

any plan to add SMA, RSI, ATR indices?

Do you have any plan to add SMI, RSI, ATR indices to the input state?

ERROR: INVALID_TIMESTEP

Traceback (most recent call last):
File "/home/arunavo/PycharmProjects/samre12Trading/trader/main.py", line 47, in
main("/home/arunavo/PycharmProjects/samre12Trading/trader/agent/config/config.cfg")
File "/home/arunavo/PycharmProjects/samre12Trading/trader/main.py", line 37, in main
agent.train()
File "/home/arunavo/PycharmProjects/samre12Trading/trader/agent/model/train.py", line 69, in train
self.replay_memory.set_history(state, supplementary)
File "/home/arunavo/PycharmProjects/samre12Trading/trader/agent/model/replay_memory.py", line 107, in set_history
self.add(state[length - 1], 0.0, 0, False, supp)
File "/home/arunavo/PycharmProjects/samre12Trading/trader/agent/model/replay_memory.py", line 53, in add
raise ValueError(INVALID_TIMESTEP)
ValueError: INVALID_TIMESTEP
ERROR:deep_trading_agent:Invalid Timestep with shapes (4,), (1,)

Process finished with exit code 1

/// In file : train.py
def train(self):
............

    state, supplementary = self.env.reset()
    self.history.set_history(state)
    self.replay_memory.set_history(state, supplementary)

#env.reset()
//returns -->> return self.historical_prices[self.current - self.history_length:self.current], np.array([1.0])

from file: cryptoenv.py
//its an np.array.shape --> 1
whereas supplementary needs 27 according to the config file

P.S: for branch adaptive_normalization

not an issue - fyi dockerfile + ssh daemon

https://gist.github.com/johndpope/b8a1f334e1d1769e73391ad6270bbff9

for easier config editing.

Nan in summary histogram for: q_network/avg_q_summary/q/2

Hi,
I tried using another datasets, one with a forex pair and another one with the BTC/USD pair but it gives me that error for every dataset. With the original dataset (btc.csv) this error doesn't happen, and the other ones has the same exact column names and data types. The only thing I did was changing the config file to take this new dataset.

Here's a look of my custom BTC dataset.

DateTime_UTC	Timestamp	price_high	price_low	price_close	price_open	volume
2017-08-05 00:00:00	1501891200	1601.0	1554.0	1592.0	1555.0	304.0
2017-08-05 00:01:00	1501891260	1592.0	1591.0	1591.0	1592.0	248.0

Also, the people on the internet say that I should decrease the learning rate, so I did, but the problem persist. I also tried messing around with the parameters from model/baseagent.py but same.
I ask here because I tried everything, including changing the order of the colums in the dataset.

Thank you!

0%|                   | 49900/100000000 [00:15<8:40:07, 3202.78it/s]Traceback (most recent call last):
  File "main.py", line 38, in <module>
    main(vars(args)['file_path'])
  File "main.py", line 29, in main
    agent.train()
  File "/deep-trading-agent/code/model/agent.py", line 71, in train
    self.observe(screen, reward, action, terminal, trade_rem)
  File "/deep-trading-agent/code/model/agent.py", line 170, in observe
    self.q_learning_mini_batch()
  File "/deep-trading-agent/code/model/agent.py", line 204, in q_learning_mini_batch
    self.learning_rate_step: self.step
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 778, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 982, in _run
    feed_dict_string, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1032, in _do_run
    target_list, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 1052, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Nan in summary histogram for: q_network/avg_q_summary/q/2
         [[Node: q_network/avg_q_summary/q/2 = HistogramSummary[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](q_network/avg_q_summary/q/2/tag, q_network/avg_q_summary/strided_slice_2)]]

Caused by op u'q_network/avg_q_summary/q/2', defined at:
  File "main.py", line 38, in <module>
    main(vars(args)['file_path'])
  File "main.py", line 28, in main
    agent = Agent(sess, logger, config, env)
  File "/deep-trading-agent/code/model/agent.py", line 44, in __init__
    self.build_dqn(params)
  File "/deep-trading-agent/code/model/agent.py", line 237, in build_dqn
    self.q.build_model((self.s_t, self.trade_rem_t))
  File "/deep-trading-agent/code/model/deepsense.py", line 205, in build_model
    self._avg_q_summary.append(tf.summary.histogram('q/{}'.format(idx), avg_q[idx]))
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/summary/summary.py", line 209, in histogram
    tag=scope.rstrip('/'), values=values, name=scope)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_logging_ops.py", line 139, in _histogram_summary
    name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 768, in apply_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2336, in create_op
    original_op=self._default_original_op, op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 1228, in __init__
    self._traceback = _extract_stack()

InvalidArgumentError (see above for traceback): Nan in summary histogram for: q_network/avg_q_summary/q/2
         [[Node: q_network/avg_q_summary/q/2 = HistogramSummary[T=DT_FLOAT, _device="/job:localhost/replica:0/task:0/cpu:0"](q_network/avg_q_summary/q/2/tag, q_network/avg_q_summary/strided_slice_2)]]

Running the docker container throws `killed` in preprocess.py

Removing intermediate container 3d5347bf9610
 ---> a84360b64f80
Step 11/16 : RUN gunzip /deep-trading-agent/data/coinbaseUSD.csv.gz
 ---> Running in 8b963696a5b5
Removing intermediate container 8b963696a5b5
 ---> 78e1cb2c844f
Step 12/16 : RUN python /deep-trading-agent/code/preprocess.py --transactions /deep-trading-agent/data/coinbaseUSD.csv --dataset /deep-trading-agent/data/btc.csv
 ---> Running in a2fc04a918b0
Killed
The command '/bin/sh -c python /deep-trading-agent/code/preprocess.py --transactions /deep-trading-agent/data/coinbaseUSD.csv --dataset /deep-trading-agent/data/btc.csv' returned a non-zero code: 137

Performance issue

I am the only one who have computation performance issue?When i run the script the requested computation time is about 250 hours...

InvalidArgumentError (see above for traceback): Input to reshape is a tensor wit h 80640 values, but the requested shape has 115200

Hi, cool project.

Getting a weird error during training when using the same config as example.cfg:

KeyError: KeyError(<weakref at 0x7f3801e2a628; to 'tqdm' at 0x7f38019d88d0>,) in
 <bound method tqdm.__del__ of   0%|                    | 24866/25000000 [00:05<
1:24:55, 4901.45it/s]> ignored
Traceback (most recent call last):
  File "main.py", line 38, in <module>
    main(vars(args)['file_path'])
  File "main.py", line 29, in main
    agent.train()
  File "/home/ted/projects/deep-trading-agent/code/model/agent.py", line 71, in 
train
    self.observe(screen, reward, action, terminal, trade_rem)
  File "/home/ted/projects/deep-trading-agent/code/model/agent.py", line 170, in
 observe
    self.q_learning_mini_batch()
  File "/home/ted/projects/deep-trading-agent/code/model/agent.py", line 186, in
 q_learning_mini_batch
    self.t_trade_rem_t: trade_rem_t_plus_1
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.
py", line 905, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.
py", line 1137, in _run
    feed_dict_tensor, options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.
py", line 1355, in _do_run
    options, run_metadata)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.
py", line 1374, in _do_call
    raise type(e)(node_def, op, message)
tensorflow.python.framework.errors_impl.InvalidArgumentError: Input to reshape i
s a tensor with 80640 values, but the requested shape has 115200
         [[Node: t_q_network/Reshape_1 = Reshape[T=DT_FLOAT, Tshape=DT_INT32, _d
evice="/job:localhost/replica:0/task:0/device:GPU:0"](t_q_network/conv_layers/co
nv_layer_2/conv_2/Relu, t_q_network/Reshape_1/shape)]]
         [[Node: t_q_network/q_values/BiasAdd/_139 = _Recv[client_terminated=fal
se, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/jo
b:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_na
me="edge_274_t_q_network/q_values/BiasAdd", tensor_type=DT_FLOAT, _device="/job:
localhost/replica:0/task:0/device:CPU:0"]()]]

Caused by op u't_q_network/Reshape_1', defined at:
  File "main.py", line 38, in <module>
    main(vars(args)['file_path'])
  File "main.py", line 28, in main
    agent = Agent(sess, logger, config, env)
  File "/home/ted/projects/deep-trading-agent/code/model/agent.py", line 44, in 
__init__
    self.build_dqn(params)
  File "/home/ted/projects/deep-trading-agent/code/model/agent.py", line 254, in
 build_dqn
    self.t_q.build_model((self.t_s_t, self.t_trade_rem_t))
  File "/home/ted/projects/deep-trading-agent/code/model/deepsense.py", line 150
, in build_model
    self.params.window_size * self.params.filter_sizes[-1]
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_array_o
ps.py", line 3903, in reshape
    "Reshape", tensor=tensor, shape=shape, name=name)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_de
f_library.py", line 787, in _apply_op_helper
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.p
y", line 3271, in create_op
    op_def=op_def)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.p
y", line 1650, in __init__
    self._traceback = self._graph._extract_stack()  # pylint: disable=protected-
access

InvalidArgumentError (see above for traceback): Input to reshape is a tensor wit
h 80640 values, but the requested shape has 115200
         [[Node: t_q_network/Reshape_1 = Reshape[T=DT_FLOAT, Tshape=DT_INT32, _d
evice="/job:localhost/replica:0/task:0/device:GPU:0"](t_q_network/conv_layers/co
nv_layer_2/conv_2/Relu, t_q_network/Reshape_1/shape)]]
         [[Node: t_q_network/q_values/BiasAdd/_139 = _Recv[client_terminated=fal
se, recv_device="/job:localhost/replica:0/task:0/device:CPU:0", send_device="/jo
b:localhost/replica:0/task:0/device:GPU:0", send_device_incarnation=1, tensor_na
me="edge_274_t_q_network/q_values/BiasAdd", tensor_type=DT_FLOAT, _device="/job:
localhost/replica:0/task:0/device:CPU:0"]()]]

Any idea for what this is from?

ERROR: No section: 'global'

Hello, I tried running the code with the config file shown in the image and I still getting that error. But the 'global' section is at the beginning of the file.

>sudo cat config/config.cfg
[global]
PARENT_DIR = /home/mmnkl9/deep-trading-agent/code/

[logging]
LOG_FILE = logs/run.log
SAVE_DIR = logs/saved_models/
TENSORBOARD_LOG_DIR = logs/tensorboard/

[preprocessing]
DATASET_PATH = data/btc.csv

[dataset]
BATCH_SIZE = 32
HISTORY_LENGTH = 180

enhancement - risk reward / position sizing

it seems without a sense of risk / reward or position sizing - the trained model is a glorified heads + tails flip calculator based off previous data.

it maybe beyond scope of project - but to build an autonomous agent that can a variety of actions - need to somehow incorporate this to win the trading game. Perhaps there's two trained models.

https://github.com/m-esgana/poor-trader-py/blob/097fbb10c217abe0db5a2cd55d73e0ad2990acd9/poor_trader/trading.py

Price gap between historical dataset and live API requests

For kaggle's btceUSD dataset we got these last 5 lines in which BTC(USD) weighted price =2130~2139:

1496188560,2130,2130,2129.999,2129.999,2.35330472,5012.53827,2129.999667
1496188620,2130,2134.8,2130,2134.8,12.03238662,25648.897767,2131.6550554
1496188680,2133.2,2134.8,2133.2,2134.8,1.40799439,3005.4756141,2134.5792536
1496188740,2134.8,2143.79,2134.8,2134.801,3.19062001,6824.2637131,2138.8519133
1496188800,2139.999,2140,2134.35,2139.999,6.73707917,14413.324651,2139.4025938

When I make a request to cryptocompare website for BTC(USD) I get this where the price suddenly jumps to 7656.98 USD at the same timestamp:

{"RAW":{"BTC":{"USD":{"TYPE":"5","MARKET":"CCCAGG","FROMSYMBOL":"BTC","TOSYMBOL":"USD","FLAGS":"1","PRICE":7656.98,"LASTUPDATE":1511013600,"LASTVOLUME":0.26175057,"LASTVOLUMETO":2002.3918605000001,"LASTTRADEID":"26754557","VOLUMEDAY":48775.74066467763,"VOLUMEDAYTO":372532531.85840243,"VOLUME24HOUR":89903.71924385922,"VOLUME24HOURTO":693640577.3317167,"OPENDAY":7699.95,"HIGHDAY":7798.12,"LOWDAY":7458.9,"OPEN24HOUR":7829.71,"HIGH24HOUR":7982.5,"LOW24HOUR":7446.1,"LASTMARKET":"Bitstamp","CHANGE24HOUR":-172.73000000000047,"CHANGEPCT24HOUR":-2.2060842611029075,"CHANGEDAY":-42.970000000000255,"CHANGEPCTDAY":-0.5580555717894304,"SUPPLY":16686375,"MKTCAP":127767239647.5,"TOTALVOLUME24H":480649.677315126,"TOTALVOLUME24HTO":3685574563.3642445}}}

What am I missing here sir?

Some Issues - Pycharm W10

Hi, I have some problems...
the first is in utils/util

def get_config_parser(filename): config = ConfigParser(allow_no_value=True) config.read(filename) return config

filename is a NoneType object, I don't udenderstand wich file i must put.

I tried with "config" instead "filename" but i receive this error:

`Traceback (most recent call last):
File "C:\Users\vadil\Anaconda3\envs\Test5\lib\configparser.py", line 1138, in _unify_values
sectiondict = self._sections[section]
KeyError: 'global'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "C:/Users/vadil/PycharmProjects/Test5/main.py", line 38, in
main(vars(args)['file_path'])
File "C:/Users/vadil/PycharmProjects/Test5/main.py", line 21, in main
config = get_config(config_parser)
File "C:\Users\vadil\PycharmProjects\Test5\utils\config.py", line 17, in get_config
config[PARENT_DIR] = config_parser.get(GLOBAL, PARENT_DIR)
File "C:\Users\vadil\Anaconda3\envs\Test5\lib\configparser.py", line 781, in get
d = self._unify_values(section, vars)
File "C:\Users\vadil\Anaconda3\envs\Test5\lib\configparser.py", line 1141, in _unify_values
raise NoSectionError(section)
configparser.NoSectionError: No section: 'global'

Process finished with exit code 1`

I love this project, in my university no one teach RL than I started this study alone.

Unable to understand the Reward Function

Hi I am having difficulty in understanding the reward function. More specifically I am not able to understand what does "amount of long currency" and "amount of short currency" mean?

I would appreciate if you could elaborate a on the reward function and the terms used in it.

Thanks.

Why On Earth would you use python 2.7 ? that's backward

Why On Earth would you use python 2.7 ? that completely makes no sense but anyway good project

Example Config

Hi @samre12 ,

It's looks cool your work.
Can you share example of the config file with us?

training issue

Hi!
Have a question about training.

After 16 hours of training, I still get average reward 0.
Will be happy if you can explain what can be wrong?
Maybe it's a problem with default setup parameters?

25%|███▊ | 6374997/25000000 [16:18:15<48:41:33, 106.25it/s]INFO:deep_trading_agent:avg_r: 0.0000, avg_l: 0.000135, avg_q: -0.001807, avg_ep_r: 0.0000, max_ep_r: 0.0911, min_ep_r: -0.0984, # game: 5000
26%|███▊ | 6399993/25000000 [16:22:11<48:18:45, 106.94it/s]INFO:deep_trading_agent:avg_r: 0.0000, avg_l: 0.000135, avg_q: -0.001469, avg_ep_r: 0.0000, max_ep_r: 0.0985, min_ep_r: -0.0641, # game: 5000
26%|███▊ | 6424989/25000000 [16:26:07<48:06:53, 107.24it/s]INFO:deep_trading_agent:avg_r: 0.0000, avg_l: 0.000138, avg_q: -0.001775, avg_ep_r: 0.0001, max_ep_r: 0.1445, min_ep_r: -0.0460, # game: 5000
26%|███▊ | 6449993/25000000 [16:30:03<48:41:25, 105.83it/s]INFO:deep_trading_agent:avg_r: -0.0000, avg_l: 0.000134, avg_q: -0.001525, avg_ep_r: -0.0000, max_ep_r: 0.0223, min_ep_r: -0.0371, # game: 5000
26%|███▉ | 6477033/25000000 [16:34:16<47:10:48, 109.06it/s]INFO:deep_trading_agent:avg_r: -0.0000, avg_l: 0.000138, avg_q: -0.002763, avg_ep_r: -0.0000, max_ep_r: 0.0302, min_ep_r: -0.0762, # game: 5000
26%|███▉ | 6499197/25000000 [16:37:41<47:10:50, 108.92it/s]INFO:deep_trading_agent:avg_r: 0.0000, avg_l: 0.000142, avg_q: -0.003163, avg_ep_r: 0.0000, max_ep_r: 0.0352, min_ep_r: -0.0225, # game: 5000
26%|███▉ | 6526765/25000000 [16:41:56<47:30:54, 108.00it/s]INFO:deep_trading_agent:avg_r: -0.0000, avg_l: 0.000135, avg_q: -0.003114, avg_ep_r: -0.0000, max_ep_r: 0.0253, min_ep_r: -0.1445, # game: 5000
26%|███▉ | 6551381/25000000 [16:45:43<47:47:03, 107.25it/s]INFO:deep_trading_agent:avg_r: 0.0000, avg_l: 0.000131, avg_q: -0.002506, avg_ep_r: 0.0000, max_ep_r: 0.0643, min_ep_r: -0.0199, # game: 5000
26%|███▉ | 6577145/25000000 [16:49:41<47:26:52, 107.85it/s]INFO:deep_trading_agent:avg_r: -0.0000, avg_l: 0.000137, avg_q: -0.001795, avg_ep_r: -0.0000, max_ep_r: 0.0300, min_ep_r: -0.1185, # game: 5000
26%|███▉ | 6599989/25000000 [16:53:14<46:38:00, 109.60it/s]INFO:deep_trading_agent:avg_r: -0.0000, avg_l: 0.000137, avg_q: -0.002334, avg_ep_r: -0.0000, max_ep_r: 0.0495, min_ep_r: -0.1122, # game: 5000

TypeError: unhashable type: 'numpy.ndarray'

Hello,
I am running main.py and got the following error:

Traceback (most recent call last):
  File "main.py", line 36, in <module>
    main(vars(args)['file_path'])
  File "main.py", line 27, in main
    agent.train()
  File "deep-trading-agent/code/model/agent.py", line 70, in train
    screen, reward, terminal = self.env.act(action)
  File "deep-trading-agent/code/model/environment.py", line 62, in act
    if self.action_dict[action] is LONG:
TypeError: unhashable type: 'numpy.ndarray'

Could you tell me what goes wrong?
Thanks

Testing module

Hi samre12,

After the training using the main.py, how can I test on another set of data? Thanks.

Thanks,
Lam

AttributeError: 'Agent' object has no attribute 'q_gru_keep_prob'

Problems with Python 3.6 - Unable to get past this point

2018-05-21 08:58:16.038248: I T:\src\github\tensorflow\tensorflow\core\platform\cpu_feature_guard.cc:140] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 2018-05-21 08:58:16.278040: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1356] Found device 0 with properties: name: GeForce GTX 1050 Ti major: 6 minor: 1 memoryClockRate(GHz): 1.4425 pciBusID: 0000:02:00.0 totalMemory: 4.00GiB freeMemory: 3.29GiB 2018-05-21 08:58:16.278375: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1435] Adding visible gpu devices: 0 2018-05-21 08:58:16.779930: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:923] Device interconnect StreamExecutor with strength 1 edge matrix: 2018-05-21 08:58:16.780127: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:929] 0 2018-05-21 08:58:16.780253: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:942] 0: N 2018-05-21 08:58:16.780467: I T:\src\github\tensorflow\tensorflow\core\common_runtime\gpu\gpu_device.cc:1053] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 3025 MB memory) -> physical GPU (device: 0, name: GeForce GTX 1050 Ti, pci bus id: 0000:02:00.0, compute capability: 6.1) INFO: Making new env: UnRealizedPnLEnv-v0 Traceback (most recent call last): File "D:/deep-trading-agent-dev/code/main.py", line 44, in <module> main(vars(args)['file_path']) File "D:/deep-trading-agent-dev/code/main.py", line 34, in main agent = Agent(sess, logger, config, env) File "D:\deep-trading-agent-dev\code\model\agent.py", line 252, in __init__ self.build_dqn(params) File "D:\deep-trading-agent-dev\code\model\agent.py", line 448, in build_dqn print (self.q_gru_keep_prob) AttributeError: 'Agent' object has no attribute 'q_gru_keep_prob'

Add trading activity as an input

I am a very beginner in ML and so have little understanding on what this model capable of.
Only want to suggest and, hopefully, get a feedback.

If we analyze only raw exchange data, as trades in order books and closed ones, we may have:

current buy and sell price
buy and sell volumes in some period of time
number of buy and sell orders closed (they are not always correlated with volume)
buy and sell order book (for example, volume needed to move price for particular percent, or measure activity in order book)

As post-processing of these values, we can divide them by SMA, measure speed they are changing.