Giter VIP home page Giter VIP logo

mlstm-fcn's Introduction

Multivariate LSTM-FCNs for Time Series Classification

MLSTM FCN models, from the paper Multivariate LSTM-FCNs for Time Series Classification, augment the squeeze and excitation block with the state of the art univariate time series model, LSTM-FCN and ALSTM-FCN from the paper LSTM Fully Convolutional Networks for Time Series Classification. The code for the LSTM-FCN and ALSTM-FCN models can be found at LSTM-FCN.

Installation

Download the repository and apply pip install -r requirements.txt to install the required libraries.

Keras with the Tensorflow backend has been used for the development of the models, and there is currently no support for Theano or CNTK backends. The weights have not been tested with those backends.

Note : The input to the Input layer of all models will be pre-shuffled to be in the shape (Batchsize, Number of variables, Number of timesteps), and the input will be shuffled again before being applied to the CNNs (to obtain the correct shape (Batchsize, Number of timesteps, Number of variables)). This is in contrast to the paper where the input is of the shape (Batchsize, Number of timesteps, Number of variables) and the shuffle operation is applied before the LSTM to obtain the input shape (Batchsize, Number of variables, Number of timesteps). These operations are equivalent.

Multivariate Benchmark Datasets

Note

The multivariate datasets are now available in the Release Tab. Please cite this paper and the original source of the appropriate dataset when using these datasets for academic purposes.

Training and Evaluation

Various multivariate benchmark datasets can be evaluated with the provided code and weight files. Refer to the weights directory for clarification.

There is 1 script file for each dataset, and 2 major sections in the code. For each of these code files, please keep the line below uncommented.

  • To use the MLSTM FCN model : model = generate_model()
  • To use the MALSTM FCN model : model = generate_model_2()
  • To use the LSTM FCN model : model = generate_model_3()
  • To use the ALSTM FCN model : model = generate_model_4()

Training

To train the a model, uncomment the line below and execute the script. Note that '???????' will already be provided, so there is no need to replace it. It refers to the prefix of the saved weight file. Also, if weights are already provided, this operation will overwrite those weights.

train_model(model, DATASET_INDEX, dataset_prefix='???????', epochs=250, batch_size=128)

Evaluate

To evaluate the performance of the model, simply execute the script with the below line uncommented.

evaluate_model(model, DATASET_INDEX, dataset_prefix='???????', batch_size=128)

Results

Citation

@misc{Karim2018,
  Author = {Fazle Karim and Somshubra Majumdar and Houshang Darabi and Samuel Harford},
  Title = {Multivariate LSTM-FCNs for Time Series Classification},
  Year = {2018},
  Eprint = {arXiv:1801.04503},
}

mlstm-fcn's People

Contributors

fazlekarim avatar stevenwtolbert avatar titu1994 avatar

Stargazers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

Watchers

 avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar  avatar

mlstm-fcn's Issues

License

Greetings,
I was wondering if there is a specific license for using the code provided in this repository
Thanks in advance!

Confusion for The Number of variables in Action 3D dataset

Hi ,titu~ @titu1994

I have read your papers about time series classification and your great work sparked my interest!
So I want figure out the detail of this master work.

I noticed that the number of variables in Action 3D dataset is 570. And I am confused how this number is calculated.
As far as I know, The data structure of Action 3D dataset is 20*4 (in single time frame) . 20 means there are 20 skeleton points, 4 means each skeleton point is described by 4 dimension:(x,y,d,c). x,y represent the position of skeleton point in the picture,and d represent depth,c represents the score confidence.

Could you nicely tell me how you extract 570 variables form the inital datasets?
It confused me for a long time, hope you can help.
Thanks anyway!

Another Way of Ensembling with LSTM

@titu1994 What do you think of the following way of ensembling with LSTM? (Feeds CNN outputs in FCN to both GAP and LSTM, and concat their outputs before the final Linear or Softmax layer).

I have read some papers that has CNN followed by LSTM and came up with this version of FCN. I unfortunately doesn't have time and resources to experiment on it as detailed as you have done.

In addition to that, have you ever tried utilizing GRU instead of the LSTM module in your experiments?

Hence, I would love to hear your idea about this and maybe results if you would try this as well.

class out_layer(nn.Module):
    def __init__(self, in_channels, out_channels = 1, gap_size = 1200, cl = False, lstm_size = 0):
        super(out_layer, self).__init__()
        self.in_channels = in_channels
        self.lstm_size = lstm_size
        self.gap = nn.AvgPool1d(kernel_size = gap_size, padding = 1)
        if lstm_size > 0:
            self.lin = nn.Linear(in_channels * 2, out_channels)
            self.lstm = nn.LSTM(input_size = gap_size, hidden_size = lstm_size, batch_first = True)
        else:
            self.lin = nn.Linear(in_channels, out_channels)
        self.activation = nn.Softmax() if cl else lambda x: x
    def forward(self, x):
        gap = self.gap(x).view(-1, self.in_channels)
        if self.lstm_size > 0:
            out_lstm, _ = self.lstm(x)
            gap = torch.cat((gap, out_lstm[:,:,-1]), 1)
        return self.activation(self.lin(gap))

The input shape for LSTM

Why the input shape for LSTM in Keras is not (Batchsize, Number of timesteps, Number of variables)?

Data shape for Ozone dataset

Thank you for sharing the code. I am trying to follow the code but confused about the data shape for example the ozone data set. The UCI ozone data set contains 2,534 days of data and 72 independent variables. Looking at the dataset which is derived from a matlab matrix the resulting data set shape is:

Train dataset :  (173, 72, 291) 
Test dataset :  (173, 72, 291) 

I see the 72 features (N) and the 291 time steps (M) but not sure about 173 and how this shape was derived. In the "generate_ozone_dataset.py" code, it looks like there is some zero padding for multi-length data but I don't understand how/why timesteps would vary. In other words, why wouldn't timesteps be a loopback window of previous ozone observations? The data in the matlab matrix is already normalized so I can't see any logic as to a multi-length time step and reason to pad with zeros. Also not sure about the first dimension of 173 and how it was derived?

Thanks for any insight.

About dataset

Hello, you did a great job, powerful work!
When I try to study the multivariate time series classification, I find the forms of dataset are so complicated and I don't know how to begin without data.
Most of your dataset's formats are 'mat', but I can't find the data files in the project. I don't know matlab, and the data in UCI are so complicated.
Can you show me the data?
I am particularly grateful for your help.

Input shape confusion

Thanks for providing the code
I'm a little bit confused about the input shape of the models.
In the keras documentation it specifically says that the input shape for LSTM needs to be input_shape=(timesteps, data_dim) but in lots of models I'm seeing you have used shape=(MAX_NB_VARIABLES, MAX_TIMESTEPS) which is the transpose of what keras documentation says.
Do you have any comment on this?

class_weight error

hello,

When I ran the code I encountered this error:
class_weight is causing a ValueError: The truth value of an array with more than one element is ambiguous. use a.any() or a.all().

I tried to fix it, but without success.
I saw someone's solution online:

from sklearn.utils import class_weight
weight = class_weight.compute_class_weight('balanced', np.unique(y_train), y_train)
class_weights = {l:c for l,c in zip(np.unique(labels[i]), class_weights)}

But this solution doesn't work for me, can you help me?
Thank you very much!

Serving in Production

Hi!

I was able to get my data to run through Model 2 and had excellent luck. I am trying to get this ready to run in a mock production environment. I ran the model and in keras_utils turned
save_weights_only = False
to get the full model, but I am getting the error:
ValueError: Unknown layer: AttentionLSTM
When I run:

weightfile ='./MLSTM-FCN/weights/stage_10epoch_weights.h5'
my_file = Path(weightfile)
if my_file.is_file():
  print("Weight file exisits")
  model = load_model(weightfile)  #THIS IS THE LINE THROWING THE ERROR
else:
  print("Weight file do not exist")
  # train the model and save it
  history =model.fit(X_train, Y_train,batch_size=32, epochs=20, 
                   callbacks=[plot],
                   validation_data=(X_test, Y_test),
                   verbose=1)
  model.save(weightfile)

Have you deployed these models in a "production" environment, and if so, do you have any suggestions? Thanks!

Can't find Dataset on UCI

Hi @titu1994 and @fazlekarim ,
I was reading through your paper, but can't find most of the datasets online that you've provided with the code (as well as mentioned in the paper about). For instance, I can't find the EEG2 and EEG dataset on UCI, the EEG dataset that I find there is - https://archive.ics.uci.edu/ml/datasets/Epileptic+Seizure+Recognition , which is substantially different from the one's that you've used.

Could you please point me to specific links from the web for those 2 datasets (EEG and EEG2)? Thanks! 👍

License

Are you planning to make -- or am I missing -- the license for this repo?

I know that it's not a necessarily normal to care; but legally speaking, we cannot open the code, or even view it, if it is unlicensed.

It's not that I want to be pedantic; but I don't see one and I was wondering why?

Question for AttentionLSTM

AttentionLSTM code is implemented by the author extending Recurrent layer module in Keras?

below, what i,f,c,o mean? What do they stand for?

        self.kernel_i = self.kernel[:, :self.units]
        self.kernel_f = self.kernel[:, self.units: self.units * 2]
        self.kernel_c = self.kernel[:, self.units * 2: self.units * 3]
        self.kernel_o = self.kernel[:, self.units * 3:]

The input shape for LSTM

Why the input shape for LSTM is not (Batchsize, Number of timesteps, Number of variables) for Keras?

i am facing problem while running the code for wafer data set

OSError Traceback (most recent call last)
in
2 model = generate_model()
3
----> 4 train_model(model, DATASET_INDEX, dataset_prefix='wafer_', epochs=1000, batch_size=128)
5
6 evaluate_model(model, DATASET_INDEX, dataset_prefix='wafer_', batch_size=128)

~\Downloads\tcs\MLSTM-FCN-master\utils\keras_utils.py in train_model(model, dataset_id, dataset_prefix, dataset_fold_id, epochs, batch_size, val_subset, cutoff, normalize_timeseries, learning_rate, monitor, optimization_mode, compile_model)
160
161 if val_subset is not None:
--> 162 X_test = X_test[:val_subset]
163 y_test = y_test[:val_subset]
164

~\Anaconda3\envs\deeplearning\lib\site-packages\keras\engine\training.py in fit(self, x, y, batch_size, epochs, verbose, callbacks, validation_split, validation_data, shuffle, class_weight, sample_weight, initial_epoch, steps_per_epoch, validation_steps, validation_freq, max_queue_size, workers, use_multiprocessing, **kwargs)
1237 steps_per_epoch=steps_per_epoch,
1238 validation_steps=validation_steps,
-> 1239 validation_freq=validation_freq)
1240
1241 def evaluate(self,

~\Anaconda3\envs\deeplearning\lib\site-packages\keras\engine\training_arrays.py in fit_loop(model, fit_function, fit_inputs, out_labels, batch_size, epochs, verbose, callbacks, val_function, val_inputs, shuffle, initial_epoch, steps_per_epoch, validation_steps, validation_freq)
214 epoch_logs['val_' + l] = o
215
--> 216 callbacks.on_epoch_end(epoch, epoch_logs)
217 if callbacks.model.stop_training:
218 break

~\Anaconda3\envs\deeplearning\lib\site-packages\keras\callbacks\callbacks.py in on_epoch_end(self, epoch, logs)
150 logs = logs or {}
151 for callback in self.callbacks:
--> 152 callback.on_epoch_end(epoch, logs)
153
154 def on_train_batch_begin(self, batch, logs=None):

~\Anaconda3\envs\deeplearning\lib\site-packages\keras\callbacks\callbacks.py in on_epoch_end(self, epoch, logs)
715 self.best = current
716 if self.save_weights_only:
--> 717 self.model.save_weights(filepath, overwrite=True)
718 else:
719 self.model.save(filepath, overwrite=True)

~\Anaconda3\envs\deeplearning\lib\site-packages\keras\engine\saving.py in save_wrapper(obj, filepath, overwrite, *args, **kwargs)
447 os.remove(tmp_filepath)
448 else:
--> 449 save_function(obj, filepath, overwrite, *args, **kwargs)
450
451 return save_wrapper

~\Anaconda3\envs\deeplearning\lib\site-packages\keras\engine\network.py in save_weights(self, filepath, overwrite)
1181 if not proceed:
1182 return
-> 1183 with h5py.File(filepath, 'w') as f:
1184 saving.save_weights_to_hdf5_group(f, self.layers)
1185 f.flush()

~\Anaconda3\envs\deeplearning\lib\site-packages\h5py_hl\files.py in init(self, name, mode, driver, libver, userblock_size, swmr, rdcc_nslots, rdcc_nbytes, rdcc_w0, track_order, **kwds)
406 fid = make_fid(name, mode, userblock_size,
407 fapl, fcpl=make_fcpl(track_order=track_order),
--> 408 swmr=swmr)
409
410 if isinstance(libver, tuple):

~\Anaconda3\envs\deeplearning\lib\site-packages\h5py_hl\files.py in make_fid(name, mode, userblock_size, fapl, fcpl, swmr)
177 fid = h5f.create(name, h5f.ACC_EXCL, fapl=fapl, fcpl=fcpl)
178 elif mode == 'w':
--> 179 fid = h5f.create(name, h5f.ACC_TRUNC, fapl=fapl, fcpl=fcpl)
180 elif mode == 'a':
181 # Open in append mode (read/write).

h5py_objects.pyx in h5py._objects.with_phil.wrapper()

h5py_objects.pyx in h5py._objects.with_phil.wrapper()

h5py\h5f.pyx in h5py.h5f.create()

OSError: Unable to create file (unable to open file: name = './weights/wafer__weights.h5', errno = 2, error message = 'No such file or directory', flags = 13, o_flags = 302)



I am getting this as error ,pls help

the detailed information about Attention LSTM

Thank you for your excellent work. It actually inspires me a lot. Here is one question:
Could you give a detailed word description about the implemention of Attention LSTM layer in the layer
_utils.py ? In my opinion, the paper Neural Machine Translation by Jointly Learning to Align and Translate introduce the attention mechanism in the encoder-decoder networks. However, you give a function that indicates a layer with LSTM.

how can I use my own data

Hi Titu,

I want to ask that what should I do to use my own data, and test on it? I found some py files for each dataset you provided, but could you please tell me the pipeline for using my own data?
Thanks!

Questions about the data set in this paper

Hello, I am Du Wang, a student from China. I have been reading a paper recently and I feel that I have benefited a lot. There is a question I want to ask you. I downloaded the datasets you provided, but found that these datasets are not the same as the UCI database. Most of the data sets you provide are smaller than the original data sets. What is the status of this data set?

OSError: Unable to open file (unable to open file: name = './weights/action_3d_weights.h5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)

Hello,
First of all, thank you for your sharing ,it's a nice job.
I was trying to run the action_3d_model but I'm getting the following error:
Traceback (most recent call last):
File "E:/program/MLSTM-FCN-master/action_3d_model.py", line 199, in
evaluate_model(model, DATASET_INDEX, dataset_prefix='action_3d', batch_size=128)
File "E:\program\MLSTM-FCN-master\utils\keras_utils.py", line 195, in evaluate_model
model.load_weights(weight_fn)
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\site-packages\keras\engine\network.py", line 1157, in load_weights
with h5py.File(filepath, mode='r') as f:
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\site-packages\h5py_hl\files.py", line 394, in init
swmr=swmr)
File "C:\Users\Administrator\AppData\Local\Programs\Python\Python37\lib\site-packages\h5py_hl\files.py", line 170, in make_fid
fid = h5f.open(name, flags, fapl=fapl)
File "h5py_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
File "h5py_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
File "h5py\h5f.pyx", line 85, in h5py.h5f.open
OSError: Unable to open file (unable to open file: name = './weights/action_3d_weights.h5', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)

By the way, I am not sure that I added data in a right way. i download the data from the release tab, and copied it to "MLSTM-FCN-master\data\Action3D", as shown as below:
image

Could you please help me out with this?
Thanks in Advance!

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.