atik-ahamed / timemachine Goto Github PK

View Code? Open in Web Editor NEW

141.0 4.0 17.0 236 KB

TimeMachine: A Time Series is Worth 4 Mambas for Long-term Forecasting

Home Page: https://arxiv.org/abs/2403.09898

License: Apache License 2.0

Python 78.54% Shell 21.46%

time-series-forecasting

timemachine's Introduction

TimeMachine

Welcome to the official repository of: TimeMachine: A Time Series is Worth 4 Mambas for Long-term Forecasting.

🚩 TimeMachine is accepted to ECAI

Usage

Install requirements. pip install -r requirements.txt
Navigate through our example scripts located at ./scripts/TimeMachine. You'll find the core of TimeMachine in models/TimeMachine.py. For example, to get the multivariate forecasting results for weather dataset, just run the following command, and you can open ./result.txt to see the results once the training is completed. Moreover, the results will also be available at csv_results, which can be utilized to make queries in the dataframe:

sh ./scripts/TimeMachine/weather.sh

Hyper-paramters can be tuned based upon needs (e.g. different look-back windows and prediction lengths). TimeMachine is built on the popular PatchTST framework.

Acknowledgement

We are deeply grateful for the valuable code and efforts contributed by the following GitHub repositories. Their contributions have been immensely beneficial to our work.

Mamba (https://github.com/state-spaces/mamba)
PatchTST (https://github.com/yuqinie98/PatchTST)
iTransformer (https://github.com/thuml/iTransformer)
RevIN (https://github.com/ts-kim/RevIN)
Reformer (https://github.com/lucidrains/reformer-pytorch)
Informer (https://github.com/zhouhaoyi/Informer2020)
FlashAttention (https://github.com/shreyansh26/FlashAttention-PyTorch)
Autoformer (https://github.com/thuml/Autoformer)
Stationary (https://github.com/thuml/Nonstationary_Transformers)
Time-Series-Library (https://github.com/thuml/Time-Series-Library)

Citation

If you find this repo useful in your research, please consider citing our paper as follows:

@article{timemachine,
  title     = {TimeMachine: A Time Series is Worth 4 Mambas for Long-term Forecasting},
  author    = {Ahamed, Md Atik and Cheng, Qiang},
  journal   = {arXiv preprint arXiv:2403.09898},
  year      = {2024}
}

timemachine's People

Contributors

Stargazers

Watchers

Forkers

valeman jacktang qxzsilver1 visionmamba sean-bin-yang jeongwhanchoi levi-ackman anacronic-io zhuolinli-shu everythingforai liujun0715 zhonghaoaustin longlive1234 maateussilva kangmincho1 lepremiere khsibr

timemachine's Issues

where to download the data?

when i run the code, i get the error, No such file or directory: '../data/weather/weather.csv'. So how i should download the data? can you give some reference link?

How to install causal conv1d and mamba-ssm

How to install causal conv1d and mamba-sms

What data shape could be mamba's correct input?

import torch
from mamba_ssm import Mamba

batch, length, dim = 2, 64, 16
x = torch.randn(batch, length, dim).to("cuda")
model = Mamba(
# This module uses roughly 3 * expand * d_model^2 parameters
d_model=dim, # Model dimension d_model
d_state=16, # SSM state expansion factor
d_conv=4, # Local convolution width
expand=2, # Block expansion factor
).to("cuda")
y = model(x)
assert y.shape == x.shape
This is the official introduction of mamba-ssm,so i think the input shape should be B L d,while in timemachine,the input data's shape is B M ni
I am confused at this about this,could you please provide some guidance for me.Thank you so much for your time and assistance.

debug

When I was running the code, I encountered the following problems, could you help me solve them? I used python=3.10, torch=2.0.1,cuda=11.8. Thank you very much import selective_scan_cuda ImportError: /home/hp/miniforge3/envs/mamba/lib/python3.10/site-packages/selective_scan_cuda.cpython-310-x86_64-linux-gnu.so: undefined symbol: _ZN3c104cuda9SetDeviceEi

The training time of the model

Hi, I recently run the code and the MSE results(and so on) is OK. However, although the model costs relatively low gpu memory, its training speed is really really slow, have you ever encountered a similar situation? Take Weather_96_96 as an example, TimeMachine costs 380s for one epoch while PatchTST only needs 40s...

Can TimeMachine be applied to multivariate time series classification(MTSC)?

Hi!Thank you for your excellent work! I've been working on an MTSC task recently, and is TimeMachine suitable for this kind of task?

Nan loss！

I run the script "sh ./scripts/TimeMachine/weather.sh"，and at the second training epoch, it occured nan training loss problem.

I print the per step loss, can you help me solve this problem?

TimeMachine_weather_96_96_128_16_0.1_1_1_256_2_1.log

not able to achieve the same excellent results

Hello, I appreciate the compliment on your ideas, but I have run the etth1.sh file using the parameters you provided without any modifications, and I'm not able to achieve the same excellent results as described in your paper.

$MTOBUUYSF)7F){S_O2R$C$

Multi-GPU Execution Error: Invalid device id

First off, thank you for your excellent work on this project! I've encountered an issue while attempting to utilize the multi-GPU functionality. Specifically, the program runs successfully when GPUs are specified in a sequential order starting from zero using the --devices parameter (e.g., "0,1,2"). However, errors occur when the GPUs are specified in a non-sequential order or do not start with zero (e.g., "0,2" or "1,2,3").

Full error message:

Traceback (most recent call last):
  File "/home/mateus/repos/forks/TimeMachine/TimeMachine_supervised/run_longExp.py", line 117, in <module>
    exp = Exp(args)  # set experiments
  File "/home/mateus/repos/forks/TimeMachine/TimeMachine_supervised/exp/exp_main.py", line 24, in __init__
    super(Exp_Main, self).__init__(args)
  File "/home/mateus/repos/forks/TimeMachine/TimeMachine_supervised/exp/exp_basic.py", line 10, in __init__
    self.model = self._build_model().to(self.device)
  File "/home/mateus/repos/forks/TimeMachine/TimeMachine_supervised/exp/exp_main.py", line 33, in _build_model
    model = nn.DataParallel(model, device_ids=self.args.device_ids)
  File "/home/mateus/miniconda3/envs/TimeMachine_supervised/lib/python3.10/site-packages/torch/nn/parallel/data_parallel.py", line 159, in __init__
    _check_balance(self.device_ids)
  File "/home/mateus/miniconda3/envs/TimeMachine_supervised/lib/python3.10/site-packages/torch/nn/parallel/data_parallel.py", line 26, in _check_balance
    dev_props = _get_devices_properties(device_ids)
  File "/home/mateus/miniconda3/envs/TimeMachine_supervised/lib/python3.10/site-packages/torch/_utils.py", line 734, in _get_devices_properties
    return [_get_device_attr(lambda m: m.get_device_properties(i)) for i in device_ids]
  File "/home/mateus/miniconda3/envs/TimeMachine_supervised/lib/python3.10/site-packages/torch/_utils.py", line 734, in <listcomp>
    return [_get_device_attr(lambda m: m.get_device_properties(i)) for i in device_ids]
  File "/home/mateus/miniconda3/envs/TimeMachine_supervised/lib/python3.10/site-packages/torch/_utils.py", line 713, in _get_device_attr
    return get_member(torch.cuda)
  File "/home/mateus/miniconda3/envs/TimeMachine_supervised/lib/python3.10/site-packages/torch/_utils.py", line 734, in <lambda>
    return [_get_device_attr(lambda m: m.get_device_properties(i)) for i in device_ids]
  File "/home/mateus/miniconda3/envs/TimeMachine_supervised/lib/python3.10/site-packages/torch/cuda/__init__.py", line 452, in get_device_properties
    raise AssertionError("Invalid device id")
AssertionError: Invalid device id
Traceback (most recent call last):
  File "/home/mateus/repos/forks/TimeMachine/TimeMachine_supervised/run_longExp.py", line 117, in <module>
    exp = Exp(args)  # set experiments
  File "/home/mateus/repos/forks/TimeMachine/TimeMachine_supervised/exp/exp_main.py", line 24, in __init__
    super(Exp_Main, self).__init__(args)
  File "/home/mateus/repos/forks/TimeMachine/TimeMachine_supervised/exp/exp_basic.py", line 10, in __init__
    self.model = self._build_model().to(self.device)
  File "/home/mateus/repos/forks/TimeMachine/TimeMachine_supervised/exp/exp_main.py", line 33, in _build_model
    model = nn.DataParallel(model, device_ids=self.args.device_ids)
  File "/home/mateus/miniconda3/envs/TimeMachine_supervised/lib/python3.10/site-packages/torch/nn/parallel/data_parallel.py", line 159, in __init__
    _check_balance(self.device_ids)
  File "/home/mateus/miniconda3/envs/TimeMachine_supervised/lib/python3.10/site-packages/torch/nn/parallel/data_parallel.py", line 26, in _check_balance
    dev_props = _get_devices_properties(device_ids)
  File "/home/mateus/miniconda3/envs/TimeMachine_supervised/lib/python3.10/site-packages/torch/_utils.py", line 734, in _get_devices_properties
    return [_get_device_attr(lambda m: m.get_device_properties(i)) for i in device_ids]
  File "/home/mateus/miniconda3/envs/TimeMachine_supervised/lib/python3.10/site-packages/torch/_utils.py", line 734, in <listcomp>
    return [_get_device_attr(lambda m: m.get_device_properties(i)) for i in device_ids]
  File "/home/mateus/miniconda3/envs/TimeMachine_supervised/lib/python3.10/site-packages/torch/_utils.py", line 713, in _get_device_attr
    return get_member(torch.cuda)
  File "/home/mateus/miniconda3/envs/TimeMachine_supervised/lib/python3.10/site-packages/torch/_utils.py", line 734, in <lambda>
    return [_get_device_attr(lambda m: m.get_device_properties(i)) for i in device_ids]
  File "/home/mateus/miniconda3/envs/TimeMachine_supervised/lib/python3.10/site-packages/torch/cuda/__init__.py", line 452, in get_device_properties
    raise AssertionError("Invalid device id")
AssertionError: Invalid device id
Traceback (most recent call last):
  File "/home/mateus/repos/forks/TimeMachine/TimeMachine_supervised/run_longExp.py", line 117, in <module>
    exp = Exp(args)  # set experiments
  File "/home/mateus/repos/forks/TimeMachine/TimeMachine_supervised/exp/exp_main.py", line 24, in __init__
    super(Exp_Main, self).__init__(args)
  File "/home/mateus/repos/forks/TimeMachine/TimeMachine_supervised/exp/exp_basic.py", line 10, in __init__
    self.model = self._build_model().to(self.device)
  File "/home/mateus/repos/forks/TimeMachine/TimeMachine_supervised/exp/exp_main.py", line 33, in _build_model
    model = nn.DataParallel(model, device_ids=self.args.device_ids)
  File "/home/mateus/miniconda3/envs/TimeMachine_supervised/lib/python3.10/site-packages/torch/nn/parallel/data_parallel.py", line 159, in __init__
    _check_balance(self.device_ids)
  File "/home/mateus/miniconda3/envs/TimeMachine_supervised/lib/python3.10/site-packages/torch/nn/parallel/data_parallel.py", line 26, in _check_balance
    dev_props = _get_devices_properties(device_ids)
  File "/home/mateus/miniconda3/envs/TimeMachine_supervised/lib/python3.10/site-packages/torch/_utils.py", line 734, in _get_devices_properties
    return [_get_device_attr(lambda m: m.get_device_properties(i)) for i in device_ids]
  File "/home/mateus/miniconda3/envs/TimeMachine_supervised/lib/python3.10/site-packages/torch/_utils.py", line 734, in <listcomp>
    return [_get_device_attr(lambda m: m.get_device_properties(i)) for i in device_ids]
  File "/home/mateus/miniconda3/envs/TimeMachine_supervised/lib/python3.10/site-packages/torch/_utils.py", line 713, in _get_device_attr
    return get_member(torch.cuda)
  File "/home/mateus/miniconda3/envs/TimeMachine_supervised/lib/python3.10/site-packages/torch/_utils.py", line 734, in <lambda>
    return [_get_device_attr(lambda m: m.get_device_properties(i)) for i in device_ids]
  File "/home/mateus/miniconda3/envs/TimeMachine_supervised/lib/python3.10/site-packages/torch/cuda/__init__.py", line 452, in get_device_properties
    raise AssertionError("Invalid device id")
AssertionError: Invalid device id
Traceback (most recent call last):
  File "/home/mateus/repos/forks/TimeMachine/TimeMachine_supervised/run_longExp.py", line 117, in <module>
    exp = Exp(args)  # set experiments
  File "/home/mateus/repos/forks/TimeMachine/TimeMachine_supervised/exp/exp_main.py", line 24, in __init__
    super(Exp_Main, self).__init__(args)
  File "/home/mateus/repos/forks/TimeMachine/TimeMachine_supervised/exp/exp_basic.py", line 10, in __init__
    self.model = self._build_model().to(self.device)
  File "/home/mateus/repos/forks/TimeMachine/TimeMachine_supervised/exp/exp_main.py", line 33, in _build_model
    model = nn.DataParallel(model, device_ids=self.args.device_ids)
  File "/home/mateus/miniconda3/envs/TimeMachine_supervised/lib/python3.10/site-packages/torch/nn/parallel/data_parallel.py", line 159, in __init__
    _check_balance(self.device_ids)
  File "/home/mateus/miniconda3/envs/TimeMachine_supervised/lib/python3.10/site-packages/torch/nn/parallel/data_parallel.py", line 26, in _check_balance
    dev_props = _get_devices_properties(device_ids)
  File "/home/mateus/miniconda3/envs/TimeMachine_supervised/lib/python3.10/site-packages/torch/_utils.py", line 734, in _get_devices_properties
    return [_get_device_attr(lambda m: m.get_device_properties(i)) for i in device_ids]
  File "/home/mateus/miniconda3/envs/TimeMachine_supervised/lib/python3.10/site-packages/torch/_utils.py", line 734, in <listcomp>
    return [_get_device_attr(lambda m: m.get_device_properties(i)) for i in device_ids]
  File "/home/mateus/miniconda3/envs/TimeMachine_supervised/lib/python3.10/site-packages/torch/_utils.py", line 713, in _get_device_attr
    return get_member(torch.cuda)
  File "/home/mateus/miniconda3/envs/TimeMachine_supervised/lib/python3.10/site-packages/torch/_utils.py", line 734, in <lambda>
    return [_get_device_attr(lambda m: m.get_device_properties(i)) for i in device_ids]
  File "/home/mateus/miniconda3/envs/TimeMachine_supervised/lib/python3.10/site-packages/torch/cuda/__init__.py", line 452, in get_device_properties
    raise AssertionError("Invalid device id")
AssertionError: Invalid device id

My etth1.sh file:

if [ ! -d "./logs" ]; then
    mkdir ./logs
fi

if [ ! -d "./logs/LongForecasting" ]; then
    mkdir ./logs/LongForecasting
fi
if [ ! -d "./csv_results" ]; then
    mkdir ./csv_results
fi
if [ ! -d "./results" ]; then
    mkdir ./results
fi
if [ ! -d "./test_results" ]; then
    mkdir ./test_results
fi
model_name=TimeMachine

root_path_name=../data/ETT-small
data_path_name=ETTh1.csv
model_id_name=ETTh1
data_name=ETTh1

rin=1
random_seed=2024
one=96
two=192
three=336
four=720
residual=1
fc_drop=0.7
dstate=256
dconv=2
for seq_len in 96
do
    for pred_len in 96 192 336 720
    do  
        for e_fact in 1
        do

            if [ $pred_len -eq $one ]
            then
                n1=512
                n2=32
            fi
            if [ $pred_len -eq $two ]
            then
                n1=512
                n2=64
            fi
            if [ $pred_len -eq $three ]
            then
                n1=512
                n2=128
            fi
            if [ $pred_len -eq $four ]
            then
                n1=128
                n2=16
            fi
            python -u run_longExp.py \
            --random_seed $random_seed \
            --is_training 1 \
            --root_path $root_path_name \
            --data_path $data_path_name \
            --model_id $model_id_name_$seq_len'_'$pred_len \
            --model $model_name \
            --data $data_name \
            --features M \
            --seq_len $seq_len \
            --pred_len $pred_len \
            --enc_in 7 \
            --n1 $n1 \
            --n2 $n2 \
            --dropout $fc_drop\
            --revin 1\
            --ch_ind 1\
            --residual $residual\
            --dconv $dconv \
            --d_state $dstate\
            --e_fact $e_fact\
            --des 'Exp' \
            --train_epochs 100\
            --itr 1 --batch_size 64 --learning_rate 0.001 --use_multi_gpu --devices  "1,2" >logs/LongForecasting/$model_name'_'$model_id_name'_'$seq_len'_'$pred_len'_'$n1'_'$n2'_'$fc_drop'_'$rin'_'$residual'_'$dstate'_'$dconv'_'$e_fact.log 
        
        done        
    done
done

I appreciate any guidance or updates you can provide on this issue. Thank you!

Do you only apply transpose MAMBA under channel independent settings?

Thank you for your outstanding work! I was wondering if the transpose mamba is only applied with variable independent settings. If so, what are your considerations for designing it this way? It seems to me that the mamba transpose is modeling relationships between variables. Looking forward to your answers, thanks again!

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.