Comments (3)
WenWeiTHU
commented on May 23, 2024
您好，感谢您的关注！
您的理解是正确的。在实现上，Transformer是可以接受变长Token输入的，除了在Embedding和FFN层，在Attetion中的W_k, W_q, W_v也是独立做在每个Token上的（映射：d_model -> d_k），和Token个数N无关。
您可以尝试Encoder-only的传统Transformer，其在推理时就可用不同于训练时的时间步(Time Step as Token)。
在Inverted Transformer中，这种推理任意Token的能力则反映在了变量维（Variate as Token）。实际上Attetion中的W_k, W_q, W_v旨在从各种Token中学习如何提取特征，服务Token之间的相关性建模，这是一种变量无关的学习能力，从而这些参数具有泛化到未知的变量上的潜力，我们的实验正是验证了这一点！
代码实现可以参考scripts/variate_generalization下的脚本，其中训练时用到的enc_in都是小于这个数据集的变量数的。而在推理时，推理的是全量变量。
from itransformer.
WenWeiTHU
commented on May 23, 2024
您还提到一个更高维度的问题，这也是目前我们正在尝试的的方向！这个问题的关键一般是如何组织这些Token以及利用他们的先验，或许可以借鉴一时空预测模型的思路，例如 Corrformer
from itransformer.
yangguang760
commented on May 23, 2024
明白了，感谢解惑！
from itransformer.
Related Issues (20)
- How to get prediction without normalized? HOT 1
- Regarding the figures in the article HOT 3
- StandardScaler vs iTransformer normalization HOT 1
- encoder模块和decoder模块 是否做过对比 HOT 1
- Dimension issue for backtesting HOT 1
- 怎么添加自己的数据集 HOT 1
- About the radar figure HOT 1
- 关于输入数据的标准化处理 HOT 1
- 预测结果很差？ HOT 8
- i have a question HOT 1
- Market dataset HOT 1
- 预测结果不好 HOT 3
- Memory Footprint? HOT 1
- 当运行到训练的循环时，内存消耗骤增
- train_loader invocation encountered "RuntimeError: stack expects each tensor to be equal size, but got [35, 34] at entry 0 and [0, 34] at entry 1" HOT 1
- ValueError: could not convert string to float: '2020-01-01 00:20:00' HOT 1
- Error when using a Custom dataset with weekly frequency HOT 2
- Question: Support for Dynamic Categorical Inputs in iTransformer HOT 2
- 无法重现论文中的结果 HOT 2
- How to visualize the results? HOT 5
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from itransformer.