cutoff-len-is-context-len's Introduction

cutoff-len-is-context-len

Demonstration that finetuning RoPE model on larger sequences than the pre-trained model adapts the model context limit

A Length-Extrapolatable Transformer

Position modeling plays a critical role in Transformers. In this paper, we focus on length extrapolation, i.e., training on short texts while evaluating longer sequences. We define attention resolution as an indicator of extrapolation. Then we propose two designs to improve the above metric of Transformers. Specifically, we introduce a relative position embedding to explicitly maximize attention resolution. Moreover, we use blockwise causal attention during inference for better resolution. We evaluate different Transformer variants with language modeling. Experimental results show that our model achieves strong performance in both interpolation and extrapolation settings. The code will be available at this https URL.

RoFormer: Enhanced Transformer with Rotary Position Embedding

https://arxiv.org/abs/2104.09864

Position encoding recently has shown effective in the transformer architecture. It enables valuable supervision for dependency modeling between elements at different positions of the sequence. In this paper, we first investigate various methods to integrate positional information into the learning process of transformer-based language models. Then, we propose a novel method named Rotary Position Embedding(RoPE) to effectively leverage the positional information. Specifically, the proposed RoPE encodes the absolute position with a rotation matrix and meanwhile incorporates the explicit relative position dependency in self-attention formulation. Notably, RoPE enables valuable properties, including the flexibility of sequence length, decaying inter-token dependency with increasing relative distances, and the capability of equipping the linear self-attention with relative position encoding. Finally, we evaluate the enhanced transformer with rotary position embedding, also called RoFormer, on various long text classification benchmark datasets. Our experiments show that it consistently overcomes its alternatives. Furthermore, we provide a theoretical analysis to explain some experimental results. RoFormer is already integrated into Huggingface: this https URL.

RoPE with xPos implementation

from Lucidrains (Phil Wang)

x-transformers

Adapted for HuggingFace Transformers LLaMa

Dilated RoPE Explanation

https://kaiokendev.github.io/til#extending-context-to-8k

cutoff-len-is-context-len's People

Contributors

Stargazers

Watchers

cutoff-len-is-context-len's Issues

Extend it more

you don't need to change max_position_embeddings here, you can instead add the variable to the model config.json, but you should set it to >2048 to get the correct sinusoidal

class LlamaRotaryEmbedding(torch.nn.Module):
    def __init__(self, dim, max_position_embeddings=2048, base=10000, device=None):
        super().__init__()
        max_position_embeddings = 8192
        inv_freq = 1.0 / (base ** (torch.arange(0, dim, 2).float().to(device) / dim))
        self.register_buffer("inv_freq", inv_freq)
       
        # Build here to make `torch.jit.trace` work.
        self.max_seq_len_cached = max_position_embeddings
        t = torch.arange(self.max_seq_len_cached, device=self.inv_freq.device, dtype=self.inv_freq.dtype)

        # Add these two lines of code, everything else remains the same
        scale = 1/4
        t *= scale
        ....

My current guess is that any fastchat-based code would work: create the model, apply all the patching you need, and then train?
The only question I have is about the dataset: does it have to contain long sequences only?

Recommend Projects