Problem converting Phi3-instruct-128k; "su" rope scaling in Phi-3 about ctranslate2 HOT 3 CLOSED

Did some additional legwork on this "su" scalilng and here's what I came up with...hope it helps, and hope that implementing it still allows someone to use the new flash attention. And as I'm learning, apparently useful when working with large language models to be knowledgeable about a little thing called "math..."

Link to su rope scaling as a jumping off point for ya...

https://huggingface.co/microsoft/Phi-3-mini-128k-instruct/blob/810c2e601a82110f6b709183a8ab5416d4a27f75/modeling_phi3.py#L142

Here's a summary of how it's implemented overall in the script, unless I'm mistaken...

Phi3SuScaledRotaryEmbedding Class

• Inheritance: Inherits from Phi3RotaryEmbedding.
• Initialization:
  • Initializes self.short_factor and self.long_factor from config.rope_scaling.
    • These factors are used to scale the frequency of the rotary embeddings based on the sequence length.
  • Initializes self.original_max_position_embeddings from config.original_max_position_embeddings.
    • This value is used as a threshold to determine whether to apply the short_factor or long_factor scaling.
• Method Overrides:
  • Overrides the forward method to apply "su" rope scaling based on sequence length:
    • If the sequence length is greater than self.original_max_position_embeddings, it applies the long_factor scaling.
    • Otherwise, it applies the short_factor scaling.
    • The scaling is done by multiplying the inverse frequency (self.inv_freq) by the respective factor.
    • The scaled inverse frequency is then used to compute the rotary embeddings.
    • The embeddings are further scaled by a scaling_factor that depends on the ratio of max_position_embeddings to original_max_position_embeddings.
    • The resulting scaled cosine and sine embeddings are returned.

Phi3Attention Class

• Method Details:
  • In the _init_rope method:
    • Checks if self.rope_scaling is not None.
    • If rope scaling configuration is provided, it determines the scaling type based on self.config.rope_scaling["type"].
  • If scaling_type == "su":
    • Initializes self.rotary_emb as an instance of Phi3SuScaledRotaryEmbedding.
    • This ensures that the "su" rope scaling is applied to the rotary embeddings during the attention computation.
  • The Phi3SuScaledRotaryEmbedding instance is created with the appropriate configuration, including the dim (head dimension) and config (model configuration).

from ctranslate2.

Thank you for your information. We don't have time to implement it now. Will try support su rope scaling in the future.

from ctranslate2.

Closing due to it successfully being implemented in release 4.3

from ctranslate2.