Giter VIP home page Giter VIP logo

Comments (4)

fedor-intercom avatar fedor-intercom commented on August 12, 2024 1

We figured out that it's due to not using the new Llama3 tokens. See AWS docs to invoke Llama3

We monkey patched _convert_one_message_to_text_llama (resolving the issue):

from typing import List
from langchain_aws.chat_models import BedrockChat 
import langchain_aws.chat_models.bedrock

def _convert_one_message_to_text_llama(message: BaseMessage) -> str:
    if isinstance(message, ChatMessage):
        message_text = f"<|begin_of_text|><|start_header_id|>{message.role}<|end_header_id|>{message.content}<|eot_id|>"
    elif isinstance(message, HumanMessage):
        message_text = f"<|begin_of_text|><|start_header_id|>user<|end_header_id|>{message.content}<|eot_id|>"
    elif isinstance(message, AIMessage):
        message_text = f"<|begin_of_text|><|start_header_id|>assistant<|end_header_id|>{message.content}<|eot_id|>"
    elif isinstance(message, SystemMessage):
        message_text = f"<|begin_of_text|><|start_header_id|>system<|end_header_id|>{message.content}<|eot_id|>"
    else:
        raise ValueError(f"Got unknown type {message}")
    return message_text

def convert_messages_to_prompt_llama(messages: List[BaseMessage]) -> str:
    """Convert a list of messages to a prompt for llama."""

    return "\n".join(
        [_convert_one_message_to_text_llama(message) for message in messages] + ["<|start_header_id|>assistant<|end_header_id|>\n\n"]
    )

langchain_aws.chat_models.bedrock._convert_one_message_to_text_llama = _convert_one_message_to_text_llama
langchain_aws.chat_models.bedrock.convert_messages_to_prompt_llama = convert_messages_to_prompt_llama

from langchain-aws.

anaszil avatar anaszil commented on August 12, 2024 1

I think "<|begin_of_text|>" should only appear in the begining of the prompt, not in the begining of each message.

Check the template here

from langchain-aws.

bqmackay avatar bqmackay commented on August 12, 2024 1

@anaszil I can confirm that putting "<|begin_of_text|>" at the beginning does work as intended.

When using the Llama3 8b model without the monkey patch, the model will continue a fictitious conversation.

from langchain-aws.

fedor-intercom avatar fedor-intercom commented on August 12, 2024

Thanks, @anaszil and @bqmackay, you are right :).
It's reflected in the PR.

from langchain-aws.

Related Issues (20)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.