Comments (4)
We figured out that it's due to not using the new Llama3 tokens. See AWS docs to invoke Llama3
We monkey patched _convert_one_message_to_text_llama
(resolving the issue):
from typing import List
from langchain_aws.chat_models import BedrockChat
import langchain_aws.chat_models.bedrock
def _convert_one_message_to_text_llama(message: BaseMessage) -> str:
if isinstance(message, ChatMessage):
message_text = f"<|begin_of_text|><|start_header_id|>{message.role}<|end_header_id|>{message.content}<|eot_id|>"
elif isinstance(message, HumanMessage):
message_text = f"<|begin_of_text|><|start_header_id|>user<|end_header_id|>{message.content}<|eot_id|>"
elif isinstance(message, AIMessage):
message_text = f"<|begin_of_text|><|start_header_id|>assistant<|end_header_id|>{message.content}<|eot_id|>"
elif isinstance(message, SystemMessage):
message_text = f"<|begin_of_text|><|start_header_id|>system<|end_header_id|>{message.content}<|eot_id|>"
else:
raise ValueError(f"Got unknown type {message}")
return message_text
def convert_messages_to_prompt_llama(messages: List[BaseMessage]) -> str:
"""Convert a list of messages to a prompt for llama."""
return "\n".join(
[_convert_one_message_to_text_llama(message) for message in messages] + ["<|start_header_id|>assistant<|end_header_id|>\n\n"]
)
langchain_aws.chat_models.bedrock._convert_one_message_to_text_llama = _convert_one_message_to_text_llama
langchain_aws.chat_models.bedrock.convert_messages_to_prompt_llama = convert_messages_to_prompt_llama
from langchain-aws.
I think "<|begin_of_text|>" should only appear in the begining of the prompt, not in the begining of each message.
Check the template here
from langchain-aws.
@anaszil I can confirm that putting "<|begin_of_text|>" at the beginning does work as intended.
When using the Llama3 8b model without the monkey patch, the model will continue a fictitious conversation.
from langchain-aws.
Thanks, @anaszil and @bqmackay, you are right :).
It's reflected in the PR.
from langchain-aws.
Related Issues (20)
- test
- test HOT 1
- raise TypeError( TypeError: Additional kwargs key generation_token_count already exists in left dict and value has unsupported type <class 'int'> HOT 1
- ChatBedrockConverse.with_structured_output does not allow for snake case parameters
- Ai Messages with Tool Calls do not include the text of Anthropic LLM responses.
- Problems using agent type "Tool calling", with Bedrock LLMs "Claude 3.5 Sonnet" and "Llama 3 70b"
- botocore.errorfactory.ValidationException: An error occurred (ValidationException) when calling the Converse operation: The model returned the following errors: Your API request included an `assistant` message in the final position, which would pre-fill the `assistant` response. When using tools, pre-filling the `assistant` response is not supported. HOT 2
- ChatBedrockConverse Pydantic example working with Haiku but not with Sonnet? HOT 5
- SageMaker streaming not working HOT 2
- ChatBedrock with Converse enabled returns consecutive user messages
- Add ChatBedrockConverse support for document content blocks
- v0.1.14 breaks streaming response rendering from Claude 3.5 Sonnet HOT 5
- Request for Pydantic v2 support HOT 3
- Streaming tool calling with ChatBedrockConverse throws an error with some models HOT 4
- Converse operation: messages.2.content: Conversation blocks and tool result blocks cannot be provided in the same turn. HOT 4
- Llama3 models intermittently don't give responses
- Support numpy v2
- Usage tokens are double counted (Anthropic Sonnet) HOT 4
- RunnableWithMessageHistory -- Error in RootListenersTracer.on_chain_end callback: KeyError('answer') Error in callback coroutine: KeyError('answer')
- Incomplete tool calls with Mistral large 2 via bedrock converse (AWS issue)
Recommend Projects
-
React
A declarative, efficient, and flexible JavaScript library for building user interfaces.
-
Vue.js
🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
-
Typescript
TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
-
TensorFlow
An Open Source Machine Learning Framework for Everyone
-
Django
The Web framework for perfectionists with deadlines.
-
Laravel
A PHP framework for web artisans
-
D3
Bring data to life with SVG, Canvas and HTML. 📊📈🎉
-
Recommend Topics
-
javascript
JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
-
web
Some thing interesting about web. New door for the world.
-
server
A server is a program made to process requests and deliver data to clients.
-
Machine learning
Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
-
Visualization
Some thing interesting about visualization, use data art
-
Game
Some thing interesting about game, make everyone happy.
Recommend Org
-
Facebook
We are working to build community through open source technology. NB: members must have two-factor auth.
-
Microsoft
Open source projects and samples from Microsoft.
-
Google
Google ❤️ Open Source for everyone.
-
Alibaba
Alibaba Open Source for everyone
-
D3
Data-Driven Documents codes.
-
Tencent
China tencent open source team.
from langchain-aws.