See https://ai.google.dev/api/rest/v1beta/files and https://ai.google.dev/api/rest/v1beta/media/upload
See https://github.com/google-gemini/cookbook/blob/main/quickstarts/Video.ipynb

See langchain-ai/langchainjs#5075

Would be great to have support for the RAG API documented at https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/rag-api and exposed within the vertex.preview.rag module in the python SDK.

See https://ai.google.dev/docs/function_calling#function_calling_mode

See langchain-ai/langchainjs#5072

See https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/grounding

See langchain-ai/langchainjs#5073

With ChatGoogleGenerativeAI requests, I need to call my custom AI client.
Currently, I am able to provide dedicated client endpoint with client_options, but I cannot pass additional headers, that are required by this client.

What I want to do:

from langchain_core.messages import HumanMessage
from langchain_google_genai import ChatGoogleGenerativeAI

REQUIRED_HEADERS = {
    "X-User-Header": "Coco",
    "X-User-Header2": "Jambo",
}

chat = ChatGoogleGenerativeAI(
    model="gemini-pro",
    google_api_key="[secret]",
    client_options={"api_endpoint": "http://127.0.0.1:8000/ai"},
    transport="rest",
    additional_headers=REQUIRED_HEADERS,  # new field, currently not exists
)
message = HumanMessage(content=[{"text": "Who is Maria Skłodowska-Curie?", "type": "text"}])
response = chat.invoke([message])

Technically, this change is very simple, as all the pieces are almost in place.
In fact, all that needs to be done is to pass the additional parameter to genai.congigure() method, as shown below:

genai.configure(
    api_key=google_api_key,
    transport=values.get("transport"),
    client_options=values.get("client_options"),
    default_metadata=default_metadata,  # <--------
)

Please check out an example of smallest working change, implementing the mentioned functionality: #112

Also, I prepared a small repository when I am testing current implementation and the proposed change against some simplified use case. Please check out the repository:
https://github.com/Hunter71/langchain-headers

Hello,

I propose a langchain Google Drive integration.
May be, we can merge our projects.

Hi there, I am trying to run the code below using the langchain-google-vertexai Python package where I want to do function calling using the gemini-pro model. But I am getting the following error message AttributeError: 'dict' object has no attribute 'schema' at line 22 of the libs/vertexai/langchain_google_vertexai/functions_utils.py file when I run code below:

from langchain import hub
from langchain.agents import AgentExecutor, create_openai_functions_agent
from langchain_community.tools.tavily_search import TavilySearchResults
from langchain_google_vertexai import ChatVertexAI

tavily_tool = TavilySearchResults(max_results=1)
tools = [tavily_tool]

instructions = """You are an assistant."""
base_prompt = hub.pull("langchain-ai/openai-functions-template")
prompt = base_prompt.partial(instructions=instructions)

llm = ChatVertexAI(model_name="gemini-pro", temperature=0.0, convert_system_message_to_human=True)

agent = create_openai_functions_agent(llm, tools, prompt)

agent_executor = AgentExecutor(agent=agent, tools=tools, verbose=True)

agent_executor.invoke({"input": "what is LangChain?"})

Can you please help me resolve this error? The above code runs fine with the langchain-google-genai package using ChatGoogleGenerativeAI instead of ChatVertexAI. Thanks!

Checked other resources

• I added a very descriptive title to this issue.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

import asyncio
from langchain_google_vertexai import ChatVertexAI
from langchain_openai import ChatOpenAI
from langchain.callbacks.base import BaseCallbackHandler
from langchain.schema import HumanMessage

class MyCallbackHandler(BaseCallbackHandler):

    def on_llm_start(self, serialized, prompts, **kwargs):
        print("  on_llm_start")

    def on_llm_end(self, repsonse, **kwargs):
        print("  on_llm_end")

    def on_llm_new_token(self, token, **kwargs):
        print("  on_llm_new_token", token)


async def main():
    # works

    print("openai: start")
    await ChatOpenAI(streaming=True).agenerate(
        [[HumanMessage(content="Hello, how are you?")]],
        callbacks=[MyCallbackHandler()],
    )
    print("openai: done")

    # fails
    print("vertex: start")
    await ChatVertexAI(streaming=True, model_name="gemini-pro").agenerate(
        [[HumanMessage(content="Hello, how are you?")]],
        callbacks=[MyCallbackHandler()],
        # stream=True ## <--- works if you add this, but isn't necessary for OpenAI
    )
    print("vertex: done")


if __name__ == "__main__":
    asyncio.run(main())

Error Message and Stack Trace (if applicable)

No response

Description

I expect ChatVertexAI to behave like other providers: accept streaming=True in the constructor and then invoke the on_llm_new_token callback for streaming operations.

The provided example gives the following output, which shows the inconsistency:

openai: start
  on_llm_start
  on_llm_new_token
  on_llm_new_token Hello
  on_llm_new_token !
  on_llm_new_token  I
  on_llm_new_token 'm
  on_llm_new_token  just
  on_llm_new_token  a
  on_llm_new_token  computer
  on_llm_new_token  program
  on_llm_new_token ,
  on_llm_new_token  so
  on_llm_new_token  I
  on_llm_new_token  don
  on_llm_new_token 't
  on_llm_new_token  have
  on_llm_new_token  feelings
  on_llm_new_token ,
  on_llm_new_token  but
  on_llm_new_token  I
  on_llm_new_token 'm
  on_llm_new_token  here
  on_llm_new_token  to
  on_llm_new_token  help
  on_llm_new_token  you
  on_llm_new_token .
  on_llm_new_token  How
  on_llm_new_token  can
  on_llm_new_token  I
  on_llm_new_token  assist
  on_llm_new_token  you
  on_llm_new_token  today
  on_llm_new_token ?
  on_llm_new_token
  on_llm_end
openai: done
vertex: start
  on_llm_start
  on_llm_end
vertex: done

If you add stream=True to the agenerate call, the callbacks are correctly invoked. So it seems it's the handling of the constructor param that is inconsistent with other providers.

System Info

System Information

OS: Darwin
OS Version: Darwin Kernel Version 22.6.0: Thu Nov 2 07:43:57 PDT 2023; root:xnu-8796.141.3.701.17~6/RELEASE_ARM64_T6000
Python Version: 3.11.6 (main, Nov 2 2023, 04:39:43) [Clang 14.0.3 (clang-1403.0.22.14.1)]

Package Information

langchain_core: 0.1.36
langchain: 0.1.13
langchain_community: 0.0.29
langsmith: 0.1.22
langchain_google_vertexai: 0.1.2
langchain_openai: 0.0.8
langchain_text_splitters: 0.0.1

Packages not installed (Not Necessarily a Problem)

The following packages were not found:

langgraph
langserve

With release v0.5.0 of google-generativeai, there is a new argument for system messages. Therefore,

ValueError: SystemMessages are not yet supported!

doesn't have to be overcome using convert_system_message_to_human anymore.

Can you add the access_token parameter to the class and pass it to the Anthropic classes in libs/vertexai/langchain_google_vertexai/model_garden.py

    async_client: Any = None  #: :meta private:
    model_name: Optional[str] = Field(default=None, alias="model")  # type: ignore[assignment]
    "Underlying model name."
    max_output_tokens: int = Field(default=1024, alias="max_tokens")
    access_token: Optional[str] = None

    class Config:
        """Configuration for this pydantic object."""

        allow_population_by_field_name = True

    @root_validator()
    def validate_environment(cls, values: Dict) -> Dict:
        from anthropic import (  # type: ignore[import-not-found]
            AnthropicVertex,
            AsyncAnthropicVertex,
        )

        values["client"] = AnthropicVertex(
            project_id=values["project"],
            region=values["location"],
            max_retries=values["max_retries"],
            access_token=values["access_token"],
        )
        values["async_client"] = AsyncAnthropicVertex(
            project_id=values["project"],
            region=values["location"],
            max_retries=values["max_retries"],
            access_token=values["access_token"],
        )
        return values

The recently introduced telemetry context managers in #31 by @holtskinner causes issue with parallelisation.

Given that the generation is an io task, concurrency is a crucial aspect that cannot be overlooked.

If multiple models are awaited in parallel, they all seem to append to same list causing exception on exit

here's an reproducible example

from langchain_google_vertexai import ChatVertexAI
from langchain_core.prompts import ChatPromptTemplate
import asyncio


model1 = ChatVertexAI(model_name="chat-bison@002")
model2 = ChatVertexAI(model_name="gemini-1.0-pro")


chain1 = ChatPromptTemplate.from_messages(["Why is the sky blue?"]) | model1

chain2 = ChatPromptTemplate.from_messages(["Why is the sky blue?"]) | model2


async def main():
    async with asyncio.TaskGroup() as tg:
        tg.create_task(chain1.ainvoke({}))
        tg.create_task(chain2.ainvoke({}))


asyncio.run(main())

Exception:

 | Traceback (most recent call last):
    |   File "/Users/sufiyan/micromamba/envs/nlp-cloud-server/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 2536, in ainvoke
    |     input = await step.ainvoke(
    |             ^^^^^^^^^^^^^^^^^^^
    |   File "/Users/sufiyan/micromamba/envs/nlp-cloud-server/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 179, in ainvoke
    |     llm_result = await self.agenerate_prompt(
    |                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Users/sufiyan/micromamba/envs/nlp-cloud-server/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 570, in agenerate_prompt
    |     return await self.agenerate(
    |            ^^^^^^^^^^^^^^^^^^^^^
    |   File "/Users/sufiyan/micromamba/envs/nlp-cloud-server/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 530, in agenerate
    |     raise exceptions[0]
    |   File "/Users/sufiyan/micromamba/envs/nlp-cloud-server/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 715, in _agenerate_with_cache
    |     result = await self._agenerate(
    |              ^^^^^^^^^^^^^^^^^^^^^^
    |   File "/Users/sufiyan/micromamba/envs/nlp-cloud-server/lib/python3.11/site-packages/langchain_google_vertexai/chat_models.py", line 635, in _agenerate
    |     with telemetry.tool_context_manager(self._user_agent):
    |   File "/Users/sufiyan/micromamba/envs/nlp-cloud-server/lib/python3.11/contextlib.py", line 144, in __exit__
    |     next(self.gen)
    |   File "/Users/sufiyan/micromamba/envs/nlp-cloud-server/lib/python3.11/site-packages/google/cloud/aiplatform/telemetry.py", line 48, in tool_context_manager
    |     _pop_tool_name(tool_name)
    |   File "/Users/sufiyan/micromamba/envs/nlp-cloud-server/lib/python3.11/site-packages/google/cloud/aiplatform/telemetry.py", line 57, in _pop_tool_name
    |     raise RuntimeError(
    | RuntimeError: Tool context error detected. This can occur due to parallelization.
    +------------------------------------

User report of

Have you seen this with the google stuff?
ImportError: cannot import name 'telemetry' from 'google.cloud.aiplatform' (/usr/local/lib/python3.10/dist-packages/google/cloud/aiplatform/init.py)
on import of from langchain_google_vertexai import VertexAIEmbeddings, ChatVertexAI

Curious if maybe telemetry isn't available in a version of google-cloud-aiplatform that is acceptable in the version string in vertex pyproject.

Could also add minimum version release testing for that package in CI (along with langchain-core)

Checked other resources

• I added a very descriptive title to this issue.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

Taken from the following discussion

Description
I am trying to pass a PDF document to a gemini-1.5-pro in multimodal mode, following a process similar to the one explained here. The documentation illustrates how to pass an image and query Gemini Pro Vision, but I want to pass a PDF directly instead.

Here is my attempt:

from langchain_core.messages import HumanMessage
from langchain_google_vertexai import ChatVertexAI
import base64

file_location = "/path/to/my/document.pdf"

# Initialize the LangChain LLM
llm = ChatVertexAI(model_name="gemini-1.5-pro-preview-0409")

# Open and read the PDF file
with open(file_location, "rb") as pdf_file:
    pdf_bytes = pdf_file.read()

# Create a message containing the Base64-encoded PDF
pdf_message = {
    "type": "image_url",  # Assuming the LLM accepts PDF under this key, you might need to verify this
    "image_url": {
        "url": f"data:application/pdf;base64,{base64.b64encode(pdf_bytes).decode('utf-8')}"
    },
}

# Create a text message asking what the PDF contains
text_message = {
    "type": "text",
    "text": "What does this PDF contain?",
}

# Combine the messages into a HumanMessage object
message = HumanMessage(content=[text_message, pdf_message])

# Send the message to the LLM and print the response
output = llm.invoke([message])
print(output.content)

Unfortunately this code fails.
However, if I use the official Vertex AI library, I am able to do it. Here is part of my code:

from vertexai.generative_models import GenerativeModel, Part

model = GenerativeModel("gemini-1.5-pro-preview-0409")

prompt = "please summarise the provided pdf document"

pdf_file_uri = "gs://my_bucket/my_document.pdf"
pdf_file = Part.from_uri(pdf_file_uri, mime_type="application/pdf")
contents = [pdf_file, prompt]

response = model.generate_content(contents)
print(response.text)

This approach works, but I was hoping to make the LangChain method function similarly.

System Info
System Information
OS: Darwin
OS Version: Darwin Kernel Version 23.4.0: Fri Mar 15 00:11:05 PDT 2024; root:xnu-10063.101.17~1/RELEASE_X86_64
Python Version: 3.11.5 (main, Sep 11 2023, 08:19:27) [Clang 14.0.6 ]

Package Information

langchain_core: 0.1.40
langchain: 0.1.14
langchain_community: 0.0.31
langsmith: 0.1.40
langchain_google_genai: 0.0.5
langchain_google_vertexai: 0.1.2
langchain_openai: 0.1.1
langchain_text_splitters: 0.0.1

Packages not installed (Not Necessarily a Problem)
The following packages were not found:

langgraph
langserve
111

### Error Message and Stack Trace (if applicable)

_No response_

### Description

I am trying to pass a PDF document to a gemini-1.5-pro in multimodal mode, following a process similar to the one explained [here](https://python.langchain.com/docs/integrations/llms/google_vertex_ai_palm/#multimodality). The documentation illustrates how to pass an image and query Gemini Pro Vision, but I want to pass a PDF directly instead.

This native Vertex AI library feature is not implemented within LangChain. I am happy to contribute and implement; would appreciate some pointers on where to start as a new contributer.

### System Info

System Info
System Information
OS: Darwin
OS Version: Darwin Kernel Version 23.4.0: Fri Mar 15 00:11:05 PDT 2024; root:xnu-10063.101.17~1/RELEASE_X86_64
Python Version: 3.11.5 (main, Sep 11 2023, 08:19:27) [Clang 14.0.6 ]

I'm happy to contribute to the repo btw after getting validation that this makes sense as a feature.

I want to be able to upload text files once and be able to reference them.

Part.from_uri is only referenced in image utils here: libs/vertexai/langchain_google_vertexai/_image_utils.py#L80-L97

but in the vertexai module, that supports arbitrary mime types like the following example.

import vertexai
from vertexai.generative_models import GenerativeModel, Part

gemini_model = GenerativeModel(model_name="gemini-1.5-pro-preview-0409")

model_response = gemini_model.generate_content([
   "summarize the readme",
   Part.from_uri("gs://example-bucket/some-directory/README.md", "text/markdown"),
])

print(model_response)

https://ai.google.dev/api/python/google/ai/generativelanguage/Message

Will need to bump langchain-core minimum version to 0.1.26 in order to make sure BaseMessage.name attribute exists

When I run the code below, always got server 500 error.

def multiply(first_int: int, second_int: int) -> int:
    """Multiply two integers together."""
    return first_int * second_int


@tool
def add(first_int: int, second_int: int) -> int:
    "Add two integers."
    return first_int + second_int

@tool
def exponentiate(base: int, exponent: int) -> int:
    "Exponentiate the base to the exponent power."
    return base**exponent


tools = [multiply, add, exponentiate]
tool_executor = ToolExecutor(tools)
functions = [convert_to_openai_tool(t)["function"] for t in tools]
model = ChatGoogleGenerativeAI(model='gemini-pro', temperature=0)
model_with_tools = model.bind(functions=functions)

message = "What's four times 23"
result = model_with_tools.invoke(message)
check = hasattr(result, 'additional_kwargs')

try:
    action = ToolInvocation(
        tool=result.additional_kwargs["function_call"]["name"],
        tool_input=json.loads(
            result.additional_kwargs["function_call"]["arguments"]),
    )
    response = tool_executor.invoke(action)
    print(response)
except KeyError:
    result = model.invoke(message)
    print(result.content)

I think the code below

glm.Tool(
            function_declarations=[_convert_to_genai_function(fc) for fc in function_calls],
        )

eg.

calculator = glm.Tool(
    function_declarations=[
      glm.FunctionDeclaration(
        name='add',
        description="Returns the sum of two numbers.",
        parameters=glm.Schema(
            type=glm.Type.OBJECT,
            properties={
                'a': glm.Schema(type=glm.Type.NUMBER),
                'b': glm.Schema(type=glm.Type.NUMBER)
            },
            required=['a','b']
        )
      ),
      glm.FunctionDeclaration(
        name='multiply',
        description="Returns the product of two numbers.",
        parameters=glm.Schema(
            type=glm.Type.OBJECT,
            properties={
                'a':glm.Schema(type=glm.Type.NUMBER),
                'b':glm.Schema(type=glm.Type.NUMBER)
            },
            required=['a','b']
        )
      )
    ])

Issue: genai Semantic Retrieval example missing required import for GoogleVectorStore
File: libs/genai/README.md
Required Import:
from langchain_google_genai import GoogleVectorStore

Additional Info:

• after resolving the import issue I started getting "DefaultCredentialsError" errors.
• I suspect this is due to multiple Credential types existing on the target Google project
  • API keys, OAuth, Service Account

Description

Since version 1.43.0, google-vertexai-aiplatform has added transport override to enable the use of REST instead of GRPC (6ab4084).
This is a key feature for enterprise where security setup prevent gRPC communications, and thus for large scale adoption.

Current behavior

from langchain_google_vertexai import VertexAI
llm = VertexAI(model_name="text-bison@002", temperature=0, location="europe-west4", api_transport="rest")

leads to a gRPC error message (expected here as gRPC is filtered by organisation)

E0329 15:25:35.266053000 10916507648 ssl_transport_security.cc:1519]   Handshake failed with fatal error SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED.
I0329 15:25:35.266485000 10916507648 subchannel.cc:744]                subchannel 0x105f0b7d0 {address=ipv4:1.2.4.5:3128, args={grpc.client_channel_factory=0x16c067ac0, grpc.default_authority=europe-west4-aiplatform.googleapis.com:443, grpc.dns_enable_srv_queries=1, grpc.http2_scheme=https, grpc.http_connect_headers=Proxy-Authorization:Basic d-----------------j, grpc.http_connect_server=europe-west4-aiplatform.googleapis.com:443, grpc.internal.channel_credentials=0x16c068c90, grpc.internal.event_engine=0x105f09d30, grpc.internal.security_connector=0x105f0afe0, grpc.internal.subchannel_pool=0x28a36f530, grpc.internal.transport=(nil), grpc.max_receive_message_length=-1, grpc.max_send_message_length=-1, grpc.primary_user_agent=grpc-python/1.62.0, grpc.resource_quota=0x28a343f50, grpc.server_uri=dns:///europe-west4-aiplatform.googleapis.com:443}}: connect failed (UNKNOWN:Ssl handshake failed: SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED {created_time:"2024-03-29T15:25:35.266102+01:00", tsi_error:"TSI_PROTOCOL_FAILURE", tsi_code:10}), backing off for -4543 ms

Expected behavior

api_transport="rest"is transparently passed as transport to the vertexai.language_models Models

With any error message (if any) being related to HTTP, not gRPC.

Each text should be able to have a list of metadatas, however, current version only accepts List[dict]

Our Chat LLM doesn't support tuned models yet.

Source

There are several properties in aiplatform.MatchingEngineIndexEndpoint that are not supported in langchain-google. These need to be added for support in the VectorSearchVectorStore.from_components() method. The more critical properties for enterprise adoption are the private-service-connect properties shown below.

• private-service-connect
• MatchingEngineIndexEndpoint_private_service_connect_ip_address

Checked other resources

• I added a very descriptive title to this issue.
• I searched the LangChain documentation with the integrated search.
• I used the GitHub search to find a similar question and didn't find it.
• I am sure that this is a bug in LangChain rather than my code.
• The bug is not resolved by updating to the latest stable version of LangChain (or the specific integration package).

Example Code

import os
os.environ['GOOGLE_API_KEY'] = '<api key>'

import requests
import base64

image_url = "https://picsum.photos/seed/picsum/300/300"
content = requests.get(image_url).content

image_bytes = base64.b64encode(requests.get(image_url).content).decode("utf-8")

image_url = f"data:image/jpg;base64,{image_bytes}"

from langchain_core.messages import HumanMessage
from langchain_google_genai import ChatGoogleGenerativeAI

llm = ChatGoogleGenerativeAI(model="gemini-pro-vision")
# example

content = [
        {
            "type": "text",
            "text": "What's in this image?",
        },  # You can optionally provide text parts
        {"type": "image_url", "image_url": image_url},
    ]

message = HumanMessage(
    content=content
)

stream = True

if stream:
    for chunk in llm.stream([message]):
        print(chunk.content)
        print('---')
else:
    res = llm.invoke([message])
    print(res)

Error Message and Stack Trace (if applicable)

google.api_core.exceptions.InternalServerError: 500 An internal error has occurred. Please retry or report in https://developers.generativeai.google/guide/troubleshooting

Description

streaming with gemini-pro-vision fails with internal server error.

System Info

Using latest google packages

langchain==0.0.354
langchain-anthropic==0.1.3
langchain-community==0.0.8
langchain-core==0.1.6
langchain-experimental==0.0.47
langchain-google-genai==0.0.11
langchain-groq==0.0.1
langchain-mistralai==0.0.2

google-ai-generativelanguage==0.4.0
google-api-core==2.17.1
google-auth==2.28.2
google-generativeai==0.4.1
google-search-results==2.4.2
googleapis-common-protos==1.63.0
langchain-google-genai==0.0.11

Linux Ubutnu 22
python 3.10

but new langchain does same.

I have an AI Agent with an OpenAI LLM using LCEL, I'm trying to replace the LLM with Gemini Pro but both ChatVertexAI and ChatGoogleGenerativeAI are unable to convert a custom tool I have, and that previously worked fine with OpenAI Chat, apparently it errors when it tries to get the parameters from the schema since one of the input fields is an ENUM type.

The error that I get is:

.venv\Lib\site-packages\langchain_google_genai\_function_utils.py", line 74, in <dictcomp>
    "type_": TYPE_ENUM[v["type"]],
                       ~^^^^^^^^
KeyError: 'type'

Note that

I'm assuming the problem is that when _convert_tool_to_genai_function is running and tries to define the parameters for the FunctionDeclaration looping through the schema properties it tries to access they "type" Key which the ENUM field doesn't have directly accessible and instead has a "allOf" Key that then contains those values.

Here's my code.
First I initialize the LLM for my Agent and I bind the tools to it:

llm = ChatGoogleGenerativeAI(  # type: ignore
    model="gemini-1.0-pro-latest",
    temperature=0,
    google_api_key=GOOGLE_API_KEY,
    convert_system_message_to_human=True,
    verbose=True,
)

llm_with_tools = llm.bind(functions=tools)

My custom tool looks kinda like this:

class LogTool(BaseTool):
    "Tool for generating a log"

    name = "generate_log"
    description = """
            Useful when you want to generate a log. 
            P = Present,
            A = Person was absent,
            H = Holiday,
            TN = You do not have your Temporary person for the day.
            """

    args_schema: Type[BaseModel] = Log

    @staticmethod
    def generate_log(name: str, status: str):
        "Create log data"

        attendance_log = {
            "name": name,
            "date": datetime.now().strftime("%m/%d/%Y"),
            "status": status,
        }

        return [attendance_log]

    def _run(self, name: str, status: str):
        response = self.generate_log(name, status)
        return response

The BaseModel for the input schema looks like this:

class Log(BaseModel):
    "Input data for generating a log"

    name: str = Field(description="Name of the person")
    status: StatusEnum = Field(description="Status of the log for the person")

And the ENUM:

class StatusEnum(str, Enum):
    "Enum for status of the log: P, A, H, TN"

    P = "Present"
    A = "Absent"
    H = "Holiday"
    TN = "Temporary person"

Langchain core recently introduced the new API method bind_tools on the BaseChatModel class and all the internal tool handling that comes with it.

AIMessage and AIMessageChunk instances are also expected to contain the tool call instructions exclusively in "tool_calls" or "tool_call_chunk" attributes.

Google GenAI chat models do not comply with the new API yet, which makes building model agnostic, generic applications cumbersome. It would be nice if all chat models encapsulate tool calls and result handing in the same way.

It's only on ChatVertexAI currently

https://python.langchain.com/docs/guides/structured_output

I have found a bug while matching api key on kaggle.

This is the getting error on kaggle notebook

Here defualt metadata is tuple datatype

###Here default metadata is of list datatype

May I fix this issue? Thanks

Right now, a user receives an empty string of text is a Safety filter blocks the response rather than the information about the safety filters (such as the probability, etc.) It would be great to have the safety filter dictionary returned as part of prompt feedback, similar to the non-VertexAI Gemini implementation

I am using ChatGoogleGenerativeAI (langchain=0.1.11, langchain-google-genai=0.0.9). Once I get the response from Google Gemini Pro I see the below error:

.venv\lib\site-packages\langchain_google_genai\chat_models.py", line 414, in _response_to_result
generation_info["finish_reason"] = candidate.finish_reason.name
AttributeError: 'int' object has no attribute 'name'

When I print candidate.finish_reason I get the value: 1 (Expecting STOP)

It appears that credentials are not passed/working when using ChatAnthropicVertex. They appear to work correctly with ChatVertexAI.

Working code

    encoded_json_credentials = get_secret("GOOGLE_VERTEX_SERVICE_ACCOUNT_CREDENTIALS")
    decoded_json_credentials = json.loads(base64.b64decode(encoded_json_credentials).decode("utf-8"))
    credentials = service_account.Credentials.from_service_account_info(decoded_json_credentials)
    return ChatVertexAI(
        model=model,
        location="us-central1",
        credentials=credentials,
        project=get_secret("GOOGLE_VERTEX_PROJECT_ID"),
        **kwargs
    )

Not working code

    encoded_json_credentials = get_secret("GOOGLE_VERTEX_SERVICE_ACCOUNT_CREDENTIALS")
    decoded_json_credentials = json.loads(base64.b64decode(encoded_json_credentials).decode("utf-8"))
    credentials = service_account.Credentials.from_service_account_info(decoded_json_credentials)
    return ChatAnthropicVertex(
        model=model,
        location="us-central1",
        credentials=credentials,
        project=get_secret("GOOGLE_VERTEX_PROJECT_ID"),
        **kwargs
    )

The following error message occurs when trying to use ChatAnthropicVertex

ERROR:root:Your default credentials were not found. To set up Application Default Credentials, see https://cloud.google.com/docs/authentication/external/set-up-adc for more information.

Hi all, I have been using GoogleGenerativeAI in my code. However, in my environment I am using google credentials instead of an api key and the GoogleGenerativeAI model validation does not allow me to not specify an api key. I would be interested in contributing this feature but would like some guidance.

My personal workaround has been to override the root validator (mostly copy-paste) as follows:

import google.generativeai as genai
from langchain_google_genai import ChatGoogleGenerativeAI
from overrides import overrides
from pydantic.v1 import root_validator

from myapp.clients.google_client import get_creds


# overriding so we can use credentials to build instead of api key
class MyGoogleGenerativeAI(ChatGoogleGenerativeAI):
    @root_validator()
    def validate_environment(cls, values: dict) -> dict:
        #google_api_key = get_from_dict_or_env(
        #    values, "google_api_key", "GOOGLE_API_KEY"
        #)
        #if isinstance(google_api_key, SecretStr):
        #    google_api_key = google_api_key.get_secret_value()

        genai.configure(
            credentials=get_creds(), # <<< type google.auth.credentials.Credentials
            # api_key=google_api_key,
            transport=values.get("transport"),
            client_options=values.get("client_options"),
        )
        if values.get("temperature") is not None and not 0 <= values["temperature"] <= 1:
            raise ValueError("temperature must be in the range [0.0, 1.0]")

        if values.get("top_p") is not None and not 0 <= values["top_p"] <= 1:
            raise ValueError("top_p must be in the range [0.0, 1.0]")

        if values.get("top_k") is not None and values["top_k"] <= 0:
            raise ValueError("top_k must be positive")
        model = values["model"]
        values["client"] = genai.GenerativeModel(model_name=model)
        return values


llm = MyGoogleGenerativeAI(model="gemini-pro")

result = llm.invoke("Write a ballad about LangChain")
print(result.content)

My plan to contribute this feature was as follows:

• Add credentials: Optional[Credentials] to the _BaseGoogleGenerativeAI class
• In ChatGoogleGenerativeAI and GoogleGenerativeAI root validator, check if credentials attribute exists and prioritize using it for genai.configure() over the google_api_key logic

However, in my testing the Credentials object is not schema-able in pydantic:

from google.auth import credentials as ga_credentials
from pydantic import BaseModel
class A(BaseModel):
    creds: ga_credentials.Credentials
    
Traceback (most recent call last):
  File "/Users/aperson/Library/Caches/pypoetry/virtualenvs/myapp-YeMax3Mq-py3.11/lib/python3.11/site-packages/IPython/core/interactiveshell.py", line 3577, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-6-d64b82d4a250>", line 1, in <module>
    class A(BaseModel):
  File "/Users/aperson/Library/Caches/pypoetry/virtualenvs/myapp-YeMax3Mq-py3.11/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 183, in __new__
    complete_model_class(
  File "/Users/aperson/Library/Caches/pypoetry/virtualenvs/myapp-YeMax3Mq-py3.11/lib/python3.11/site-packages/pydantic/_internal/_model_construction.py", line 517, in complete_model_class
    schema = cls.__get_pydantic_core_schema__(cls, handler)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  [...]
  File "/Users/aperson/Library/Caches/pypoetry/virtualenvs/myapp-YeMax3Mq-py3.11/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 851, in match_type
    return self._unknown_type_schema(obj)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/aperson/Library/Caches/pypoetry/virtualenvs/myapp-YeMax3Mq-py3.11/lib/python3.11/site-packages/pydantic/_internal/_generate_schema.py", line 397, in _unknown_type_schema
    raise PydanticSchemaGenerationError(
pydantic.errors.PydanticSchemaGenerationError: Unable to generate pydantic-core schema for <class 'google.auth.credentials.Credentials'>. Set `arbitrary_types_allowed=True` in the model_config to ignore this error or implement `__get_pydantic_core_schema__` on your type to fully support it.
If you got this error by calling handler(<some type>) within `__get_pydantic_core_schema__` then you likely need to call `handler.generate_schema(<some type>)` since we do not call `__get_pydantic_core_schema__` on `<some type>` otherwise to avoid infinite recursion.
For further information visit https://errors.pydantic.dev/2.6/u/schema-for-unknown-type                                                           

Does anyone have advice on how to implement this so we can instantiate _BaseGoogleGenerativeAI using a google.auth.credentials.Credentials credentials object instead of an api key?

Hi!

Thank you for integrating gemini models into langchain.

However, I noticed the following:

1. for gemini models, we set the name directly (e.g. "gemini-pro")
2. For some other models, we need to prefix it with "models/" (e.g. "models/text-bison-001")
3. Similarly, for embedding it uses the prefix (e.g. "models/embedding-001")

Can we make the model naming convention more consistent to avoid confusion?

Hi.

For the ChatGoogleGenerativeAI, we pass the following paramters:

 chat = ChatGoogleGenerativeAI(
        google_api_key=api_key,
        max_tokens=max_tokens,
        model=model,
    )

However, for openai and antrhopic, we use model_name and not model:

 chat = ChatAnthropic(
        temperature=0,
        anthropic_api_key=api_key,
        model_name=model,
    )
 chat = ChatOpenAI(
        openai_api_key=api_key,
        max_tokens=max_tokens,
        model_name=model,
    )

Can we make the interface of chat objects consistent across providers?
We can add a parameter model_name and deprecate model going forward.

I'm using a simple RAG chain (sample code below). When using ChatVertexAI as my LLM I get the following error only for certain question prompts. It works fine for some prompts but throws the IndexError for others. Using the same prompt always results in the error below. Changing the LLM for AzureAI solves the problem.

Error Log:
IndexError('list index out of range')Traceback (most recent call last):

File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 1979, in _atransform_stream_with_config
chunk: Output = await asyncio.create_task( # type: ignore[call-arg]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/langchain_core/tracers/log_stream.py", line 237, in tap_output_aiter
async for chunk in output:

File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 2853, in _atransform
async for output in final_pipeline:

File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 4734, in atransform
async for item in self.bound.atransform(

File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 2883, in atransform
async for chunk in self._atransform_stream_with_config(

File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 1979, in _atransform_stream_with_config
chunk: Output = await asyncio.create_task( # type: ignore[call-arg]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/langchain_core/tracers/log_stream.py", line 237, in tap_output_aiter
async for chunk in output:

File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 2853, in _atransform
async for output in final_pipeline:

File "/usr/local/lib/python3.11/site-packages/langchain_core/output_parsers/transform.py", line 60, in atransform
async for chunk in self._atransform_stream_with_config(

File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 1979, in _atransform_stream_with_config
chunk: Output = await asyncio.create_task( # type: ignore[call-arg]
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/langchain_core/tracers/log_stream.py", line 237, in tap_output_aiter
async for chunk in output:

File "/usr/local/lib/python3.11/site-packages/langchain_core/output_parsers/transform.py", line 38, in _atransform
async for chunk in input:

File "/usr/local/lib/python3.11/site-packages/langchain_core/utils/aiter.py", line 97, in tee_peer
item = await iterator.anext()
^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/langchain_core/runnables/base.py", line 1333, in atransform
async for output in self.astream(final, config, **kwargs):

File "/usr/local/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 315, in astream
raise e

File "/usr/local/lib/python3.11/site-packages/langchain_core/language_models/chat_models.py", line 293, in astream
async for chunk in self._astream(

File "/usr/local/lib/python3.11/site-packages/langchain_google_vertexai/chat_models.py", line 768, in _astream
message = _parse_response_candidate(chunk.candidates[0], streaming=True)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^

File "/usr/local/lib/python3.11/site-packages/langchain_google_vertexai/chat_models.py", line 325, in _parse_response_candidate
first_part = response_candidate.content.parts[0]
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^
IndexError: list index out of range

Sample Code

#  Vertex AI
llm = ChatVertexAI(model_name="gemini-1.5-pro-preview-0409")
retriever = get_retriever()
answer_chain = create_chain(
    llm,
    retriever,
)
return answer_chain

Library Versions
langchain 0.1.16
langchain-google-vertexai 1.0.1

Maybe related
langchain-ai/langchain#17800

Hello!

I'd like to use langchain with the Claude models from Anthropic coming from Vertex AI.

To leverage these models in Python, we have to use a function from anthropic called AnthropicVertex but this cannot be integrated as a langchain model.
AFAIK we cannot bring the Claude models from langchain-anthropic as the auth method is different: https://github.com/langchain-ai/langchain/tree/master/libs/partners/anthropic

Could this be created here with the VertexAI function? Or is there any alternative to bring this model with langchain?

Thank you!

Hi everyone,

I started to work with the GoogleDriveLoader() and I recognized that there might be an issue with the oauth flow. Let's assume the following setup:

.env file:
GOOGLE_APPLICATION_CREDENTIALS=""

main.py:

loader = GoogleDriveLoader(
        folder_id="XXXXX",
        credentials_path="XXXX/google_credentials.json",
        token_path="XXXX/google_token.json",
        recursive=False,
    )

With this setup, the GoogleDriveLoader() creates a local auth server during local development that always uses a random port. This leads to the problem that my redirect url that I configured in the Google Cloud never works. So I'm wondering why this class creates a local server and if so, why can't we set the port ourselves?

The code which creates this behavior is in line 191:

        if not creds or not creds.valid:
            if creds and creds.expired and creds.refresh_token:
                creds.refresh(Request())
            elif "GOOGLE_APPLICATION_CREDENTIALS" not in os.environ:
                creds, project = default()
                creds = creds.with_scopes(SCOPES)
                # no need to write to file
                if creds:
                    return creds
            else:
                flow = InstalledAppFlow.from_client_secrets_file(
                    str(self.credentials_path), SCOPES
                )
              # Why do we set port 0 -> leads to random port? 
             # Why do we create a local server?
                creds = flow.run_local_server(port=0) 
            with open(self.token_path, "w") as token:
                token.write(creds.to_json())

I can solve this issue by setting the port to my preffered one eg 8080. Anyone who has an idea?

I'm currently trying to add support for Gemma to my LLM playground application.
I'm using Langserve & Langchain to host this playground.
Gemma is being hosted in VertexAI with a public endpoint.

Currently, I'm trying to add support to my Langserve endpoints to serve Gemma, however, I'm using RunnablePassthrough to route to which specific LLMs I'd like to use at any given time.

When I try to do this, it seems that the langchain_google_vertexai implementation isn't aware of the currently event loop available in the thread.

This works fine for a multitude of other LLMs, even the ones available from VertexAI out of the box (chat-bison, gemini-pro, etc...) but for whatever reason the Gemma object exported by this package doesn't seem to be agreeable

Description:

To align with other APIs such as VertexAI and VertexModelGarden, it may be useful in some cases to pass google.auth.Credentials directly to the GoogleDriveLoader constructor instead of passing the service account file.

It should provide more flexibility. It should not change the other log in features.

Proposed solution:
Instantiate the credentials in the GoogleDriveLoader attributes. Replace the credentials calls with an attribute reference.

I would be happy to contribute if you validate this feature.

from google.auth.credentials import Credentials

class GoogleDriveLoader(BaseLoader, BaseModel):
    """Load Google Docs from `Google Drive`."""
    credentials : Credentials | None

Reference:
https://github.com/langchain-ai/langchain-google/blob/3c9efa1741431e44b0a9b061ba18a4bc51e137a9/libs/community/langchain_google_community/drive.py#L21C7-L21C24

For a period of time, Vertex AI Search engine_data_type=3 (Blended search) doesn't support some search params which are defaulted to on.

    status = StatusCode.INVALID_ARGUMENT
details = "Setting query_expansion_spec, spell_correction_spec, content_search_spec proto fields is not allowed for multi-datastore search."

FR 1:
Ability to explicitly remove the search settings.
(perhaps value = -1 means don't pass in the param)

FR 2:
Default to remove for now.
While we are waiting on those features to be available, let's just get the default settings to work, and we can improve the defaults in the future.

NotImplementedError Traceback (most recent call last)
Cell In[8], line 21
18 # LLM with function call
19 # llm = ChatOpenAI(model="gpt-3.5-turbo-0125", temperature=0)
20 llm = GoogleGenerativeAI(model="gemini-pro", temperature=0.7)
---> 21 structured_llm_grader = llm.with_structured_output(GradeDocuments)
23 # Prompt
24 system = """You are a grader assessing relevance of a retrieved document to a user question. \n
25 If the document contains keyword(s) or semantic meaning related to the question, grade it as relevant. \n
26 Give a binary score 'yes' or 'no' score to indicate whether the document is relevant to the question."""

File c:\Users\Acer\AppData\Local\Programs\Python\Python311\Lib\site-packages\langchain_core\language_models\base.py:208, in BaseLanguageModel.with_structured_output(self, schema, **kwargs)
204 def with_structured_output(
205 self, schema: Union[Dict, Type[BaseModel]], **kwargs: Any
206 ) -> Runnable[LanguageModelInput, Union[Dict, BaseModel]]:
207 """Implement this if there is a way of steering the model to generate responses that match a given schema.""" # noqa: E501
--> 208 raise NotImplementedError()

NotImplementedError:

Here's my chain:

from langchain_google_genai import ChatGoogleGenerativeAI, HarmBlockThreshold, HarmCategory
from langchain.schema.output_parser import StrOutputParser
from langchain.prompts import HumanMessagePromptTemplate, ChatPromptTemplate
from langchain import hub
import os 

import dotenv
dotenv.load_dotenv()

model = ChatGoogleGenerativeAI(model="gemini-pro", temperature=0.8,
                               safety_settings={
                                   HarmCategory.HARM_CATEGORY_HARASSMENT: HarmBlockThreshold.BLOCK_NONE,
                                   HarmCategory.HARM_CATEGORY_HATE_SPEECH: HarmBlockThreshold.BLOCK_NONE,
                                   HarmCategory.HARM_CATEGORY_SEXUALLY_EXPLICIT: HarmBlockThreshold.BLOCK_NONE,
                                   HarmCategory.HARM_CATEGORY_DANGEROUS_CONTENT: HarmBlockThreshold.BLOCK_NONE,
                               },)
prompt = hub.pull("[private repo]", api_key=os.getenv("LANGCHAIN_API_KEY"))

chain = prompt | model | StrOutputParser()

When I call the invoke method, with the parameter json, the code runs without any errors:

chain.invoke({"character_name": "Stitch from Lilo and Stitch", 
              "character_description": "",
              "user_info": "Name: John Smith\nAge: 30\nOccupation: Programmer",
              "chat_history":[
                  {"role": "ai", "content": "Aloha! I'm doing great, thank you. How about you?"},
                  {"role": "user", "content": "Hello, Stitch! How are you today?"},
              ]})

However, when I try to call the invoke method through requests, I'm getting an error.

import requests

url = "http://127.0.0.1:8000/character/invoke"
payload = {
    "input": {
        "character_name": "Stitch from Lilo and Stitch", 
              "character_description": "",
              "user_info": "Name: John Smith\nAge: 30\nOccupation: Programmer",
              "chat_history":[
                  {"role": "ai", "content": "Aloha! "},
                  {"role": "user", "content": "Hello, Stitch! How are you today?"},
              ]}
            }


response = requests.post(url, json=payload)

print(response.json()['output'])

The error is this:

File "C:\Users\n1ckq\anaconda3\envs\offline-ai\lib\site-packages\langchain_google_genai\chat_models.py", line 180, in _achat_with_retry
    raise ChatGoogleGenerativeAIError(
langchain_google_genai.chat_models.ChatGoogleGenerativeAIError: Invalid argument provided to Gemini: 400 Please ensure that multiturn requests alternate between user and model.

Upon further investigation, I noticed that the messages get converted to AIMessage regardless of what role I'm passing in the json. This is what the dictionary looks like after I send the API call.

{'character_description': '', 'character_name': 'Stitch from Lilo and Stitch', 'chat_history': [AIMessage(content="Aloha!", role='model'), AIMessage(content='Hello, Stitch! How are you today?', role='user')], 'user_info': 'Name: John Smith\nAge: 30\nOccupation: Programmer'}

Could you please help me fix this bug? Am I doing something wrong here?

ChatAnthropicVertex doesn't support with_structured_output although ChatVertexAI support it.

System messages don't work. I tried both gemini-1.5 and 1.0. I also tried convert_system_message_to_human=True and convert_system_message_to_human=False. None of these work, it doesn't follow system message.

See documentation for responseMimeType at https://cloud.google.com/vertex-ai/generative-ai/docs/model-reference/gemini#request_body

See langchain-ai/langchainjs#5071

Model response from VertexAIModelGarden is unexpected. It is in one of two formats:

Prompt
<prompt>
Output:
<prompt>
<model_response>

or

Prompt
<prompt>
Output:
<model_response>

In either case this response format is unexpected for Langchain components and will break them (e.g. ReAct agents).

pydantic json schema will include an allOf if a nested object has a description. for example

from langchain_core.pydantic_v1 import BaseModel, Field

class Node(BaseModel):
    id: str
    type: str
    
class Relationship(BaseModel):
    source: Node
    target: Node = Field(..., description="foo")


Relationship.schema()

generates

{
  'title': 'Relationship',
  'type': 'object',
  'properties': {
    'source': {'$ref': '#/definitions/Node'},
    'target': {
      'title': 'Target',
      'description': 'foo',
      'allOf': [{'$ref': '#/definitions/Node'}]
    }
  },
  'required': ['source', 'target'],
  'definitions': {
    'Node': {
      'title': 'Node',
      'type': 'object',
      'properties': {
        'id': {'title': 'Id', 'type': 'string'},
        'type': {'title': 'Type', 'type': 'string'}
      },
     'required': ['id', 'type']
    }
  }
}

notice difference between 'source' and 'target'.

When trying to use function-calling, the allOf leads to a protobuf error

File ~/anaconda3/lib/python3.11/site-packages/proto/marshal/rules/message.py:36, in MessageRule.to_proto(self, value)
     31 if isinstance(value, dict) and not self.is_map:
     32     # We need to use the wrapper's marshaling to handle
     33     # potentially problematic nested messages.
     34     try:
     35         # Try the fast path first.
---> 36         return self._descriptor(**value)
     37     except TypeError as ex:
     38         # If we have a type error,
     39         # try the slow path in case the error
     40         # was an int64/string issue
     41         return self._wrapper(value)._pb

ValueError: Protocol message Schema has no "allOf" field.

using

protobuf==4.25.3
google-cloud-aiplatform==1.44.0

Hi there, I'm trying to call a gemini-pro model using langchain-google-vertexai. However, as shown in the code below, if temperature=0, the temperature setting does not seem to be reflected.

from langchain_google_vertexai import VertexAI

model1 = VertexAI(model_name="gemini-pro", temperature=0.5)
print(model1._prepare_params())
# {'temperature': 0.5, 'candidate_count': 1, 'stop_sequences': None}

model2 = VertexAI(model_name="gemini-pro", temperature=0)
print(model2._prepare_params())
# {'candidate_count': 1, 'stop_sequences': None}

I think the cause may be in the code below:

# https://github.com/langchain-ai/langchain-google/blob/main/libs/vertexai/langchain_google_vertexai/llms.py
# from line 216 of langchain_google_vertexai/llms.py::_VertexAICommon._default_params

    @property
    def _default_params(self) -> Dict[str, Any]:
        ...
        # params is a dictionary of parameters held by the instance, such as temperature.
        for param_name, param_value in params.items():
            default_value = default_params.get(param_name)
            if param_value or default_value:  # if param_value is a value determined to be false, it will be skipped.
                updated_params[param_name] = (
                    param_value if param_value else default_value
                )
        return updated_params

I would like to ask why it is implemented this way. I'm a beginner, but if possible, I'm thinking of creating a pull request for the fix. Thanks!

System Info

aiohttp==3.9.1
aiosignal==1.3.1
annotated-types==0.6.0
anyio==4.1.0
attrs==23.1.0
cachetools==5.3.2
certifi==2023.11.17
charset-normalizer==3.3.2
colorama==0.4.6
dataclasses-json==0.6.3
frozenlist==1.4.0
google-ai-generativelanguage==0.4.0
google-api-core==2.15.0
google-auth==2.25.2
google-generativeai==0.3.1
googleapis-common-protos==1.62.0
greenlet==3.0.2
grpcio==1.60.0
grpcio-status==1.60.0
idna==3.6
jsonpatch==1.33
jsonpointer==2.4
langchain-community==0.0.3
langchain-core==0.1.0
langchain-google-genai==0.0.3
langsmith==0.0.70
marshmallow==3.20.1
multidict==6.0.4
mypy-extensions==1.0.0
numpy==1.26.2
packaging==23.2
Pillow==10.1.0
proto-plus==1.23.0
protobuf==4.25.1
pyasn1==0.5.1
pyasn1-modules==0.3.0
pydantic==2.5.2
pydantic_core==2.14.5
PyYAML==6.0.1
requests==2.31.0
rsa==4.9
sniffio==1.3.0
SQLAlchemy==2.0.23
tenacity==8.2.3
tqdm==4.66.1
typing-inspect==0.9.0
typing_extensions==4.9.0
urllib3==2.1.0
yarl==1.9.4

Who can help?

No response

Information

• The official example notebooks/scripts
• My own modified scripts

Related Components

• LLMs/Chat Models
• Embedding Models
• Prompts / Prompt Templates / Prompt Selectors
• Output Parsers
• Document Loaders
• Vector Stores / Retrievers
• Memory
• Agents / Agent Executors
• Tools / Toolkits
• Chains
• Callbacks/Tracing
• Async

Reproduction

from typing import Any, Optional
from uuid import UUID

from langchain.chains import LLMChain
from langchain_community.chat_models import ChatOpenAI
from langchain_core.callbacks import BaseCallbackHandler
from langchain_core.outputs import LLMResult
from langchain_core.prompts import PromptTemplate
from langchain_google_genai import ChatGoogleGenerativeAI

prompt_template = "Tell me details about the Company {name} with 2 bullet point?"


class StreamingLLMCallbackHandler(BaseCallbackHandler):

    def on_llm_new_token(self, token: str, **kwargs: Any) -> None:
        print(token, end='')

    def on_llm_end(self, response: LLMResult, *, run_id: UUID, parent_run_id: Optional[UUID] = None, **kwargs: Any) -> Any:
        print('\n\n---------------------LLM_END--------------------------')
        print(response.generations[0][0].text)


def gemini_stream_with_cb():
    llm = ChatGoogleGenerativeAI(model="gemini-pro", google_api_key='AIzaSPVggdfgerweselzbVCHH', streaming=True, callbacks=[StreamingLLMCallbackHandler()])
    llm_chain = LLMChain(llm=llm, prompt=PromptTemplate.from_template(prompt_template), verbose=True)
    llm_chain.run(name="Google")


def openai_stream_with_cb():
    llm = ChatOpenAI(openai_api_key='sk-pSHx9qACbzdgsdgyuuyNCXpZ0', model='gpt-4', streaming=True, callbacks=[StreamingLLMCallbackHandler()])
    llm_chain = LLMChain(llm=llm, prompt=PromptTemplate.from_template(prompt_template), verbose=True)
    llm_chain.run(name="Open AI")


if __name__ == '__main__':
    gemini_stream_with_cb()
    # openai_stream_with_cb()

ChatGoogleGenerativeAI: on_llm_new_token not triggered, but on_llm_end triggered
ChatOpenAI: Both triggered

Expected behavior

on_llm_new_token should trigger for ChatGoogleGenerativeAI

This seems to fail 50% with no completions from the server

from langchain_core.messages import AIMessage, HumanMessage, ToolMessage
from langchain_google_vertexai import ChatVertexAI


model = ChatVertexAI(model_name="gemini-pro")


messages = [
    HumanMessage(
        content=(
            "Repeat the given string using the provided tools. "
            "Do not write anything else or provide any explanations. "
            "For example, if the string is 'abc', you must print the "
            "letters 'a', 'b', and 'c' one at a time and in that order. "
            "\n\n type this: 'a'"
        )
    ),
    AIMessage(
        content="",
        additional_kwargs={
            "function_call": {
                "name": "type_letter",
                "arguments": '{"letter": "a"}',
            }
        },
        tool_calls=[
            {
                "name": "type_letter",
                "args": {"letter": "a"},
                "id": "abc123",
            },
        ],
    ),
    ToolMessage(
        name="type_letter",
        content="OK",
        tool_call_id="abc123",
    ),
]


result = list(model.invoke(messages))

Currently, the VertexAIEmbeddings only supports embeddings containing textembedding-gecko and multimodalembedding in the model names. However, there are newer preview models that have recently been released - text-embedding-preview-0409 and text-multilingual-embedding-preview-0409. According to [1], these models offer better performance than the recent stable versions. These newer models should be supported by langchain too for trying them out.

[1] - https://cloud.google.com/vertex-ai/generative-ai/docs/embeddings/get-text-embeddings#latest_models

With older package versions, used to get "empty response" error, even if that was also bad from google. Not get this instead. Even further back error handling was also bad for this situation.

  File "/home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/langchain_core/language_models/chat_models.py", line 65, in generate_from_stream
    for chunk in stream:
  File "/home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/langchain_google_genai/chat_models.py", line 600, in _stream
    _chat_result = _response_to_result(chunk, stream=True)
  File "/home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/langchain_google_genai/chat_models.py", line 421, in _response_to_result
    message=_parse_response_candidate(candidate, stream=stream),
  File "/home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/langchain_google_genai/chat_models.py", line 383, in _parse_response_candidate
    first_part = response_candidate.content.parts[0]
  File "/home/jon/miniconda3/envs/h2ogpt/lib/python3.10/site-packages/proto/marshal/collections/repeated.py", line 125, in __getitem__
    return self._marshal.to_python(self._pb_type, self.pb[key])
IndexError: list index out of range