amithkoujalgi / ollama4j Goto Github PK

I felt there was a lack of test cases for OllamaChatRequestBuilder, so I added tests for a few methods.
If there are any mistakes, please comment🙇‍♂️

PR. #37

When sending messages directly to the Mistral API (similar to OpenAI API), then one can send messages with different roles, whereas see

• https://platform.openai.com/docs/guides/text-generation/chat-completions-api
• https://cookbook.openai.com/examples/azure/chat_with_your_own_data#code

Is this possible somehow with ollama4j respectively Ollama itself?

Could you please do some update to support the Gemma 2B and 7B?

Issue

I get an empty response from the library, but my tcpflow logs show that the response does come later.

{
    "role" : "assistant",
    "content" : "",
    "images" : null
  }

Details

I have an old laptop with only Dual-Core i5 2.7Ghz and am running llama3 on Ollama.
I created a simple app to test ollama4j with llama3, but getting empty responses back, even though I set a huge request timeout.

Here is my code

String host = "http://localhost:11434/";
      String model = "llama3";
      OllamaAPI ollamaAPI = new OllamaAPI(host);
      ollamaAPI.setRequestTimeoutSeconds(600000);
      ollamaAPI.setVerbose(true);
      OllamaChatRequestBuilder builder = OllamaChatRequestBuilder.getInstance(model);
      Options options =
          new OptionsBuilder()
              .setTemperature(0.2f)
              .setNumCtx(example.getModel().getCompletionLength())
              .setTopK(1)
              .setTopP(0.9F)
              .build();
      OllamaChatRequestModel requestModel = builder.withMessage(OllamaChatMessageRole.SYSTEM, example.getSystemPrompt())
          .withMessage(OllamaChatMessageRole.USER, "What can you help me with?")
          .withOptions(options)
          .build();
      System.out.println("Ollama request: " + requestModel.toString());
      OllamaChatResult chatResult = ollamaAPI.chat(requestModel);
      System.out.println("Ollama answer: " + chatResult.getHttpStatusCode() + " in seconds: " + chatResult.getResponseTime() + ":\n" + chatResult.getResponse());

And my logs show this:

Ollama request: {
  "model" : "llama3",
  "options" : {
    "top_p" : 0.9,
    "top_k" : 1,
    "temperature" : 0.2,
    "num_ctx" : 1024
  },
  "stream" : false,
  "messages" : [ {
    "role" : "system",
    "content" : "You are a helpful customer service representative for a credit card company who helps answer customer questions about their past transactions and spending history. Today's date is January 18th, 2024. You provide precise answers and use functions to look up information...",
    "images" : null
  }, {
    "role" : "user",
    "content" : "What can you help me with?",
    "images" : null
  } ]
}
Ollama answer: 200 in seconds: 108372:

If I look into the message history, I basically see

{
    "role" : "assistant",
    "content" : "",
    "images" : null
  }

But if I check my tcpflow logs, I can see a Response:

HTTP/1.1 200 OK
Content-Type: application/json; charset=utf-8
Date: Tue, 14 May 2024 10:29:15 GMT
Content-Length: 801

{
   "model":"llama3",
   "created_at":"2024-05-14T10:29:15.284946Z",
   "message":{
      "role":"assistant",
      "content":"I'm happy to assist you with any questions or concerns you may have about your credit card account. I can help you:\n\n* Review your transaction history and spending habits\n* Check your available credit limit and current balance\n* Provide information on rewards and benefits associated with your card\n* Help you track your spending by category (e.g., groceries, entertainment, etc.)\n* Offer suggestions for managing your debt or improving your financial situation\n\nWhat specific area would you like me to help you with today?"
   },
   "done_reason":"stop",
   "done":true,
   "total_duration":54914791839,
   "load_duration":15113419,
   "prompt_eval_duration":804784000,
   "eval_count":101,
   "eval_duration":54082687000
}

Trying the streaming API works, but not the synchronous one.
Do you have any idea what the problem is?

I sometimes get the error

UnrecognizedPropertyException: Unrecognized field "eval_count" (class io.github.amithkoujalgi.ollama4j.core.models.OllamaResponseModel), not marked as ignorable (11 known properties: "response", "done", "evalCount", "eval_duration", "model", "created_at", "prompt_eval_duration", "load_duration", "context", "prompt_eval_count", "total_duration"])

does this 1.0.64 version support LLaMa3?

After the latest update of the Ollama client (version: 6.2.2, May 12, 2024), I encounter the following exceptions:

ERROR io.github.amithkoujalgi.ollama4j.core.models.request.OllamaChatEndpointCaller - Error parsing the Ollama chat response!
com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "done_reason" (class io.github.amithkoujalgi.ollama4j.core.models.chat.OllamaChatResponseModel), not marked as ignorable (12 known properties: "done", "message", "error", "model", "created_at", "prompt_eval_duration", "load_duration", "context", "eval_duration", "eval_count", "total_duration", "prompt_eval_count"])
at [Source: REDACTED (StreamReadFeature.INCLUDE_SOURCE_IN_LOCATION disabled); line: 1, column: 142] (through reference chain: io.github.amithkoujalgi.ollama4j.core.models.chat.OllamaChatResponseModel["done_reason"])
at com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:61)
at com.fasterxml.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:1153)
at com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:2241)
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1793)
at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownVanilla(BeanDeserializerBase.java:1771)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:316)
at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
at com.fasterxml.jackson.databind.deser.DefaultDeserializationContext.readRootValue(DefaultDeserializationContext.java:342)
at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4905)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3848)
at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3816)
at io.github.amithkoujalgi.ollama4j.core.models.request.OllamaChatEndpointCaller.parseResponseAndAddToBuffer(OllamaChatEndpointCaller.java:40)
at io.github.amithkoujalgi.ollama4j.core.models.request.OllamaEndpointCaller.callSync(OllamaEndpointCaller.java:99)
at io.github.amithkoujalgi.ollama4j.core.OllamaAPI.chat(OllamaAPI.java:521)
at io.github.amithkoujalgi.ollama4j.core.OllamaAPI.chat(OllamaAPI.java:498)

Hello, i see your example of how to generate embeddings, but it is not evident how to use the response afterwards.

Do we need to supply the response as in input parameter to generate method?
How to procced with the embeddings between LLama server restarts ?

Could you please advise on it?

https://amithkoujalgi.github.io/ollama4j/docs/apis-generate/generate-embeddings
https://ollama.com/blog/embedding-models

According to the Ollama documentation one can set the temperature using options

https://github.com/jmorganca/ollama/blob/main/docs/api.md#parameters
https://github.com/jmorganca/ollama/blob/main/docs/modelfile.md#valid-parameters-and-values

{ "model": "mistral", "stream":false, "prompt":"How many moons does earth have?", "options":{"temperature":0} }

IIUC one cannot set the temperature / options yet using ollama4j.

I would like to suggest to introduce another "ask" method, where one can set options as well

Options options = new Options();
options.setTemperature(0.7);
ollamaAPI.ask(String, String, Options)

WDYT?

Thanks

Michael

Ollama4j uses logback as a logging backend and has a transitive dependency on it.

Libraries should not have transitive dependencies on logging backends.

If you need the backend during testing, best use the test scope on the dependency.

In general, the application which uses the ollama4j library should define a logging backend for ollama4j to use.

Description

The generate method of PromptBuilder has undergone changes due to recent updates. Consequently, the code provided in the example no longer functions correctly.

Previously, the method was called as follows:

OllamaResult response = ollamaAPI.generate(model, promptBuilder.build());

However, in the latest version, additional options need to be passed to the generate method. Therefore, the code needs to be modified as follows:

OllamaResult response = ollamaAPI.generate(model, promptBuilder.build(), new Options());

Updating the code to reflect this change should prevent any further compatibility issues.

Do you have plans to support

• Either Basic Auth
• Or an Authorization Header Bearer Token

?

I understand Ollama does not support this, but when using a ReverseProxy in front of Ollama, then this would help a lot re security :-)

Thanks

Michael

It appears that certain requests generated by the ask() method trigger a 400 - Bad request on the side of ollama.

Trying the same request manually works though (i.e. copying the JSON generated by ollamaRequestModel.toString() and posting it manually).

Friendly suggestion that it would be better to use the Java Duke logo instead of the TM Java logo. I speak out of experience :) Otherwise very interesting project, keep it up!

Hi @amithkoujalgi ,

Thanks for creating this library! I've just implemented it in an experimental project for our Open Source library, Datafaker.net.

You can find the repo here should you be interested: https://github.com/datafaker-net/datafaker-experimental/tree/main/ollama-api

Cheers,
Erik

In addition to the /api/chat endpoint, the system prompt parameter (and other parameters that override model behaviours) can also be provided to requests at /api/generate.

Thus these should also be made available for the ollama4j API.

See:
Advanced parameters (optional):

format: the format to return a response in. Currently the only accepted value is json

system: system message to (overrides what is defined in the Modelfile)

template: the prompt template to use (overrides what is defined in the Modelfile)

context: the context parameter returned from a previous request to /generate, this can be used to keep a short conversational memory

raw: if true no formatting will be applied to the prompt. You may choose to use the raw parameter if you are specifying a full templated prompt in your request to the API

keep_alive: controls how long the model will stay loaded into memory following the request (default: 5m)

Originally posted by @AgentSchmecker in #20 (comment)