Giter VIP home page Giter VIP logo

Comments (3)

wordware-ai avatar wordware-ai commented on May 28, 2024 1

I turned this into a package https://www.npmjs.com/package/mistral-edge

from client-js.

xezpeleta avatar xezpeleta commented on May 28, 2024

Fixed thanks to https://github.com/haverstack/axios-fetch-adapter/

Instructions

  1. Install mistralai js client: npm install @mistralai/mistralai
  2. Install axios-fetch-adapter: npm install @haverstack/axios-fetch-adapter
  3. Edit the file node_modules/@mistralai/mistralai/src/client.js:

Import the module:

import fetchAdapter from "@haverstack/axios-fetch-adapter";  //import fetchAdapter (line 3)

Specify the adapter:

    ...
    const response = await axios({
      adapter: fetchAdapter, //Use the fetchAdapter (line 47)
      method: method,
      ...

from client-js.

nyacg avatar nyacg commented on May 28, 2024

I was also facing this issue and couldn't work out how to get the adapter to work so I just wrote my own client for the rest API.

Might be useful for some people:

interface Message {
  role: "user" | "system" | "assistant";
  content: string;
}

interface MistralConfig {
  model: "mistral-tiny" | "mistral-small" | "mistral-medium";
  temperature?: number;
  maxTokens?: number;
  topP?: number;
  randomSeed?: number;
  safeMode?: boolean;
}

export async function* streamMistralChat(
  messages: Message[],
  config: MistralConfig,
  apiKey: string | undefined = undefined,
): AsyncGenerator<string, void, void> {
  const r = await fetch("https://api.mistral.ai/v1/chat/completions", {
    method: "post",
    headers: {
      Authorization: `Bearer ${apiKey ?? process.env.MISTRAL_API_KEY}`,
      "Content-Type": "application/json",
    },
    body: JSON.stringify({
      model: config.model,
      messages: messages,
      temperature: config.temperature,
      max_tokens: config.maxTokens,
      top_p: config.topP,
      random_seed: config.randomSeed,
      stream: true,
      safe_prompt: config.safeMode,
    }),
  });

  if (!r.ok) {
    console.error(`Error fetching from Mistral API ${r.status}`);
    console.error(await r.text());
    throw new Error(`Mistral API error, status: ${r.status}`);
  }

  const reader = r.body.getReader();
  const decoder = new TextDecoder();
  let buffer: string[] = [];

  while (true) {
    const { done, value: bytes } = await reader.read();
    if (done) {
      break;
    }

    const chunk = decoder.decode(bytes, { stream: true });

    for (let i = 0, len = chunk.length; i < len; ++i) {
      // We've got newline delimited JSON which has a double newline to separate chunks
      const isChunkSeparator = chunk[i] === "\n" && buffer[buffer.length - 1] === "\n";

      // Keep buffering unless we've hit the end of a data chunk
      if (!isChunkSeparator) {
        buffer.push(chunk[i]);
        continue;
      }

      const chunkLine = buffer.join("");
      if (chunkLine.trim() === "data: [DONE]") {
        break;
      }

      if (chunkLine.startsWith("data:")) {
        const chunkData = chunkLine.substring(6).trim();
        if (chunkData !== "[DONE]") {
          const chunkObject = JSON.parse(chunkData);
          // We just stream the completion text
          yield chunkObject.choices[0].delta.content ?? "";
        }
      } else {
        throw Error(`Invalid chunk line encountered: ${chunkLine}`);
      }

      buffer = [];
    }
  }
}

Just put your API key in the MISTRAL_API_KEY env var and run

const response = streamMistralChat([{ role: "user", content: "Hello!" }], {
      model: "mistral-tiny",
      temperature: 0.7,
      // etc
});

for await (const chunk of response) {
    // Do whatever you want with the streamed text
    console.log(chunk); 
}

from client-js.

Related Issues (18)

Recommend Projects

  • React photo React

    A declarative, efficient, and flexible JavaScript library for building user interfaces.

  • Vue.js photo Vue.js

    🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.

  • Typescript photo Typescript

    TypeScript is a superset of JavaScript that compiles to clean JavaScript output.

  • TensorFlow photo TensorFlow

    An Open Source Machine Learning Framework for Everyone

  • Django photo Django

    The Web framework for perfectionists with deadlines.

  • D3 photo D3

    Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

  • javascript

    JavaScript (JS) is a lightweight interpreted programming language with first-class functions.

  • web

    Some thing interesting about web. New door for the world.

  • server

    A server is a program made to process requests and deliver data to clients.

  • Machine learning

    Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.

  • Game

    Some thing interesting about game, make everyone happy.

Recommend Org

  • Facebook photo Facebook

    We are working to build community through open source technology. NB: members must have two-factor auth.

  • Microsoft photo Microsoft

    Open source projects and samples from Microsoft.

  • Google photo Google

    Google ❤️ Open Source for everyone.

  • D3 photo D3

    Data-Driven Documents codes.