matlab-deep-learning / llms-with-matlab Goto Github PK

per toshiakit/MatGPT#30

Hi,

I'm exploring MatGPT's integration capabilities with LLMs and am interested in extending its utility to custom models, particularly those deployed locally.

In Python projects, customizing the base_url, as seen in openai-python openai/openai-python#913, is a straightforward approach to support custom API endpoints.

Although I'm not familiar with the specific construction of MatGPT, could a similar method be applied here to enable the use of in-house or custom LLMs via user-defined API endpoints? Your insights or guidance on this possibility would be greatly appreciated.

This refers to the use of custom assistants
https://community.openai.com/t/custom-assistant-api-endpoints/567107

GPT-4 Turbo is currently available as "gpt-4-1106-preview"

When does llms-with-matlab support Google Gemini?

Currently you cannot select "none" for ToolChoice

function toolChoice = convertToolChoice(this, toolChoice)
    % if toolChoice is empty
    if isempty(toolChoice)
        % if Tools is not empty, the default is 'auto'.
        if ~isempty(this.Tools) 
            toolChoice = "auto";
        end
    elseif ToolChoice ~= "auto"
        % if toolChoice is not empty, then it must be in the format
        % {"type": "function", "function": {"name": "my_function"}}
        toolChoice = struct("type","function","function",struct("name",toolChoice));
    end

end

We should fix this

function toolChoice = convertToolChoice(this, toolChoice)
    % if toolChoice is empty
    if isempty(toolChoice)
        % if Tools is not empty, the default is 'auto'.
        if ~isempty(this.Tools) 
            toolChoice = "auto";
        end
    elseif ~ismember(ToolChoice,["auto","none"])
        % if toolChoice is not empty, then it must be "none" or in the format
        % {"type": "function", "function": {"name": "my_function"}}
        toolChoice = struct("type","function","function",struct("name",toolChoice));
    end

end

Would it be possible to add the ability of using LLMs-with-MATLAB with an an Azure OpenAI API key, instead of a regular OpenAI API key ?

"gpt-4o" is now available from OpenAI.

mustBeValidSize in openAIImages is unused. Shall we use it in the argument validation of nvpSize which we do in generate, edit and createVariation methods?

If we should use it and at the moment Size validation is wrong, then this is a bug. Otherwise, I would categorize it as an enhancement.

Error for ExampleChatBot.mxl

When I excute this, it show errors as
Error using openAIMessages/addResponseMessage
Input struct must contain field 'role' with value 'assistant', and field 'content'.

I am not sure where these errors are coming from the disconnect of my API key or something else

similar errors are found in other mxl files .

Streaming function errors when a tool is called

UsingDALLEToEditImages:

[images,resp] = createVariation(mdl, imagePath, NumImages=4);

Error using [imread](matlab:matlab.lang.internal.introspective.errorDocCallback('imread', 'E:\jobarchive\Btext\2024_04_22_h17m05s46_job2591482_pass\matlab\toolbox\matlab\imagesci\imread.m', 352)) ([line 352](matlab: opentoline('E:\jobarchive\Btext\2024_04_22_h17m05s46_job2591482_pass\matlab\toolbox\matlab\imagesci\imread.m',352,0)))
Can't read URL "https://oaidalleapiprodscus.blob.core.windows.net/private/org-pq0MGRBYHualPO5l0BarTWqW/user-qV2F18SvbhogHzXIkc70zO8l/img-QWk9Dz7ye9dye7G36Po5g2YQ.png?st=2024-05-14T14%3A42%3A29Z&se=2024-05-14T16%3A42%3A29Z&sp=r&sv=2021-08-06&sr=b&rscd=inline&rsct=image/png&skoid=6aaadede-4fb3-4698-a8f6-684d7786b067&sktid=a48cca56-e6da-484e-a814-9c849652bcb3&skt=2024-05-13T23%3A03%3A20Z&ske=2024-05-14T23%3A03%3A20Z&sks=b&skv=2021-08-06&sig=aGDLoCV%2B47a5o3S5wEg7Y4Ki%2BVcEED33Lrb3W1cvPtg%3D".
Reason: Could not find file : https://oaidalleapiprodscus.blob.core.windows.net/private/org-pq0MGRBYHualPO5l0BarTWqW/user-qV2F18SvbhogHzXIkc70zO8l/img-QWk9Dz7ye9dye7G36Po5g2YQ.png?st=2024-05-14T14%3A42%3A29Z&se=2024-05-14T16%3A42%3A29Z&sp=r&sv=2021-08-06&sr=b&rscd=inline&rsct=image/png&skoid=6aaadede-4fb3-4698-a8f6-684d7786b067&sktid=a48cca56-e6da-484e-a814-9c849652bcb3&skt=2024-05-13T23%3A03%3A20Z&ske=2024-05-14T23%3A03%3A20Z&sks=b&skv=2021-08-06&sig=aGDLoCV%2B47a5o3S5wEg7Y4Ki%2BVcEED33Lrb3W1cvPtg%3D

Error in [openAIImages>extractImages](matlab:matlab.lang.internal.introspective.errorDocCallback('openAIImages>extractImages', 'C:\Users\vpapanas\OneDrive - MathWorks\Documents\GitHub\llms-with-matlab\openAIImages.m', 301)) ([line 301](matlab: opentoline('C:\Users\vpapanas\OneDrive - MathWorks\Documents\GitHub\llms-with-matlab\openAIImages.m',301,0)))
images = arrayfun(@imread,urls,UniformOutput=false);

Error in [openAIImages/createVariation](matlab:matlab.lang.internal.introspective.errorDocCallback('openAIImages/createVariation', 'C:\Users\vpapanas\OneDrive - MathWorks\Documents\GitHub\llms-with-matlab\openAIImages.m', 266)) ([line 266](matlab: opentoline('C:\Users\vpapanas\OneDrive - MathWorks\Documents\GitHub\llms-with-matlab\openAIImages.m',266,0)))
images = extractImages(response);

To be able to render the examples live scripts
https://www.mathworks.com/matlabcentral/fileexchange/163796-large-language-models-llms-with-matlab