writingmate / chatlabs Goto Github PK

Problem: currently /api/chat/tools/route.ts only support function calling supported by OpenAI library, but Anthropic's function calling is done differently. We need to change the code to support both OpenAI and Anthropic function calling.

https://platform.openai.com/docs/guides/function-calling
https://docs.anthropic.com/claude/docs/functions-external-tools

Currently, premium users (profile.plan = premium_yearly, premium_monthly) have access to all features - in the same way like Pro users. Only Pro users should have access to Pro features

• Paid models (model.paid = true)
• Plugins
• Assistants

Steps to reproduce:

1. Try to address the question to any web page several times in a row, using any AI model and Web Search plugin turned On.
2. I've caught the bug on GPT-4 Turbo and Mistral Large models, trying to translate/summarize the article (https://barmstrong.medium.com/how-to-hire-executives-e2ee8e05cad3)

Current behaviour: see on the screen below
Expected behaviour: a user is able to get a reply to their message from an AI.

Found the bug with Mistral Large model. Not sure if it is reproduced with other LLMs.

Steps to reproduce

1. Pick Mistral Large model
2. Turn on "Image Generation" plugin. Do not turn on 'Web Search' Plugin.
3. Put the prompt "Create a wide image showing a robot looking into the window with the picturesque view outside"

Current result:

• LLM provides a blurred image
• Besides the image, LLM shows results similar to what Web Search plugin usually provides (but it's not turned On), with broken links to YouTube videos and resources.

Expected result:
LLM provides image and nothing more.

Video:
https://www.loom.com/share/6661b12b3e9a44369ac05f3f3295b5bd?sid=6f8ccd20-e459-4b19-8d1d-b1f53cfedd68

Icons of Assistants (image) should be changed to more clear, simple and understandable

Google drive with new icons: link

I don't know if the service key inside file "20240108234540_setup.sql" is valid or not, but I recommend removing it and replacing it with something like the variable 'SUPABASE_SERVICE_ROLE_KEY. ' When needed, you can change that variable manually to the actual key inside Supabase UI --> Database --> Functions --> delete_storage_object.

@vood

Add a cross button to the internal paywall (image). There should be two ways of how to close the paywall

• by tapping on the area outside the paywall (as now)
• by clicking the cross icon

Button to read aloud each message. We need to add a play button icon next to each assistant message. Upon clicking the button the message should be played via whisper or elevenlabs api.

1. Pick Article Rewriter assistant
2. Ask something

Current result: no reply

I wanted to bring to your attention a serious security issue I discovered with the ChatLabs API. It seems that any user can access any model thought using the ChatLabs API directly, without any rate limits. This means that even with a free account, users can access Anthoipic or any other model, and have unlimited usage of any model they want.

Upon discovering this issue, I attempted to find a way to report it safely, but was unable to do so. As this is a serious security concern, I felt it was important to bring it to your attention as soon as possible. @vood

The original repo we forked is open source, we would like to contribute back and make our repo open source too

It is not clear what is ‘Source count’ (image). How to locate this parameter: upload file, choose file, click file Settings
Not clear what file retrieval option (image)
You cannot remove attached files from chat without using the Left menu (image)

Structurization of the left side menu.

The main idea is to keep Chats section in the tier-1 of UX hierarchy and move Plugins, Files, Assistants, Prompts libraries to the tier-2 (one step 'deeper').

1. Remove left vertical sub-menu with icons
2. Remove the Chats icon
3. Put icons of sections Prompts, Assistants, Files, Plugins horizontally underneath the New Chat button.
4. Put icons of Profile Settings and Premium Crystal at the bottom
   

Interaction with sections
5. Files sub-section

• Is opened by tapping the Files icon
• the sub-section slides out from the left side of the screen (the same way Profile Settings appear now)
• when opened,
  a) the name of the section "Files" is placed at the top.
  b) the "Close" button is placed in the top right corner that closes the sub-section "Files"
  

6. Assistants - same logic as for Files
7. Plugins - same logic as for Files
8. Prompts - same logic as for Files UNTIL #19 is not done.

Link to the mockup in Figma: https://www.figma.com/file/3cWnVt9tSMoczyUEEKacb1/%F0%9F%94%AE-Writingmate-Design?type=design&node-id=3498-4710&mode=design&t=i18K3vDavkpkXV6c-4

Currently the first call inside the /api/chat/tools endpoint is non streaming, which is bad experience for the user.

There are two ways to do this:

1. Use streaming variables from Vercel SDK
2. Use Stream functions from Vercel SDK. We will also have to implement our custom AnthropicStream since official one does not support function calling.

Add empty state description to the Plugins section (example)

Add text: Assistants are special AI characters instructed to provide information, solve specific problems, simulate conversations or offer creative content based on user queries.

Despite the attempted resolution in Issue #8, the application still permits users to send messages while a file is in the process of uploading. This behavior is inconsistent with the expected functionality where message sending should be disabled during uploads to ensure sequential data processing and user experience integrity.

Add Prompts (Doc)

AI, while translating the the web links, glues, cuts and distorts words

Steps to reproduce:

1. Choose any of Mistral Large, GPT4 Turbo, Claude 3 LLMs
2. Turn Web Search On
3. Ask AI to translate the information from the link

Current behaviour: AI glues, cuts and distorts words
Expected behaviour: Proper translation provided

Vercel released new AI SDK https://vercel.com/blog/ai-sdk-3-generative-ui with Generative AI support.

Specifically, it has a very interesting approach to interact with tools and render custom output.

We'd like to consider to use this for:

• Image Generation tool
• Web Browsing tool

Reproduced with account [email protected]

Two accounts in stripe: image

Gmail login

• Sometimes an error “access blocked” appears (image)
• Sometimes a user is moved back to the Login page (video)
  Email login
• Login link in email doesn’t work

Steps to reproduce:

1. Choose any AI
2. Choose a file and ask AI sth about this file

Current behaviour: endless loading, no result provided

Expected behaviour: AI processes the command and provides a result

Video: https://www.loom.com/share/db9f6a927cb6424a988042017c1013ed?sid=16f58ffe-77b9-4bde-9dc2-a08fa0053982

Use standard browser API for voice input:

Use IconMicrophone from "@tabler/icons-react" and put as a button next to the send button in the text area. When pressed, request voice recording permissions from the browser, record voice and put text into the textearea.

getUserMedia is the API to access microphone.

https://developer.mozilla.org/en-US/docs/Web/API/MediaDevices/getUserMedia

Currently, if a user uploads a file and sends a message to AI before the file is uploaded, AI replies with no answer (image). To exclude cases like this, I propose
to block the ability to send a message until a file/image is uploaded
When a file is being uploaded and it’s not possible to send a message, display Send Message button as disabled (image2)

Currently, all users go through an onboarding process, in the /pages/setup/page.tsx, this triggered when profiles.has_onboarded is set to false
We'd like to remove this step completely, since none of the information on the profile is required for operation.
To do this, we will need to make sure that the logic inside /pages/setup/page.tsx lives somewhere else and gets executed on first user login.

The conversation is divided into question-answer pairs and dumped into separate threads in the left panel under the "Chats" tab. I think this problem started yesterday. I thought it might be caused by using different assistants during one conversation, but now I see that this error occurs even when I use only one assistant throughout the conversation. During the conversation everything appears normal in main panel, but after refreshing or switching to different/new thread it becomes impossible to continue previous discussions.

Currently, our Image Generation plugin shows small generated images (previews) with barely recognizable details on it.

Would be great to generate wide images that take 100% of the chat width, as Dalle does.

Examples form ChatGPT and ChatLabs attached

ChatGPT

ChatLabs

Now, users, while trying to create or edit an assistant, see a 1-line description field regardless of the length of the entered text.

1. The text should be wrapped.
2. The description field should be resizable: the longer text, the more lines of text are shown.
3. Max number of input lines is 6. After that, scrolling should be available.

Dalle doesn’t show created images, providing a link instead (image). - An image should be shown right in a chat.

• After a user clicks on an image a full-sized image should be opened in a new tab

Steps to reproduce:

1. Open new chat
2. Choose assistant
3. Remove assistant
4. Start chatting

Current result: user is talking to assistant that was removed
Expectation: user is talking to LLM without assitants

GPT4 Turbo + Web Search plugin
Asked AI to summarize the video: https://www.youtube.com/watch?v=jvqFAi7vkBc

Current result: summary includes links with time codes like https://youtube.com/watch?v=jvqFAi7vkBc&s=95 (&s=95 at the end). This time code doesn't work.

Expected result: links are like: https://www.youtube.com/watch?v=jvqFAi7vkBc&t=95s (t=95s at the end) and moves user to a specific time code of the video

Users should understand what an Assistant can do for them before starting the first conversation. Also, clarify ‘Source count’ and the file retrieval option and address the issue of not being able to remove attached files from chat without using the Left menu.

Asked a question to GPT 4 Turbo with Web Search plugin turned on

Got the result:
An error occurred with your deployment
EDGE_FUNCTION_INVOCATION_TIMEOUT

Screenshot attached

Our app is PWA, we can publish directly to Play store and with some magic to AppStore

Research this

In one of the previous versions of the app (about a month ago), there was a place where you could save the main prompt ("User instructions"). Now there is no access to that. There is an "Override System Prompt" option, but the text entered there is not saved permanently.

TBD, we need to create a single point of entry for searching all entities

Add empty state description to the Chat section (example)
Replace text from ‘No Chats.’ to “Your chat history will be displayed here.”’

Sort the order of assistants

• In the left menu (image)
• In the dropdown menu of Assistants - only for zero state (when a user never used any of assistants - after usage the last used assistant is automatically located at the top of the list in this menu)

Suggested order from the top to the bottom of the list: AI Image assistant, SEO Article rewriter, TLDR Article Summarizer, AI text detector, SMM Captions, Video Script Generator, Market scout, Medical Students Helper, Cooking helper.

It takes more than 3 seconds to initially load the screen. Need to figure out how to improve it

It would be nice to have an option (in global user settings) to disable autoscroll. It's really annoying to be constantly moved to the bottom when AI generates a longer output.

Currently,

• Sonar Medium Online is available for free users
• Sonar Small Online is not available for free users

It should be the opposite way:

• Sonar Medium Online is not available for free users
• Sonar Small Online is available for free users

Steps to reproduce.

1. Pick ChatGPT 4, select any tool/plugin (i.e. WebSearch)
2. Pick Gemini 1.0 Pro Vision model
3. Upload an image (tried jpeg, jpg, webp formats)
4. Sent a prompt

Current result: the error "Provider not supported: google" pops up.

Expected result: AI sends a proper reply.

Explanation:
We should not send selectedTools and should not use /api/chat/tools to API when models that do not support tools selected. The models that support tools/plugins marked with model.tools = true.

Add empty state description to the Files section (example)

Replace text from ‘No files.’ to “Upload files to enrich conversations and assistants with context, data analysis, feedback, or customization. Uploaded files will be displayed here.”’

1. If you select an image, you can't return to an initial state (with no image selected);
2. If you click to replace it and cancel, it shows as No image, and an error appears, but the previous image preview is still visible;
3. Once you get the error, the only way to proceed is to refresh the page

See video:
https://github.com/writingmate/chatlabs/assets/50916363/8199baad-b42f-465d-83bc-867300eb9ccb

Steps to reproduce:

1. Create prompt with double braces. For instance: "Draw a wide plain and minimalistic image related to {{Topic}}, with few elements on it and without text, in warm trending colors"
2. Try to edit description or name of the prompt after the prompt is created

Current result:
User cannot edit prompt name & description, "Enter Prompt Variables" menu pops up

Expected result:
User can edit prompt name & description, "Enter Prompt Variables" menu does not pop up

Add empty state description to the Prompts section (example)

Replace text from ‘No prompts.’ to “Prompts are pre-saved text inputs designed to generate specific responses and communicate with AI quicker. Prompts you create will be displayed here.”’

I’ve reproduced the bug with GPT3.5 and GPT4 Turbo models. Highly possible, that it’s not related to a specific model

The bug is reproduced not every time. Sometimes, especially if it’s 2nd or the following request, Dalle provides an image by a link.

Steps to reproduce:

1. choose gpt 3.5 turbo or gpt4 turbo model
2. turn on Dalle plugin
3. try to generate image

Currently, I catch two behaviours

• GPT 3.5 is unable to generate an image (example)
• GPT 3.5 accepts request but is unable to display an image (example)

Carbon.ai (documentation) provides integration with Slack, Google Drive, etc

Add empty state description to the Plugins section (example)

Add the text: “Plugins are special add-ons that allow you to do extra things beyond just chatting, such as using up-to-date information from the web or checking the weather by connecting to external services or databases.”

The description disappears after a user creates the first new plugin by themself.