GPT-4 powered chat about vscode-copilot-release HOT 33 CLOSED

I'll second that. I was excited about being invited to beta test but quickly discovered the model is inferior to GPT-4. Copilot Chat is a non-starter for me until the response quality is at least as good as GPT-4.

from vscode-copilot-release.

I thought Github Copilot X products (Chat, PRs) were meant to be powered by GPT-4.

from vscode-copilot-release.

Yes, both models are in play. Each model is being used where it is most effective. We'll have more nitty-gritty details in an upcoming blogpost, stay tuned.

In your logs, the second request is used to generate the follow-up question that shows up in chat panel.

from vscode-copilot-release.

Considering that the whole ideia about the chat inside the IDE is to get the context from the project in order to give better predictions/answers, GPT-4 is extremely needed. This need is way beyond what GPT-3 (or even 3.5) can do.

Looks like Microsoft have to make some agreements with OpenAI in order to update Codex to use GPT-4 instead of GPT-3 or 3.5.
Let's put some pressure on it!

from vscode-copilot-release.

I use Chat GPT 4 daily and pay for this product because I usually get very good results after about 3 questions and answers. Before I tried my luck with Chat GPT 3 and the quality was not satisfying. I had high expectations for GitHub Copilot X, but of course I would never work with an outdated model. I get significantly better results on the OpenAI website!

from vscode-copilot-release.

Any news about that upcoming blog post?

from vscode-copilot-release.

I'm surprised that it isn't GPT-4, and surprised at how immediately I could tell that it isn't. For a programming assistant, or any professional assistant, anything less just doesn't cut it.

GPT-4 is much slower.

I've actually missed the slower speed of GPT-4, which feels more conversational

from vscode-copilot-release.

I'll second that. I was excited about being invited to beta test but quickly discovered the model is inferior to GPT-4. Copilot Chat is a non-starter for me until the response quality is at least as good as GPT-4.

Completely agree. After having tried GPT-4 for debugging and code suggestions, it is obvious that Copilot Chat is way behind.

from vscode-copilot-release.

Also, the database seems to be outdated. In case of programming, being up to date is essential...

GCP chat: "glib2 is a library, what is the latest version of it?"

Yes, GLib is a library that provides core application building blocks for C. The latest stable version of GLib is 2.70.0, which was released on September 22, 2021.

Google: "glib2 stable version"

Overview of changes in GLib 2.76.2, 2023-04-21 [....]

from vscode-copilot-release.

This will probably not happen in the near future because

• GPT-4 is much more expensive to run than Codex
• GPT-4 is much slower.

You will have to wait a few months/years until hardware gets cheaper or huge algorithmic advancements are made.

EDIT: I guess the future is now.

from vscode-copilot-release.

Yea I came here to say that it's quite disappointing that this is clearly still GPT3.5
I'd happily pay extra or put in my own GPT4 keys if they had the option but atm the answers are wrong in nearly every test I've done and in some cases its argued with me when I explain how its misunderstood the code

from vscode-copilot-release.

Completely agreed with this. GPT3.5 is clearly not good enough as a programming assistant.

Example with rust on my side, I asked Copilot chat to migrate a few functions that were using async-channel to crossbeam-channel. The code proposed by Copilot Chat seem to have the broad idea but does not run and is in fact close to useless given the number of bugs (as expected from GPT3.5).

from vscode-copilot-release.

I thought Github Copilot X products (Chat, PRs) were meant to be powered by GPT-4.

Agree. That's where my confusion arose from as well

from vscode-copilot-release.

If you have questions with repeatedly lower quality as ChatGPT, we'd love issues on those to include them in our benchmark suite.

You can see which model is being used in the Output for GitHub Copilot Chat. Look for [chat fetch] chat model *model will be here*

The asking-copilot-tests don't work. This is from the OpenAI playground with gpt-4 as model:

AI also nicely explains its own problem with self-awareness:

from vscode-copilot-release.

Strange but in vscode plugin, copilot chat says that it runs on GPT-4 for, while my Jetbrains' copilot chat plugin says it runs on GPT-3.5

from vscode-copilot-release.

bump

from vscode-copilot-release.

This will probably not happen in the near future because

• GPT-4 is much more expensive to run than Codex
• GPT-4 is much slower.

You will have to wait a few months/years until hardware gets cheaper or huge algorithmic advancements are made.

I'm not sure about that, based on a few things:

1. As mentioned by @brianbruggeman the official marketing says: " With chat and terminal interfaces, support for pull requests, and early adoption of OpenAI’s GPT-4, GitHub Copilot X is our vision for the future of AI-powered software development. ".
2. Yes, sure GPT-4 is more expensive than his older brothers 3 and 3.5, but we are talking about a paid product, right? Doesn't look good when you offer a paid product that it's not delivering the best stuff your company have available.
3. Lots of competitors are arriving to this market; having an excellent Copilot would help Microsoft to maintain developers inside its ecosystem (VS Code + Github) while generating revenue from them.
4. About GPT-4 being slower, maybe by implementing GPT-4 in Codex they could drop some parts and parameters, making the product a bit faster (and maybe less power hungry) compared with the GPT-4 used by ChatGPT (that is designed for wider and more general usage).

from vscode-copilot-release.

In my recent tests, I've found that compared to Github Copilot X, I prefer using ChatGPT-4 to help improve my coding skills, as its accuracy is significantly higher.

from vscode-copilot-release.

gpt chat widget in phpstorm says it's using GPT 3.

I suggest to use the feedback mechanism in JetBrains for Chat, as this repo is just for VS Code's Copilot Chat.

from vscode-copilot-release.

from vscode-copilot-release.

Also, the database seems to be outdated. In case of programming, being up to date is essential...

GCP chat: "glib2 is a library, what is the latest version of it?"

Yes, GLib is a library that provides core application building blocks for C. The latest stable version of GLib is 2.70.0, which was released on September 22, 2021.

Google: "glib2 stable version"

Overview of changes in GLib 2.76.2, 2023-04-21 [....]

That's a good point.

I think you (or someone else) may could open a new issue here suggesting the Copilot Chat Team to make a way to keep libs and stuff like that always up to date, maybe tracking releases on npm and other package managers when possible (and apt, snap an others for linux based OSs), and even at github itself, since a huge part of libs/frameworks developers hosts their projects on github

from vscode-copilot-release.

Second this as well. It sure is convenient to have things directly in the IDE. But for any big code scaffolding, I just can't do without GPT-4. Would love a GPT-4 variant in the mix sometime.

(Though the speed is nice!)

from vscode-copilot-release.

I've been rigorously testing the Chat GPT beta version daily since my acceptance into the program, and I must say it's an incredibly fascinating piece of technology. It's noteworthy to mention the intriguing balance it strikes between response time and quality of answers. The speed at which it constructs a response is indeed impressive.

However, it seems that some challenges persist in terms of the overall user experience. At times, the responses may lack precision or come across as misaligned with the asked query, which does, in certain situations, affect my productivity. The ideal goal of an AI such as this is to enhance user productivity, so there's room for improvement in aligning the responses more accurately with the user's needs, especially around the context of a project's contents.

I've also noticed that despite the marketing claims (link), it appears the chat is not entirely powered by Chat GPT-4. This brings up some valid questions about its efficacy as a beta/test product for users beyond just giving general feedback. For reference, the response I received when asked if the chat was powered by Chat GPT was: Yes, I am powered by OpenAI's GPT-3 language model.

In terms of comparison, Chat GPT-4 often requires 2-3 requests for delivering the desired answer. And yet, Chat GPT-3 or its variant GPT-3.5 seems to have different metrics altogether, suggesting that a bench-marking exercise might be beneficial. There's so many anecdotal responses across the various social media that indicates a transparent OpenAI bechmarking about the quality of answers would be really helpful as a metric.

Overall, the progress and potential of Chat GPT is genuinely exciting and I'm enthusiastically looking forward to future improvements that will enhance the user experience and improve its utility.

from vscode-copilot-release.

if you, like me, are happy to pay for gpt4 using openai, then you can simply setup a local proxy and redirect the requests there. proof of concept: https://github.com/ClaraLeigh/proxy-for-copilot
this breaks streamed responses & extension searching, I might fix that if it annoys me too much but atm its not on my radar

from vscode-copilot-release.

if you, like me, are happy to pay for gpt4 using openai, then you can simply setup a local proxy and redirect the requests there. proof of concept: https://github.com/ClaraLeigh/proxy-for-copilot
this breaks streamed responses & extension searching, I might fix that if it annoys me too much but atm its not on my radar

This is a great alternative, and if someone is in hurry for GPT4 powered chat, he/she should use it.

from vscode-copilot-release.

Yeah, the GPT model is too dumb. After chatting with Copilot Chat and working for a few days - I returned to GPT-4. More context awareness, better knowledge base. With my workflow, Copilot Chat is not good enough to be used on a daily basis.

from vscode-copilot-release.

Here's an alternative which lets you use GPT4 with your own API key

Can I use some API key from my Github Copilot subscription or do I need an API key from an ChatGPT Plus subscription?

from vscode-copilot-release.

@ericktreichel You bring your own key from the Open AI API

from vscode-copilot-release.

Closing as all responses in panel/quick chat are served by GPT-4 since Copilot Chat 0.10 (part of VS Code's October release). 3.5 is still used for other messages.

Please give it a try and file new issues for any follow-up feedback!

from vscode-copilot-release.

@digitarald I'm still seeing GPT-3 with v0.10.0:

from vscode-copilot-release.

Same

from vscode-copilot-release.

@digitarald Thanks for the clarification! Interestingly both models are used:

2023-11-02T18:22:21.699Z [INFO] [chat fetch] engine https://api.githubcopilot.com/chat
2023-11-02T18:22:21.699Z [INFO] [chat fetch] modelMaxTokenWindow 4096
2023-11-02T18:22:21.699Z [INFO] [chat fetch] chat model gpt-4
2023-11-02T18:22:21.699Z [INFO] [chat fetch] intent model tinybert-0.1.1.onnx
2023-11-02T18:22:21.699Z [INFO] [chat fetch] intent tokenizer prajjwal1/bert-tiny
...
2023-11-02T18:22:24.698Z [INFO] [chat fetch] engine https://api.githubcopilot.com/chat
2023-11-02T18:22:24.698Z [INFO] [chat fetch] modelMaxTokenWindow 8192
2023-11-02T18:22:24.698Z [INFO] [chat fetch] chat model gpt-3.5
2023-11-02T18:22:25.082Z [INFO] [chat fetch] request.response: [https://api.githubcopilot.com/chat/completions], took 384 ms
2023-11-02T18:22:25.082Z [INFO] [streamMessages] message 0 returned. finish reason: [stop]

Can you provide details when the 3.5 model is used? Apparently there was some context used in the chat. Does it have to do with that?

Thanks!

from vscode-copilot-release.

Just bought the yearly plan and the gpt chat widget in phpstorm says it's using GPT 3. WHY?

from vscode-copilot-release.