Hi, I have this error while running the local-models/Phi, may I know how to resolve it?

My system is Ubuntu 22.04 and runs as CPU. I have tried to change to the latest version of TorchSharp 0.102.4, the error was the same.

Loading Phi2 from huggingface model weight folder
Unhandled exception. System.ArgumentOutOfRangeException: Length of JSON exceeded int.MaxValue, not supported yet (Parameter 'length')
at TorchSharp.PyBridge.Safetensors.LoadIndex(Stream stream)
at TorchSharp.PyBridge.PyBridgeModuleExtensions.load_safetensors(Module module, Stream stream, Boolean strict, IList1 skip, Dictionary2 loadedParameters, Boolean leaveOpen)
at TorchSharp.PyBridge.PyBridgeModuleExtensions.load_safetensors(Module module, String location, Boolean strict, IList1 skip, Dictionary2 loadedParameters)
at TorchSharp.PyBridge.PyBridgeModuleExtensions.load_checkpoint(Module module, String path, String checkpointName, Boolean strict, IList1 skip, Dictionary2 loadedParameters)
at PhiForCasualLM.FromPretrained(String modelFolder, String configName, String checkPointName, ScalarType defaultDType, String device) in /home/xxxx/ai-samples/src/local-models/Phi/Phi.cs:line 45
at Program.

Thank you.

Hi team,

LLamaSharp has already supported LLaMA2 and LLaMA3. Would you like to accept a PR which adds examples of LLaMA2 & LLaMA3 with it?

Should we add the model's response in as a ChatRequestAssistantMessage? This way we can show it creating the chatbot-like experience?

Create a full-stack app that showcases Aspire using components like:

• Hosted Model Services
  • Text / Chat Inference (i.e. HuggingFace Text Inference)
  • Embedding Inference (i.e. HuggingFace Embedding Inference)
• Vector Database (i.e. Milvus / Qdrant)
• Chat UI (i.e. HuggingFace Chat UI)
• Use SK for orchestration

Dependencies

• dotnet/aspire#2869
• dotnet/aspire#2737
• dotnet/aspire#2189

We need to have a location that can support both gpt-35-turbo and dall-e-3
we also need a gpt model that support tools call.

After looking at the docs https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models#gpt-35-turbo-model-availability

We will use
location: AusEast
gpt-35-turbo version: 1106

Sample 1

Spell error in initilization

Sample 2

• spell error: compilation -> completion
  

• spell error: whilw -> while
  

• resonse -> response
  

Sample 3

• The LLM invokes the funciton call before I even start asking my age when I use gpt-3.5-turbo
  

Sample 4

• initizing -> initializing and kerenel -> kernel
  

Sample 5

• Directoty -> Directory
  

• Whitespace missing and --prerelease tag is missing
  

• The highlighted using statement has already been added in previous steps
  

• Need to disable SKEXP warning between step 4 and 5
  

Sample 6

• Maksure -> make sure
  

• missing suppress skexp warning in source
  

• missing Kernel after Semantic
  

Sample 7

• The Complete sample project section is still for Sample 6
  

• Sevice -> Service
  

• kerel -> kernel
  

• Miss semi-colomn at the end of code snippet
  

• Not clear on how to invoke web api from console. Do we miss the setup for openapi page?
  

Great Idea!

Open GitHub discussion to Speed Up Iteration.

Privide links to Direct Users to Discords.

Work on CodeGen tokenizer needed for Phi-2 will be complete in Microsoft.ML.Tokenizers in the next few days.

Once it's available, consider replacing the current tokenizer implementation with the one provided by Microsoft.ML.Tokenizers.

https://github.com/dotnet/ai-samples/blob/main/src/local-models/Phi/Tokenizer.cs

During the azd up command you may encounter and error with like this:

'pwsh' is not recognized as an internal or external command

It's because no recent (v7+) PowerShell version has been installed on the Windows computer.

To fix the issue you need to execute the command dotnet tool update --global PowerShell

Let's update the documentation to prevent this error.

The sample(s) should focus on only basic chat with a web UI, which can serve as a starting point template for future samples or new exploratory projects.

With this, we should provide a few different options for front-end UIs:

• .NET back-end + JavaScript front-end
• .NET back-end + Blazor front-end
• .NET back-end + .NET MAUI front-end

We should help make sure folks create resources in the correct region and know how to choose model version.

Related PR Comment:
#26 (comment)

Model/Region Docs:
https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/models#gpt-35-turbo-model-availability

Today, the Python Evaluation building block can be used against a .NET backend that uses the Chat Protocol (Azure Search supports this). However, we know from customer feedback (A top request from MVPs) that they want a .NET native solution for this. The goal is to deliver .NET-specific building block documentation and sample code to support this. Currently this is still in a design phase, but the latest thinking is that we may adapt a community DotNet LLM Eval sample that was built by a fellow Microsoft employee (Maho Pacheco).

Work to be done:
• Evaluate options for LLM evaluation and land on the best practice guidance we’d give to devs. Decide if we’ll lead with the Python implementation for Build.
• Continue investigating what is needed for a .NET native evaluation block
• If continuing with adapting the DotNet LLM Eval sample, we’d need to:
o Likely provide an analysis UI that isn’t Grafana, for now aiming for parity with Pamela’s output.
o Figure out a solution for generating questions and answers to use as “grounded truth” – currently the Azure AI synthetics library, used by the Python block, does not have a .NET equivalent.
o Update to Support Semantic Kernel 1.x.
• Create guidance documentation

Documentation Still Refers to 'Getting Started' folder, which has been Renamed

Local dev environment setup could be simplified using Winget configuration files.

Add Winget configuration to install the following prerequisites

https://github.com/dotnet/ai-samples/blob/main/src/quickstarts/README.md#prerequisites

https://github.com/dotnet/ai-samples/blob/u/jmatthiesen/addWingetConfig/configurations/configuration.dsc.yml

• Hello LLM
• Add Chat History
• Add Context/Data
• Add Function Call

The link Getting-started: Deploying the Azure resources is broken

• Hello LLM
• Add Chat History
• Add Context/Data
• Add Function Call

Describe the issue:
Word spelling error.
Hike Images samples (link1, link2) in the Semantic Kernel and Azure OpenAI SDK directories have this issue.

Suggestion:
Change to dall.

Describe the issue:
Grammatical errors.
All samples in the Semantic Kernel and Azure OpenAI SDK directories have this issue.


Suggestion:

1. Change to other samples or another sample.
2. Remove redundant minute.

@hemarina and @FBoucher for notification.

In samples that use models, provide similar samples that use hosted model providers:

• Azure AI Model Catalog
  • Phi
  • Llama
  • Mixtral
  • Stable Diffusion
  • CLIP
• HuggingFace Inference API
• Ollama
• Llamafile
• LM Studio