This Virtual Reality project is a demonstration of a number of Artificial Interlligence elements brought together in Unity for the purpose of training and developement focusing on Soft Skills in the area of Restorative Justice for use in Teacher Training.
Set up for use on the Meta/Oculus Quest 2, this project can be used as a template for any training application in VR. Follow the below guidance on how to amend the project to your needs.
- What you'll need to do
- Essential elements
The Microsoft TTS setup is based on Active Nick's Unity-Test-To-Speech project. By adapting his Speech Manager script, it is possible to call realtime TTS from the Microsoft Azure Cognitive Services.
To implement this in your project, you will need to set up (at least) a free account with the Azure services which will permit up to 5000 requests per month.
Once your account has been set up, you will then find your 'Speech Service API Key' and 'Speech Service Region' in the Keys and Endpoint section of your account portal.
This will then need to be added to each Speech Manager script in the Inspector window.
You will find that a number of the voices referenced by the script are no longer available since Microsoft have upgraded the voices to 'Nueral' (this will be tidied up at a later time). A number of availabe voices have been added at the top of the list, but should you wish to add others from the non-preview voices, open the script 'TTSClient', add a new reference to the enum 'VoiceName' in the format shown below and a new case to the public string 'ConvertVoiceNametoString', as shown below.
If we wanted to add the South African English voice Luke to the app...
...we would add the new entry to the enum formatted as shown here...
...and add a new case as show here.
The new voice is then available to select on the Speech Manager in the Inspector window.
The voice recognition is handled by Wit.ai, which is part of the Meta/Oculus Integration Package. There is thorough documentation on their website on how to use the system, but in short, you will need to log in either via your Facebook account.
You then would need to create a new app, and retrieve your Service Access token, which is to be put into Unity to link the two projects.
Retrive the Wit.ai Service Token from your app settings on their website here...
...and add it here within Unity.
You are then able to train the Wit.ai system to recognise key words and perform custom functions within Unity, though in this application we are only taking the string and using a string comparison algorithm to select our desired option.
Yarn Spinner is the pulse of the app, waiting for options, holding the voicelines for the characters, calling the voice recogition and the order of the flow. There is excellent documentaion on their website as well as great help on their Discord.
Any amendments to the script need to be made to the Yarn script which is attached to the current Yarn Project selected in the Dialogue Runner.
If being used for your own individual project, create a new Yarn project and Yarn script within the Project window.
When adding the line ending to the script, ensure to add the character name to the line ending, and make sure this matches exactly with the character name on the NPC prefab scripts, 'Character Yarn Line Handler' and 'Speech Manager'. Be sure to create a tag with the name as well.
This will link the script lines to the correct NPC, and send them to the correct TTS and Audio Source. Also ensure that the 'characterToTalk' variable within the Yarn script is set to the NPC's name.
The Meta/Oculus Lip Sync is also now included with the Oculus Integration package. Your character models will need to include blend shapes for the facial expressions of letter pronunciations. The Rocket Box collection now includes facial blend shapes, and can be used for free.
Be sure to drag the Skinned Mesh Renderer for your desired model on to the 'OVR Lip Sync Contect Morph Target'.
React
Typescript
Django
Laravel
Facebook
Microsoft
Google
Alibaba
D3
Tencent