Develop and Troubleshoot Nuance Mix AI Powered Chat Bots locally and on Microsoft Azure.

• Overview
• About the Client
• Setup
• Functionality
• Pre-Requisites
• Quick Start 🚀
• Getting Started
• Conventions for Rich UI
• Conventions for Link Handling
• Data Access Integrations
• Publishing to Azure
• License

This demo client offers a sample integration to Nuance Mix's Conversational AI Runtime Services, specifically

• Dialog (DLGaaS)
• Natural Language Understanding (NLUaaS)
• Text to Speech (TTSaaS) and the
• Runtime Event Log API

The client uses the HTTP/1.1 transcoded version of the APIs (rather than the native HTTP/2 gRPC versions).

The prime purpose of this tool is to assist in Bot development and troubleshooting, including offering controlled hosted availability using Azure StaticWebApps.

Azure Functions with Core Tools are used throughout the development and deployment lifecycle.

For more information on how to leverage the client across various environments, see networking options, and be sure to secure your functions. Monitoring is made available through Application Insights.

Disclaimer: This sample client is not intended to illustrate a production-ready implementation; it is intended to aid development, demonstration, troubleshooting, and other such use cases.

• Gatsby, React and React-Bootstrap for the Client/Frontend
• Azure Functions Python for the API
  • Acts as a Proxy for Nuance Services, with Data Integrations for Mix.dialog/DLGaaS

Please see the Exchanging data section in the Mix documentation. It is crucial to understand the concepts as you design your Mix projects and leverage this client -- it acts more like a gateway.

Use Data Access nodes with the client_fetch configuration and write the integration layer locally with the intention of separate hosted Functions (through external_fetch). These Functions would be referenced and configured within Mix.dialog and Mix.dashboard respectively.

The following illustrates a scenario where the client is deployed to Azure, and Data Access nodes have been configured to use external_fetch within Mix.

This simplifies the client handling, deferring to the Functions themselves, and offers lifecycle controls within Mix.

⚠️ There may be scenarios where client_fetch is appropriate; this has been set up such that dlgaas.js will invoke local ClientFetchHandlers.

• Bot engagements using Nuance Mix's DLGaaS HTTP/1.1 API (not HTTP/2 gRPC)
  • Text-Only (no Streams, ie. ASR/TTS orchestration yet)
  • PRESENTATION LAYER: Rich UI through conventions
  • DATA: Functions for development through Client Fetch, and when deployed, enables External Fetch usage
  • DATA: Client Fetch handler for Location Data supplied in userData
  • DATA: Log Events, Filtering and Timeline Visualization
• Natural Language Understanding using NLUaaS HTTP/1.1 API (not HTTP/2 gRPC)
• Text to Speech Synthesis using TTSaaS HTTP/1.1 API (not HTTP/2 gRPC)
• Event data using the Log Events HTTP/1.1 API

• Azure Functions Core Tools
• Node.js 14.3+ (Client Toolchain)
• Python 3.7+ (Functions)
• Docker Engine 20.10.0+ (Local development)

Skip this if you already have certificates.

brew install mkcert

mkcert localhost 127.0.0.1 ::1
mkcert -install

openssl pkcs12 -export -out certificate.pfx -inkey localhost+2-key.pem -in localhost+2.pem

Download pre-build mkcert.exe and openssl.exe binaries.

mkcert.exe localhost 127.0.0.1 ::1
certutil -addstore "Root" "C:\Users\<USERNAME>\AppData\Local\mkcert\rootCA.pem"

openssl.exe pkcs12 -export -out certificate.pfx -inkey localhost+2-key.pem -in localhost+2.pem

Download pre-build mkcert binary.

mkcert localhost 127.0.0.1 ::1
mkcert -install

openssl pkcs12 -export -out certificate.pfx -inkey localhost+2-key.pem -in localhost+2.pem

• Generated certificate files certificate.pfx, localhost+2.pem and localhost+2-key.pem should be present in the project's base folder.
• Supply the password used when generating certificates in a file .password.

docker-compose build
docker-compose up -d

Launch https://localhost:8000

Note:

• Override environment variable values in .env file as needed.
• If facing issues, try with docker-compose restart.

npm install

cd api/
python3 -m venv env
source env/bin/activate
pip install -r requirements.txt

cp sample/local.settings.json api/local.settings.json

# Process 1 - Start the API
cd api/
func start --useHttps --cert ../certificate.pfx --password "<REPLACE_ME>"

# Process 2 - Start the Client
npm run develop -- --https --cert-file localhost+2.pem --key-file localhost+2-key.pem

Launch https://localhost:8000

This set up recommends using HTTPS. A self signed cert will be necessary.

⚠️ Take note of your password when creating the cert, you will need it when running the function.

For macOS users:

brew install mkcert

mkcert localhost 127.0.0.1 ::1
mkcert -install

openssl pkcs12 -export -out certificate.pfx -inkey localhost+2-key.pem -in localhost+2.pem

For Windows users:

Download pre-build binaries of mkcert.exe and openssl.exe.

mkcert.exe localhost 127.0.0.1 ::1

openssl.exe pkcs12 -export -out certificate.pfx -inkey localhost+2-key.pem -in localhost+2.pem

Import generated rootCA certificate to Windows Trusted Root Certification Authorities store.

# Run in elevated command prompt window
certutil -addstore "Root" "C:\Users\<USERNAME>\AppData\Local\mkcert\rootCA.pem"

See the official Azure Functions Reference - Python for more information.

Update the following environment variables to override the default values:

export oauth_server_url="https://auth.crt.nuance.com/oauth2"
export base_url_dlgaas="https://dlg.api.nuance.com/dlg/v1"
export base_url_nluaas="https://nlu.api.nuance.com/nlu/v1"
export base_url_ttsaas="https://tts.api.nuance.com/api/v1"
export base_url_logapi="https://log.api.nuance.com"
export oauth_scope="dlg nlu tts log"

Note: When deployed, these can be set per deployment environment in the Azure Portal. Override values as needed in .env when using docker-compose

⚠️ You must add this file before running your function.

In the api directory, create a file local.settings.json, and populate with the following snippet.

This will enable CORS for the function, and set a worker count as desired. Be mindful of the http vs https distinction. By default, the application is set up to use https.

{
  "IsEncrypted": false,
  "Host": {
    "CORS": "https://localhost:8000"
  },
  "Values": {
    "AzureWebJobsStorage": "",
    "FUNCTIONS_WORKER_RUNTIME": "python",
    "FUNCTIONS_WORKER_PROCESS_COUNT": 10
  }
}

For convenience, a sample is available to copy:

cp sample/local.settings.json api/local.settings.json

This sample offers a stub integration with SendGrid for email capabilities. This can be used if creating a Data Access node with the name Server_Send_Email, which in turn will call the email-api-send locally.

To configure this integration, provide the following environment variables:

export sendgrid_api_key="<REPLACE_ME>"
export sendgrid_from_email="<REPLACE_ME>"
export sendgrid_custom_token="<REPLACE_ME>"

Note: Override values as needed in .env when using docker-compose ⚠️ The custom token must be provided in the Mix project named SENDGRID_TOKEN. This is to thwart unintentional usage.

The following instructions assume the creation of a fresh virtual environment, followed by installing those dependencies in it.

cd api/
# Create the virtual environment
python3 -m venv env
# Use the environment
source env/bin/activate
# Install dependencies
pip install -r requirements.txt

⚠️ Ensure you are in the virtual environment when running. (ref. source env/bin/activate)

func start --useHttps --cert ../certificate.pfx --password "<REPLACE_ME>"

By default, the Functions should be served on https://localhost:7071.

func start

By default, the Functions should be served on http://localhost:7071.

⚠️ Check the client's shared.js to change ports during development.

Perform the following in the root directory:

npm install

Reference the Certificates section earlier. The following assumes the creation of a cert with localhost+2.

npm run develop -- --https --cert-file localhost+2.pem --key-file localhost+2-key.pem

By default, the webapp should be served on https://localhost:8000 - launch this in your browser.

⚠️ Location capabilities will not be usable (ie. navigation.geolocation).

npm run develop

By default, the webapp should be served on http://localhost:8000 - launch this in your browser.

This client employs certain conventions within Mix.dialog to offer special types of rendering in the presentation layer.

To take advantage of these, navigate to:

• QA node > Node Properties
  • User Input > Optional > Interactivity

Provide the following strings in the "type" field.

Special Input Types for QA Nodes:

• email
• phone
• currency
• date

To provide hints (aka placeholders), the sendData must include a variable with the type followed by 'Hint', ie. 'emailHint', 'phoneHint' and so on.

For Interactivity (aka selectables):

• carousel
• buttons
• colorpicker (description should contain the HEX value)

Note: when using Rich Text markup in messages, if elements contains classes from Bootstrap they will be rendered accordingly in this client. See chat.js to see the allowed DOM elements and attributes.

Images can be rendered when using selectables and the carousel type. Edit STUB_SELECTABLE_IMAGES in shared.js to experiment with resources.

This client employes certain conventions within Mix.dialog to offer special types of event handling in the presentation layer.

To take advantage of these, simply mark up your Rich Text messages appropriately with the data attributes, depending on the intended interaction.

To trigger an intent or entity selection, leverage the following attributes:

• data-mix-action="selectable"
• data-mix-selectable-id - can be INTENT or the entity name
• data-mix-selectable-value

<a href="#"
  data-mix-action="selectable"
  data-mix-selectable-id="INTENT"
  data-mix-selectable-value="iBuyPhone">
    Buy a Phone
</a>

<a href="#"
  data-mix-action="selectable"
  data-mix-selectable-id="ePhoneCapacity"
  data-mix-selectable-value="128GB">
    128GB
</a>

To trigger a simulated user input, leverage the following attributes:

• data-mix-action="input"
• data-mix-input

<a href="#"
  data-mix-action="input"
  data-mix-input="$500">
    $500
</a>

This client offers developers the ability to use the client_fetch mode of the Data Accss node in Mix.dialog and integrate with an Azure Function.

This pattern would apply in a gateway-style integration, however the intention of this set up, is to eventually have the integrations use external_fetch pointing to deployed Functions.

Create a new Function:

./scripts/create-da-handler.sh "weather-api-city-conditions"

Navigate to api/weather-api-city-conditions/__init__.py and start integrating.

By default, POST requests are expected with the body containing the sendData payload. Update api/weather-api-city-conditions/function.json if other methods are desired.

Once the function has been set up, add a handler to ExternalFetchHandlers in dlgaas.js named with the Data Access node's name, pointing to the newly defined endpoint.

There are cases where the use of Mix.dialog Data Access node's client fetch is intended for the end client (vs. the gateway pattern employed here).

Add a handler to ClientFetchHandlers in dlgaas.js named with the Data Access node's name.

At this time, geolocation (lat/lng) is supported through the HTML5 API, when the client is running securely.

To request the client to provide it's location, set the Data Access node's ID to Client_Location_LatLng, and this will return a location object.

The data integration layer in this client is handled through Functions, and complimented by sample integrations in api/providers.py.

Each data access access is represented by it's own Function, in preparation for use when deployed and hosted.

This client illustrates integrations with: Yahoo Finance, AccuWeather, SendGrid and a Mock Phone Store.

This API does not require authentication, in this case: the requests are made with no further checks.

• POST /api/finance-api-asset-price

Third party service authentication is required and done through passing the token in the payload of the request. In this case: apiKeyAccuweather is used with the service and must be saved in the session (stored in a variable).

• POST /api/weather-api-city-search
• POST /api/weather-api-city-conditions

Third party service authentication is configured as part of the Function, but a SENDGRID_TOKEN is required to execute calls from known sources.

In this case: the token must been assigned in Mix at the PROJECT level.

• POST /api/email-api-send

This integration illustrates the use of Dynamic Wordsets, consistently filtering options based on the user's selection, derived from static data.

• POST /api/store-api-filter-phone-wordset
• POST /api/store-api-request-purchase
• POST /api/store-api-purchase

To deploy this client, follow the Azure StaticWebApps deployment guide to publish. Update static/staticwebapp.config.json as needed.

Essentially:

1. Create a GitHub Repo
2. Create an Azure StaticWebApp pointing to the GitHub Repo (use: gatsby, api/)
3. Configure accordingly in Azure
   • Create a Application Insights resource and link to StaticWebApp (APPINSIGHTS_INSTRUMENTATIONKEY)
   • Add environment variables (sengrid_api_key, sendgrid_from_email)

This source code is licensed under the Apache-2.0 license found in the LICENSE.md file in the root directory of this source tree.