microsoft / botfunctionaltestingservice Goto Github PK

A service for end-to-end (functional) testing of a bot. Programmatically simulate a user’s back-and-forth conversation with a bot, to test whether the bot behaves as expected.

License: MIT License

JavaScript 100.00%

botfunctionaltestingservice's Introduction

Bot Functional Testing Service

A service to enable functional testing of a Microsoft Bot Framework bot. A call to this service programmatically simulates a user’s back-and-forth conversation with a bot, to test whether the bot behaves as expected.

When calling the service, a Test is given as input. A Test is basically a “recording” of a user’s conversation with a bot. The Test is run against a given bot to check whether the conversation occurs as expected.

The service exposes a RESTful API for running Tests. The HTTP response code of an API call indicates whether the conversation had occurred as expected or not. If not, the response body contains information regarding the Test failure.

Creating a Test

In order to create Tests, you should work with the Bot Framework Emulator.

Note: After going through installation basics, make sure you configure ngrok as detailed here.

The simplest way to create a Test is to have a conversation with your bot within the emulator, then save the Transcript (.transcript file) as explained here. The Transcript can be used by itself as a Test for the service. The service will relate to the relevant information in the Transcript (ignoring conversation-specific details like timestamps, for example) and attempt to conduct a similar conversation, sending the user utterances to the bot and expecting the same bot replies.

Deployment

The service is a Node.js application. It can be installed using npm (npm install).

It can be easily deployed to Azure as an App Service:

Environment Variables

The service communicates with a bot, therefore it needs to know the bot's Web Chat secret key.

The service may communicate with multiple bots, so each bot should be identified by a logical name. All bots' secrets need to be defined in a single environment variable named SECRETS, which should include a string representing a JSON object. In this JSON object, each key (bot's name) is mapped to a value (bot's secret).

For example, let's assume we give the logical name 'samplebot' to the bot we would like to test, and that its Web Chat secret is '123'. Then we should have an environment variable named SECRETS set to the following string:

{"samplebot" : "123"}

In case you would like to test a single bot most of the time, you can define an environment variable called DefaultBot to specify the logical name of your default bot.

Note: If you are deploying the code sample using the "Deploy to Azure" option, you should set the variables in the Application Settings of your App Service.

Running a Test

There are several options for calling the service to run a Test, using HTTP GET or POST. There are also several ways to pass Test parameters to the service.

In all cases, the service needs to be aware of the bot to test. The target bot is identified by a logical name. This name can be passed as a 'bot' HTTP query parameter, e.g. '…?bot=bot-logical-name'. If no bot name is specified as a query parameter, the service uses the DefaultBot environment variable.

Using HTTP `POST` with a Transcript

The simplest way to run a test is to POST an HTTP request to the /test route of the service. The request body should contain the contents of a Transcript in JSON (application/json) format.

Assuming our target bot is named 'samplebot' and our service was deployed to Azure as 'testing123', the request query may look like:

https://testing123.azurewebsites.net/test?bot=samplebot

In case you have DeafultBot set to 'samplebot', the request may look like:

https://testing123.azurewebsites.net/test

Using HTTP `GET` with a Transcript URL

Instead of POST-ing the Transcript as the request body, you can store it somewhere and give its URL to the service as a 'url' HTTP query parameter in a GET HTTP request.

Let's assume that we have a Blob Storage account on Azure called 'samplestorageaccount', and we uploaded a Transcript file called 'Sample.transcript' to a container called 'tests'. The corresponding request query may look like:

https://testing123.azurewebsites.net/test?bot=samplebot&url=https://samplestorageaccount.blob.core.windows.net/tests/Sample.transcript

Contributing

This project welcomes contributions and suggestions. Most contributions require you to agree to a Contributor License Agreement (CLA) declaring that you have the right to, and actually do, grant us the rights to use your contribution. For details, visit https://cla.microsoft.com.

When you submit a pull request, a CLA-bot will automatically determine whether you need to provide a CLA and decorate the PR appropriately (e.g., label, comment). Simply follow the instructions provided by the bot. You will only need to do this once across all repos using our CLA.

This project has adopted the Microsoft Open Source Code of Conduct. For more information see the Code of Conduct FAQ or contact [email protected] with any additional questions or comments.

botfunctionaltestingservice's People

Contributors

Stargazers

Watchers

Forkers

iboonz jayd2446 vdedyukhin guytau mosabsamara memassamara bhaskers-blu-org2 taffywrinkle claudiusgonzo standardgalactic test-mass-forker-org-1

botfunctionaltestingservice's Issues

My transcript file contains two message interactions however direct line activities sis only returning one message

In directlineclient.js line 106, with the following transcript json my messages.length is always less than my nExpectedActivities.
My messages.length = 1 and my nExpectedActivities = 2

This call
https://directline.botframework.com/v3/directline/conversations/3FXT8Ynl90pfTNpNuAF4O-6/activities
only return one activity. I noticed that the user answer/response was not being pulled but the question was being pulled.
I would expect a messages.length of 2.

I am using the latest BOT composer and Emulator.
What is the expectation of the transcript file? Was directlineclient.js written to handle a different transcript format?

Thank you!

Got 1 total activities (including user's response)
We have less than expected 2 activities - retry number 18...



pollMessages messages: [
    {
        "type": "message",
        "id": "2cTuGxy7T2BDk7MnymzbHf-j|0000000",
        "timestamp": "2021-06-23T16:10:28.6162545Z",
        "channelId": "directline",
        "from": {
            "id": "cw-first-bot-poc",
            "name": "cw-first-bot-poc"
        },
        "conversation": {
            "id": "2cTuGxy7T2BDk7MnymzbHf-j"
        },
        "text": "What is your age?",
        "speak": "What is your age?",
        "inputHint": "expectingInput",
        "replyToId": "HNiTisR5DB3"
    }
]

failure: "Test #0: User message 'undefined' response failed - Could not obtain 2 responses"

TRANSCRIPT JSON:

[
  {
    "type": "conversationUpdate",
    "membersAdded": [
      {
        "id": "83d3e8b0-d42a-11eb-8404-09f92c9ce3b2",
        "name": "Bot"
      },
      {
        "id": "3b224ec6-5943-49cb-9e56-88974ea58c56",
        "name": "User"
      }
    ],
    "membersRemoved": [],
    "channelId": "emulator",
    "conversation": {
      "id": "83e0e100-d42a-11eb-8492-297aa9d16fb6|livechat"
    },
    "id": "84214560-d42a-11eb-8492-297aa9d16fb6",
    "localTimestamp": "2021-06-23T09:54:23-04:00",
    "recipient": {
      "id": "83d3e8b0-d42a-11eb-8404-09f92c9ce3b2",
      "name": "Bot",
      "role": "bot"
    },
    "timestamp": "2021-06-23T13:54:23.540Z",
    "from": {
      "id": "3b224ec6-5943-49cb-9e56-88974ea58c56",
      "name": "User",
      "role": "user"
    },
    "locale": "en-US",
    "serviceUrl": "https://072328c54bb5.ngrok.io"
  },
  {
    "type": "message",
    "serviceUrl": "https://072328c54bb5.ngrok.io",
    "channelId": "emulator",
    "from": {
      "id": "83d3e8b0-d42a-11eb-8404-09f92c9ce3b2",
      "name": "Bot",
      "role": "bot"
    },
    "conversation": {
      "id": "83e0e100-d42a-11eb-8492-297aa9d16fb6|livechat"
    },
    "recipient": {
      "id": "3b224ec6-5943-49cb-9e56-88974ea58c56",
      "role": "user"
    },
    "locale": "en-US",
    "text": "Welcome to your bot.",
    "speak": "<speak version='1.0' xmlns='http://www.w3.org/2001/10/synthesis' xml:lang='en-US'><voice name='en-US-AriaNeural'>Welcome to your bot.</voice></speak>",
    "replyToId": "84214560-d42a-11eb-8492-297aa9d16fb6",
    "id": "87f9ebb0-d42a-11eb-8492-297aa9d16fb6",
    "localTimestamp": "2021-06-23T09:54:29-04:00",
    "timestamp": "2021-06-23T13:54:29.995Z"
  },
  {
    "type": "message",
    "serviceUrl": "https://072328c54bb5.ngrok.io",
    "channelId": "emulator",
    "from": {
      "id": "83d3e8b0-d42a-11eb-8404-09f92c9ce3b2",
      "name": "Bot",
      "role": "bot"
    },
    "conversation": {
      "id": "83e0e100-d42a-11eb-8492-297aa9d16fb6|livechat"
    },
    "recipient": {
      "id": "3b224ec6-5943-49cb-9e56-88974ea58c56",
      "role": "user"
    },
    "locale": "en-US",
    "text": "What is your age?",
    "speak": "<speak version='1.0' xmlns='http://www.w3.org/2001/10/synthesis' xml:lang='en-US'><voice name='en-US-AriaNeural'>What is your age?</voice></speak>",
    "inputHint": "expectingInput",
    "replyToId": "84214560-d42a-11eb-8492-297aa9d16fb6",
    "id": "880a3f60-d42a-11eb-8492-297aa9d16fb6",
    "localTimestamp": "2021-06-23T09:54:30-04:00",
    "timestamp": "2021-06-23T13:54:30.101Z"
  },
  {
    "type": "trace",
    "serviceUrl": "https://072328c54bb5.ngrok.io",
    "channelId": "emulator",
    "from": {
      "id": "83d3e8b0-d42a-11eb-8404-09f92c9ce3b2",
      "name": "Bot",
      "role": "bot"
    },
    "conversation": {
      "id": "83e0e100-d42a-11eb-8492-297aa9d16fb6|livechat"
    },
    "recipient": {
      "id": "3b224ec6-5943-49cb-9e56-88974ea58c56",
      "role": "user"
    },
    "locale": "en-US",
    "replyToId": "84214560-d42a-11eb-8492-297aa9d16fb6",
    "label": "Bot State",
    "valueType": "https://www.botframework.com/schemas/botState",
    "value": {
      "dialog": {
        "eventCounter": 1,
        "_adaptive": {
          "actions": [
            {
              "dialogId": "ActionScope[TextInput,SendActivity(${SendActivity_YazHvo()})]",
              "options": null,
              "dialogStack": [
                {
                  "id": "TextInput",
                  "state": {
                    "options": null,
                    "turnCount": 1,
                    "value": null
                  },
                  "StackIndex": null,
                  "Version": null
                },
                {
                  "id": "ActionScope[TextInput,SendActivity(${SendActivity_YazHvo()})]",
                  "state": {
                    "offset": 0
                  },
                  "StackIndex": null,
                  "Version": null
                }
              ]
            }
          ]
        }
      },
      "this": {
        "options": null,
        "turnCount": 1,
        "value": null
      },
      "conversation": {
        "DialogState": {
          "dialogStack": [
            {
              "id": "FirstBot",
              "state": {
                "eventCounter": 2,
                "_adaptive": {
                  "actions": [
                    {
                      "dialogId": "ActionScope[Foreach(=`turn.Activity.membersAdded`),BeginDialog.../HMn7IuEwg1pzJR1C1ESjIlSTHKboW2OK+VcPrjmxIc=]",
                      "options": null,
                      "dialogStack": [
                        {
                          "id": "Age",
                          "state": {
                            "eventCounter": 1,
                            "_adaptive": {
                              "actions": [
                                {
                                  "dialogId": "ActionScope[TextInput,SendActivity(${SendActivity_YazHvo()})]",
                                  "options": null,
                                  "dialogStack": [
                                    {
                                      "id": "TextInput",
                                      "state": {
                                        "options": null,
                                        "turnCount": 1,
                                        "value": null
                                      },
                                      "StackIndex": null,
                                      "Version": null
                                    },
                                    {
                                      "id": "ActionScope[TextInput,SendActivity(${SendActivity_YazHvo()})]",
                                      "state": {
                                        "offset": 0
                                      },
                                      "StackIndex": null,
                                      "Version": null
                                    }
                                  ]
                                }
                              ]
                            }
                          },
                          "StackIndex": null,
                          "Version": "FNSPQcBzNtm+Z9NANQxtvRTSLobamkBkUwsO1Na1zKA="
                        },
                        {
                          "id": "BeginDialog[Age]",
                          "state": {},
                          "StackIndex": null,
                          "Version": null
                        },
                        {
                          "id": "ActionScope[Foreach(=`turn.Activity.membersAdded`),BeginDialog.../HMn7IuEwg1pzJR1C1ESjIlSTHKboW2OK+VcPrjmxIc=]",
                          "state": {
                            "offset": 1
                          },
                          "StackIndex": null,
                          "Version": null
                        }
                      ]
                    }
                  ]
                },
                "foreach": {
                  "index": 1,
                  "value": {
                    "id": "3b224ec6-5943-49cb-9e56-88974ea58c56",
                    "name": "User"
                  }
                }
              },
              "StackIndex": null,
              "Version": "8NSC3+47aKLKSiw7atczY4839850/2guTQz+9s4vHSY="
            }
          ]
        }
      },
      "user": {}
    },
    "name": "BotState",
    "id": "881562f0-d42a-11eb-8492-297aa9d16fb6",
    "localTimestamp": "2021-06-23T09:54:30-04:00",
    "timestamp": "2021-06-23T13:54:30.175Z"
  },
  {
    "channelData": {
      "clientActivityID": "1624456475268uei7iztlgn",
      "clientTimestamp": "2021-06-23T13:54:35.269Z"
    },
    "text": "263",
    "textFormat": "plain",
    "type": "message",
    "channelId": "emulator",
    "from": {
      "id": "3b224ec6-5943-49cb-9e56-88974ea58c56",
      "name": "User",
      "role": "user"
    },
    "locale": "en-US",
    "timestamp": "2021-06-23T13:54:35.315Z",
    "entities": [
      {
        "requiresBotState": true,
        "supportsListening": true,
        "supportsTts": true,
        "type": "ClientCapabilities"
      }
    ],
    "conversation": {
      "id": "83e0e100-d42a-11eb-8492-297aa9d16fb6|livechat"
    },
    "id": "8b25b030-d42a-11eb-8492-297aa9d16fb6",
    "localTimestamp": "2021-06-23T09:54:35-04:00",
    "recipient": {
      "id": "83d3e8b0-d42a-11eb-8404-09f92c9ce3b2",
      "name": "Bot",
      "role": "bot"
    },
    "serviceUrl": "https://072328c54bb5.ngrok.io"
  },
  {
    "type": "trace",
    "timestamp": "2021-06-23T13:54:36.054Z",
    "serviceUrl": "https://072328c54bb5.ngrok.io",
    "channelId": "emulator",
    "from": {
      "id": "83d3e8b0-d42a-11eb-8404-09f92c9ce3b2",
      "name": "Bot",
      "role": "bot"
    },
    "conversation": {
      "id": "83e0e100-d42a-11eb-8492-297aa9d16fb6|livechat"
    },
    "recipient": {
      "id": "3b224ec6-5943-49cb-9e56-88974ea58c56",
      "role": "user"
    },
    "locale": "en-US",
    "replyToId": "8b25b030-d42a-11eb-8492-297aa9d16fb6",
    "label": "LuisV3 Trace",
    "valueType": "https://www.luis.ai/schemas/trace",
    "value": {
      "recognizerResult": {
        "text": "263",
        "alteredText": null,
        "intents": {
          "None": {
            "score": 0.91405714
          }
        },
        "entities": {}
      },
      "luisModel": {
        "ModelID": "5f0d3bcb-4adc-4f41-8538-3e5284e984bb"
      },
      "luisOptions": {
        "IncludeAllIntents": false,
        "IncludeInstanceData": false,
        "IncludeAPIResults": false,
        "Log": true,
        "DynamicLists": null,
        "ExternalEntities": null,
        "PreferExternalEntities": true,
        "DateTimeReference": null,
        "Slot": "production",
        "Version": null
      },
      "luisResult": {
        "query": "263",
        "prediction": {
          "topIntent": "None",
          "intents": {
            "None": {
              "score": 0.91405714
            }
          },
          "entities": {}
        }
      }
    },
    "name": "LuisRecognizer",
    "id": "8b967360-d42a-11eb-8492-297aa9d16fb6",
    "localTimestamp": "2021-06-23T09:54:36-04:00"
  },
  {
    "type": "message",
    "serviceUrl": "https://072328c54bb5.ngrok.io",
    "channelId": "emulator",
    "from": {
      "id": "83d3e8b0-d42a-11eb-8404-09f92c9ce3b2",
      "name": "Bot",
      "role": "bot"
    },
    "conversation": {
      "id": "83e0e100-d42a-11eb-8492-297aa9d16fb6|livechat"
    },
    "recipient": {
      "id": "3b224ec6-5943-49cb-9e56-88974ea58c56",
      "role": "user"
    },
    "locale": "en-US",
    "text": "You are still young!",
    "speak": "<speak version='1.0' xmlns='http://www.w3.org/2001/10/synthesis' xml:lang='en-US'><voice name='en-US-AriaNeural'>You are still young!</voice></speak>",
    "replyToId": "8b25b030-d42a-11eb-8492-297aa9d16fb6",
    "id": "8ba5b5a0-d42a-11eb-8492-297aa9d16fb6",
    "localTimestamp": "2021-06-23T09:54:36-04:00",
    "timestamp": "2021-06-23T13:54:36.154Z"
  },
  {
    "type": "trace",
    "serviceUrl": "https://072328c54bb5.ngrok.io",
    "channelId": "emulator",
    "from": {
      "id": "83d3e8b0-d42a-11eb-8404-09f92c9ce3b2",
      "name": "Bot",
      "role": "bot"
    },
    "conversation": {
      "id": "83e0e100-d42a-11eb-8492-297aa9d16fb6|livechat"
    },
    "recipient": {
      "id": "3b224ec6-5943-49cb-9e56-88974ea58c56",
      "role": "user"
    },
    "locale": "en-US",
    "replyToId": "8b25b030-d42a-11eb-8492-297aa9d16fb6",
    "label": "Bot State",
    "valueType": "https://www.botframework.com/schemas/botState",
    "value": {
      "conversation": {
        "DialogState": {
          "dialogStack": []
        }
      },
      "user": {}
    },
    "name": "BotState",
    "id": "8bafc7c0-d42a-11eb-8492-297aa9d16fb6",
    "localTimestamp": "2021-06-23T09:54:36-04:00",
    "timestamp": "2021-06-23T13:54:36.220Z"
  }
]

Deploy to Azure not working

The deploy to Azure returns "site could not be reached"

This repo is missing important files

There are important files that Microsoft projects should all have that are not present in this repository. A pull request has been opened to add the missing file(s). When the pr is merged this issue will be closed automatically.

Microsoft teams can learn more about this effort and share feedback within the open source guidance available internally.

Merge this pull request

Recommend Projects

React

A declarative, efficient, and flexible JavaScript library for building user interfaces.
Vue.js

🖖 Vue.js is a progressive, incrementally-adoptable JavaScript framework for building UI on the web.
Typescript

TypeScript is a superset of JavaScript that compiles to clean JavaScript output.
TensorFlow

An Open Source Machine Learning Framework for Everyone
Django

The Web framework for perfectionists with deadlines.
Laravel

A PHP framework for web artisans
D3

Bring data to life with SVG, Canvas and HTML. 📊📈🎉

Recommend Topics

javascript

JavaScript (JS) is a lightweight interpreted programming language with first-class functions.
web

Some thing interesting about web. New door for the world.
server

A server is a program made to process requests and deliver data to clients.
Machine learning

Machine learning is a way of modeling and interpreting data that allows a piece of software to respond intelligently.
Visualization

Some thing interesting about visualization, use data art
Game

Some thing interesting about game, make everyone happy.

Recommend Org

Facebook

We are working to build community through open source technology. NB: members must have two-factor auth.
Microsoft

Open source projects and samples from Microsoft.
Google

Google ❤️ Open Source for everyone.
Alibaba

Alibaba Open Source for everyone
D3

Data-Driven Documents codes.
Tencent

China tencent open source team.