microsoft / botframework-functionaltests Goto Github PK

Make test for a bot resume conversation from where it left off, after leaving on a tangent

Currently in botbuilder-dotnet/FunctionalTests/Skills/DialogToDialog there are bots that the team runs manually to tests the dialog features in skills.

We should instead automate this process, as done already for Skills with send a message and receiving an echo in the BotFramework-FunctionalTests repo.

Skills Functional Test Scenarios:

• Multi-turn aspect of dialogs (state properly maintained between multiple bots) #50
  • Also test for a bot that is simultaneously both a Host and a Skill
• Tangent #75
• Cancelling #76

[enhancement]

I don't know if this is something we should have or not, but it came up in a discussion with @stevengum

Add functional tests from Composer bots (for now we just test them against each other)

Several waterfall steps are failing after the latest updates to the host and the skill, we need to fix them to make sure we can merge.

To expedite failure investigations, we should have a quick and easy way to determine what version of Bot Builder is running in the pipeline.

Add tags for host and skill separately:

• BotBuilderVersionHost=4.11.0-daily.20200907.164245 (preview)
• BotBuilderVersionSkill=4.10.2 (stable)

DotNet21DotNet21SkillBotFunctionalTest
DotNet21DotNet31SkillBotFunctionalTest
DotNet31DotNet31SkillBotFunctionalTest
https://fuselabs.visualstudio.com/SDK_v4/_build?definitionScope=%5CSkillBots

TODO: Add description and resolution here.

A clean up for the project is needed.

• Update Skill Manifests
• Remove unnecessary markdowns and DotNet .sln files
• Add BotNamesSuffix variable to the pipelines
• Add CleanUp Pipeline

There're missing tests for Waterfall Bot that should be added.

Tests missing:

• Cards
• Proactive Messages
• Attachments

Make tests for "remote cancelling" in skills

Currently in botbuilder-dotnet/FunctionalTests/Skills/DialogToDialog there are bots that the team runs manually to tests the dialog features in skills.

We should instead automate this process, as done already for Skills with send a message and receiving an echo in the BotFramework-FunctionalTests repo.

Skills Functional Test Scenarios:

• Multi-turn aspect of dialogs (state properly maintained between multiple bots) #50
  • Also test for a bot that is simultaneously both a Host and a Skill
• Tangent #75
• Cancellting #76

[enhancement]

We implemented a workaround for this issue in EchoTests.

Once the but is fixed in the SDK, we should remove the workaround code (highlighted below) from the test and update the script to stop using the ${ExpectedEocCode} token (the value should be always completedSuccessfully).

Some things we need to know:
How long does it take to deploy?
How long does it take to build?
How many tests were run?
How long each test takes?
How long it takes to reconfigure?
Which ones are the slowest tests?
What combination of tests failed? (good error reporting)

Notes:
number pipeline banes in azure devops so they sort better
why can't a see stack traces and failure messages in failure reports?
where do I got to see pipeline failures?

We use MemoryStorage in most of our unit tests in the product and I think it may be better to use cosmosDB for state for end to end testing. It will help us catch serialization and other storage related issues that we don't normally see in our tests based on MemoryStorage (it happened in the past).

Move the channel creation to deployBotResources.yml.

This enables types other than just 'message'.

Sample 80 in all three languages has been changed this way:

https://github.com/microsoft/BotBuilder-Samples/blob/master/samples/csharp_dotnetcore/80.skills-simple-bot-to-bot/SimpleRootBot/Bots/RootBot.cs#L57

https://github.com/microsoft/BotBuilder-Samples/blob/master/samples/javascript_nodejs/80.skills-simple-bot-to-bot/simple-root-bot/rootBot.js#L32

https://github.com/microsoft/BotBuilder-Samples/blob/master/samples/python/80.skills-simple-bot-to-bot/simple-root-bot/bots/root_bot.py#L41

Describe the bug

DotNet21JsSkillBotFunctionalTest fails due to time out errors.
2021-01-20T13:22:40.7926957Z Error Message: 2021-01-20T13:22:40.7927543Z System.TimeoutException : operation timed out while waiting for a response from the bot

To Reproduce

Run the host bot locally or deploy to Azure and test in WebChat.

On June 17th, an invalid template error started failing all the pipelines. A workaround was applied with PR 118 and PR 117.

The fix has been identified and implemented for the JS-to-JS pipeline: PR 121.

We need to:

1. Reverse the workarounds.
2. Implement the fix in all pipelines.

The test scripts are intermittently timing out when the web apps are just deploying, linux deployments are especially problematic.

As part of the deployment we need to add a step that sends a rest request to the web app and waits until it is running before attempting to start any tests.

The following powershell example show how to use the Invoke-RestMethod to issue a REST OPTIONS request and wait for a status code that will tell us the web app is ready:

$httpStatus = ""
do {
    try {
        Invoke-RestMethod -Uri https://somepythonskillbot.azurewebsites.net/api/messages -Method OPTIONS
    } catch {
        $httpStatus = $_.Exception.Response.StatusCode.value__
        # Dig into the exception to get the Response details.
        # Note that value__ is not a typo.
        Write-Host "StatusCode:" $_.Exception.Response.StatusCode.value__ 
        Write-Host "StatusDescription:" $_.Exception.Response.StatusDescription

        # Wait a sec before trying again
        Start-Sleep 1
    }
# wait for "405 Method not allowed"
} Until ($httpStatus -eq "405")

Write-Host "done"

This will work for dotnet, js and python skills. It will also work for dotnet and js hosts but it will not work for python hosts, in that case the expected https status to wait for is 500 (I'll open an issue in the Python repo to make sure we get a 405 when we send an options request)

Sample failures in pipelines JsJsSkillBotFunctionalTest and PyJsSkillBotFunctionalTest.

Tracking

• C#
• JS PR TODO
• Python PR TODO

In order to be able to run transcript based function tests against a given bot we will need to create a test runner, this issue involves creating the initial spec.

Failures started on Friday 9/4.
Pipelines failing: Python hosts:

Deployment completes without errors. Below is a sample output of the test failure:

2020-09-04T10:22:25.2815177Z Starting test execution, please wait... 2020-09-04T10:22:25.8373485Z 2020-09-04T10:22:25.8375087Z A total of 1 test files matched the specified pattern. 2020-09-04T10:23:15.1514642Z X Skill_OAuthCard_SignInSuccessful [41s 648ms] 2020-09-04T10:23:15.1516045Z Error Message: 2020-09-04T10:23:15.1516988Z Assert.IsTrue failed. Expected: Echo: skill 2020-09-04T10:23:15.1517798Z Stack Trace: 2020-09-04T10:23:15.1518932Z at SkillFunctionalTests.Bot.TestBotClient.AssertReplyAsync(String expected, CancellationToken cancellationToken) in D:\a\1\s\SkillsFunctionalTests\tests\SkillFunctionalTests\Bot\TestBotClient.cs:line 148 2020-09-04T10:23:15.1521826Z at FunctionalTests.OAuthSkillTest.Skill_OAuthCard_SignInSuccessful() in D:\a\1\s\SkillsFunctionalTests\tests\SkillFunctionalTests\OAuthSkillTest.cs:line 37 2020-09-04T10:23:15.1522676Z 2020-09-04T10:23:15.1523429Z Standard Output Messages: 2020-09-04T10:23:15.1524162Z Sent to bot: Hello 2020-09-04T10:23:15.1524895Z Messages sent from bot: 2020-09-04T10:23:15.1525655Z Type:message; Text:Hello and welcome! 2020-09-04T10:23:15.1526498Z Type:message; Text:Me no nothin'. Say "skill" and I'll patch you through 2020-09-04T10:23:15.1527403Z Type:message; Text:Me no nothin'. Say "skill" and I'll patch you through 2020-09-04T10:23:15.1528396Z Sent to bot: skill 2020-09-04T10:23:15.1528845Z Messages sent from bot: 2020-09-04T10:23:15.1529296Z Type:message; Text:Got it, connecting you to the skill... 2020-09-04T10:23:15.1529802Z Type:message; Text:The skill encountered an error or bug. 2020-09-04T10:23:15.1530337Z Type:message; Text:To continue to run this bot, please fix the bot source code. 2020-09-04T10:23:15.1530880Z Type:message; Text:The skill encountered an error or bug. 2020-09-04T10:23:15.1531606Z Type:message; Text:To continue to run this bot, please fix the bot source code.

We need to refactor the Simple host bots to allow to prompt the user for a delivery mode to use and also to prompt for what skill to invoke.

The possible delivery modes are

• Normal
• ExpectReplies

The target that a user can select should be pulled from app settings and the default config should include:

• EchoSkillBotDotNet
• EchoSkillBot21DotNet
• EchoSkillBotV3DotNet
• EchoSkillBotJS
• EchoSkillBotV3JS
• EchoSkillBotPython

This task also involves updating the HostReceivesEndOfConversation.transcript and ShouldRedirectToSkill.transcript (with the related json files) to apply the selections based on the test case defintion

Tracking

• SimpleHostBotDotNet
• SimpleHostBotDotNet21
• SimpleHostBotJS
• SimpleHostBotPython

We need to reorganize the repo folders in order to:

• Be able to store the source code for different skill consumers in different languages.
• Be able to store the source code for different skill in different languages.
• Separate the tests to be executed from the source code for the bots

The new folder structure will follow the pattern below:

RepoRoot
 |
 |-> Bots (contains the source code for the bots used for testing)
 |    |
 |    |-> DotNet (contains dotnet bots)
 |    |    |
 |    |    |-> Consumers
 |    |    |    |
 |    |    |    |-> CodeFirst (source for consumers created using code)
 |    |    |    |    |
 |    |    |    |    |-> SimpleHostBot (a version of the host bot created with the latest dotnet core version supported by BF)
 |    |    |    |    |
 |    |    |    |     -> SimpleHostBot-2.1 (a dotnet core 2.1 version of SimpleHostBot)
 |    |    |    |
 |    |    |    |-> Composer (source for consumers created using Composer)
 |    |    |    |    |
 |    |    |    |     -> ComposerHost
 |    |    |    |    
 |    |    |     -> PVA (placeholder for storing assets related to PVA bots)
 |    |    |
 |    |     -> Skills
 |    |         |
 |    |         |-> CodeFirst (source for skills created using code)				
 |    |         |    |
 |    |         |    |-> DialogSkillBot
 |    |         |    |
 |    |         |    |-> EchoSkillBot (a version of the echo skill created with the latest dotnet core version supported by BF)
 |    |         |    |
 |    |         |    |-> EchoSkillBot-2.1 (a dotnet core 2.1 version of EchoSkillBot)				
 |    |         |    |
 |    |         |    |-> EchoSkillBot-v3 (a version of EchoSkillBot implemented using BF 3.x)				
 |    |         |    |
 |    |         |     -> TeamsSkillBot
 |    |         |    				
 |    |          -> Composer (source for skills created using Composer)
 |    |              |
 |    |              |-> ComposerSkill1
 |    |              |
 |    |               -> ComposerSkill2
 |    |
 |    |-> JavaScript (contains JavaScript bots)
 |    |    |
 |    |    |-> Consumers
 |    |    |    |
 |    |    |    |-> CodeFirst
 |    |    |    |    |
 |    |    |    |    |-> SimpleHostBot
 |    |    |    |    |
 |    |    |    |    |-> ...
 |    |    |    |    |
 |    |    |    |     -> Other hosts
 |    |    |    |
 |    |    |     -> Composer (placeholder for composer consumers with a JS runtime)
 |    |    |    
 |    |    |-> Skills
 |    |         |
 |    |         |-> CodeFirst (source for skills created using code)				
 |    |         |    |
 |    |         |    |-> DialogSkillBot
 |    |         |    |
 |    |         |    |-> ...
 |    |         |    |
 |    |         |     -> Other skills
 |    |         |
 |    |          -> Composer (placeholder for composer consumers with a JS runtime)
 |    |
 |     -> Python (repeats pattern above)
 |    
 |-> Build (yaml files and other assets used by CI/CD)
 |    
 |-> Libraries (reusable libraries for tests)
 |    |
 |     -> TranscriptTestRunner
 |
 |-> Tests (test projects)
 |    |
 |     -> SkillFunctionalTests
 |    
  -> Specs (spec documents)

Note: we will use PascalCase for folder names in this repo.

Tests are failing in the "Create resource group" stage with a template validation error. There is an ongoing investigation that has escalated to the Azure support team.

While the issue is being investigated, we need to update the tests with a workaround shown in this PR.

I am trying to create a dotnet solution that contains all the projects under Bots/DotNet and I can't because some projects have the same name. The ask here is to rename some csproj in the functionaltestsstaging branch as follows:

• Rename Bots\DotNet\Consumers\CodeFirst\SimpleHostBot-2.1\SimpleHostBot.csproj to Bots\DotNet\Consumers\CodeFirst\SimpleHostBot-2.1\SimpleHostBot-2.1.csproj
• Rename DotNet\Skills\CodeFirst\EchoSkillBot-2.1\EchoSkillBot.csproj to DotNet\Skills\CodeFirst\EchoSkillBot-2.1\EchoSkillBot-2.1.csproj
• Rename Bots\DotNet\Skills\CodeFirst\EchoSkillBot-v3\EchoSkillBot.csproj Bots\DotNet\Skills\CodeFirst\EchoSkillBot-v3\EchoSkillBot-v3.csproj
• Update related yaml files as needed.

This issue involves updating the host bots to display a not supported message when the user selects expectReplies for V3 skills (JS or dotnet).

Composer 1.3 has been released today. As part of this issue we need to update SimpleComposerHostBot and SimpleComposerSkillBot to make sure they work with the latest version of composer.

We also need to update the runtime for these bots to the latest one shipped with composer

This issue involves:

• Updating the waterfall host to ask for delivery mode
• Updating existing tests
• Updating pipeline
• Adding SignIn and FileUpload tests

Describe the bug

When you run all the tests together, the Proactive tests ends up sending a proactive message to all the active conversations and causes test to randomly fail.

To Reproduce

Run all the tests from TestExcorer and you see a failure like:

    The bot's response was different than expected. The assertion: "text == 'Done with "WaterfallSkillBot". 
    
     What skill would you like to call?'" was evaluated as false.
    Expected Activity:
    {
      "type": "message",
      "role": "bot",
      "text": "Done with \"WaterfallSkillBot\". \n\n What skill would you like to call?",
      "assertions": [
        "type == 'message'",
        "from.role == 'bot'",
        "recipient.role == 'user'",
        "text == 'Done with \"WaterfallSkillBot\". \n\n What skill would you like to call?'",
        "speak == 'Done with \"WaterfallSkillBot\". \n\n What skill would you like to call?'",
        "inputHint == 'expectingInput'",
        "suggestedActions.actions[0].type == 'imBack'",
        "suggestedActions.actions[0].title == 'EchoSkillBot'",
        "suggestedActions.actions[0].value == 'EchoSkillBot'",
        "suggestedActions.actions[1].type == 'imBack'",
        "suggestedActions.actions[1].title == 'WaterfallSkillBot'",
        "suggestedActions.actions[1].value == 'WaterfallSkillBot'"
      ]
    }
    Actual Activity:
    {
      "type": "message",
      "id": "IuifW5CT2QGLsOH0A8LYGs-6|0000010",
      "timestamp": "2021-01-25T07:36:34.0490657+00:00",
      "localTimestamp": null,
      "localTimezone": null,
      "serviceUrl": null,
      "channelId": "directline",
      "from": {
        "id": "GGWaterfallHostBotDotNet",
        "name": "GGWaterfallHostBotDotNet",
        "aadObjectId": null,
        "role": "bot"
      },
      "conversation": {
        "isGroup": null,
        "conversationType": null,
        "id": "IuifW5CT2QGLsOH0A8LYGs-6",
        "name": null,
        "aadObjectId": null,
        "role": null,
        "tenantId": null
      },
      "recipient": {
        "id": null,
        "name": null,
        "aadObjectId": null,
        "role": "user"
      },
      "textFormat": null,
      "attachmentLayout": null,
      "membersAdded": null,
      "membersRemoved": null,
      "reactionsAdded": null,
      "reactionsRemoved": null,
      "topicName": null,
      "historyDisclosed": null,
      "locale": null,
      "text": "Got proactive message with value: 6eaed445-05d4-41fd-afaa-4ecbd20f5683",
      "speak": null,
      "inputHint": "acceptingInput",
      "summary": null,
      "suggestedActions": null,
      "attachments": null,
      "entities": null,
      "channelData": null,
      "action": null,
      "replyToId": "8c5e1110-8650-4029-b0a1-f62c6b3e00a7",
      "label": null,
      "valueType": null,
      "value": null,
      "name": null,
      "relatesTo": null,
      "code": null,
      "expiration": null,
      "importance": null,
      "deliveryMode": null,
      "listenFor": null,
      "textHighlights": null,
      "semanticAction": null,
      "callerId": null
    }
    Expected: True
    Actual:   False

Expected behavior

Proactive tests should only send activities to the active conversation, not all of them.

We need to be able to assert variable bot's answers as dates for some test scenarios.

Example: Flight Booking bot, when the user says: "I want to book a flight for tomorrow.", the bot will answer with a variable date depending on the day the test is running. We need to assert these responses.

Hi, it would be good to update the testing project to use XUnit rather than MSTests. XUnit has better support for data driven tests and I think we will be able to take advantage of that as the tests become more complex.

In addition to that Microsoft.Bot.Builder.Testing and our testing samples are also based on XUnit. If we use XUnit, we and come up with good helpers we would be able to refactor those helpers into the product down the road.

Let me know if you need me to do the port and I can do a PR with the changes.

Gabo

Describe the bug

All python bots fail to deploy.

To Reproduce

The error is visible in the any of the pipelines that deploy Python bots. A couple of samples:

• DotNet21PySkillBotFunctionalTest
• JsPySkillBotFunctionalTest

Error Information

The error happens in this line: pythonDeployStepsNewRG.yml#L109

The error message: fatal: unable to access 'https://NightlyDotNet21PySkillBot-192768.scm.azurewebsites.net:443/NightlyDotNet21PySkillBot-192768.git/': The requested URL returned error: 403

Let's explore the option to remove the git deployment and instead, use the recommended method here.

Refactor DirectLineTestClient class in TranscriptTestRunner to move the SignIn method to a different class allowing reusability with other test clients.

This issue tracks small cleanup tasks needed to merge stagingScenarios into main.

• Fix legacy tests
• Fix app service plans
• Update BotBuilder package versions
• Add task for deleting pre-existing bots in Deploy pipeline
• Connect Bots with AppInsights
• Keep consistency between variable names
• Improve test names
• Apply fix for App Service Plans default delete
• Move JSV3 skill up in the folder structure
• Add dotnet v3 to the bots solution
• Merge main updates into scenariosStaging and solve conflicts

This change involves updating the TestClientOptions sestion in SkillsFunctionalTests/appSettings.json to be able to store the bot appId in config rather than in code.

It also involves changing the structure of appsettings so the directline client options can be loaded in a dictionary with well known bot names rather than in an array.

We are currently using the arm templates located in each bot folder, this task involve consolidating them in a single place to simplify maintenance.

these pipeline pairs still share the same AppId, and are chained: JsDotNet21SkillBotFunctionalTest --> JsDotNet31SkillBotFunctionalTest, etc.

Building bots to ensure that the multi-turn aspect in skills works with JS bots.

See #50, the C# ticket, for more context

This issue involves:

• Implement a SimpleComposerHostBot that does the same thing that SimpleHostBot does.
• Implement a EchoComposerSkillBot that does the same thing as EchoSkillBot.

Updating pipelines and writing tests for these new bots should be addressed in another issue.

We have some scenarios where we would like to replace some tokens in the script before executing it (e.g.: DeliveryMode and TargetSkill are defined in the TestData parameters)

So, given the following a json script fragment:

    {
      "type": "message",
      "role": "user",
      "text": "skill"
    },

I should be able to change it to:

    {
      "type": "message",
      "role": "user",
      "text": "${TargetSkill}"
    },

And then execute the script passing a parameter dictionary with as follows:

 var scriptParams = new Dictionary<string, string>() { { "TargetSkill", testCase.TargetSkill } };
 await runner.RunTestAsync(Path.Combine(_testScriptsFolder, testCase.Script), scriptParams);

This issue involves

• Updating the yaml files to deploy the composer based SimpleComposerHostBot and EchoComposerSkillBot
• Update the tests so these bots are added to the test permutations for the SIngleTurn scenario

More details:
Out of the box, the composer organizes files in a way that does not allow for a straight web deploy, this task involves updating (or creating) a yaml file that rearranges the files in a way where they can be used for deployment and deploying and configuring the web apps.

Ben has a sample yaml that shows what needs to be done (it involves copying a bunch of files that are in some parent folders to a ComposerDialogs folder under runtime/azurewebapp):

Once the bot is deployed, the script should also update the web app settings in azure with the following values:

For SimpleComposerHostBot:

• MicrosoftAppId
• MicrosoftAppPassword
• skill__EchoSkillBot__endpointUrl: set to the URL of the deployed EchoComposerSkillBot
• skill__EchoSkillBot__msAppId: set to the appId of the deployed EchoComposerSkillBot
• skillHostEndpoint: set to the URL of SimpleComposerHostBot + api/skills (e.g.: https://contoso.azurewebsites.net/api/skills)

For EchoComposerSkillBot :

• MicrosoftAppId
• MicrosoftAppPassword

The DirectLineTestClient constructor takes an IConfiguration parameter and it reads the values for BotId and DirectLineClient directly from appsettings.json assuming a specific name for the settings.

Going forward we will need to support multiple DirectLinetClients (one for each consumer bot) so this won't work.

This issue involves:

• Create a DirectLinetClientOptions POCO class with a BotId and DirectLineClientSecreate property
• Modify DirectLinetClient to take a DirectLinetClientOptions in the constructor.
• Modify TestClientFactory to take the options object and use it to construct the DirectLineTestClient instance (this may need some thought based on other clients that we may need to support in the future).
• Modify the SkillFunctionalTests appsettings.json file to support setting multiple clients and secrets.
• Modify instantiation logic to read the config values, initialize an options object and pass it to the factory.

We need to refactor the CI/CD pipeline so we can:

1. Have a manually triggered pipeline that deploys and configures a shared resource group with common components used by all the bots:
   App Service Plan
   CosmosDB
   AppInsights

2. Have a pipeline that provisions all the consumer and skills at once and then executes the test cases in a configure/test loop that should run for each scenario.

Here is a draft diagram that outlines the deployment once is done:

Test Multi-Turn conversation in Skills

Currently in botbuilder-dotnet/FunctionalTests/Skills/DialogToDialog there are bots that the team runs manually to tests the dialog features in skills.

We should instead automate this process, as done already for Skills with send a message and receiving an echo in the BotFramework-FunctionalTests repo.

Skills Functional Test Scenarios:

• Multi-turn aspect of dialogs (state properly maintained between multiple bots) #50
  • Also test for a bot that is simultaneously both a Host and a Skill
• Tangent #75
• Cancelling #76

[enhancement]

To be addressed once #373 is completed.

Once the AppIds and Passwords for gov cloud are provision in keyvault we will need to update the 02.B pipeline to configure the bots and run the tests using the gov cloud credentials.

TranscriptConverter is part of the test runner effort meant to revamp the FunctionalTests capabilities in this repo. The spec is in development here.

This component is still in development, but preliminary, the following parts are needed:

Methods

• Convert - Inputs: and Emulator transcript; - Output: a "test script"

Tentatively, the test script will contain all properties of the Emulator transcript minus variable GUIDs like Id fields and date/time fields.

The Convert methods should deserialize the input, remove the undesired fields and serialize again.

Draft scenario:

1. I create a new skill (in all 3 languages)
2. I add the skill to a consumer (or many)
3. I update the yamls so the skill gets deployed
4. I create tests for my new skill
5. I merge with main and everything works

Issue:

There is not enough infrastructure to speed up the addition of scenario coverage in the functional test build.

Proposed Change

Add a test runner that takes a bot and a chat file and automatically asserts the conversation remains the same.

Create a host bot based on waterfall dialogs and connect it to the following skills:

• AttachmentsSkillBot
• AuthSkillBot
• CardSkillBot
• DialogSkillBot
• EchoSkillBot
• ProactiveSkillBot
• SsoSkillBot
• TeamsSkillBot
• WeatherSkillBot