A common testing issue is running out of workspaces, particularly when on the lite plan.
The error message for running out of workspaces is well-defined, we should detect it and present a useful remediation to the user.

intentmetrics.py can run standalone but is also naturally "chained" to a k-folds or blind run.
Issue suggestion is to add configuration in config.ini which runs intentmetrics right after those tests if the user sets some configuration value.

Build a confusion matrix, either per fold or as summary of all folds.
This helps quickly identify poorly-performing intents and those intents that get confused for each other.

The configuration file has a variety of required/optional parameters and these parameters are dependent on what mode the tool is run in.

Update the sample configuration file so that it provides guidance without having to go out to the README.

Blind testing requires that the blind_figure_title exist in the config file. Need to add the following to the config.ini.sample file.

blind_figure_title =

add config parser to requirements.txt

1. Normal execution modifies the data/workspace_base.json file
2. If a user has multiple config files, ie config.ini.mode1 and config.ini.mode2, these are by default tracked in gitignore

For 1), move the data/workspace_base.json file to data/workspace_base.json.sample, as it appears only created in Git to hold a reference to the data/ directory in the first place
For 2, update .gitignore files to allow the pattern described above

When running the Blind Test - the test-out.csv file does not get created. I am attaching my config file here. Any recommendations are welcome. Thank you - Rebecca James

[DEFAULT]
mode = BLIND
workspace_id =
test_input_file = ./data/test.csv
temporary_file_directory = ./data
; previous_blind_out = ./data/previous_blind_out.csv
test_output_path = ./data-test-out.csv
; Figure path for kfold and blind
out_figure_path= ./data/figure.png
keep_workspace_after_test = no
blind_figure_title='title'
; partial_credit_table = ./data/partial-credit-table.csv

[ASSISTANT CREDENTIALS]
username = apikey
password = 

Received a divide by zero error during analysis
https://github.com/cognitive-catalyst/WA-Testing-Tool/blob/master/createPrecisionCurve.py#L121

On windows it can fail reading the intent .csv file due to incorrect encodings

As reported from a user email:

The problem is that your configuration parameters don't include the URL of the service. This means that the default of https://gateway.watsonplatform.net/assistant/api is used which is fine for all WA instances hosted in US South. However, in other regions the URL of the WA API is different e.g if the instance is based in Germany the URL of the API will be https://gateway-fra.watsonplatform.net/assistant/api and unless you give users a way to specify this they wont be able to use their code.

Solution would be add the URL to the config file, and pass it as an input parameter to all scripts that call the API

The API version used in the API calls is coded in utils/__ init__.py

WCS_VERSION = '2018-07-10'

As a workaround the users can edit the __ init__.py and change the WCS_VERSION variable but they may forget or not know about it.

Ideally this should be exposed to the config files so that users and match/configure the testing version with the same version they use in their application

Something along the lines of:

def send_message(text, counter = 0):
    try:
        ## get the response and return
    except:
        if counter < 5:
            return send_message(text, counter+1):
        else:
            ## return actual failure

Around the call to Watson Assistant to be more robust to random bad responses.

Add the y-axis parameter to createPrecisionCurve script in order to keep the y-axis constant. Additionally, expose the value for the y-axis parameter within the config file.

New python installs may see errors like the following:

File "WA-Tool/venv/lib/python3.7/site-packages/matplotlib/backend_bases.py", line 1956, in _get_output_canvas
.format(fmt, ", ".join(sorted(self.get_supported_filetypes()))))
ValueError: Format 'jpg' is not supported (supported formats: eps, pdf, pgf, png, ps, raw, rgba, svg, svgz)

'jpg' support varies by platform, some platforms require installing the additional pillow library for jpg support.

Half of the visualizations in this tool already use 'png' which does not cause the same issues. Workarounds include:

• Add 'pillow' as dependency (requirements.txt, pip install)
• Change any 'jpg' exports to 'png'

The results in each of the k-fold test results does not currently include the "golden intent" column, they only have the "predicted intent" column. For ease of analysis the "golden intent" (the intent that the utterance belongs to in the training data) should also be output as part of the experiment.

due to an error in the python sdk, the username:password (pre-IAM) no longer works. Suggest removing support for username:password until the sdk is fixed.

Used the standard test on a system with dialog, and dialog column in the results was completely blank.

Several parameters need not be provided by user - the tool can assume sensible defaults.
Ideally the user is only required to provide connection information to their workspace and the mode to run the tool in as that is information the tool truly cannot know otherwise.

This increases the ease of first running the tool.

From the config file

; (Required) Test request rate (maximum number of API calls per second)
max_test_rate = 100

; (Required) All temporary files will be stored here
temporary_file_directory = ./data

; (Required) yes/no on whether to keep(yes) or delete(no) workspaces created by this tool after the testing phase
keep_workspace_after_test = no

; (Required for blind and test) Test output path
test_output_path = ./data/test-out.csv

; (Required for blind and kfold) Output figure path
out_figure_path= ./data/figure.png

; (Required for kfold) Number of folds.  If on LITE plan use 3.
; Each fold creates a workspace (make sure you have enough workspaces available, LITE plans are restricted to 5)
fold_num = 5

; (Required for blind) Title for blind testing output figure
blind_figure_title = 'Blind Test Results'

max_test_rate, temporary_file_directory, and keep_workspace_after_test already have sensible defaults, we need not require the user to provide them.

test_output_path: k-folds defaults a version of this parameter, the other modes can default to data/blind_out.csv and data/test_out.csv (k-folds should thus be updated to read this parameter.). The tool already prints out the location of all files written, so the user will not be surprised by this result.

out_figure_path: A good default would be test_output_path + '.jpg'

blind_figure_title: The default above is fine, we should not require the user to provide it.

The tool is failing with the following message while running on MacOS:

    raise ClientConnectorSSLError(req.connection_key, exc) from exc
aiohttp.client_exceptions.ClientConnectorSSLError: Cannot connect to host gateway.watsonplatform.net:443 ssl:None [[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:777)]

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/Users/victorpovar/Desktop/WA-Testing-Tool-master/utils/testConversation.py", line 194, in <module>
    func(ARGS)
  File "/Users/victorpovar/Desktop/WA-Testing-Tool-master/utils/testConversation.py", line 131, in func
    loop.run_until_complete(asyncio.gather(*tasks))
  File "/Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/asyncio/base_events.py", line 467, in run_until_complete
    return future.result()
  File "/Users/victorpovar/Desktop/WA-Testing-Tool-master/utils/testConversation.py", line 72, in fill_df
    'alternate_intents': True}, url, sem)
RETRY
  File "/Users/victorpovar/Desktop/WA-Testing-Tool-master/utils/testConversation.py", line 55, in post
3
    print(response.status)
RETRY
UnboundLocalError: local variable 'response' referenced before assignment

F-score is computed but not reported, it is a useful addition to precision and recall metrics already created.

Need the capability so that users can run this tool from Watson Studio using Python Notebook. The test result files can be exported (if needed) and/or analyzed in the Notebook itself.

https://www.ibm.com/support/knowledgecenter/en/SS4U29/api.html defines all legal Voice Gateway commands. Checking against this list verifies commands are not mis-spelled.

Users can modify the python code to set BASE_URL but this too should be exposed in config.ini.

Per WA release notes, the authentication method will be changing for new services instances created on or after 10/30/18:

"API authentication changes
On 30 October 2018, the US South and Germany regions will transition from using Cloud Foundry to using token-based Identity and Access Management (IAM) authentication. (See Authenticating with IAM tokens External link icon for more information.)

The method used to authenticate with IAM service instances is different from the method used to authenticate with Cloud Foundry instances. Existing applications that use Cloud Foundry will continue to work. However, if you migrate a service instance or create a new service instance in a region that uses IAM, you must update the code that handles authentication. All regions are transitioning to IAM, but on a rolling schedule. For more details, see Data centers."

Given a input file for a partial credit intent scoring mapping that has the following 3 columns, where a golden intent can be on more than one line:

Golden Intent | Partial Credit Intent | Partial Credit Intent score

Where Golden Intent is an intent from your WA, and Partial Credit Intent is another intent in your WA that when served in place of the Golden Intent is going to be give partial credit for being correct, and the partial credit (in the range [0,1]) is in the column Partial Credit Intent score
Change the scoring of the

• Add the partial credit mapping file as an optional parameter to run.py
• Intent Metrics to use accept the partial credit table as an optional parameter to calculate TPR and PPV
• Add a column to the Schema of Results after yes/no for test output, called score that is a numeric value between [0,1] that is created with the partial credit table, if no partial credit table then the values are just 1 and 0.
• Precision Curve and here change correct from being the count of yes to be the sum of the scores (scores are [0,1])

In user testing it was reported that if there was no previous blind test available, the script could not continue even though this should be an optional parameter.

suggestion: if the string for prev result is empty, or the string does not resolve to a valid file, fall back to the default action of no previous results after a friendly message to the terminal.

Intent metrics report provides nice summarization in table form however a picture is worth 1000 words (how many tables is that?)

A treemap is a logical way to visualize the metrics data:

• SIZE of box relates to number of samples for that intent
• COLOR of box relates to the accuracy for that intent

With a visual summary ala a tree map it becomes visually obvious to focus on the largest, most-red boxes.

Ala #38 , new Mac users need to initialize their SSL store. This should be documented.

Additionally, recommend use of python3 ala https://python3statement.org/ with Python 2's eventual end of life.

Firstly - great tool thank you.
The WA classifier now supports using entities within intent training. For instance "Can I get a @PhoneModelName? "
https://console.bluemix.net/docs/services/conversation/intents.html#defining-intents

This is very useful when reusing bot training across multiple bots for instance "Hello @me" where the intent training can be ported across and @me @otherpeople be controlled as entities.

At the moment when creating test data, the WA-Testing-Tool, submits "Hello @me" without replacing the @me with a literal from the entity training. This means that these values almost always fail, even though they would pass if @me replaced with one of the literal values.

Ask is to download the entity training with the intent training. When the intent training trains the workspace, @entity should be passed as currently. But when the test set is created, the @entity should be replaced with a random synonym/literal from the entity training set.

Watson Assistant provides an API to review conversation logs:
https://cloud.ibm.com/apidocs/assistant-v1#list-log-events-in-a-workspace

This log output is easily filtered and scraped for various information.

We encourage chatbot developers to monitor their application in production and to review the way users interact with the bot. One point to monitor is the responses to open-ended intent gathering questions. (This open-ended intent is usually the first input from the user)

The user utterances containing intent responses is useful for testing the performance of a chatbot or improving its training.

Given a list of intent-based utterances it is possible to
Test accuracy: Create a "blind" test file by adding a column with a "golden intent" for each utterance, then running WA-Testing-Tool in blind mode.
Improve training: Inspect utterances for new intents/entities that may need to be added.

Main README and examples do not describe how to use intentmetrics.py

@victor-zhao-ibm ... similar to the internal bug, but this one in the intent metrics file

After including updates for #33 and #48 the examples need to be updated showing how to provide IAM apikey and configurable URL

Currently, the default WA confidence score to fallback on "I don't know" is 0.2. Add this threshold value to your config settings as the variable "tau", and indicate on all of the precision test results at what point on the curve corresponds to the "tau" confidence score. Indicate with a * or o on the line, clear enough to distinguish and put the value of tau in the legend.

If a node is doing a "Wait for user input", it should have at least one of the following:

• Text output to the user
• A context action defined
• A webhook defined

If there is no text output, the end condition should be a jump or a "skip user input".
Else, this node is likely in error.

Watson Assistant is frequently paired with Watson Discovery Service for handling short-tail (WA) and long-tail (WDS) questions. A common pattern is for WA to handle questions classified above a certain confidence, and to hand low-confidence utterances to WDS. The current k-folds tool does not handle this.

Challenges:

1. Low confidence (<20%) results should not be considered correct.
   If intent[0].confidence < 0.20 then WA returns "Irrelevant". k-folds/blind should reflect this

2. Chatbot may route low confidence (ie <50%) to "out of scope" intent
   Especially in case of chatbots using Discovery

This should be implemented as an optional post-processor like the intentmetrics.py and confusionmatrix.py

A user request was submitted to calculate the following per intent metrics given the results of the k-fold-union or a golden (or blind) set:

• intent name
• intent true positive rate = TP/(TP+FN)
• intent positive predictive value = TP/(TP +FP)
• intent number of samples (relevant elements)

Using the definitions here

I think we could add this as another sub-directory, like the intent descriptions, that would just contain a simple python script that inputs the "out" file from the k-fold or golden test, and outputs the per-intent metrics above in a csv.

Had to force requirements.txt to use wdc 1.3.5 because the 2.0 release changed the signatures of the calls. Best thing to do here is update our code to use the latest SDK, so adding a ticket to remind us to do that

The input of the testing tool currently accepts the intent and entity spreadsheets that are created from exporting the information from a workspace. For usability, have the configuration only contain the user/password/workspace_id for the Watson Assistant workspace and have the test tool export information from there and create new workspaces as necessary

Also, update the README.md file to reflect this change

Consolidate all of the k-fold-test-out.csv results to one file as standard procedure for easier analysis.

I keep getting this error even though assistant credentials is in the config file

'ASSISTANT CREDENTIALS' is missing in config file

Right now configured to use workspace ID, add an option to run "offline" by passing in a WA workspace

Watson Assistant workspaces often integrate with other systems:
Service Orchestration Engine (SOE) layers are used to coordinate with additional APIs. A syntax pattern exists for directing SOEs how to handle a Watson Assistant response.
IBM Voice Gateway is used in voice bots with Speech to Text and Text to Speech services handling vocalizations and user response transcription. Voice Gateway expects a specific syntax. Additionally, best practices exist such as setting speech customization parameters on each dialog node where a user response is collected.

Watson Assistant does not natively validate the syntax used to integrate with these systems. We need methodology to validate the syntax and patterns used.

watson-developer-cloud package functionality should instead use ibm-watson package namespace

When running the standard test, the top result (line 2) was the utterance "utterance" and this was not part of the input.

add another column in the k-fold detailed results that indicates the fold in which that test data was held out for testing. could be useful in deciding if a particular failure was caused by bad luck of all utterances in the vicinity of that utterance were held out on the same fold