leafle / newman-reporter-teamcity Goto Github PK

any usage instructions would be very handy

Hello. Thanks for teamcity reporter.
Have some encryption problems. Did you have any ideas, how to fix it?

CLI reporter is dominant. It means if there are two dominant reporters they will fight against each other and make output almost useless.

// Mark the CLI reporter as dominant, so that no two dominant reporters are together
PostmanCLIReporter.prototype.dominant = true;

I can't use -r cli,teamcity together - common output is awful. I need teamcity diagnostics (start/stop/success/failure) in 'real time' while everything else I can read from cli output in a pretty form.

In my TeamCity Build logs, I see the following:

##teamcity[testStarted name='xxxxTest Namexxxx' captureStandardOutput='true']
[11:41:22]
Incorrect property name.
Valid property list format is (name( )=( )'escaped_value'( )) where escape symbol is "|"

I was using newman and newman-reporter-teamcity in Powershell to run the API testing.
The newman command line is :
newman run $DirectAGCollectionURL -e $GetEnvVarAUTest --iteration-data $DataFile -n $Iterations --folder $APIFolderNames[$i] -r cli,teamcity,json --reporter-json-export $OutputFile
Below is the just copy & past of small part of the test report generated
...

└ Create Semaphore
##teamcity[testStarted name='Create Semaphore' captureStandardOutput='true']
POST https://au-test.leap.services/api/v1/semaphore [201 Created, 540B, 1028ms]
┌
│ 'Scenario 'No Semaphore Name' responseCode.code=201'
│ 'Defect API-18 is still outstanding'
└
√ 1) Check Expected RC = 201
√ 2) Validate response against schema
3. 3) Check SemaphoreId is NOT Created for 'No Semaphore Name' (Negative Scenario)
√ 4) Content-Type is application/json
##teamcity[testFailed name='Create Semaphore' message='3) Check SemaphoreId is NOT Created for |'No Semaphore Name|' (Negative Scenario)' details='3) Check SemaphoreId is NOT Created for |'No Semaphore Name|' (Negative Scenario) - Response code: 1028, reason: Created']
##teamcity[testFinished name='Create Semaphore' duration='1028']

Here the problem is, in the team city report, it gives wrong Response code by referring to the response time value(1028), not the real response code for the API (RC=201) .
The expectation is Response code: 201

My testing environment version:
In a command prompt window, type "npm list -g --depth=0 " to get some of the details related to newman:
+-- [email protected]
+-- [email protected]

First off, thank you for creating such an awesome Newman extension. We've been using it for quite a while.

Senario:

When we run test cases in parallel, sometimes TeamCity will mismatch 'test case 1' to 'test suite 2'
##teamcity[testSuiteStarted name='test suite 1']
##teamcity[testSuiteStarted name='test suite 2']
##teamcity[testStarted name='test case 1' captureStandardOutput='true']
##teamcity[testFinished name='test case 1' duration='100']
##teamcity[testStarted name='test case 2' captureStandardOutput='true']
##teamcity[testFinished name='test case 2' duration='100']
##teamcity[testSuiteFinished name='test suite 2']

TeamCity would treat both 'test case 1' and 'test case 2' as part of 'test suite 2'.

Solution:

Use flowId

I have given it a go. It works.
But I'm not happy with the fact that I required a third party library 'uuid/v4' to get the unique id.

Any thoughts on that?

Cheers,

When running Postman tests via Newman, the Teamcity reporter shows tests that have failed test-scripts as Passed.

Example:
Build log:
Report:

following there instructions:
https://github.com/leafle/newman-reporter-teamcity

Tried to run my collection as per the instructions and got the following error:
warning: "teamcity" reporter could not be loaded.

teamcity was installed as per the instructions, so why can't it find it?

Note I'm using windows 10.

Any tests which are marked as Skipped in Postman collection and when running these test using Newman on TeamCity pass the tests, this creates confusion in the overall result.
Please suggest if there is a workaround to handle this case or kindly provide a fix.

Right now it's not obvious in the Teamcity UI what directory a Postman test is in a certain group or directory. It only reports the test name. The workaround is to include the directory name in the test name.