Will need a dedicated page for Newman on getpostman.com/docs/

Just tried running Newman 1.0.2 with node.js v0.10.28 on a 64-bit Windows 7 machine and received a "TypeError" error on a fairly common ASP.NET MVC Web API 2 error message.

D:\code>newman -c MVCWebApp03_test.json

Iteration 1 of 1
200 20ms Basic value to root http://localhost:26069/values/5
404 21ms Basic value http://localhost:26069/api/values/5

TypeError: Cannot read property 'two' of undefined
at getKey (C:\Users\tehuser\AppData\Roaming\npm\node_modules\newman\src\uti
lities\VariableProcessor.js:62:23)
at String.replace (native)
at Object.jsface.Class._findReplace (C:\Users\tehuser\AppData\Roaming\npm\n
ode_modules\newman\src\utilities\VariableProcessor.js:64:31)
at Object.jsface.Class._processPathVariable (C:\Users\tehuser\AppData\Roami
ng\npm\node_modules\newman\src\utilities\VariableProcessor.js:33:23)
at Object.jsface.Class.processRequestVariables (C:\Users\tehuser\AppData\Ro
aming\npm\node_modules\newman\src\utilities\VariableProcessor.js:107:8)
at Object.jsface.Class._processUrlUsingEnvVariables (C:\Users\tehuser\AppDa
ta\Roaming\npm\node_modules\newman\src\runners\RequestRunner.js:128:21)
at Object.jsface.Class.execute (C:\Users\tehuser\AppData\Roaming\npm\node
modules\newman\src\runners\RequestRunner.js:46:9)
at Object.jsface.Class._onRequestExecuted (C:\Users\tehuser\AppData\Roaming
\npm\node_modules\newman\src\runners\RequestRunner.js:80:8)
at EventEmitter.emit (events.js:117:20)
at Object.jsface.Class.emit (C:\Users\tehuser\AppData\Roaming\npm\node_modu
les\newman\src\utilities\EventEmitter.js:34:16)

Where “Basic value to root” returns:
"value"
…and “Basic value” returns:
{
"Message": "No HTTP resource was found that matches the request URI 'http://localhost:26069/api/values/5'.",
"MessageDetail": "No type was found that matches the controller named 'api'."
}

More info:

[email protected] C:\Users\jonathal\AppData\Roaming\npm\node_modules\newman
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected]
├── [email protected] ([email protected], [email protected], [email protected])
├── [email protected] ([email protected], [email protected], [email protected], reque
[email protected], [email protected], [email protected])
├── [email protected] ([email protected], [email protected])
└── [email protected] ([email protected], [email protected], [email protected], [email protected])

This is more of a question but wanted to see if there was any insight.

Just installed release 1.07 and I am able to run my entire collection from the command line with no problems.

However, I am attempting to execute directly from code (included below)
I noticed a couple things.

1. I will run into an error with the Helper.js
   TypeError: Cannot read property 'ok' of undefined
   at Object.jsface.Class.testCaseSuccess c:\NewmanPOC\node_modules\newman\src\utilities\Logger.js:60:40)
   at jsface.Class._logTestResults (c:\NewmanPOC\node_modules\newman\src\responseHandlers\TestResponseHandler.js:195:9)

I worked around this by changing my the local code in the Helper.js
exports.symbols = {
err: (process.platform === "win32") ? "\u00D7 " : "✗ ",
ok: (process.platform === "win32") ? "\u221A " : "✔ "
};
To
Helpers.symbols = {
err: (process.platform === "win32") ? "\u00D7 " : "✗ ",
ok: (process.platform === "win32") ? "\u221A " : "✔ "
};

2. Past this temp workaround I am able to run and see console messages for successful tests. However, once I hit a failure (anything that is supposed to pass through ErrorHandler.js) I am getting the following error:

Iteration 1 of 25
200
TypeError: Object # has no method 'testCaseError'
at Object.jsface.Class.testCaseError (c:\NewmanPOC\node_modules\newman\src\utilities\ErrorHandler.js:24:13)
at c:\NewmanPOC\node_modules\newman\src\responseHandlers\TestResponseHandler.js:197:18

Or another example
Iteration 2 of 25
200 505ms Get Auth Token https://####.com/v1/requestToken
√ Status code is 200

TypeError: Object # has no method 'error'
at Object.jsface.Class.responseError (c:\NewmanPOC\node_modules\newman\src\utilities\ErrorHandler.js:16:7)
at Object.jsface.Class._printResponse (c:\NewmanPOC\node_modules\newman\src\responseHandlers\AbstractResponseHandler.js:39:17)
at Object.jsface.Class._onRequestExecuted (c:\NewmanPOC\node_modules\newman\src\responseHandlers\AbstractResponseHandler.js:30:9)
at Object.jsface.Class._onRequestExecuted (c:\NewmanPOC\node_modules\newman\src\responseHandlers\TestResponseHandler.js:27:46)

Code File
var Newman = require('newman');
var JSON5 = require('jju');
var fs = require('fs');
// read the collectionjson file
var collectionJson = JSON5.parse(fs.readFileSync("POSTGoalCollection.json", 'utf8'));
// define Newman options
newmanOptions = {
envJson: JSON5.parse(fs.readFileSync("HerokuSanityEnvironment.json", "utf-8")), // environment file (in parsed json format)
iterationCount: 10, // define the number of times the runner should run
dataFile: "CreateGoalTests.json", // data file if required
outputFile: "outfile.json", // the file to export to
responseHandler: "TestResponseHandler", // the response handler to use
stopOnError: false
}
Newman.execute(collectionJson, newmanOptions);

Suite of tests run fine from Postman. When I try and run them from Newman using: **newman -c testcollection1.json -e Staging.postman_environment

the following error is shown in the console:

Iteration 1 of 1
EXCEPTION - SyntaxError: Unexpected token u
RequestError: e9a710e9-486f-3fc6-2838-d70082368c68 terminated. Error: undefined
RequestError: 41713c16-f095-8c35-1629-eba1f52eea20 terminated. Error: undefined
x Successful POST request
EXCEPTION - TypeError: Cannot call method 'has' of undefined
RequestError: 92d62b99-1cea-e6d2-bba1-ff5c7a2cc5df terminated. Error: undefined
x Status code is 200
RequestError: 41026d39-43b9-168d-b94f-263836e3cbab terminated. Error:      undefined
RequestError: 743f8ce7-94f0-2639-7420-eac88e28369e terminated. Error: undefined

Image attached containing the Json

Looks like its not handling the access_token variable

I can see everything except the collection name:

Trying to work out how best to structure my environments/globals in postman to make re-use in newman as easy as possible. I have a number of scenarios to test which require different input data which up till now I've used environments for. I also use globals for common variables across all scenarios. There's no explicit support in newman afaics to allow me to pass both.

One way around this I thought was to make each environment a data file and consider it an iteration and use the globals as an environment file to pass in however the environment file format is not directly usable as a set of variables in a data file iteration which means some pre-processing is required.

Another thought I had was to merge the globals with each environment file and just run newman for each environment but this would be a manual process (there's no ability to export an environment and merge the globals in a single step). Even better would be for newman to support multiple environment files and treat it as a flat variable list, that way I could pass the environment and the globals at the same time.

Has anyone else faced this issue and how did you deal with it? Any thoughts appreciated.

Right now if you have newman is installed in /node_modules you can't import as it is.

var newman = require('newman'); // doesnt work
var newman = require('src/Newman'); // works

I think it will be good if we add a index.js file in root folder exposing the Newman module so that its more intuitive to import.

Index.js

module.exports = require('./src/Newman');

This allows a simple import

var newman = require('newman');

Let me know if this makes sense and I can quickly push a fix. Will help us in writing a grunt plugin as well.

I'm writing very simple tests to pass against some calls made to an https ASP.Net Web API. These tests run and pass as predicted in Postman, but I get different results when I attempt the same in the commandline. I've checked the Collection json file that I am using; all the appropriate header declarations are present and nothing is different.

for comparison:

and

The specific tests that I am running are also very basic:

tests["Response time is less than 500ms"] = responseTime < 500;
tests["Status code is 200"] = responseCode.code === 200;
tests["Content-Type is present"] = responseHeaders.hasOwnProperty("Content-Type");

(Edited to add additional screenshots. I've also tried using both Content-Type and content-type.)

Any ideas? Apologies in advance if I have just overlooked something completely obvious.

Currently, the environment variables set by one iteration will be carried over to the next iteration. Each iteration should have clean environment variables or you will see some strange behavior.

Running from command line or code does not honor the the iterationCount if a data file is also used. It is always running every record of the datafile.

My data file has 25 records.

newman -c POSTGoalCollection.json -e HerokuSanityEnvironment.json -d CreateGoalTests.json -n 10

or in code:
newmanOptions = {
envJson: JSON5.parse(fs.readFileSync("HerokuSanityEnvironment.json", "utf-8")), // environment file (in parsed json format)
iterationCount: 10, // define the number of times the runner should run
dataFile: "CreateGoalTests.json", // data file if required
outputFile: "outfile.json", // the file to export to
responseHandler: "TestResponseHandler", // the response handler to use
stopOnError: false
}
When this run 25 iterations are always executed. Only 10 should have occurred.

This is what I would expect as test cases:
Iteration is set and a data file provided = runs the collection the number of iterations
Iteration is not set and data file provided = runs the collection the number of records in data file
Iteration is set and no data file provided = runs the collection the number of iterations (Currently works)
iteration is not set and no data file provided = runs the collection once. (Currently works)

Noticed that POST calls with form-data fields does not work with Newman(command line), but the same works with collection Runner.
I have confirmed the behavior, if I move the form-fields to the URL-params section, it works with Newman(command line).

Can not read the request name as the text is black. Screenshot:

Tests perform correctly using collection runner however executing tests with Newman result in an error EXCEPTION - SyntaxError: Unexexpected token u
Test case failed: (test case name)

I am passing in a data file, an environment file and a collection.

@a85 Need your input here to finalize JSON export format.

Url of the form {{url}}/blog/users goes into infinite recursion when not provided with an environment file.
However, url of the form http://{{url}}/blog/users ends gracefully.

Running a collection using the usual
newman -c xyz.json.postman_collection -e abc.postman_environment
was giving me a request error.

Couldn't figure why it wasnt able to construct the request. Turned out that my environment variables didn't have the http:// prefixed to them. Postman would understand that and execute the requests anyway, but Newman wouldn't.

I have noticed a recurring pattern in my tests whereas specs related specifically to Content-Type seems to fail when run with Newman but pass in Postman. Please observe the failing in Newman:

But they (the same suite and same env) pass in Postman:

Specifically, the tests take the form of things like:

tests["Content-Type is  UTF-8 application/json"] = responseHeaders.hasOwnProperty("Content-Type") && 'application/json; charset=utf-8' === responseHeaders['Content-Type'];

tests["Content-Type is text/html"] = responseHeaders.hasOwnProperty("Content-Type") && 'text/html' === responseHeaders['Content-Type'];

Have I done something wrong? Can I provide more information or help diagnose this issue?

Some basics about my system:

OSX 10.9.3

Darwin Concurrent-Chickpea.local 13.2.0 Darwin Kernel Version 13.2.0: Thu Apr 17 23:03:13 PDT 2014; root:xnu-2422.100.13~1/RELEASE_X86_64 x86_64

node v0.10.28 installed from homebrew

newman 1.0.6

Quick edit: The collection and environment files ->

https://github.com/SexualHealthInnovations/PrivateResults/blob/master/spec/data/Private-Results-API.json.postman_collection

https://github.com/SexualHealthInnovations/PrivateResults/blob/master/spec/data/Private-Results-Local.postman_environment

Add iteration count

Add data variables in postman.

Refactor code based on comments here - #74

@viig99 Should we validate the collection JSON format if that complies to a schema?

Newman, by default exits with a status code of 0 if everything runs well
i.e. without any exceptions. Continuous integration tools respond to these exit
codes and correspondingly pass or fail a build.

Unfortunately - this is incorrect. If I run a collection and some tests fail inside, newman returns a return code of 0.

On running newman, filename checks for the postman collection files provide a well handled error message.

However, for the same check in the environment variable filename, I see what looks like a trace.

For the same type of check, the error handling should be uniform.

In Windows. Below I ran the test without the path (sitting in the correct directory) and it ran fine, then I added the directory to each file and it failed. Works fine both ways on Mac.

1st run-----------
C:\Users\jpumph\Desktop\REST Automation\Postman>newman -c ATD-JetPacks.json.postman_collection -e ATD-FT.postman_environment > C:\Users\jpumph\Desktop\REST Automation\Postman\Postman.log

2nd run-------------
C:\Users\jpumph\Desktop\REST Automation\Postman>newman -c C:\Users\jpumph\Desktop\REST Automation\Postman\ATD-JetPacks.json.postman_collection -e C:\Users\jpumph\Desktop\REST Automation\Postman\ATD-FT
.postman_environment > C:\Users\jpumph\Desktop\REST Automation\Postman\Postman.log

C:\Users\jpumph\AppData\Roaming\npm\node_modules\newman\node_modules\json5\lib\json5.js:44
throw error;
^
SyntaxError: Unexpected ''
at JSON5.parse.error (C:\Users\jpumph\AppData\Roaming\npm\node_modules\newman\node_modules\json5\lib\json5.js:40:25)
at JSON5.parse.word (C:\Users\jpumph\AppData\Roaming\npm\node_modules\newman\node_modules\json5\lib\json5.js:340:13)
at JSON5.parse.value (C:\Users\jpumph\AppData\Roaming\npm\node_modules\newman\node_modules\json5\lib\json5.js:443:56)
at Object.JSON5.stringify as parse
at main (C:\Users\jpumph\AppData\Roaming\npm\node_modules\newman\bin\newman:68:33)
at Object. (C:\Users\jpumph\AppData\Roaming\npm\node_modules\newman\bin\newman:104:1)
at Module._compile (module.js:456:26)
at Object.Module._extensions..js (module.js:474:10)
at Module.load (module.js:356:32)
at Function.Module._load (module.js:312:12)

C:\Users\jpumph\Desktop\REST Automation\Postman>

Hey - I was experimenting with Postman/Newman for the first time today.

I had a couple of requests in a collection. However when I ran them, the only output I saw was "Iteration 1 of 1". For a while I wondered why they weren't running correctly. Did some more digging and looking at the collections of others I noticed the JSON produced from Postman didn't contain an order array like the others had.

In Postman, if you drag and drop the requests, this will now produce an order array in the JSON for anything you upload / download. This is a workaround in Postman. If you like I'll file a bug report over there too.

But perhaps Newman should run the requests in the order of the requests array if there's an absence of an order array?

Thanks for Postman & Newman - they're great so far.

After running the test, I think Newman should show some stats. At the very least, how many tests failed. It is annoying to have to scroll to know how many failed.

These tests results overview could show test success or failure by subfolder. I work with a long test collection, and I organise it with folders (like registration, permissions, ...). It would be great to see the test outcome by folder. Something like this:

Overview
62 passed, 2 failed

-> registration
20 passed, 0 failed
-> permissions
42 passed, 2 failed

Automatically read collection and env from folder given it follows a format?

Any chance of adding support for custom templates for rendering test reports? A Junit test reporter would be particularly usefull for example to allow jenkins and similar CI tools process and render test results effectively .

I call newman with -s and -o outputfile.json, a test fails and the run halts.
No output file is ever written to allow me to diagnose the collection run failure.

If using this with Jenkins it will be very important to rely on the output file for determining failures.

Shouldn't we have a callback function as a third parameter on Newman that is run when the iteration gets over - so that users can hook in their functionality. Although we are emitting an event when this happens, I'm not sure how others can know when Newman gets over. This is also required for the Grunt plugin. Something like this -

Newman.execute(collectionJson, optionsDict, callback); 

Usage: https://github.com/prakhar1989/grunt-newman/blob/master/tasks/newman.js#L10-L19

Or is there a better way to handle this?

I've got a few testcases and make relatively heavy use of the SugarJS Date extensions. These work fine in Postman and in the collection runner, but when trying to execute them in Newman it bails out with messages like this:

EXCEPTION - TypeError: Object function Date() { [native code] } has no method 'create'

I assume this is due to SugarJS not being included? Or perhaps some kind of node.js compatibility issue.

Need to carry out through testing if this is working. Note to self: Check mail for detailed issue.

When you live run the newman collection runner on screen results look like this:
200 669ms Delete Account
√ Status code is 200
On a Mac, if you redirect that output (newman -c .... > Postman.log) to a file for later use (like you would using cron), the output looks the same when you "cat" the file. But in Widows if you redirect > then "type" the file contents you get a bunch of extra characters.
←[39m←[32m200←[39m←[36m 669ms←[39m Delete Account
←[24m←[39m←[32m ←[32mΓêÜ Status code is 200←[39m

Is there a way around this? I need to see the runtime output rather than the json you get with the -o option, but I need it to be readable in Windows like it is on the Mac.

Hi, Newman

I just tried to use this tool in command, it always no infromation, i don't know what happened. 

The steps i used to do test:

• Upload collection to postman, and got https://www.getpostman.com/collections/910fd2e72c4ac2f3add9
• Down load one environment from PostMan, as role.json :

{"id":"f5732ad5-9225-aef6-adf5-0d9de3fed909","name":"Role-owner","values":[{"key":"role","value":"owner","type":"text"}],"timestamp":1399973640780}

• Use new man to test result newman -u https://www.getpostman.com/collections/910fd2e72c4ac2f3add9 -e role.json -s

And i got nothing output except Iteration 1 of 1. So i was confused on how to use this tool.
So if i want to run only one specific request, is it possible?

Thanks!

Group tags per test

This is a feature request for extending the current JSON format for a test used in both Postman and Newman to include group tags.

The idea is similar to phpunit's @group where you can tag any test in any number of groups and run tests against either an individual group or multiple groups.

JSON format

{
    ...
    "time": 1402331140943,
    "version": 3???,
    "responses": [],
    "tests": "",
    "groups": [],
    "collectionId": "",
    "synced": false
}

Example usage:

Only run the user tests in a collection

newman -c collection.json -g "user"

Run the user and admin tests in a collection

newman -c collection.json -g "user,admin"

Run everything apart from the admin tests

newman -c collection.json -g "-admin"

Run everything apart from the user and the admin tests

newman -c collection.json -g "-user,-admin"

caveats

The JSON format version might need to be upgraded to V3

Add a "method" parameter in request object for parity with postman.

I'm trying to run a collection as a file through Newman. Everything works fine on the app, and I've saved the collection file correctly onto my desktop. However, when I type in the command to run my file; "newman -c UCRT_Test.json" I get the message "Please specify a Postman Collection either as a file or a URL". I'm pretty confused by this. Any help?

This as discussed would help in signalling errors for integration with CI/build systems.
If it's present already then probably help needs to be updated.

Provide a better error message in case the user runs a collection without the provided env json file.

Note: I'm on tip of tree, including the offending check in:
2ed4724

If I have a variable inside of my data for a POST or PUT, and I run multiple iterations, the variables inside will only be expanded once. In the above checkin, you assign the transformed data to the ACTUAL request, meaning next time this request is evaluated, the variables are already expanded, but only with the values assigned for the first iteration.

Enable exception handling in vm.runInThisContext to avoid, poor user code from breaking the application.

See screenshot below. Need to handle a case when response is not defined (as done in RequestRunner). Have added a hack at the moment. Please fix this. [Above is present, below is ideal case]

Iteration count.

I am attempting to test a service that exposes its API over https.
Everything works fine hitting this URL in Postman but when executing this in Newman I receive an error.

EXCEPTION - Syntax Error: Unexpected token u
RequestError: dfaa2798-66f5-00df-89f277629098 terminated. Error: undefined.

I'm trying to schedule my collection to run with crontab but even with full paths I still get an error about "node".

jpumph0414mac:Documents jpumph$ crontab -l
30 11 * * * /usr/local/bin/newman -c /Users/JPUMPH/Documents/ATD-JetPacks.json.postman_collection -e /Users/JPUMPH/Documents/ATD-FT.postman_environment > /Users/JPUMPH/Documents/Postman.log 2>&1

jpumph0414mac:Documents jpumph$ cat Postman.log
env: node: No such file or directory

There is an idea to have each test loop over a dataset and record the results of each pass.

This would work something like this, and the tests would be executed against each row of data.

supply data to the test, one object per row

var data = {
    "rows": [
        {"username":"someone1", "password":"somepass1","shouldPass":true},
        {"username":"someone2", "password":"somepass2","shouldPass":fail}
    ]
}

The test runner would use those values in the url like so:
http://somesite/login/user={$username} etc

testing responses

Each row of values should be available to the tests in its iteration. Sp the tests for this could be something like: (sudo code for ideas)

var data = JSON.parse(responseBody);
tests["User logged in"]  = data.loggedIn === row.shouldPass; //current row value

getting data into the test

There is a data field in the test json object, the data could be passed into that with a "dataMode" of "rows" or objectArray etc.

Possible extensions

The ability to process the results of multiple API calls including AVG response times etc.

Personally, I would like this in newman but can see the advatages of using it in postman too. Feel free to add/extend/run away with the idea :)