Based on the most recent interop checklists, the following items remain to be automated if possible.
NB: Not all test cases are applicable for all Device Classes.

AMWA NMOS Discovery & Registration (IS-04) Checklist

Version 2.2 (17th July 2018)

Section 1: Discovery & Registration

NB: Each of the below items should be checked off against each API version implemented.

• 1.1 Node can discover network registration service via mDNS
• 1.2 Node can discover network registration service via unicast DNS
• 1.3 Node re-queries unicast/multicast DNS when its Registration API connection fails and the DNS TTL expires
• 1.4 Node does not re-query unicast/multicast DNS for known records unless the TTL expires
• 1.5 Node does not register with registries set to DNS-SD TXT priority>=100 unless explicitly configured to
• 1.6 Node registers with registries based upon their DNS-SD TXT priority order (lowest first)
• 1.7 Node can register a valid Node resource with the network registration service, matching its Node API self resource
• 1.8 Node maintains itself in the registry via periodic calls to the health resource
• 1.9 Node correctly handles HTTP 4XX and 5XX codes from the registry, re-registering or trying alternative Registration APIs as required
• 1.10 Node can register a valid Device resource with the network registration service, matching its Node API Device resource
• 1.11 Node can register a valid Source resource with the network registration service, matching its Node API Source resource
• 1.12 Node can register a valid Flow resource with the network registration service, matching its Node API Flow resource
• 1.13 Node can register a valid Sender resource with the network registration service, matching its Node API Sender resource
• 1.14 Node can register a valid Receiver resource with the network registration service, matching its Node API Receiver resource
• 1.15 Node advertises a Node type mDNS announcement with no ver_* TXT records in the presence of a Registration API
• 1.16 Node advertises a Node type mDNS announcement with ver_* TXT records in the absence of a Registration API
• 1.17 Node does not re-issue mDNS query answers periodically (ie. against DNS-SD spec)
• 1.18 **Node can discover other Nodes’ resources via mDNS, populating a local interface as specific by the P2P discovery scheme
• 1.19 Node successfully re-registers with the Registration API if its network link is unplugged for 15 seconds (assuming default timeouts and 404 heartbeat response)
• 1.20 Node does not re-register itself with the Registration API if its network link is unplugged for between 5 and 10 seconds and then re-plugged (assuming default timeouts and 200 heartbeat response)
• 1.21 Node successfully discovers and heartbeats with a second registry if the first one goes offline

**Local registry population is only of use to devices which can display a list of sources to connect to via a user interface. This targets display and control devices (including camera viewfinders which can pick up PGM out).

Test 26 (immediate receiver activations) takes 48 seconds to test my 80 receivers (and mine is a small system!). Maybe some of these tests just configure a random sampling of the receivers and time out after, say, 10 receivers? Possibly there could be an option on the front page to control this.

The current flat file structure is getting a little messy. I'd suggest we adopt something like the following:

ReadMe.md
setup.py
requirements.txt
nmos-test.py
nmos_testing/(Specification.py|TestHelper.py|TestResult.py...)
nmos_testing/test_sets/(GenericTest.py|IS0401.py|IS0501.py...)
nmos_testing/spec_utils/(NMOSUtils.py|IS04Utils.py...)
nmos_testing/mocks/(Registry.py|Node.py)

Imports will need a bit of re-jigging as a result.

Best to do this when there aren't any pending PRs!

IS-04 testing fails on the first tests with
zeroconf.BadTypeInNameException: Service name (nmos-registration) must be <= 15 bytes.

The exception appears here and here.

It is a known issue of the Registration API specification but how doesn't it fail in your case?

Follow-up to #3 / #22.
Would it now be nice if we changed the function names from test_01, test_02, etc. to test_discover_registry, test_discover_registry_unicast, etc.?

Query API WebSocket tests would be great... and if we can share tooling with Event & Tally tests, so much the better.

(I'll being trying to contribute some of these feature requests I'm adding now!)

Based on the most recent interop checklists, the following items remain to be automated if possible.
NB: Not all test cases are applicable for all Device Classes.

AMWA NMOS Connection Management (IS-05) Checklist

Version 1.2 (2nd July 2018)

Section 7: SDP Files

• 7.1 SDP file and /transportfile endpoint must pass a sdpoker test, run with default settings: https://github.com/Streampunk/sdpoker
• 7.2 Receivers should tolerate SDP fmtp lines which:
  • Have no additional termination character before the carriage return.
  • Use a semicolon ';' before the carriage return.
  • Use a semicolon and space '; ' before the carriage return.
• 7.3 Receivers should tolerate SDP files that contain attribute parameters detailing the mapping of RTP header extensions.

As of commit c18cc3f, nmos-cpp-node from the sony/nmos-cpp project started failing several unit tests, starting with test_04 ("Node can register a valid Node..."), with the message "Node API JSON does not match data in registry...". Investigating further, it appears that left-hand side of the comparison has a whole lot more data than the right-hand side and that limiting the data being compared make the tests succeed. In particular, if I change line 195 from elif reg_resource != node_resources[resource]: to elif reg_resource[1]['payload']['data'] != node_resources[resource]:, the tests pass.

Does that seem like a reasonable fix, or is there something more subtle going on here?

[edit: my original proposed fix would break the test at line 193]

Follow-up to #3 / #22.
We would like to tease apart the tests/do more per-test setup/teardown if possible so more of them can be run in isolation.

I don't think the validate_CORS testing implemented by GenericTest is correct. (Interestingly this seemingly only bites me in IS0402 for the Registration API "Registration API rejects an invalid X resource with a 400 HTTP code" tests, because most of the other tests don't use check_response.)

I think this is largely down to the test being based on the flawed description of CORS in the APIs: Server Side Implementation Notes.

For example, I think Access-Control-Allow-Headers should only be used in the response to a CORS preflight (OPTIONS) request.

On the other hand, the response to a GET request should have the Access-Control-Expose-Headers header, e.g. for Content-Length.

Based on the most recent interop checklists, the following items remain to be automated if possible.
NB: Not all test cases are applicable for all Device Classes.

AMWA NMOS Discovery & Registration (IS-04) Checklist

Version 2.2 (17th July 2018)

Section 3: Identifiers & Timestamps

NB: Each of the below items should be checked off against each API version implemented.

• 3.1 Node advertises its Node, Device, Source, Flow, Sender and Receiver resources via the Node API
• 3.2 Identifiers persist through a Node reboot (for physical Nodes with a fixed processing layout)

Based on the most recent interop checklists, the following items remain to be automated if possible.
NB: Not all test cases are applicable for all Device Classes.

AMWA NMOS Connection Management (IS-05) Checklist

Version 1.2 (2nd July 2018)

Section 2: API Operation with RTP Senders and Receivers

• 2.1 The sr-ctrl type is advertised against the IS-04 device with the correct HREF for each device the API represents
• 2.2 When a sender or receiver is registered to an IS-04 device, it is populated in the connection management API
• 2.3 When a new sender or receiver is added to the API, its staged and active transport parameters are set to reflect their current state
• 2.4 When a sender or receiver is removed from a device, it is removed from the connection management API
• 2.5 Changes to staged parameters have no impact on the behaviour of streams from/to senders/receivers
• 2.6 When new parameters become active they have the expected effect on streams from/to senders/receivers
• 2.7 Senders make available a transport file for the “tuning” of receivers, which matches the one available from manifest_href advertised via the Node API
• 2.8 When senders receive updates to their transport parameters, these are correctly reflected in an update to its transport file.
• 2.9 Senders and receivers should express any restrictions on available parameters (for example available source IP addresses for senders) in their constraints
• 2.10 All instances of “auto” settings are interpreted as per the specification
• 2.11 Sender and receivers list their available interfaces under source_ip and interface_ip under constraints along with an “auto” option.
• 2.12 Senders and receivers are able to be configured for multicast operation via the API
• 2.13 Senders and receivers are able to be configured for unicast operation via the API

Downgrade queries should be tested on both list and single resource endpoints.

(I'll being trying to contribute some of these feature requests I'm adding now!)

Can you please add a column to indicate how long each test took? Some tests are being very slow for me (this doesn't surprise me), but it would be helpful to know which ones are the slow ones.

I tried running the IS-05 tests and got the following traceback (I'll try to add more data as I find it):

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/flask/app.py", line 1997, in __call__
    return self.wsgi_app(environ, start_response)
  File "/usr/lib/python3/dist-packages/flask/app.py", line 1985, in wsgi_app
    response = self.handle_exception(e)
  File "/usr/lib/python3/dist-packages/flask/app.py", line 1540, in handle_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3/dist-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/usr/lib/python3/dist-packages/flask/app.py", line 1982, in wsgi_app
    response = self.full_dispatch_request()
  File "/usr/lib/python3/dist-packages/flask/app.py", line 1614, in full_dispatch_request
    rv = self.handle_user_exception(e)
  File "/usr/lib/python3/dist-packages/flask/app.py", line 1517, in handle_user_exception
    reraise(exc_type, exc_value, tb)
  File "/usr/lib/python3/dist-packages/flask/_compat.py", line 33, in reraise
    raise value
  File "/usr/lib/python3/dist-packages/flask/app.py", line 1612, in full_dispatch_request
    rv = self.dispatch_request()
  File "/usr/lib/python3/dist-packages/flask/app.py", line 1598, in dispatch_request
    return self.view_functions[rule.endpoint](**req.view_args)
  File "/home/hbv-bld1/bt/git/nmos-testing/nmos-test.py", line 122, in index_page
    result = test_obj.run_tests()
  File "/home/hbv-bld1/bt/git/nmos-testing/GenericTest.py", line 75, in run_tests
    self.execute_tests()
  File "/home/hbv-bld1/bt/git/nmos-testing/GenericTest.py", line 65, in execute_tests
    self.result += self.basics()
  File "/home/hbv-bld1/bt/git/nmos-testing/GenericTest.py", line 178, in basics
    result = self.check_api_resource(resource, response_code, api)
  File "/home/hbv-bld1/bt/git/nmos-testing/GenericTest.py", line 229, in check_api_resource
    self.save_subresources(resource[0], response)
  File "/home/hbv-bld1/bt/git/nmos-testing/GenericTest.py", line 247, in save_subresources
    if isinstance(response.json(), list):
  File "/usr/lib/python3/dist-packages/requests/models.py", line 842, in json
    self.content.decode(encoding), **kwargs
  File "/usr/lib/python3/dist-packages/simplejson/__init__.py", line 516, in loads
    return _default_decoder.decode(s)
  File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 374, in decode
    obj, end = self.raw_decode(s)
  File "/usr/lib/python3/dist-packages/simplejson/decoder.py", line 404, in raw_decode
    return self.scan_once(s, idx=_w(s, idx).end())
simplejson.scanner.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

There was another registry running on the subnet when I started the test, which explains why it didn't register, but crashing is never good. ;-)

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/flask/app.py", line 1836, in call
return self.wsgi_app(environ, start_response)
File "/usr/lib/python3/dist-packages/flask/app.py", line 1820, in wsgi_app
response = self.make_response(self.handle_exception(e))
File "/usr/lib/python3/dist-packages/flask/app.py", line 1403, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3/dist-packages/flask/_compat.py", line 33, in reraise
raise value
File "/usr/lib/python3/dist-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/usr/lib/python3/dist-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/lib/python3/dist-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3/dist-packages/flask/_compat.py", line 33, in reraise
raise value
File "/usr/lib/python3/dist-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/lib/python3/dist-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functionsrule.endpoint
File "/home/hbv-bld1/bt/git/nmos-testing/nmos-test.py", line 218, in index_page
raise ex
File "/home/hbv-bld1/bt/git/nmos-testing/nmos-test.py", line 215, in index_page
result = test_obj.run_tests(test_selection)
File "/home/hbv-bld1/bt/git/nmos-testing/GenericTest.py", line 125, in run_tests
self.execute_tests(test_name)
File "/home/hbv-bld1/bt/git/nmos-testing/GenericTest.py", line 98, in execute_tests
self.result.append(method())
File "/home/hbv-bld1/bt/git/nmos-testing/IS0401Test.py", line 101, in test_01
return test.FAIL("Node did not attempt to register with the registry advertised on {}.".format(default_ip))
NameError: name 'default_ip' is not defined

I'm running commit 009b81b.

A new set of tests should cover the integration points between IS-04 and IS-05, ensuring that implementations indicate the correct state via both APIs (Node/Connection) as a change is made via either of them.

Based on the most recent interop checklists, the following items remain to be automated if possible.
NB: Not all test cases are applicable for all Device Classes.

AMWA NMOS Discovery & Registration (IS-04) Checklist

Version 2.2 (17th July 2018)

Section 5: Registries

NB: Each of the below items should be checked off against each API version implemented.

• 5.1 Registration API is advertised using mDNS, with appropriate TXT records
• 5.2 Query API is advertised using mDNS, with appropriate TXT records
• 5.3 Registration API accepts and stores a valid Node resource
• 5.4 Registration API rejects an invalid Node resource with a 400 HTTP code
• 5.5 Registration API accepts and stores a valid Device resource
• 5.6 Registration API rejects an invalid Device resource with a 400 HTTP code
• 5.7 Registration API accepts and stores a valid Source resource
• 5.8 Registration API rejects an invalid Source resource with a 400 HTTP code
• 5.9 Registration API accepts and stores a valid Flow resource
• 5.10 Registration API rejects an invalid Flow resource with a 400 HTTP code
• 5.11 Registration API accepts and stores a valid Sender resource
• 5.12 Registration API rejects an invalid Sender resource with a 400 HTTP code
• 5.13 Registration API accepts and stores a valid Receiver resource
• 5.14 Registration API rejects an invalid Receiver resource with a 400 HTTP code
• 5.15 Registration API accepts heartbeat requests for a Node held in the registry
• 5.16 Registration API cleans up Nodes and their sub-resources when a heartbeat doesn’t occur for the duration of a fixed timeout period
• 5.17 Query API supports browsing Node, Device, Source, Flow, Sender and Receiver resources via HTTP
• 5.18 Query API supports persistent websocket connections to each resource type, created via the /subscriptions resource
• 5.19 Query API supports basic query parameter filtering
• 5.20 Query API supports advanced query parameter filtering (v1.1+)
• 5.21 Query API supports query parameters in websocket connections
• 5.22 Two instances of the same Registry Server are shown to operate in a cluster, sharing registered data
• 5.23 A Node can be shown to register with one clustered Registration API, before moving to another Registration API in the cluster without the Registry Server losing registered data

The IS-07 spec appears to use valid RAML 1.0, but ramlfications (being a 0.8 library) can't quite handle the way they're written. As such several tests appear as 'manual' when they should work.

There are edge cases for Query API pagination, including the cursors themselves but also correct encoding of the Link header, especially when RQL queries are involved.

(I'll being trying to contribute some of these feature requests I'm adding now!)

Follow-up to #3 / #22 / #24.
We could probably auto-generate names for the RAML auto tests, e.g. auto_get_x_nmos, auto_get_x_nmos_node, ..., auto_get_x_nmos_node_v1_2_sources_sourceId.
It'd be nice to able to list and then run auto-tests in isolation too. Would mean a mode that returned the collected test names without running them.

Based on the most recent interop checklists, the following items remain to be automated if possible.
NB: Not all test cases are applicable for all Device Classes.

AMWA NMOS Connection Management (IS-05) Checklist

Version 1.2 (2nd July 2018)

Section 3: Bulk Interface

• 3.1 The bulk interface is exposed as an endpoint
• 3.2 Changes made to staged parameters in the bulk interface are correctly reflected in the single interface.
• 3.3 Activations carried out on the bulk interface result in the activation of staged parameters
• 3.4 If an individual sender/receiver rejects the request (i.e returns a 400/500 series response) the status code is reflected in the JSON response, but the response code for the bulk request is still 200.

When a v1.x-dev branch gets deleted, if you previously had it in your local cache of the spec files, the next git pull will fail. The only solution to this at present is to manually delete the cache directory. Ideally this should be detected and handled more gracefully.

test_01 in IS0401Test only allows 5 seconds for initial registration. The comment implies this is something to do with the default heartbeat interval, which doesn't make any sense to me...

Based on the most recent interop checklists, the following items remain to be automated if possible.
NB: Not all test cases are applicable for all Device Classes.

AMWA NMOS Discovery & Registration (IS-04) Checklist

Version 2.2 (17th July 2018)

Section 6: Registry (and Connection Management) Clients

NB: Each of the below items should be checked off against each API version implemented.

• 6.1 Interface discovers a Query API via mDNS
• 6.2 Interface discovers a Query API via unicast DNS
• 6.3 Interface discovers required resources (Nodes, Devices etc) via the Query API
• 6.4 Interface updates based on dynamic changes from a Query API websocket
• 6.5 Connection manager instructs Receivers to subscribe to Senders via the Node API
• 6.6 Connection manager indicates success or failure of routing via updates received from the Query API

Follow-up to #9.

CORS preflight response headers:

• Access-Control-Allow-Methods header instead of Allow
• Access-Control-Allow-Headers header, probably based on request's Access-Control-Request-Headers, e.g. Content-Type
• Access-Control-Max-Age with a sensible value (since answer is unlikely to be dynamic)

CORS response headers:

• Access-Control-Allow-Origin header for request Origin (or *)
• Access-Control-Expose-Headers header for non-safelisted headers (e.g. Content-Length). (Not all browsers support latest CORS spec which allows *.)

(I'll being trying to contribute some of these feature requests I'm adding now!)

Hi again,

I finally got to the relative activation tests and have discovered that the retry loop times out too quickly. I had to change the loop to test if retries < 5 rather than the original 3.

It would be useful to be able to trigger a test set by providing the required fields (IPs/Ports/Versions etc) from the command line as an alternative to the web interface. This would make it much more straightforward to use the test suite for automated integration testing. Output of test results would be to stdout/file, likely in a machine readable format (XML/JSON).

Based on the most recent interop checklists, the following items remain to be automated if possible.
NB: Not all test cases are applicable for all Device Classes.

AMWA NMOS Connection Management (IS-05) Checklist

Version 1.2 (2nd July 2018)

Section 1: API Implementations

• 1.1 Connection Management API structure matches the spec.
• 1.2 All API endpoints implement the correct methods (GET, PATCH etc.)
• 1.3 A transport file (e.g SDP) can be staged to the receiver API directly
• 1.4 The transport parameters in the transport file automatically propagate to the receiver’s staged transport parameters
• 1.5 Immediate activations can be called on the API
• 1.6 Relative offset and absolute offset activations can be called on the API
• 1.7 When activation occurs staged transport parameters transition to active
• 1.8 For relative and absolute offset activations the transition of parameters occurs at the correct time offset.
• 1.9 During activation all “auto” entries are translated to their actual values
• 1.10 During the time between a delayed activation being requested and it occurring, any attempt to alter staged transport files or parameters results in an HTTP 423 (locked).
• 1.11 The sender’s active transport file endpoint returns an HTTP 200 response containing the transport file.
• 1.12 The API rejects any requests that do not match the relevant schema with an HTTP 400 response code
• 1.13 If a transport file and transport parameter set are PATCHed to staged simultaneously the content of the transport parameters takes precedence over the content of the transport files.
• 1.14 The keys in the the constraints endpoint mirror the keys of the transport parameters on the staged endpoint
• 1.15 At the point of activation the 'version' attribute of the relevant IS-04 Sender or Receiver is incremented.
• 1.16 Receivers which do not support 2022-7 can stage transport parameters for the primary leg only when receiving a 2022-7 format transport file

Hi, Andrew,

I'd like to limit the tests run to a single test (since it's the one I'm trying to debug). Any suggestions on how I might do that, other than simply ripping out all the other tests?

Thanks,
Bill

Based on the most recent interop checklists, the following items remain to be automated if possible.
NB: Not all test cases are applicable for all Device Classes.

AMWA NMOS Discovery & Registration (IS-04) Checklist

Version 2.2 (17th July 2018)

Section 2: RTP Transport

• 2.1 Automated interpretation and multicast group join based on a valid SDP file for an ST.2110-20 RTP multicast stream
• 2.2 Automated interpretation and multicast group join based on a valid SDP file for an ST.2110-30 RTP multicast stream
• 2.3 Automated interpretation and multicast group join based on a valid SDP file for an ST.2110-40 RTP multicast stream
• 2.4 Automated interpretation and multicast group join based on a valid SDP file for an ST.2022-6 RTP multicast stream
• 2.5 Automated production of a valid SDP file for an ST.2110-20 RTP multicast stream
• 2.6 Automated production of a valid SDP file for an ST.2110-30 RTP multicast stream
• 2.7 Automated production of a valid SDP file for an ST.2110-40 RTP multicast stream
• 2.8 Automated production of a valid SDP file for an ST.2022-6 RTP multicast stream

Could the Registry tests create a mock Node (or a few) in order to test the Query API, rather than relying on the user having to run a Node?

Based on the most recent interop checklists, the following items remain to be automated if possible.
NB: Not all test cases are applicable for all Device Classes.

AMWA NMOS Connection Management (IS-05) Checklist

Version 1.2 (2nd July 2018)

Section 5: FEC and RTCP

These tests only apply to servers making use of FEC and/or RTCP parameters.

• 5.1 Where FEC is in use, FEC parameters are exposed in staged and active parameter sets
• 5.2 Where FEC is in use, FEC parameters are represented in the constraints endpoint as per the spec. examples
• 5.3 Where RTCP is in use, RTCP parameters are exposed in staged and active parameter sets
• 5.4 Where RTCP is in use, RTCP parameters are represented in the constraints endpoint as per the spec. examples

I just got the following error: "Got wrong response from bulk/, expected an array containing ['senders/', 'receivers/'], got ['receivers/', 'senders/']". IS-05 does not obviously specify the order of elements in the array (unless the RAML order must be considered definitive), and either order seems like it should be acceptable.

Based on the most recent interop checklists, the following items remain to be automated if possible.
NB: Not all test cases are applicable for all Device Classes.

AMWA NMOS Connection Management (IS-05) Checklist

Version 1.2 (2nd July 2018)

Section 6: API Clients

These checks only apply to API clients

• 6.1 UI can discover Query API automatically
• 6.2 UI can display Senders and Receivers from Query API
• 6.3 UI updates via websocket when Senders and Receivers change
• 6.4 UI can discover Devices with Connection Management APIs via the Query API data
• 6.5 UI can retune an RTP receiver using an SDP file
• 6.6 UI can configure an RTP transmitter for unicast sending
• 6.7 UI can configure an RTP receiver for unicast receiving
• 6.8 UI can enable/disable and configure 2022-7 on an RTP transmitter/receiver
• 6.9 UI can enable/disable and configure FEC on an RTP transmitter/receiver
• 6.10 UI can interact with both the deprecated Node API connection management, and the IS-05 Connection Management API

In order to fully test the specs, some tests must be run on an isolated network segment with the unit under test. This is fine for general testing, but in a workshop environment, a multi-user testing resource would be preferred (as was possible in the original Riedel test suite).

Ideally a workshop focused mode should also be available, even if this is incapable of testing implementations to the same degree.

If a Sender isn't enabled, the spec says the API should return 404 for its /transportfile. That understandably isn't handled by the GenericTest code for reads, so a fail is reported.

I.e. make testing v1.2 the default for IS-04.

Based on the most recent interop checklists, the following items remain to be automated if possible.
NB: Not all test cases are applicable for all Device Classes.

AMWA NMOS Discovery & Registration (IS-04) Checklist

Version 2.2 (17th July 2018)

Section 4: Connection Management

NB: Each of the below items should be checked off against each API version implemented.

• 4.1 Node advertises valid Sender and Receiver resources via its Node API
• 4.2 PUTing to a Receiver target resource with a Sender resource payload is accepted and connects the Receiver to a stream
• 4.3 Receiver resource (in Node API and registry) is correctly updated to match the subscribed Sender ID upon subscription
• 4.4 PUTing to a Receiver target resource with an empty JSON object payload is accepted and disconnects the Receiver from a stream

I have switched from the RiedelCommunications repo, successfully pip installed the additional dependencies (gitpython, netifaces, ramlfication), and started up test-nmos.py OK. I can see it's fetched the IS-04 specs etc.

Trying to execute first test, "IS-04 Node API", I get error in the browser and console:

jsonref.JsonRefError: URLError: <urlopen error [WinError 2] The system cannot find the file specified: '\\resource_core.json'>

Since it has successfully downloaded resource_core.json and the other schema files, I assume this is about resolving the first '$ref' it processes, presumably in schemas/node.json?

Any ideas?

IS0402Test custom tests are written to work with separate reg_url and query_url., but GenericTest assumes both apis are on the same base_url and the web-app UI and nmos-test driver assume Registration API and Query API are served on the same port.

Is it best practice to serve Reg and Query APIs on the same port?
And does the same go for Node API and Connection API?

I just got bitten by this -- my staged parameters violate the constraints, and since the API client didn't specify any transport parameters, the server rejected the request as invalid. Oops.

I don't think you can do the same thing for the active parameters, however -- a parameter might be constrained to "auto", but the active parameter has to reflect the actual value.

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/flask/app.py", line 1836, in call
return self.wsgi_app(environ, start_response)
File "/usr/lib/python3/dist-packages/flask/app.py", line 1820, in wsgi_app
response = self.make_response(self.handle_exception(e))
File "/usr/lib/python3/dist-packages/flask/app.py", line 1403, in handle_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3/dist-packages/flask/_compat.py", line 33, in reraise
raise value
File "/usr/lib/python3/dist-packages/flask/app.py", line 1817, in wsgi_app
response = self.full_dispatch_request()
File "/usr/lib/python3/dist-packages/flask/app.py", line 1477, in full_dispatch_request
rv = self.handle_user_exception(e)
File "/usr/lib/python3/dist-packages/flask/app.py", line 1381, in handle_user_exception
reraise(exc_type, exc_value, tb)
File "/usr/lib/python3/dist-packages/flask/_compat.py", line 33, in reraise
raise value
File "/usr/lib/python3/dist-packages/flask/app.py", line 1475, in full_dispatch_request
rv = self.dispatch_request()
File "/usr/lib/python3/dist-packages/flask/app.py", line 1461, in dispatch_request
return self.view_functionsrule.endpoint
File "/home/hbv-bld1/bt/git/nmos-testing/nmos-test.py", line 134, in index_page
result = test_obj.run_tests()
File "/home/hbv-bld1/bt/git/nmos-testing/GenericTest.py", line 90, in run_tests
self.execute_tests()
File "/home/hbv-bld1/bt/git/nmos-testing/IS0401Test.py", line 44, in execute_tests
super(IS0401Test, self).execute_tests()
File "/home/hbv-bld1/bt/git/nmos-testing/GenericTest.py", line 86, in execute_tests
self.result.append(method())
File "/home/hbv-bld1/bt/git/nmos-testing/IS0401Test.py", line 65, in test_01
zeroconf.register_service(info)
File "/usr/lib/python3/dist-packages/zeroconf.py", line 1427, in register_service
self.check_service(info)
File "/usr/lib/python3/dist-packages/zeroconf.py", line 1533, in check_service
raise NonUniqueNameException
zeroconf.NonUniqueNameException

Hi, Andrew,

I'm using ea91716 and noticed a bug in check_matching_resource: It doesn't return if it finds a resource ID on the node that isn't in the registry. Fix with:

diff --git IS0401Test.py IS0401Test.py
index b8cd8bf..91fd6a1 100644
--- IS0401Test.py
+++ IS0401Test.py
@@ -164,7 +164,7 @@ class IS0401Test(GenericTest):
 
                     for resource in node_resources:
                         if resource not in reg_resources:
-                            test.FAIL("{} {} was not found in the registry.".format(res_type.title(), resource))
+                            return test.FAIL("{} {} was not found in the registry.".format(res_type.title(), resource))
                         elif reg_resources[resource] != node_resources[resource]:
                             return test.FAIL("Node API JSON does not match data in registry for "
                                              "{} {}.".format(res_type.title(), resource))

Recognizing that we haven't agreed a core set of RQL operators, AMWA-TV/is-04#20, we could do with at least some tests to encourage implementors.

(I'll being trying to contribute some of these feature requests I'm adding now!)

A global fallback timeout for requests should be included in case of this sort of error condition.

This should avoid Fails or N/As against things which an implementation may choose to implement. It should include a note to indicate why it may/may not be important to implement a given optional feature.

Based on the most recent interop checklists, the following items remain to be automated if possible.
NB: Not all test cases are applicable for all Device Classes.

AMWA NMOS Connection Management (IS-05) Checklist

Version 1.2 (2nd July 2018)

Section 4: SMPTE 2022-7 Operation

This section only applied to API servers making use SMPTE 2022-7 “dual legged” operation.

• 4.1 Staged and active transport parameters are able to have two “legs” with different parameter sets for each
• 4.2 During activations both “legs” are activated simultaneously
• 4.3 When operating with SMPTE 2022-7, senders provide transport files compliant with SDP group semantics (see VSF TR-04)
• 4.4 Transport files specifying parameters for both legs (as per VSF TR-04) are correctly parsed by receivers and applied to staged transport parameters
• 4.5 When a SMPTE 2022-7 receiver is activated with a SDP file that only specifies one leg it only populates one leg of the transport parameters, and returns the other leg to null/default values.
• 4.6 Node API correctly advertises two interface_bindings for Senders and Receivers which are 2022-7 capable.
• 4.7 Constraints endpoint has two separate elements with constraints for each leg, even if they are identical.

The channel mapping /activations resource uses a new key/value pair format, with sub-resources accessed via a GET to /activations/{key}. The test suite doesn't support this format yet.

Requires a new test case in the IS0501 suite for v1.0 APIs. As the /transporttype endpoint doesn't exist at v1.0 it will likely need to be based upon presence of a transport parameter such as rtp_enabled.