The following guide indicates how to run the tool against an implementation of one of the NMOS APIs.
|Open a browser and go to the address of the server you deployed.|
|Select a test suite from the dropdown.|
|Provide the IP or hostname and port of the relevant API under test, along with the preferred API version.|
|Optionally, select a subset of the tests to run. This should default to ‘all’.|
|Once all of the field have been completed, press ‘Run’. The result of the tests will be shown after a few seconds.|
Understanding the Results
The result of each test case will be one of the following:
|Successful test case.|
|Required feature of the specification has been found to be implemented incorrectly.|
|Not a failure, but the API being tested is responding or configured in a way which is not recommended in most cases.|
|Test is disabled due to test suite configuration; change the config or test manually.|
|Test was not run due to prior responses from the API, which may be OK, or indicate a fault.|
|Recommended/optional feature of the specifications has been found to be not implemented.|
|Test suite does not currently test this feature, so it must be tested manually.|
|Test is not applicable, e.g. due to the version of the specification being tested.|
The testing tool has a number of general configuration settings that affect its operation, defined in
To alter these, copy
nmostesting/UserConfig.py and make your changes there.
MAX_TEST_ITERATIONS can be set to restrict the maximum number of resources or test points that time-consuming tests run against.
The default of
0 means unlimited (all available resources or test points) for a really thorough test!
The following pages provide information on additional instructions for testing specific features:
The testing tool supports two modes of operation that support usage in larger scale or continuous integration tests.
Some of the tests contained within this tool perform a number of steps which may not be obvious without viewing the source code. To help with debugging, descriptions of such behaviour is covered by the advanced testing pages.