When the Tempest suite is executed, each test duration is measured and the full
console output is stored to a *log* file for further analysis.
-The Tempest testcases are distributed accross two
+The Tempest testcases are distributed across two
Tiers:
* Smoke Tier - Test Case 'tempest_smoke_serial'
A basic SLA (stop test on errors) has been implemented.
-The Rally testcases are distributed accross two Tiers:
+The Rally testcases are distributed across two Tiers:
* Smoke Tier - Test Case 'rally_sanity'
* Components Tier - Test case 'rally_full'
NOTE: Test case 'rally_sanity' executes a limited number of Rally smoke test
cases. Test case 'rally_full' executes the full defined set of Rally tests.
+
+Refstack-client to run Defcore testcases
+-----------------------------------------
+
+Refstack-client `[8]`_ is a command line utility that allows you to
+execute Tempest test runs based on configurations you specify.
+It is the official tool to run Defcore `[9]`_ testcases,
+which focuses on testing interoperability between OpenStack clouds.
+
+Refstack-client is integrated in Functest, consumed by Dovetail, which
+intends to define and provide a set of OPNFV related validation criteria
+that will provide input for the evaluation of the use of OPNFV trademarks.
+This progress is under the guideline of Compliance Verification Program(CVP).
+
+Defcore testcases
+^^^^^^^^^^^^^^^^^^
+
+*Danube Release*
+
+Set of DefCore tempest test cases not flagged and required.
+According to `[10]`_, some tests are still flagged due to outstanding bugs
+in the Tempest library, particularly tests that require SSH. Refstack developers
+are working on correcting these bugs upstream. Please note that although some tests
+are flagged because of bugs, there is still an expectation that the capabilities
+covered by the tests are available. It only contains Openstack core compute
+(no object storage). The approved guidelines (2016.08) are valid for Kilo,
+Liberty, Mitaka and Newton releases of OpenStack.
+The list can be generated using the Rest API from RefStack project:
+https://refstack.openstack.org/api/v1/guidelines/2016.08/tests?target=compute&type=required&alias=true&flag=false
+
+Running methods
+^^^^^^^^^^^^^^^
+
+Two running methods are provided after refstack-client integrated into
+Functest, Functest command line and manually, respectively.
+
+By default, for Defcore test cases run by Functest command line,
+are run followed with automatically generated
+configuration file, i.e., refstack_tempest.conf. In some circumstances,
+the automatic configuration file may not quite satisfied with the SUT,
+Functest also inherits the refstack-client command line and provides a way
+for users to set its configuration file according to its own SUT manually.
+
+*command line*
+
+Inside the Functest container, first to prepare Functest environment:
+
+::
+
+ functest env prepare
+
+then to run default defcore testcases by using refstack-client:
+
+::
+
+ functest testcase run refstack_defcore
+
+In OPNFV Continuous Integration(CI) system, the command line method is used.
+
+*manually*
+
+Inside the Functest container, first to prepare the refstack virtualenv:
+
+::
+
+ cd /home/opnfv/repos/refstack-client
+ source .venv/bin/activate
+
+then prepare the tempest configuration file and the testcases want to run with the SUT,
+run the testcases with:
+
+::
+
+ ./refstack-client test -c <Path of the tempest configuration file to use> -v --test-list <Path or URL of test list>
+
+using help for more information:
+
+::
+
+ ./refstack-client --help
+ ./refstack-client test --help
+
+Reference tempest configuration
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+*command line method*
+
+When command line method is used, the default tempest configuration file
+is generated by Rally.
+
+*manually*
+
+When running manually is used, recommended way to generate tempest configuration
+file is:
+
+::
+
+ cd /usr/lib/python2.7/site-packages/functest/opnfv_tests/openstack/refstack_client
+ python tempest_conf.py
+
+a file called tempest.conf is stored in the current path by default, users can do
+some adjustment according to the SUT:
+
+::
+
+ vim refstack_tempest.conf
+
+a reference article can be used `[15]`_.
+
+
snaps_smoke
------------
This test case contains tests that setup and destroy environments with VMs with
and without Floating IPs with a newly created user and project. Set the config
value snaps.use_floating_ips (True|False) to toggle this functionality. When
-the config value of snaps.use_keystone is True, functest must have access
+the config value of snaps.use_keystone is True, Functest must have access
the cloud's private network.
This suite consists in 38 tests (test duration < 10 minutes)
* Delete operations
* Delete the port previously created via OpenStack
- * Check that the port has been also succesfully deleted in OpenDaylight
+ * Check that the port has been also successfully deleted in OpenDaylight
* Delete previously subnet created via OpenStack
* Check that the subnet has also been successfully deleted in OpenDaylight
* Delete the network created via OpenStack
- * Check that the network has also been succesfully deleted in OpenDaylight
+ * Check that the network has also been successfully deleted in OpenDaylight
Note: the checks in OpenDaylight are based on the returned HTTP status
code returned by OpenDaylight.
Features
--------
-In Danube, functest supports the integration of:
+In Danube, Functest supports the integration of:
* barometer
* bgpvpn
* domino
* fds
* multisite
- * netready
* odl-sfc
* promise
* security_scan
-Note: copper is not supported in Danube.
-
Please refer to the dedicated feature user guides for details.
.. _`[5]`: https://github.com/Orange-OpenSource/opnfv-cloudify-clearwater/blob/master/openstack-blueprint.yaml
.. _`[6]`: https://scap.nist.gov/
.. _`[7]`: https://github.com/OpenSCAP/openscap
+.. _`[8]`: https://github.com/openstack/refstack-client
+.. _`[9]`: https://github.com/openstack/defcore
+.. _`[10]`: https://github.com/openstack/interop/blob/master/2016.08/procedure.rst
.. _`[11]`: http://robotframework.org/
.. _`[12]`: http://artifacts.opnfv.org/parser/colorado/docs/userguide/index.html
.. _`[13]`: https://wiki.opnfv.org/display/PROJ/SNAPS-OO
.. _`[14]`: https://github.com/oolorg/opnfv-functest-vrouter
+.. _`[15]`: https://aptira.com/testing-openstack-tempest-part-1/
`OPNFV main site`_
Code Review / functest.git / blobdiff