By internal, we mean that this particular test cases have been
developped and/or integrated by functest contributors and the associated
code is hosted in the Functest repository.
-An internal case can be fully developped or a simple integration of
+An internal case can be fully developed or a simple integration of
upstream suites (e.g. Tempest/Rally developped in OpenStack are just
integrated in Functest).
The structure of this repository is detailed in `[1]`_.
The code to run these test cases may be directly in the repository of
the project. We have also a **features** sub directory under opnfv_tests
-directory that may be used (it can be usefull if you want to reuse
+directory that may be used (it can be useful if you want to reuse
Functest library).
Functest framework
Functest can be considered as a framework.
Functest is release as a docker file, including tools, scripts and a CLI
-to prepare the environement and run tests.
+to prepare the environment and run tests.
It simplifies the integration of external test suites in CI pipeline
and provide commodity tools to collect and display results.
Since Colorado, test categories also known as tiers have been created to
-group similar tests, provide consistant sub-lists and at the end optimize
+group similar tests, provide consistent sub-lists and at the end optimize
test duration for CI (see How To section).
The definition of the tiers has been agreed by the testing working group.
`-- openstack_utils.py
Note that for Openstack, keystone v3 is now deployed by default by compass,
-fuel and joid in Danube. All installers still support keysone v2 (deprecated in
+fuel and joid in Danube. All installers still support keystone v2 (deprecated in
next version).
Test collection framework
===================
An automatic reporting page has been created in order to provide a
- consistant view of the scenarios.
+ consistent view of the scenarios.
In this page, each scenario is evaluated according to test criteria.
The code for the automatic reporting is available at `[8]`_.
os-odl_l2-nofeature scenarios.
If no result is available or if all the results are failed, the test
case get 0 point.
- If it was succesfull at least once but not anymore during the 4 runs,
+ If it was successful at least once but not anymore during the 4 runs,
the case get 1 point (it worked once).
If at least 3 of the last 4 runs were successful, the case get 2 points.
If the last 4 runs of the test are successful, the test get 3 points.
Dashboard
=========
-Dashboard is used to provide a consistant view of the results collected
+Dashboard is used to provide a consistent view of the results collected
in CI.
The results showed on the dashboard are post processed from the Database,
which only contains raw results.
security_scan before the creation of the security repository)
-How test constraints are defined?
+How are test constraints defined?
=================================
-Test constraints are defined according to 2 paramaters:
+Test constraints are defined according to 2 parameters:
* The scenario (DEPLOY_SCENARIO env variable)
* The installer (INSTALLER_TYPE env variable)
scenario: '(ocl)|(nosdn)|^(os-odl)((?!bgpvpn).)*$'
-How to write and check constaint regex?
+How to write and check constraint regex?
=======================================
Regex are standard regex. You can have a look at `[11]`_
You can use the API `[13]`_. The static declaration is in git `[5]`_
If you are in a Functest docker container (assuming that the
-environement has been prepared): just use the CLI.
+environment has been prepared): just use the CLI.
You can get the list per Test cases or by Tier::
This command will run all the test cases of the first 2 tiers, i.e.
healthcheck, connection_check, api_check, vping_ssh, vping_userdata,
-snaps_somke, tempest_smoke_serial and rally_sanity.
+snaps_smoke, tempest_smoke_serial and rally_sanity.
How to push your results into the Test Database
Please note that each exclusion must be justified. the goal is not to exclude
test cases because they do not pass. Several scenarios reached the 100% criteria.
-So it is expected in the patch submited to exclude the cases to indicate the
+So it is expected in the patch submitted to exclude the cases to indicate the
reasons of the exclusion.
CATEGORIES/TIERS description:
+----------------+-------------------------------------------------------------+
-| healthcheck | Simple OpenStack healtcheck tests case that validates the |
+| healthcheck | Simple OpenStack healthcheck tests case that validates the |
| | basic operations in OpenStack |
+----------------+-------------------------------------------------------------+
| Smoke | Set of smoke test cases/suites to validate the most common |
| | Those come from Feature projects and need a bit of support |
| | for integration |
+----------------+-------------------------------------------------------------+
-| Components | Advanced Openstack tests: Full Tempest, Full Rally |
+| Components | Advanced OpenStack tests: Full Tempest, Full Rally |
+----------------+-------------------------------------------------------------+
| Performance | Out of Functest Scope |
+----------------+-------------------------------------------------------------+
VNF category is really dedicated to test including:
* creation of resources
- * deployement of an orchestrator/VNFM
+ * deployment of an orchestrator/VNFM
* deployment of the VNF
* test of the VNFM
* free resources
Create your own VnfOnboarding file
-you must create your entry point through a python clase as referenced in the
+you must create your entry point through a python class as referenced in the
configuration file
e.g. aaa => creation of the file <Functest repo>/functest/opnfv_tests/vnf/aaa/aaa.py
When the Tempest suite is executed, each test duration is measured and the full
console output is stored to a *log* file for further analysis.
-The Tempest testcases are distributed accross two
+The Tempest testcases are distributed across two
Tiers:
* Smoke Tier - Test Case 'tempest_smoke_serial'
A basic SLA (stop test on errors) has been implemented.
-The Rally testcases are distributed accross two Tiers:
+The Rally testcases are distributed across two Tiers:
* Smoke Tier - Test Case 'rally_sanity'
* Components Tier - Test case 'rally_full'
* Delete operations
* Delete the port previously created via OpenStack
- * Check that the port has been also succesfully deleted in OpenDaylight
+ * Check that the port has been also successfully deleted in OpenDaylight
* Delete previously subnet created via OpenStack
* Check that the subnet has also been successfully deleted in OpenDaylight
* Delete the network created via OpenStack
- * Check that the network has also been succesfully deleted in OpenDaylight
+ * Check that the network has also been successfully deleted in OpenDaylight
Note: the checks in OpenDaylight are based on the returned HTTP status
code returned by OpenDaylight.