* connection_check
* vping_ssh
* vping_userdata
- * odl
* rally_full
* rally_sanity
* tempest_smoke
integrated by functest contributors and the associated code is hosted in the
Functest repository.
An internal case can be fully developed or a simple integration of
-upstream suites (e.g. Tempest/Rally developed in OpenStack, or odl suites are
+upstream suites (e.g. Tempest/Rally developed in OpenStack are
just integrated in Functest).
The structure of this repository is detailed in `[1]`_.
The main internal test cases are in the opnfv_tests subfolder of the
repository, the internal test cases can be grouped by domain:
- * sdn: odl, odl_fds
* openstack: connection_check, vping_ssh, vping_userdata, tempest_*, rally_*
* vnf: cloudify_ims
jinja2 templates `[3]`_.
-Dashboard
-=========
-
-Additional dashboarding is managed at the testing group level, see `[5]`_ for
-details.
-
-
==========
References
==========
_`[4]`: https://wiki.opnfv.org/display/functest/2017+Beijing?preview=%2F11699623%2F11700523%2FTestAPI+-+test+results+collection+service.pptx
-_`[5]`: https://lfanalytics.io/projects/lfn%2Fopnfv/dashboard
-
IRC support chan: #opnfv-functest
EXTERNAL_NETWORK=XXX # if not first network with router:external=True
DASHBOARD_URL=XXX # else tempest_horizon will be skipped
NEW_USER_ROLE=XXX # if not member
- SDN_CONTROLLER_IP=XXX # if odl scenario
VOLUME_DEVICE_NAME=XXX # if not vdb
FLAVOR_EXTRA_SPECS=hw:mem_page_size:large # if fdio scenarios
| cinder_test | functest | healthcheck | 01:05 | PASS |
| tempest_smoke | functest | healthcheck | 05:39 | PASS |
| tempest_horizon | functest | healthcheck | 01:05 | PASS |
- | odl | functest | healthcheck | 00:00 | SKIP |
+--------------------------+------------------+---------------------+------------------+----------------+
NOTE: the duration is a reference and it might vary depending on your SUT.
The format for the DEPLOY_SCENARIO env variable can be described as follows:
* vim: (os|k8s) = OpenStack or Kubernetes
- * controller is one of ( nosdn | odl )
+ * controller: nosdn
* nfv_feature is one or more of ( ovs | kvm | sfc | bgpvpn | nofeature )
* ha_mode (high availability) is one of ( ha | noha )
test_overview.rst
test_details.rst
test_results.rst
- reporting.rst
troubleshooting.rst
+++ /dev/null
-.. http://creativecommons.org/licenses/by/4.0
-
-Test reporting
-==============
-
-An automatic reporting page has been created in order to provide a consistent
-view of the Functest tests on the different scenarios.
-
-In this page, each scenario is evaluated according to test criteria.
-
-The results are collected from the centralized database every day and, per
-scenario. A score is calculated based on the results from the last 10 days.
-This score is the addition of single test scores. Each test case has a success
-criteria reflected in the criteria field from the results.
-
-As an illustration, let's consider the scenario
-os-odl_l2-nofeature-ha scenario, the scenario scoring is the addition of the
-scores of all the runnable tests from the categories (tiers, healthcheck, smoke
-and features) corresponding to this scenario.
-
- +---------------------+---------+---------+---------+---------+
- | Test | Apex | Compass | Fuel | Joid |
- +=====================+=========+=========+=========+=========+
- | vPing_ssh | X | X | X | X |
- +---------------------+---------+---------+---------+---------+
- | vPing_userdata | X | X | X | X |
- +---------------------+---------+---------+---------+---------+
- | tempest_smoke | X | X | X | X |
- +---------------------+---------+---------+---------+---------+
- | rally_sanity | X | X | X | X |
- +---------------------+---------+---------+---------+---------+
- | odl | X | X | X | X |
- +---------------------+---------+---------+---------+---------+
- | promise | | | X | X |
- +---------------------+---------+---------+---------+---------+
- | doctor | X | | X | |
- +---------------------+---------+---------+---------+---------+
- | security_scan | X | | | |
- +---------------------+---------+---------+---------+---------+
- | parser | | | X | |
- +---------------------+---------+---------+---------+---------+
- | copper | X | | | X |
- +---------------------+---------+---------+---------+---------+
-
- src: os-odl_l2-nofeature-ha Colorado (see release note for the last matrix
- version)
-
-All the testcases (X) listed in the table are runnable on os-odl_l2-nofeature
-scenarios.
-Please note that other test cases (e.g. sfc_odl, bgpvpn) need ODL configuration
-add-ons and, as a consequence, specific scenario.
-There are not considered as runnable on the generic odl_l2 scenario.
-
-
-If no result is available or if all the results are failed, the test case get 0
-point.
-If it was successful at least once but not anymore during the 4 runs, the case
-get 1 point (it worked once).
-If at least 3 of the last 4 runs were successful, the case get 2 points.
-If the last 4 runs of the test are successful, the test get 3 points.
-
-In the example above, the target score for fuel/os-odl_l2-nofeature-ha is
-3 x 8 = 24 points and for compass it is 3 x 5 = 15 points .
-
-The scenario is validated per installer when we got 3 points for all individual
-test cases (e.g 24/24 for fuel, 15/15 for compass).
-
-Please note that complex or long duration tests are not considered yet for the
-scoring. In fact the success criteria are not always easy to define and may
-require specific hardware configuration.
-
-Please also note that all the test cases have the same "weight" for the score
-calculation whatever the complexity of the test case. Concretely a vping has
-the same weight than the 200 tempest tests.
-Moreover some installers support more features than others. The more cases your
-scenario is dealing with, the most difficult to rich a good scoring.
-
-Therefore the scoring provides 3 types of indicators:
-
- * the richness of the scenario: if the target scoring is high, it means that
- the scenario includes lots of features
- * the maturity: if the percentage (scoring/target scoring * 100) is high, it
- means that all the tests are PASS
- * the stability: as the number of iteration is included in the calculation,
- the percentage can be high only if the scenario is run regularly (at least
- more than 4 iterations over the last 10 days in CI)
-
-In any case, the scoring is used to give feedback to the other projects and
-does not represent an absolute value of the scenario.
-
-See `reporting page`_ for details. For the status, click on the version,
-Functest then the Status menu.
-
-.. _`reporting page`: http://testresults.opnfv.org/reporting/
-
-.. figure:: ../../../images/functest-reporting-status.png
- :align: center
- :alt: Functest reporting portal Fuel status page
| | | | benchmarking OpenStack modules |
| | | | See the Rally documents `[3]`_ |
+-------------+---------------+------------+----------------------------------+
-| Controllers | smoke | odl | Opendaylight Test suite |
-| | | | Limited test suite to check the |
-| | | | basic neutron (Layer 2) |
-| | | | operations mainly based on |
-| | | | upstream testcases. See below |
-| | | | for details |
-+-------------+---------------+------------+----------------------------------+
| VNF | vnf | cloudify | Example of a real VNF deployment |
| | | \_ims | to show the NFV capabilities of |
| | | | the platform. The IP Multimedia |
| | +------------+----------------------------------+
| | | vyos | vRouter testing |
| | | \_vrouter | |
-| | +------------+----------------------------------+
-| | | juju_epc | Validates deployment of a complex|
-| | | | mobility VNF on OPNFV Platform. |
-| | | | Uses Juju for deploying the OAI |
-| | | | EPC and ABot for defining test |
-| | | | scenarios using high-level DSL. |
-| | | | VNF tests reference 3GPP |
-| | | | Technical Specs and are executed |
-| | | | through protocol drivers provided|
-| | | | by ABot. |
+-------------+---------------+------------+----------------------------------+
| Kubernetes | healthcheck | k8s_smoke | Test a running Kubernetes |
| | | | cluster and ensure it satisfies |
.. _`[3]`: https://rally.readthedocs.io/en/latest/index.html
-Controllers
------------
-
-Opendaylight
-^^^^^^^^^^^^
-
-If the Basic Restconf test suite fails, check that the ODL controller is
-reachable and its Restconf module has been installed.
-
-If the Neutron Reachability test fails, verify that the modules
-implementing Neutron requirements have been properly installed.
-
-If any of the other test cases fails, check that Neutron and ODL have
-been correctly configured to work together. Check Neutron configuration
-files, accounts, IP addresses etc.).
-
-
Features
--------
import os_client_config
import shade
-from tempest.lib.common.utils import data_utils
+from tempest.lib.common.utils import data_utils # pylint: disable=import-error
from xtesting.core import testcase
from functest.utils import config
mock.call().decode("utf-8"),
mock.call(['rally', 'deployment', 'check']),
mock.call().decode("utf-8")]
- mock_exec.assert_has_calls(calls)
+ mock_exec.assert_has_calls(calls, any_order=True)
@mock.patch('functest.opnfv_tests.openstack.rally.rally.os.path.exists')
@mock.patch('functest.opnfv_tests.openstack.rally.rally.os.makedirs')
'DEBUG': env.INPUTS['DEBUG'],
'DEPLOY_SCENARIO': env.INPUTS['DEPLOY_SCENARIO'],
'INSTALLER_TYPE': env.INPUTS['INSTALLER_TYPE'],
- 'SDN_CONTROLLER_IP': None,
- 'SDN_CONTROLLER_USER': 'admin',
- 'SDN_CONTROLLER_PASSWORD': 'admin',
- 'SDN_CONTROLLER_WEBPORT': '8080',
- 'SDN_CONTROLLER_RESTCONFPORT': '8181',
'BUILD_TAG': env.INPUTS['BUILD_TAG'],
'NODE_NAME': env.INPUTS['NODE_NAME'],
'POD_ARCH': None,
[testenv:perm]
basepython = python3.9
-whitelist_externals = bash
+allowlist_externals = bash
path=. -not -path './.tox/*' -not -path './.git/*' -not -path './docs/com/pres/reveal.js/*' -not -path './elements/functest/install.d/*'
commands =
bash -c "\
git+https://opendev.org/openstack/rally-openstack.git@2.3.0#egg=rally-openstack
git+https://github.com/xrally/xrally-kubernetes.git@2ffa85af2bff3438b6b23034b6ec6ee1de481090#egg=xrally-kubernetes
pylint===2.11.1
+astroid==2.8.0
flake8===4.0.1
nose===1.3.7
ruamel.yaml===0.17.17
Code Review / functest.git / commitdiff