X-Git-Url: https://gerrit.opnfv.org/gerrit/gitweb?a=blobdiff_plain;f=docs%2Ftesting%2Fdeveloper%2Fdevguide%2Findex.rst;h=d700ac1fe9743cd0cb87fa12cb37e39d54ce486a;hb=920e8d3cd4386e4d9085b5557b0c381d5f043830;hp=722d0a7d2fa0d0709bbb48e3ed86d94a054ebb4f;hpb=83a3d18e1d62f5caff94619a33a2ed4a094cf0d9;p=functest.git diff --git a/docs/testing/developer/devguide/index.rst b/docs/testing/developer/devguide/index.rst index 722d0a7d2..d700ac1fe 100644 --- a/docs/testing/developer/devguide/index.rst +++ b/docs/testing/developer/devguide/index.rst @@ -1,29 +1,34 @@ -****************************** -OPNFV FUNCTEST developer guide -****************************** +.. SPDX-License-Identifier: CC-BY-4.0 + +************************ +Functest Developer Guide +************************ .. toctree:: :numbered: :maxdepth: 2 - ============ Introduction ============ Functest is a project dealing with functional testing. -Functest produces its own internal test cases but can also be considered +The project produces its own internal test cases but can also be considered as a framework to support feature and VNF onboarding project testing. -Functest developed a test API and defined a test collection framework -that can be used by any OPNFV project. Therefore there are many ways to contribute to Functest. You can: * Develop new internal test cases * Integrate the tests from your feature project * Develop the framework to ease the integration of external test cases - * Develop the API / Test collection framework - * Develop dashboards or automatic reporting portals + +Additional tasks involving Functest but addressing all the test projects +may also be mentioned: + + * The API / Test collection framework + * The dashboards + * The automatic reporting portals + * The testcase catalog This document describes how, as a developer, you may interact with the Functest project. The first section details the main working areas of @@ -39,97 +44,125 @@ Functest developer areas Functest High level architecture ================================ -Functest is project delivering a test container dedicated to OPNFV. +Functest is a project delivering test containers dedicated to OPNFV. It includes the tools, the scripts and the test scenarios. +In Euphrates Alpine containers have been introduced in order to lighten the +container and manage testing slicing. The new containers are created according +to the different tiers: + + * functest-core: https://hub.docker.com/r/opnfv/functest-core/ + * functest-healthcheck: https://hub.docker.com/r/opnfv/functest-healthcheck/ + * functest-smoke: https://hub.docker.com/r/opnfv/functest-smoke/ + * functest-features: https://hub.docker.com/r/opnfv/functest-features/ + * functest-components: https://hub.docker.com/r/opnfv/functest-components/ + * functest-vnf: https://hub.docker.com/r/opnfv/functest-vnf/ + * functest-parser: https://hub.docker.com/r/opnfv/functest-parser/ + * functest-restapi: https://hub.docker.com/r/opnfv/functest-restapi/ + +Standalone functest dockers are maintained for Euphrates but Alpine containers +are recommended. Functest can be described as follow:: - +----------------------+ - | | - | +--------------+ | +-------------------+ - | | | | Public | | - | | Tools | +------------------+ OPNFV | - | | Scripts | | | System Under Test | - | | Scenarios | +------------------+ | - | | | | Management | | - | +--------------+ | +-------------------+ - | | - | Functest Docker | - | | - +----------------------+ + +----------------------+ + | | + | +--------------+ | +-------------------+ + | | | | Public | | + | | Tools | +------------------+ OPNFV | + | | Scripts | | | System Under Test | + | | Scenarios | | | | + | | | | | | + | +--------------+ | +-------------------+ + | | + | Functest Docker | + | | + +----------------------+ Functest internal test cases ============================ -The internal test cases in Danube are: +The internal test cases in Euphrates are: + - * healthcheck - * connection_check * api_check + * connection_check + * snaps_health_check * vping_ssh * vping_userdata * odl - * snaps_smoke - * tempest_smoke_serial + * rally_full * rally_sanity + * tempest_smoke_serial * tempest_full_parallel - * rally_full * cloudify_ims -By internal, we mean that this particular test cases have been -developped and/or integrated by functest contributors and the associated -code is hosted in the Functest repository. -An internal case can be fully developped or a simple integration of -upstream suites (e.g. Tempest/Rally developped in OpenStack are just -integrated in Functest). +By internal, we mean that this particular test cases have been developed and/or +integrated by functest contributors and the associated code is hosted in the +Functest repository. +An internal case can be fully developed or a simple integration of +upstream suites (e.g. Tempest/Rally developed in OpenStack, or odl suites are +just integrated in Functest). + The structure of this repository is detailed in `[1]`_. The main internal test cases are in the opnfv_tests subfolder of the -repository, the internal test cases are: +repository, the internal test cases can be grouped by domain: - * sdn: odl, onos - * openstack: healthcheck, vping_ssh, vping_userdata, tempest_*, rally_*, connection_check, api_check, snaps_smoke + * sdn: odl, odl_fds + * openstack: api_check, connection_check, snaps_health_check, vping_ssh, + vping_userdata, tempest_*, rally_* * vnf: cloudify_ims -If you want to create a new test case you will have to create a new -folder under the testcases directory. +If you want to create a new test case you will have to create a new folder +under the testcases directory (See next section for details). Functest external test cases ============================ -The external test cases are inherited from other OPNFV projects, -especially the feature projects. +The external test cases are inherited from other OPNFV projects, especially the +feature projects. The external test cases are: - * promise - * doctor - * onos + * barometer * bgpvpn + * doctor + * domino + * fds + * parser + * promise + * refstack_defcore + * snaps_smoke + * functest-odl-sfc + * orchestra_clearwaterims + * orchestra_openims + * vyos_vrouter + * juju_vepc + +External test cases integrated in previous versions but not released in +Euphrates: + * copper + * moon + * netready * security_scan - * sfc-odl - * sfc-onos - * parser - * domino - * multisite - * opera_ims - * orchestra_ims -The code to run these test cases may be directly in the repository of -the project. We have also a **features** sub directory under opnfv_tests -directory that may be used (it can be usefull if you want to reuse -Functest library). +The code to run these test cases is hosted in the repository of the project. +Please note that orchestra test cases are hosted in Functest repository and not +in orchestra repository. Vyos_vrouter and juju_vepc code is also hosted in +functest as there are no dedicated projects. + Functest framework ================== -Functest can be considered as a framework. -Functest is release as a docker file, including tools, scripts and a CLI -to prepare the environement and run tests. -It simplifies the integration of external test suites in CI pipeline -and provide commodity tools to collect and display results. +Functest is a framework. + +Historically Functest is released as a docker file, including tools, scripts +and a CLI to prepare the environment and run tests. +It simplifies the integration of external test suites in CI pipeline and +provide commodity tools to collect and display results. -Since Colorado, test categories also known as tiers have been created to -group similar tests, provide consistant sub-lists and at the end optimize +Since Colorado, test categories also known as **tiers** have been created to +group similar tests, provide consistent sub-lists and at the end optimize test duration for CI (see How To section). The definition of the tiers has been agreed by the testing working group. @@ -139,806 +172,123 @@ The tiers are: * smoke * features * components - * performance * vnf - * stress Functest abstraction classes ============================ -In order to harmonize test integration, 3 abstraction classes have been -introduced in Danube: - - * testcase_base: base for any test case - * feature_base: abstraction for feature project - * vnf_base: abstraction for vnf onboarding - -The goal is to unify the way to run test from Functest. - -feature_base and vnf_base inherit from testcase_base. - - +-----------------------------------------+ - | | - | Testcase_base | - | | - | - init() | - | - run() | - | - publish_report() | - | - check_criteria() | - | | - +-----------------------------------------+ - | | - V V - +--------------------+ +--------------------------+ - | | | | - | feature_base | | vnf_base | - | | | | - | - prepare() | | - prepare() | - | - post() | | - deploy_orchestrator() | - | - parse_results() | | - deploy_vnf() | - | | | - test_vnf() | - | | | - clean() | - | | | - execute() | - | | | | - +--------------------+ +--------------------------+ +In order to harmonize test integration, abstraction classes have been +introduced: + + * testcase: base for any test case + * unit: run unit tests as test case + * feature: abstraction for feature project + * vnf: abstraction for vnf onboarding + +The goal is to unify the way to run tests in Functest. + +Feature, unit and vnf_base inherit from testcase:: + + +----------------------------------------------------------------+ + | | + | TestCase | + | | + | - init() | + | - run() | + | - push_to_db() | + | - is_successful() | + | | + +----------------------------------------------------------------+ + | | | | + V V V V + +--------------------+ +---------+ +------------------------+ +-----------------+ + | | | | | | | | + | feature | | unit | | vnf | | robotframework | + | | | | | | | | + | | | | |- prepare() | | | + | - execute() | | | |- deploy_orchestrator() | | | + | BashFeature class | | | |- deploy_vnf() | | | + | | | | |- test_vnf() | | | + | | | | |- clean() | | | + +--------------------+ +---------+ +------------------------+ +-----------------+ + + +Testcase +-------- +.. raw:: html + :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.testcase.html + +Feature +------- +.. raw:: html + :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.feature.html + +Unit +---- +.. raw:: html + :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.unit.html + +VNF +--- +.. raw:: html + :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.vnf.html + +Robotframework +-------------- +.. raw:: html + :url: http://artifacts.opnfv.org/functest/docs/apidoc/functest.core.robotframework.html + + +see `[5]`_ to get code samples Functest util classes ===================== -In order to simplify the creation of test cases, Functest develops some -functions that can be used by any feature or internal test cases. +In order to simplify the creation of test cases, Functest develops also some +functions that are used by internal test cases. Several features are supported such as logger, configuration management and -Openstack capabilities (snapshot, clean, tacker,..). -These functions can be found under /functest/utils and can be described as -follows: - -functest/utils/ -|-- config.py -|-- constants.py -|-- env.py -|-- functest_logger.py -|-- functest_utils.py -|-- openstack_clean.py -|-- openstack_snapshot.py -|-- openstack_tacker.py -`-- openstack_utils.py - -Note that for Openstack, keystone v3 is now deployed by default by compass, -fuel and joid in Danube. All installers still support keysone v2 (deprecated in -next version). - -Test collection framework -========================= - -The OPNFV testing group created a test collection database to collect -the test results from CI: - - - http://testresults.opnfv.org/test/swagger/spec.html - - Authentication: opnfv/api@opnfv - -Any test project running on any lab integrated in CI can push the -results to this database. -This database can be used to see the evolution of the tests and compare -the results versus the installers, the scenarios or the labs. - - -Overall Architecture --------------------- -The Test result management can be summarized as follows:: - - +-------------+ +-------------+ +-------------+ - | | | | | | - | Test | | Test | | Test | - | Project #1 | | Project #2 | | Project #N | - | | | | | | - +-------------+ +-------------+ +-------------+ - | | | - V V V - +-----------------------------------------+ - | | - | Test Rest API front end | - | http://testresults.opnfv.org/test | - | | - +-----------------------------------------+ - A | - | V - | +-------------------------+ - | | | - | | Test Results DB | - | | Mongo DB | - | | | - | +-------------------------+ - | - | - +----------------------+ - | | - | test Dashboard | - | | - +----------------------+ - -Test API description --------------------- -The Test API is used to declare pods, projects, test cases and test -results. Pods are the pods used to run the tests. -The results pushed in the database are related to pods, projects and -cases. If you try to push results of test done on non referenced pod, -the API will return an error message. - -An additional method dashboard has been added to post-process -the raw results in release Brahmaputra (deprecated in Colorado). - -The data model is very basic, 4 objects are created: - - * Pods - * Projects - * Testcases - * Results - -Pods:: - - { - "id": , - "details": , - "creation_date": "YYYY-MM-DD HH:MM:SS", - "name": , - "mode": , - "role": - }, - -Projects:: - - { - "id": , - "name": , - "creation_date": "YYYY-MM-DD HH:MM:SS", - "description": - }, - -Testcases:: - - { - "id": , - "name":, - "project_name":, - "creation_date": "YYYY-MM-DD HH:MM:SS", - "description": , - "url": - }, - -Results:: - - { - "_id": , - "case_name": , - "project_name": , - "pod_name": , - "installer": , - "version": , - "start_date": "YYYY-MM-DD HH:MM:SS", - "stop_date": "YYYY-MM-DD HH:MM:SS", - "build_tag": , - "scenario": , - "criteria": , - "trust_indicator": { - "current": 0, - "histories": [] - } - } - -The API can described as follows. For detailed information, please go to - - http://testresults.opnfv.org/test/swagger/spec.html - - Authentication: opnfv/api@opnfv - -Version: - - +--------+--------------------------+-----------------------------------------+ - | Method | Path | Description | - +========+==========================+=========================================+ - | GET | /versions | Get all supported API versions | - +--------+--------------------------+-----------------------------------------+ - - -Pods: - - +--------+----------------------------+-----------------------------------------+ - | Method | Path | Description | - +========+============================+=========================================+ - | GET | /api/v1/pods | Get the list of declared Labs (PODs) | - +--------+----------------------------+-----------------------------------------+ - | POST | /api/v1/pods | Declare a new POD | - | | | Content-Type: application/json | - | | | { | - | | | "name": "pod_foo", | - | | | "mode": "metal", | - | | | "role": "ci-pod", | - | | | "details": "it is a ci pod" | - | | | } | - +--------+----------------------------+-----------------------------------------+ - | GET | /api/v1/pods/{pod_name} | Get a declared POD | - +--------+----------------------------+-----------------------------------------+ - -Projects: - - +--------+----------------------------+-----------------------------------------+ - | Method | Path | Description | - +========+============================+=========================================+ - | GET | /api/v1/projects | Get the list of declared projects | - +--------+----------------------------+-----------------------------------------+ - | POST | /api/v1/projects | Declare a new test project | - | | | Content-Type: application/json | - | | | { | - | | | "name": "project_foo", | - | | | "description": "whatever you want" | - | | | } | - +--------+----------------------------+-----------------------------------------+ - | DELETE | /api/v1/projects/{project} | Delete a test project | - +--------+----------------------------+-----------------------------------------+ - | GET | /api/v1/projects/{project} | Get details on a {project} | - | | | | - +--------+----------------------------+-----------------------------------------+ - | PUT | /api/v1/projects/{project} | Update a test project | - | | | | - | | | Content-Type: application/json | - | | | { | - | | | | - | | | } | - +--------+----------------------------+-----------------------------------------+ - - -Testcases: - - +--------+----------------------------+-----------------------------------------+ - | Method | Path | Description | - +========+============================+=========================================+ - | GET | /api/v1/projects/{project}/| Get the list of testcases of {project} | - | | cases | | - +--------+----------------------------+-----------------------------------------+ - | POST | /api/v1/projects/{project}/| Add a new test case to {project} | - | | cases | Content-Type: application/json | - | | | { | - | | | "name": "case_foo", | - | | | "description": "whatever you want" | - | | | "url": "whatever you want" | - | | | } | - +--------+----------------------------+-----------------------------------------+ - | DELETE | /api/v1/projects/{project}/| Delete a test case | - | | cases/{case} | | - +--------+----------------------------+-----------------------------------------+ - | GET | /api/v1/projects/{project}/| Get a declared test case | - | | cases/{case} | | - +--------+----------------------------+-----------------------------------------+ - | PUT | /api/v1/projects/{project}?| Modify a test case of {project} | - | | cases/{case} | | - | | | Content-Type: application/json | - | | | { | - | | | | - | | | } | - +--------+----------------------------+-----------------------------------------+ - -Results: - - +--------+----------------------------+------------------------------------------+ - | Method | Path | Description | - +========+============================+==========================================+ - | GET | /api/v1/results | Get all the test results | - +--------+----------------------------+------------------------------------------+ - | POST | /api/v1/results | Add a new test results | - | | | Content-Type: application/json | - | | | { | - | | | "project_name": "project_foo", | - | | | "scenario": "odl-l2", | - | | | "stop_date": "2016-05-28T14:42:58.384Z", | - | | | "trust_indicator": 0.5, | - | | | "case_name": "vPing", | - | | | "build_tag": "", | - | | | "version": "Colorado", | - | | | "pod_name": "pod_foo", | - | | | "criteria": "PASS", | - | | | "installer": "fuel", | - | | | "start_date": "2016-05-28T14:41:58.384Z",| - | | | "details": | - | | | } | - +--------+----------------------------+------------------------------------------+ - | GET | /api/v1/results? | Get the test results of {case} | - | | case={case} | | - +--------+----------------------------+------------------------------------------+ - | GET | /api/v1/results? | Get the test results of build_tag | - | | build_tag={tag} | {tag}. | - +--------+----------------------------+------------------------------------------+ - | GET | /api/v1/results? | Get last {N} records of test results | - | | last={N} | | - +--------+----------------------------+------------------------------------------+ - | GET | /api/v1/results? | Get the test results of scenario | - | | scenario={scenario} | {scenario}. | - +--------+----------------------------+------------------------------------------+ - | GET | /api/v1/results? | Get the test results of trust_indicator | - | | trust_indicator={ind} | {ind}. | - +--------+----------------------------+------------------------------------------+ - | GET | /api/v1/results? | Get the test results of last days | - | | period={period} | {period}. | - +--------+----------------------------+------------------------------------------+ - | GET | /api/v1/results? | Get the test results of {project} | - | | project={project} | | - +--------+----------------------------+------------------------------------------+ - | GET | /api/v1/results? | Get the test results of version | - | | version={version} | {version}. | - +--------+----------------------------+------------------------------------------+ - | GET | /api/v1/results? | Get the test results of criteria | - | | criteria={criteria} | {criteria}. | - +--------+----------------------------+------------------------------------------+ - | GET | /api/v1/results? | get the results on pod {pod} | - | | pod={pod} | | - +--------+----------------------------+------------------------------------------+ - | GET | /api/v1/results? | Get the test results of installer {inst} | - | | installer={inst} | | - +--------+----------------------------+------------------------------------------+ - | GET | /api/v1/results? | Get the results according to combined | - | | | query conditions supported above | - +--------+----------------------------+------------------------------------------+ - | GET | /api/v1/results/{result_id}| Get the test result by result_id | - +--------+----------------------------+------------------------------------------+ - -Scenarios: - - +--------+----------------------------+-----------------------------------------+ - | Method | Path | Description | - +========+============================+=========================================+ - | GET | /api/v1/scenarios | Get the list of declared scenarios | - +--------+----------------------------+-----------------------------------------+ - | POST | /api/v1/scenario | Declare a new scenario | - +--------+----------------------------+-----------------------------------------+ - | GET | /api/v1/scenario? | Get a declared scenario | - | | | | - +--------+----------------------------+-----------------------------------------+ - - -The code of the API is hosted in the releng repository `[6]`_. -The static documentation of the API can be found at `[17]`_. -The test API has been dockerized and may be installed locally in your -lab. See `[15]`_ for details. - -The deployment of the test API has been automated. -A jenkins job manages: - * the unit tests of the test api - * the creation of a new docker file - * the deployment of the new test api - * the archive of the old test api - * the backup of the Mongo DB - -Test API Authorization -~~~~~~~~~~~~~~~~~~~~~~ - -PUT/DELETE/POST operations of the testapi now require token based authorization. The token needs -to be added in the request using a header 'X-Auth-Token' for access to the database. - -e.g:: - headers['X-Auth-Token'] - -The value of the header i.e the token can be accessed in the jenkins environment variable -*TestApiToken*. The token value is added as a masked password. - -.. code-block:: python - - headers['X-Auth-Token'] = os.environ.get('TestApiToken') - -The above example is in Python. Token based authentication has been added so that only ci pods -jenkins job can have access to the database. - -Please note that currently token authorization is implemented but is not yet enabled. - - Automatic reporting - =================== - - An automatic reporting page has been created in order to provide a - consistant view of the scenarios. - In this page, each scenario is evaluated according to test criteria. - The code for the automatic reporting is available at `[8]`_. - - The results are collected from the centralized database every day and, - per scenario. A score is calculated based on the results from the last - 10 days. This score is the addition of single test scores. Each test - case has a success criteria reflected in the criteria field from the - results. - - Considering an instance of a scenario os-odl_l2-nofeature-ha, the - scoring is the addition of the scores of all the runnable tests from the - categories (tiers healthcheck, smoke and features) - corresponding to this scenario. - - - +---------------------+---------+---------+---------+---------+ - | Test | Apex | Compass | Fuel | Joid | - +=====================+=========+=========+=========+=========+ - | vPing_ssh | X | X | X | X | - +---------------------+---------+---------+---------+---------+ - | vPing_userdata | X | X | X | X | - +---------------------+---------+---------+---------+---------+ - | tempest_smoke_serial| X | X | X | X | - +---------------------+---------+---------+---------+---------+ - | rally_sanity | X | X | X | X | - +---------------------+---------+---------+---------+---------+ - | odl | X | X | X | X | - +---------------------+---------+---------+---------+---------+ - | promise | | | X | X | - +---------------------+---------+---------+---------+---------+ - | doctor | X | | X | | - +---------------------+---------+---------+---------+---------+ - | security_scan | X | | | | - +---------------------+---------+---------+---------+---------+ - | parser | | | X | | - +---------------------+---------+---------+---------+---------+ - | copper | X | | | X | - +---------------------+---------+---------+---------+---------+ - - All the testcases listed in the table are runnable on - os-odl_l2-nofeature scenarios. - If no result is available or if all the results are failed, the test - case get 0 point. - If it was succesfull at least once but not anymore during the 4 runs, - the case get 1 point (it worked once). - If at least 3 of the last 4 runs were successful, the case get 2 points. - If the last 4 runs of the test are successful, the test get 3 points. - - In the example above, the target score for fuel/os-odl_l2-nofeature-ha - is 3x6 = 18 points. - - The scenario is validated per installer when we got 3 points for all - individual test cases (e.g 18/18). - Please note that complex or long duration tests are not considered for - the scoring. The success criteria are not always easy to define and may - require specific hardware configuration. These results however provide - a good level of trust on the scenario. - - A web page is automatically generated every day to display the status. - This page can be found at `[9]`_. For the status, click on Status menu, - you may also get feedback for vims and tempest_smoke_serial test cases. - - Any validated scenario is stored in a local file on the web server. In - fact as we are using a sliding windows to get results, it may happen - that a successful scenarios is no more run (because considered as - stable) and then the number of iterations (4 needed) would not be - sufficient to get the green status. - - Please note that other test cases (e.g. sfc_odl, bgpvpn) need also - ODL configuration addons and as a consequence specific scenario. - There are not considered as runnable on the generic odl_l2 scenario. - -Dashboard -========= - -Dashboard is used to provide a consistant view of the results collected -in CI. -The results showed on the dashboard are post processed from the Database, -which only contains raw results. +Openstack capabilities (tacker,..). +These functions can be found under /functest/utils and can be described +as follows:: -In Brahmaputra, we created a basic dashboard. -Since Colorado, it was decided to adopt ELK framework. Mongo DB results -are extracted to feed Elasticsearch database (`[7]`_). + functest/utils/ + |-- config.py + |-- constants.py + |-- decorators.py + |-- env.py + |-- functest_utils.py + |-- openstack_tacker.py + `-- openstack_utils.py -A script was developed to build elasticsearch data set. This -script can be found in `[16]`_. +It is recommended to use the SNAPS-OO library for deploying OpenStack +instances. SNAPS `[4]`_ is an OPNFV project providing OpenStack utils. -For next versions, it was decided to integrated bitergia dashboard. -Bitergia already provides a dashboard for code and infrastructure. -A new Test tab will be added. The dataset will be built by consuming -the test API. - -======= -How TOs +TestAPI ======= +Functest is using the Test collection framework and the TestAPI developed by +the OPNFV community. See `[6]`_ for details. -How Functest works? -=================== - -The installation and configuration of the Functest docker image is -described in `[1]`_. - -The procedure to start tests is described in `[2]`_ - - -How can I contribute to Functest? -================================= - -If you are already a contributor of any OPNFV project, you can -contribute to functest. If you are totally new to OPNFV, you must first -create your Linux Foundation account, then contact us in order to -declare you in the repository database. - -We distinguish 2 levels of contributors: - - * the standard contributor can push patch and vote +1/0/-1 on any Functest patch - * The commitor can vote -2/-1/0/+1/+2 and merge - -Functest commitors are promoted by the Functest contributors. - - -Where can I find some help to start? -==================================== - -This guide is made for you. You can also have a look at the project wiki -page `[10]`_. -There are references on documentation, video tutorials, tips... - -You can also directly contact us by mail with [Functest] prefix in the -title at opnfv-tech-discuss@lists.opnfv.org or on the IRC chan -#opnfv-functest. - - -What kind of testing do you do in Functest? -=========================================== - -Functest is focusing on Functional testing. The results must be PASS or -FAIL. We do not deal with performance and/or qualification tests. -We consider OPNFV as a black box and execute our tests from the jumphost -according to Pharos reference technical architecture. - -Upstream test suites are integrated (Rally/Tempest/ODL/ONOS,...). -If needed Functest may bootstrap temporarily testing activities if they -are identified but not covered yet by an existing testing project (e.g -security_scan before the creation of the security repository) - - -How test constraints are defined? -================================= - -Test constraints are defined according to 2 paramaters: - - * The scenario (DEPLOY_SCENARIO env variable) - * The installer (INSTALLER_TYPE env variable) - -A scenario is a formal description of the system under test. -The rules to define a scenario are described in `[4]`_ - -These 2 constraints are considered to determinate if the test is runnable -or not (e.g. no need to run onos suite on odl scenario). - -In the test declaration for CI, the test owner shall indicate these 2 -constraints. The file testcases.yaml `[5]`_ must be patched in git to -include new test cases. A more elaborated system based on template is -planned for next releases - -For each dependency, it is possible to define a regex:: - - name: promise - criteria: 'success_rate == 100%' - description: >- - Test suite from Promise project. - dependencies: - installer: '(fuel)|(joid)' - scenario: '' - -In the example above, it means that promise test will be runnable only -with joid or fuel installers on any scenario. - -The vims criteria means any installer and exclude onos and odl with -bgpvpn scenarios:: - - name: vims - criteria: 'status == "PASS"' - description: >- - This test case deploys an OpenSource vIMS solution from Clearwater - using the Cloudify orchestrator. It also runs some signaling traffic. - dependencies: - installer: '' - scenario: '(ocl)|(nosdn)|^(os-odl)((?!bgpvpn).)*$' - - -How to write and check constaint regex? -======================================= - -Regex are standard regex. You can have a look at `[11]`_ - -You can also easily test your regex via an online regex checker such as `[12]`_. -Put your scenario in the TEST STRING window (e.g. os-odl_l3-ovs-ha), put -your regex in the REGULAR EXPRESSION window, then you can test your rule -. - - -How to know which test I can run? -================================= - -You can use the API `[13]`_. The static declaration is in git `[5]`_ - -If you are in a Functest docker container (assuming that the -environement has been prepared): just use the CLI. - -You can get the list per Test cases or by Tier:: - - # functest testcase list - healthcheck - vping_ssh - vping_userdata - tempest_smoke_serial - rally_sanity - odl - doctor - security_scan - tempest_full_parallel - rally_full - vims - # functest tier list - - 0. healthcheck: - ['healthcheck'] - - 1. smoke: - ['vping_ssh', 'vping_userdata', 'tempest_smoke_serial', 'rally_sanity'] - - 2. sdn_suites: - ['odl'] - - 3. features: - ['doctor', 'security_scan'] - - 4. openstack: - ['tempest_full_parallel', 'rally_full'] - - 5. vnf: - ['vims'] - - -How to manually start Functest tests? -===================================== - -Assuming that you are connected on the jumphost and that the system is -"Pharos compliant", i.e the technical architecture is compatible with -the one defined in the Pharos project:: - - # docker pull opnfv/functest:latest - # envs="-e INSTALLER_TYPE=fuel -e INSTALLER_IP=10.20.0.2 -e DEPLOY_SCENARIO=os-odl_l2-nofeature-ha -e CI_DEBUG=true" - # sudo docker run --privileged=true -id ${envs} opnfv/functest:latest /bin/bash - - -Then you must connect to the docker container and source the -credentials:: - - # docker ps (copy the id) - # docker exec -ti bash - # source $creds - - -You must first check if the environment is ready:: - - # functest env status - Functest environment ready to run tests. - - -If not ready, prepare the env by launching:: - - # functest env prepare - Functest environment ready to run tests. - -Once the Functest env is ready, you can use the CLI to start tests. - -You can run test cases per test case or per tier: - # functest testcase run or # functest tier run - - -e.g:: - - # functest testcase run tempest_smoke_serial - # functest tier run features - - -If you want to run all the tests you can type:: - - # functest testcase run all - - -If you want to run all the tiers (same at the end that running all the -test cases) you can type:: - - # functest tier run all - - -How to declare my tests in Functest? -==================================== - -If you want to add new internal test cases, you can submit patch under -the testcases directory of Functest repository. - -For feature test integration, the code can be kept into your own -repository. The Functest files to be modified are: - - * functest/docker/Dockerfile: get your code in Functest container - * functest/ci/testcases.yaml: reference your test and its associated constraints - - -Dockerfile ----------- - -This file lists the repositories (internal or external) to be cloned in -the Functest container. You can also add external packages:: - - RUN git clone https://gerrit.opnfv.org/gerrit/ ${REPOS_DIR}/ - -testcases.yaml --------------- - -All the test cases that must be run from CI / CLI must be declared in -ci/testcases.yaml. - -This file is used to get the constraints related to the test:: - - name: - criteria: 'PASS', 'rate > 90%' - description: >- - - dependencies: - installer: regex related to installer e.g. 'fuel', '(apex)||(joid)' - scenario: regex related to the scenario e.g. 'ovs*no-ha' - - -You must declare your test case in one of the category (tier). - -If you are integrating test suites from a feature project, the default -category is **features**. - - -How to select my list of tests for CI? -====================================== - -Functest can be run automatically from CI, a jenkins job is usually -called after an OPNFV fresh installation. -By default we try to run all the possible tests (see `[14]` called from -Functest jenkins job):: - - cmd="python ${FUNCTEST_REPO_DIR}/ci/run_tests.py -t all ${flags}" - - -Each case can be configured as daily and/or weekly task. -Weekly tasks are used for long duration or experimental tests. -Daily tasks correspond to the minimum set of test suites to validate a scenario. - -When executing run_tests.py, a check based on the jenkins build tag will -be considered to detect whether it is a daily and/or a weekly test. - -in your CI you can customize the list of test you want to run by case or -by tier, just change the line:: - - cmd="python ${FUNCTEST_REPO_DIR}/ci/run_tests.py -t ${flags}" - -e.g.:: - - cmd="python ${FUNCTEST_REPO_DIR}/ci/run_tests.py -t healthcheck,smoke ${flags}" - -This command will run all the test cases of the first 2 tiers, i.e. -healthcheck, connection_check, api_check, vping_ssh, vping_userdata, -snaps_somke, tempest_smoke_serial and rally_sanity. +Reporting +========= +A web page is automatically generated every day to display the status based on +jinja2 templates `[3]`_. -How to push your results into the Test Database -=============================================== -The test database is used to collect test results. By default it is -enabled only for CI tests from Production CI pods. +Dashboard +========= -The architecture and associated API is described in previous chapter. -If you want to push your results from CI, you just have to call the API -at the end of your script. +Additional dashboarding is managed at the testing group level, see `[7]`_ for +details. -You can also reuse a python function defined in functest_utils.py:: - def push_results_to_db(db_url, case_name, logger, pod_name,version, payload): - """ - POST results to the Result target DB - """ - url = db_url + "/results" - installer = get_installer_type(logger) - params = {"project_name": "functest", "case_name": case_name, - "pod_name": pod_name, "installer": installer, - "version": version, "details": payload} +======= +How TOs +======= - headers = {'Content-Type': 'application/json'} - try: - r = requests.post(url, data=json.dumps(params), headers=headers) - if logger: - logger.debug(r) - return True - except Exception, e: - print "Error [push_results_to_db('%s', '%s', '%s', '%s', '%s')]:" \ - % (db_url, case_name, pod_name, version, payload), e - return False +See How to section on Functest wiki `[8]`_ ========== @@ -949,44 +299,16 @@ _`[1]`: http://artifacts.opnfv.org/functest/docs/configguide/index.html Functest _`[2]`: http://artifacts.opnfv.org/functest/docs/userguide/index.html functest user guide -_`[3]`: https://wiki.opnfv.org/opnfv_test_dashboard Brahmaputra dashboard - -_`[4]`: https://wiki.opnfv.org/display/INF/CI+Scenario+Naming - -_`[5]`: https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml - -_`[6]`: https://git.opnfv.org/cgit/releng/tree/utils/test/result_collection_api - -_`[7]`: https://git.opnfv.org/cgit/releng/tree/utils/test/scripts - -_`[8]`: https://git.opnfv.org/cgit/releng/tree/utils/test/reporting/functest +_`[3]`: https://git.opnfv.org/cgit/releng/tree/utils/test/reporting -_`[9]`: http://testresults.opnfv.org/reporting/ +_`[4]`: https://git.opnfv.org/snaps/ -_`[10]`: https://wiki.opnfv.org/opnfv_functional_testing +_`[5]` : http://testresults.opnfv.org/functest/framework/index.html -_`[11]`: https://docs.python.org/2/howto/regex.html +_`[6]`: http://docs.opnfv.org/en/latest/testing/testing-dev.html -_`[12]`: https://regex101.com/ +_`[7]`: https://opnfv.biterg.io/goto/283dba93ca18e95964f852c63af1d1ba -_`[13]`: http://testresults.opnfv.org/test/api/v1/projects/functest/cases - -_`[14]`: https://git.opnfv.org/cgit/releng/tree/jjb/functest/functest-daily.sh - -_`[15]`: https://git.opnfv.org/cgit/releng/tree/utils/test/result_collection_api/README.rst - -_`[16]`: https://git.opnfv.org/cgit/releng/tree/utils/test/scripts/mongo_to_elasticsearch.py - -_`[17]`: http://artifacts.opnfv.org/releng/docs/testapi.html - -OPNFV main site: http://www.opnfv.org - -OPNFV functional test page: https://wiki.opnfv.org/opnfv_functional_testing +_`[8]`: https://wiki.opnfv.org/pages/viewpage.action?pageId=7768932 IRC support chan: #opnfv-functest - -_`OpenRC`: http://docs.openstack.org/user-guide/common/cli_set_environment_variables_using_openstack_rc.html - -_`Rally installation procedure`: https://rally.readthedocs.org/en/latest/tutorial/step_0_installation.html - -_`config_functest.yaml` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.yaml