-You must declare your test case in one of the category (tier).
-
-If you are integrating test suites from a feature project, the default
-category is **features**.
-
-
-How to select my list of tests for CI?
-======================================
-
-Functest can be run automatically from CI, a jenkins job is usually
-called after an OPNFV fresh installation.
-By default we try to run all the possible tests (see `[14]` called from
-Functest jenkins job)::
-
- cmd="python ${FUNCTEST_REPO_DIR}/ci/run_tests.py -t all ${flags}"
-
-
-Each case can be configured as daily and/or weekly task.
-Weekly tasks are used for long duration or experimental tests.
-Daily tasks correspond to the minimum set of test suites to validate a scenario.
-
-When executing run_tests.py, a check based on the jenkins build tag will
-be considered to detect whether it is a daily and/or a weekly test.
-
-in your CI you can customize the list of test you want to run by case or
-by tier, just change the line::
-
- cmd="python ${FUNCTEST_REPO_DIR}/ci/run_tests.py -t <whatever you want> ${flags}"
-
-e.g.::
-
- cmd="python ${FUNCTEST_REPO_DIR}/ci/run_tests.py -t healthcheck,smoke ${flags}"
-
-This command will run all the test cases of the first 2 tiers, i.e.
-healthcheck, connection_check, api_check, vping_ssh, vping_userdata,
-snaps_smoke, tempest_smoke_serial and rally_sanity.
-
-
-How to push your results into the Test Database
-===============================================
-
-The test database is used to collect test results. By default it is
-enabled only for CI tests from Production CI pods.
-
-The architecture and associated API is described in previous chapter.
-If you want to push your results from CI, you just have to call the API
-at the end of your script.
-
-You can also reuse a python function defined in functest_utils.py::
-
- def push_results_to_db(db_url, case_name, logger, pod_name,version, payload):
- """
- POST results to the Result target DB
- """
- url = db_url + "/results"
- installer = get_installer_type(logger)
- params = {"project_name": "functest", "case_name": case_name,
- "pod_name": pod_name, "installer": installer,
- "version": version, "details": payload}
-
- headers = {'Content-Type': 'application/json'}
- try:
- r = requests.post(url, data=json.dumps(params), headers=headers)
- if logger:
- logger.debug(r)
- return True
- except Exception, e:
- print "Error [push_results_to_db('%s', '%s', '%s', '%s', '%s')]:" \
- % (db_url, case_name, pod_name, version, payload), e
- return False
-
-
-Where can I find the documentation on the test API?
-===================================================
-
-http://artifacts.opnfv.org/releng/docs/testapi.html
-
-
-How to exclude Tempest case from default Tempest smoke suite?
-=============================================================
-
-Tempest default smoke suite deals with 165 test cases.
-Since Colorado the success criteria is 100%, i.e. if 1 test is failed the
-success criteria is not matched for the scenario.
-
-It is necessary to exclude some test cases that are expected to fail due to
-known upstream bugs (see release notes).
-
-A file has been created for such operation: https://git.opnfv.org/cgit/functest/tree/functest/opnfv_tests/openstack/tempest/custom_tests/blacklist.txt.
-
-It can be described as follows::
-
- -
- scenarios:
- - os-odl_l2-bgpvpn-ha
- - os-odl_l2-bgpvpn-noha
- installers:
- - fuel
- - apex
- tests:
- - tempest.api.compute.servers.test_create_server.ServersTestJSON.test_list_servers
- - tempest.api.compute.servers.test_create_server.ServersTestJSON.test_verify_server_details
- - tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_list_servers
- - tempest.api.compute.servers.test_create_server.ServersTestManualDisk.test_verify_server_details
- - tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_reboot_server_hard
- - tempest.scenario.test_network_basic_ops.TestNetworkBasicOps.test_network_basic_ops
- - tempest.scenario.test_server_basic_ops.TestServerBasicOps.test_server_basic_ops
- - tempest.scenario.test_volume_boot_pattern.TestVolumeBootPattern.test_volume_boot_pattern
- - tempest.scenario.test_volume_boot_pattern.TestVolumeBootPatternV2.test_volume_boot_pattern
-
-Please note that each exclusion must be justified. the goal is not to exclude
-test cases because they do not pass. Several scenarios reached the 100% criteria.
-So it is expected in the patch submitted to exclude the cases to indicate the
-reasons of the exclusion.
-
-
-How do I know the Functest status of a scenario?
-================================================
-
-A Functest automatic reporting page is generated daily.
-This page is dynamically created through a cron job and is based on the results
-stored in the Test DB.
-You can access this reporting page: http://testresults.opnfv.org/reporting
-
-See https://wiki.opnfv.org/pages/viewpage.action?pageId=6828617 for details.
-
-
-I have tests, to which category should I declare them?
-======================================================
-
-CATEGORIES/TIERS description:
-
-+----------------+-------------------------------------------------------------+
-| healthcheck | Simple OpenStack healthcheck tests case that validates the |
-| | basic operations in OpenStack |
-+----------------+-------------------------------------------------------------+
-| Smoke | Set of smoke test cases/suites to validate the most common |
-| | OpenStack and SDN Controller operations |
-+----------------+-------------------------------------------------------------+
-| Features | Test cases that validate a specific feature on top of OPNFV.|
-| | Those come from Feature projects and need a bit of support |
-| | for integration |
-+----------------+-------------------------------------------------------------+
-| Components | Advanced OpenStack tests: Full Tempest, Full Rally |
-+----------------+-------------------------------------------------------------+
-| Performance | Out of Functest Scope |
-+----------------+-------------------------------------------------------------+
-| VNF | Test cases related to deploy an open source VNF including |
-| | an orchestrator |
-+----------------+-------------------------------------------------------------+
-
-The main ambiguity could be between features and VNF.
-In fact sometimes you have to spawn VMs to demonstrate the capabilities of the
-feature you introduced.
-We recommend to declare your test in the feature category.
-
-VNF category is really dedicated to test including:
-
- * creation of resources
- * deployment of an orchestrator/VNFM
- * deployment of the VNF
- * test of the VNFM
- * free resources
-
-The goal is not to study a particular feature on the infrastructure but to have
-a whole end to end test of a VNF automatically deployed in CI.
-Moreover VNF are run in weekly jobs (one a week), feature tests are in daily
-jobs and use to get a scenario score.
-
-Where are the logs?
-===================
-
-Functest deals with internal and external testcases. Each testcase can generate
-logs.
-
-Since Colorado we introduce the possibility to push the logs to the artifact.
-A new script (https://git.opnfv.org/releng/tree/utils/push-test-logs.sh) has
-been created for CI.
-
-When called, and assuming that the POD is authorized to push the logs to
-artifacts, the script will push all the results or logs locally stored under
-/home/opnfv/functest/results/.
-
-If the POD is not connected to CI, logs are not pushed.
-But in both cases, logs are stored in /home/opnfv/functest/results in the
-container.
-Projects are encouraged to push their logs here.
-
-Since Colorado it is also easy for feature project to integrate this feature by
-adding the log file as output_file parameter when calling execute_command from
-functest_utils library
-
- ret_val = functest_utils.execute_command(cmd, output_file=log_file)
-
-
-How does Functest deal with VNF onboarding?
-===========================================
-
-VNF onboarding has been introduced in Brahmaputra through the automation of a
-clearwater vIMS deployed thanks to cloudify orchestrator.
-
-This automation has been described at OpenStack summit Barcelona:
-https://youtu.be/Jr4nG74glmY
-
-The goal of Functest consists in testing OPNFV from a functional perspective:
-the NFVI and/or the features developed in OPNFV. Feature test suites are
-provided by the feature project. Functest just simplifies the integration of
-the suite into the CI and gives a consolidated view of the tests per scenario.
-
-Functest does not develop VNFs.
-
-Functest does not test any MANO stack.
-
-OPNFV projects dealing with VNF onboarding
-------------------------------------------
-
-Testing VNF is not the main goal however it gives interesting and realistic
-feedback on OPNFV as a Telco cloud.
-
-Onboarding VNF also allows to test a full stack: orchestrator + VNF.
-
-Functest is VNF and MANO stack agnostic.
-
-An internship has been initiated to reference the Open Source VNF: Intern
-Project Open Source VNF catalog
-
-New projects dealing with orchestrators or VNFs are candidate for Danube.
-
-The 2 projects dealing with orchestration are:
-
- * orchestra (Openbaton)
- * opera (Open-O)
-
-The Models project address various goals for promoting availability and
-convergence of information and/or data models related to NFV service/VNF
-management, as being defined in standards (SDOs) and as developed in open
-source projects.
-
-Functest VNF onboarding
------------------------
-
-In order to simplify VNF onboarding a new abstraction class has been developed
-in Functest.
-
-This class is based on vnf_base and can be described as follow:
-
- +------------+ +--------------+
- | test_base |------------>| vnf_base |
- +------------+ +--------------+
- |_ prepare
- |_ deploy_orchestrator (optional)
- |_ deploy_vnf
- |_ test_vnf
- |_ clean
-
-
-Several methods are declared in vnf_base:
-
- * prepare
- * deploy_orchestrator
- * deploy_vnf
- * test_vnf
- * clean
-
-deploy_vnf and test_vnf are mandatory.
-
-prepare will create a user and a project.
-
-How to declare your orchestrator/VNF?
--------------------------------------
-1) test declaration
-
-You must declare your testcase in the file <Functest repo>/functest/ci/testcases.yaml
-
-2) configuration
-
-You can precise some configuration parameters in config_functest.yaml
-
-3) implement your test
-
-Create your own VnfOnboarding file