+.. This work is licensed under a Creative Commons Attribution 4.0 International License.
+.. SPDX-License-Identifier: CC-BY-4.0
+
=======
License
=======
-OPNFV Colorado release note for Functest Docs
-(c) by Morgan Richomme (Orange)
-
-OPNFV Colorado release note for Functest Docs
+OPNFV Euphrates release note for Functest Docs
are licensed under a Creative Commons Attribution 4.0 International License.
You should have received a copy of the license along with this.
If not, see <http://creativecommons.org/licenses/by/4.0/>.
-===========================================
-OPNFV Colorado1.0 release note for Functest
-===========================================
+=============================================
+OPNFV Euphrates 5.1 release note for Functest
+=============================================
Abstract
========
-This document describes the release note of Functest project.
-
+This document contains the release notes of the Functest project.
-Version history
-===============
-+------------+----------+------------------+------------------------+
-| **Date** | **Ver.** | **Author** | **Comment** |
-| | | | |
-+------------+----------+------------------+------------------------+
-| 2016-08-17 | 1.0.0 | Morgan Richomme | Functest for Colorado |
-| | | (Orange) | release |
-+------------+----------+------------------+------------------------+
-
-OPNFV Colorado Release
+OPNFV Euphrates Release
======================
Functest deals with functional testing of the OPNFV solution.
-It includes test cases developed within the project and test cases developed in
-other OPNFV projects and other upstream communities.
+It includes test cases developed within the project, test cases developed in
+other OPNFV projects and it also intgrates test cases from other upstream
+communities.
The internal test cases are:
- * healthcheck
- * vPing ssh
- * vPing userdata
- * Tempest Smoke Serial
- * Rally Sanity
- * ODL
- * Tempest full parallel
- * Rally full
- * vIMS
+ * connection_check
+ * api_check
+ * snaps_health_check
+ * vping_ssh
+ * vping_userdata
+ * tempest_smoke_serial
+ * refstack_defcore
+ * snaps_smoke
+ * rally_sanity
+ * odl
+ * tempest_full_parallel
+ * rally_full
+ * cloudify_ims
+ * vyos_vrouter
The OPNFV projects integrated into Functest framework for automation are:
+ * barometer
* bgpvpn
- * Copper
- * Doctor
- * Domino
- * Moon
- * Multisite
- * ONOSFW
- * ONOS-sfc
- * ODL-sfc
- * Parser
- * Promise
- * Security scan
-
-The validation of a scenario requires a subset of these tests depending
-on the installer and the scenario.
-
-The 3 last internal test cases (tempest full parallel, Rally full and
-vIMS) are not considered for scenario validation.
+ * doctor
+ * domino
+ * fds
+ * odl-sfc
+ * odl-netvirt
+ * parser
+ * promise
+ * orchestra_openims
+ * orchestra_clearwaterims
+
Release Data
============
| **Project** | functest |
| | |
+--------------------------------------+--------------------------------------+
-| **Repo/tag** | colorado.1.0 |
+| **Repo/tag** | opnfv-5.1.0 |
| | |
+--------------------------------------+--------------------------------------+
-| **Release designation** | Colorado base release |
+| **Release designation** | Euphrates 5.1 release |
| | |
+--------------------------------------+--------------------------------------+
-| **Release date** | September 22 2016 |
+| **Release date** | December 15th 2017 |
| | |
+--------------------------------------+--------------------------------------+
-| **Purpose of the delivery** | Colorado base release |
+| **Purpose of the delivery** | Euphrates second release |
| | |
+--------------------------------------+--------------------------------------+
Software
--------
- - The Functest Docker image: https://hub.docker.com/r/opnfv/functest (tag: colorado.1.0)
+ Functest Docker images:
- - The TestAPI Docker image: https://hub.docker.com/r/opnfv/testapi (tag:colorado.1.0)
+ * https://hub.docker.com/r/opnfv/functest-healthcheck
+ * https://hub.docker.com/r/opnfv/functest-smoke
+ * https://hub.docker.com/r/opnfv/functest-features
+ * https://hub.docker.com/r/opnfv/functest-components
+ * https://hub.docker.com/r/opnfv/functest-vnf
+ * https://hub.docker.com/r/opnfv/functest-parser
+ * https://hub.docker.com/r/opnfv/functest-restapi
+ TestAPI Docker image:
+
+ * https://hub.docker.com/r/opnfv/testapi
+
+Docker tag for Euphrates 5.1 release: opnfv-5.1.0
+Docker tag for Euphrates with latest bugfixes: euphrates
Documents
---------
- - Installation/configuration guide: http://artifacts.opnfv.org/functest/colorado/docs/configguide/index.html
+ - Installation/configuration guide: http://docs.opnfv.org/en/stable-euphrates/submodules/functest/docs/testing/user/configguide/index.html
+
+ - User Guide: http://docs.opnfv.org/en/stable-euphrates/submodules/functest/docs/testing/user/userguide/index.html
- - User Guide: http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html
+ - Developer Guide: http://docs.opnfv.org/en/stable-euphrates/submodules/functest/docs/testing/developer/devguide/index.html
- - Developer Guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html
+ - API Docs: http://artifacts.opnfv.org/functest/docs/index.html
+
+ - Functest Framework presentation: http://testresults.opnfv.org/functest/framework/index.html
Version change
==============
-Feature evolution
------------------
+Functest now delivers light-weigth Docker images based on Alpine 3.7. The test cases are grouped into several categories
+or tiers and must be run from the corresponding container. For example, to run the test case healthcheck, the image
+opnfv/functest-healthcheck shall be used. The tiers and the tests within them are explained in detail in the User Guide.
+
+The former Ubuntu image is not longer maintained.
- - refactoring of ODL functional tests (with upstream modifications)
+The Parser test case has its own dedicated Docker image since it requires libraries released for OpenStack Pike and
+Euphrates is based on Ocata.
- - refactoring of TestAPI (update, swagger documentation, dockerization)
+The Docker images do not contain OS images (Cirros, Ubuntu, Centos, ..) anymore. A script has been created under the
+ci directory (download_images.sh) which contains all the needed images for all the tests. This file can be modified by
+the user since not all the images might be used. It must be executed before starting Functest and attach the needed
+images as a Docker volume. See Configuration Guide for more information.
- - jenkins logs improvement
+The requirements have been split into 3 files:
+ * requirements.txt : lists all abstract dependencies of the OPNFV packages
+ * test-requirements.txt : lists all abstract dependencies required by Functest Unit Tests
+ * upper-constraints.txt : lists all concrete upstream dependencies required by Functest Docker container
- - update integration of Doctor, Promise and SDNVPN projects
-
- - split Tempest and rally into 2 different tests: smoke and full
-
- - vIMS test suite integration
-
- - adoption of Kibana for dashboarding
-
-
-New features
-------------
-
- - Functest CLI to prepare and run the tests
-
- - creation of the healthcheck test case
-
- - support new scenarios (ocl, odl_l2-sfc, onos-sfc, lxd, moon, fdio, multisite)
-
- - integration of new OPNFV feature projects (copper, domino, multisite,
- moon, parser, onos-sfc, odl-sfc, security scan)
-
- - introduction of test tiers in functest framework
-
- - automatic reporting
-
- - introduction of a jenkins summary table
-
- - support of ARM architecture
-
-
-Scenario Matrix
-===============
-
-For Colorado 1.0, Functest was tested on the following scenarios (if not
-precised, the scenario is a HA scenario):
-
-+---------------------+---------+---------+---------+---------+
-| Scenario | Apex | Compass | Fuel | Joid |
-+=====================+=========+=========+=========+=========+
-| nosdn | X | X | X | X |
-+---------------------+---------+---------+---------+---------+
-| nosdn-noha | | | X | X |
-+---------------------+---------+---------+---------+---------+
-| kvm | | | X | |
-+---------------------+---------+---------+---------+---------+
-| kvm-noha | | | X | |
-+---------------------+---------+---------+---------+---------+
-| lxd | | | | X |
-+---------------------+---------+---------+---------+---------+
-| lxd-noha | | | | X |
-+---------------------+---------+---------+---------+---------+
-| ovs-noha (dpdk) | | | X | |
-+---------------------+---------+---------+---------+---------+
-| odl_l2 | X | X | X | X |
-+---------------------+---------+---------+---------+---------+
-| odl_l2-noha | | | X | |
-+---------------------+---------+---------+---------+---------+
-| odl_l2-bgpvpn | X | | X | |
-+---------------------+---------+---------+---------+---------+
-| odl_l2-bgpvpn-noha| | | X | |
-+---------------------+---------+---------+---------+---------+
-| odl_l2-fdio-noha | X | | | |
-+---------------------+---------+---------+---------+---------+
-| odl_l2-moon | | X | | |
-+---------------------+---------+---------+---------+---------+
-| odl_l2-sfc-noha | X | | X | |
-+---------------------+---------+---------+---------+---------+
-| odl_l3 | X | X | X | |
-+---------------------+---------+---------+---------+---------+
-| odl_l3-noha | | | X | |
-+---------------------+---------+---------+---------+---------+
-| onos | | X | X | X |
-+---------------------+---------+---------+---------+---------+
-| onos-noha | | | X | |
-+---------------------+---------+---------+---------+---------+
-| onos-sfc | | X | X | X |
-+---------------------+---------+---------+---------+---------+
-| onos-sfc-noha | | | X | |
-+---------------------+---------+---------+---------+---------+
-| multisite | | | X | |
-+---------------------+---------+---------+---------+---------+
-
-For Colorado 2.0, the following scenarios have been re-released:
-
-+---------------------+---------+---------+---------+---------+
-| Scenario | Apex | Compass | Fuel | Joid |
-+=====================+=========+=========+=========+=========+
-| onos | X | X | X | X |
-+---------------------+---------+---------+---------+---------+
-| odl_l2-sfc-ha | | | X | |
-+---------------------+---------+---------+---------+---------+
-
-
-In Colorado, the functional tests have been sliced in 6 different
-categories:
-
-+----------------+-----------------------------------------------+
-| Category | Description |
-+================+===============================================+
-| healthcheck | Basic OpenStack commands |
-+----------------+-----------------------------------------------+
-| smoke | vPings, Tempest and rally smoke tests |
-+----------------+-----------------------------------------------+
-| sdn_suites | Specific SDN feature tests |
-+----------------+-----------------------------------------------+
-| features | OPNFV feature project functional test suites |
-+----------------+-----------------------------------------------+
-| openstack | Advanced, long duration OpenStack tests |
-| | (Tempest and Rally full suite). Each test may |
-| | last several hours |
-+----------------+-----------------------------------------------+
-| vnf | Complex scenarios dealing with orchestration, |
-| | VNF deployment and tests (e.g. vIMS) |
-+----------------+-----------------------------------------------+
-
-For the scenario validation, we consider only the healthcheck, smoke,
-sdn_suites and features categories. These tests are run systematically
-in the CI daily loops.
-
-Success criteria have been defined for these test cases, they can be
-PASS/FAIL or a success rate may be declared (100%, > 90%)
-All the criteria, as well as the test dependencies are declared in the
-ci/testcases.yaml file.
-
-The scoring for the Colorado release per installer can be described as
-follows.
-
-The scoring is an indicator showing how many feature project test suites
-have been integrated on the scenario.
-
-The scoring equals the number of tests * succesful iteration of each
-test [0-3]. The scoring method is described in https://wiki.opnfv.org/pages/viewpage.action?pageId=6828617
-
- e.g.
- apex/odl_l2-nofeature-ha
- tests = vping_ssh+vping_userdata+tempest+rally+odl+doctor+copper
- Scoring = 21/21 = 7 * 3
-
-By default, if not specified, the scenarios are HA.
-HA means OpenStack High Availability (main services). Note that not
-all VIM (e.g. OpenStack) services are deployed in HA mode, as that
-depends upon support of the specific service for HA deployment.
-For example, in the Colorado release, the Congress service
-is deployed in non-HA mode even for HA OPNFV scenarios, as explicit
-support for HA operation has not yet been verified.
-See the release notes of the installers for more details.
-
-
-Apex
-----
-
-+------------------+---------+---------+-------------------+
-| Scenario | Scoring | Success | Results |
-| | | rate | |
-+==================+=========+=========+===================+
-| nosdn | 17/18 | 95% | `apex-res-174`_ |
-+------------------+---------+---------+-------------------+
-| odl_l2 | 21/21 | 100% | `apex-res-175`_ |
-+------------------+---------+---------+-------------------+
-| odl_l3 | 15/18 | 83% | `apex-res-176`_ |
-+------------------+---------+---------+-------------------+
-| odl_l2-bgpvpn | 14/18 | 78% | `apex-res-235`_ |
-+------------------+---------+---------+-------------------+
-| odl_l2-fdio-noha | 12/15 | 80% | `apex-res-6`_ |
-+------------------+---------+---------+-------------------+
-| odl_l2-sfc-noha | 18/24 | 75% | `apex-res-217`_ |
-+------------------+---------+---------+-------------------+
-| onos-nofeature-ha| 20/21 | 95% | `apex-res-423`_ |
-+------------------+---------+---------+-------------------+
-
-
-Compass
--------
-
-+------------------+---------+---------+-------------------+
-| Scenario | Scoring | Success | Results |
-| | | rate | |
-+==================+=========+=========+===================+
-| nosdn | 12/12 | 100% | `compass-res-55`_ |
-+------------------+---------+---------+-------------------+
-| odl_l2 | 15/15 | 100% | `compass-res-59`_ |
-+------------------+---------+---------+-------------------+
-| odl_l3 | 9/12 | 75% | `compass-res-73`_ |
-+------------------+---------+---------+-------------------+
-| odl_l2-moon | 15/18 | 83% | `compass-res-567`_|
-+------------------+---------+---------+-------------------+
-| onos-ha | 15/15 | 100% | `compass-res-285`_|
-+------------------+---------+---------+-------------------+
-| onos-sfc-ha | 17/18 | 95% | `compass-res-76`_ |
-+------------------+---------+---------+-------------------+
-
-Note: all the Compass tests for Colorado have been executed on virtual
-environment. Bare metal resources were used for Master branch.
-
-
-Fuel
-----
-
-+---------------------+---------+---------+-------------------+
-| Scenario | Scoring | Success | Results |
-| | | rate | |
-+=====================+=========+=========+===================+
-| nosdn | 18/18 | 100% | `fuel-res-129`_ |
-+---------------------+---------+---------+-------------------+
-| nosdn-noha | 15/15 | 100% | `fuel-res-154`_ |
-+---------------------+---------+---------+-------------------+
-| nosdn-kvm | 18/18 | 100% | `fuel-res-128`_ |
-+---------------------+---------+---------+-------------------+
-| nosdn-kvm-noha | 15/15 | 100% | `fuel-res-161`_ |
-+---------------------+---------+---------+-------------------+
-| nosdn-ovs | 12/18 | 67%* | `fuel-res-213`_ |
-+---------------------+---------+---------+-------------------+
-| nosdn-ovs-noha | 15/15 | 100% | `fuel-res-162`_ |
-+---------------------+---------+---------+-------------------+
-| odl_l2 | 21/21 | 100% | `fuel-res-123`_ |
-+---------------------+---------+---------+-------------------+
-| odl_l2-noha | 17/18 | 94% | `fuel-res-155`_ |
-+---------------------+---------+---------+-------------------+
-| odl_l2-bgpvpn | 14/18 | 78% | `fuel-res-119`_ |
-+---------------------+---------+---------+-------------------+
-| odl_l2-bgpvpn-noha | 14/15 | 93% | `fuel-res-160`_ |
-+---------------------+---------+---------+-------------------+
-| odl_l2-sfc-noha | 6/21 | 29% | `fuel-res-219`_ |
-+---------------------+---------+---------+-------------------+
-| odl_l2-sfc-ha | 16/21 | 76% | `fuel-res-376`_ |
-+---------------------+---------+---------+-------------------+
-| odl_l3 | 15/18 | 83% | `fuel-res-115`_ |
-+---------------------+---------+---------+-------------------+
-| odl_l3-noha | 12/15 | 80% | `fuel-res-164`_ |
-+---------------------+---------+---------+-------------------+
-| onos | 20/21 | 95% | `fuel-res-492`_ |
-+---------------------+---------+---------+-------------------+
-| onos-noha | 18/18 | 100% | `fuel-res-166`_ |
-+---------------------+---------+---------+-------------------+
-| onos-sfc | 24/24 | 100% | `fuel-res-124`_ |
-+---------------------+---------+---------+-------------------+
-| onos-sfc-noha | 21/21 | 100% | `fuel-res-129`_ |
-+---------------------+---------+---------+-------------------+
-| multisite | N.R | 100% | `fuel-res-8`_ |
-+---------------------+---------+---------+-------------------+
-
-*: all results passed, lacking iterations to reach the full score
-
-Results of Functest on AArch64 Colorado 3.0
-
-+---------------------+---------+---------+----------------------+
-| Scenario | Scoring | Success | Results |
-| | | rate | |
-+=====================+=========+=========+======================+
-| nosdn | 18/18 | 100% | `fuel-arm-res-128`_ |
-+---------------------+---------+---------+----------------------+
-| odl_l2 | 21/21 | 100% | `fuel-arm-res-122`_ |
-+---------------------+---------+---------+----------------------+
-| odl_l2-noha | 18/18 | 100% | `fuel-arm-res-129`_ |
-+---------------------+---------+---------+----------------------+
-| odl_l3 | 17/18 | 95% | `fuel-arm-res-135`_ |
-+---------------------+---------+---------+----------------------+
-
-
-Joid
-----
-
-+---------------------+---------+---------+-----------------+
-| Scenario | Scoring | Success | Results |
-| | | rate | |
-+=====================+=========+=========+=================+
-| nosdn | 18/18 | 100% | `joid-res-102`_ |
-+---------------------+---------+---------+-----------------+
-| nosdn-noha | 17/18 | 95% | `joid-res-93`_ |
-+---------------------+---------+---------+-----------------+
-| nosdn-lxd | 12/12 | 100% | `joid-res-104`_ |
-+---------------------+---------+---------+-----------------+
-| nosdn-lxd-noha | 12/12 | 100% | `joid-res-91`_ |
-+---------------------+---------+---------+-----------------+
-| odl_l2 | 21/21 | 100% | `joid-res-103`_ |
-+---------------------+---------+---------+-----------------+
-| onos | 21/21 | 100% | `joid-res-345`_ |
-+---------------------+---------+---------+-----------------+
-| onos-sfc | 24/24 | 100% | `joid-res-97`_ |
-+---------------------+---------+---------+-----------------+
-
-It is highly recommended to install a json viewer in your browser
-(e.g. https://addons.mozilla.org/fr/firefox/addon/jsonview/)
-
-You can get additional details through test logs on http://artifacts.opnfv.org/.
-As no search engine is available on the OPNFV artifact web site you must
-retrieve the pod identifier on which the tests have been executed (see
-field pod in any of the results) then click on the selected POD and look
-for the date of the test you are interested in.
-
-The reporting pages can be found at:
-
- * apex: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-apex.html
- * compass: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-compass.html
- * fuel: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-fuel.html
- * joid: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-joid.html
-
-Colorado known restrictions/issues
-==================================
-
-+-----------+-----------+----------------------------------------------+
-| Installer | Scenario | Issue |
-+===========+===========+==============================================+
-| any | odl_l3-* | Tempest cases related to using floating IP |
-| | | addresses fail because of a known ODL bug. |
-| | | vPing_ssh test case is excluded for the same |
-| | | reason. |
-| | | https://jira.opnfv.org/browse/APEX-112 |
-| | | https://jira.opnfv.org/browse/FUNCTEST-445 |
-+-----------+-----------+----------------------------------------------+
-| apex/fuel | *-bgpvpn | vPing_ssh (floating ips not supported) and |
-| | | vIMS excluded. Some Tempest cases related to |
-| | | floating ips also excluded. Some performance |
-| | | issues have been detected in this scenario |
-| | | (i.e. BGPVPN extension enabled) when running |
-| | | commands against the OpenStack APIs, thus |
-| | | Rally sanity test case has been disabled. |
-| | | Performance issues seem to be connected to |
-| | | the ODL version. It is planned to reintroduce|
-| | | Rally sanity in Colorado 2.0 with the |
-| | | adoption of ODL Boron release. |
-+-----------+-----------+----------------------------------------------+
-| apex | *-fdio | Due to late integration, fdio decided to |
-| | | focus on mandatory tests and exclude feature |
-| | | tests (copper, doctor, security_scan) from |
-| | | its scenarios |
-+-----------+-----------+----------------------------------------------+
-| compass | moon | First ODL test FAILS because ODL/Openstack |
-| | | federation done in moon is partial. Only |
-| | | MD-SAL is federated (not AD-SAL) |
-+-----------+-----------+----------------------------------------------+
-| fuel | any | TestServerBasicOps test case skipped |
-| | | https://gerrit.opnfv.org/gerrit/#/c/19635/ |
-+-----------+-----------+----------------------------------------------+
-| fuel | kvm | vPing_ssh and vIMS excluded (metadata related|
-| | | scenarios) |
-+-----------+-----------+----------------------------------------------+
-| fuel | multisite | random errors running multisite. A known bug |
-| | | in keystone mitaka, due to which memcache |
-| | | raises exception and keystone becomes |
-| | | unresponsive |
-| | | bugs.launchpad.net/keystone/+bug/1600394 |
-| | | workaround consists in restarting memcache on|
-| | | server |
-+-----------+-----------+----------------------------------------------+
-| joid | any | Tempest cases related to object storage |
-| | | excluded |
-| | | https://gerrit.opnfv.org/gerrit/#/c/17871/ |
-+-----------+-----------+----------------------------------------------+
-| joid | domino | Domino tests are skipped in CI. However the |
-| | | test case can be run by manually setting |
-| | | IS_IPandKEY_CONFIGURED=true after manually |
-| | | setting properly the IP addresses of the 3 |
-| | | Controller nodes in the configuration file |
-| | | /repos/domino/tests/run_multinode.sh |
-+-----------+-----------+----------------------------------------------+
+OPNFV (test-)requirements.txt have been updated according to stable/ocata global-requirements.txt.
+Functest uses (and completes) stable/ocata upper-constraints.txt in Dockerfiles and tox configuration.
+The project relies on pbr, which injects requirements into the install_requires, tests_require and/or dependency_links
+arguments to setup. It also supports conditional dependencies which can be added to the requirements (e.g. dnspython>=1.14.0;python_version=='2.7')
+
+The way to manage logging has been centralized to a configuration file (logging.ini) which might be modified by the user.
+By default, the output of executing the test cases is redirected to log files and is not displayed on the console, only result
+messages and summary tables are displayed.
+
+The framework has been refactored and all the test cases inherit from a core class TestCase. For Feature projects who develop
+test cases, 2 sub-classes have been created:
+ - Feature: it implements all the needed functions and the developer must only overwrite the method "execute" (e.g. Barometer)
+ - BashFeature: it is used if the third party test case is a shell script. This way, the execution command must be specified in
+ testcases.yaml as the argument (e.g. Domino, Doctor)
+
+An internal REST API has been introduced in Euphrates. The goal is to trigger Functest operations through an API in addition of the CLI.
+This could be considered as a first step towards a pseudo micro services approach where the different test projects could expose and
+consume APIs to the other test projects.
+
+Euphrates 5.1 improvements
+==========================
+
+* Alpine images are now supported for ARM (arm64).
+* Added Vyos_router test case.
+* Updated of Rally 0.9.1 and fixed some bugs in cinder scenarios.
+* Patch to allow building containers from a gerrit change.
+* Selection of a subset of SNAPS test cases.
+* Reorder VNF test cases and adjust timeouts in VNFs.
+
+
+
+Euphrates 5.1 known restrictions/issues
+=======================================
++--------------+-----------+----------------------------------------------+
+| Installer | Scenario | Issue |
++==============+===========+==============================================+
+| fuel@aarch64 | any | VNF tier not supported yet. |
++--------------+-----------+----------------------------------------------+
+| | | The test cases belonging to the VNF tier |
+| any | any | have been only tested on os-nosdn-nofeature |
+| | | scenarios and baremetal deployments. |
++--------------+-----------+----------------------------------------------+
+| Joid | k8 | Functest does not offer test suites for |
+| Compass | | Kubernetes scenarios yet. |
++--------------+-----------+----------------------------------------------+
Test and installer/scenario dependencies
========================================
It is not always possible to run all the test cases on all the scenarios.
-The following table details the dependencies of the test cases per
-scenario. The scenario dependencies (installer or scenario) are detailed
-in https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml
+The scenario dependencies (installer or scenario) are detailed
+in the different testcases.yaml for each tier:
+
+ * https://git.opnfv.org/functest/tree/docker/healthcheck/testcases.yaml?h=stable/euphrates
+ * https://git.opnfv.org/functest/tree/docker/smoke/testcases.yaml?h=stable/euphrates
+ * https://git.opnfv.org/functest/tree/docker/features/testcases.yaml?h=stable/euphrates
+ * https://git.opnfv.org/functest/tree/docker/components/testcases.yaml?h=stable/euphrates
+ * https://git.opnfv.org/functest/tree/docker/vnf/testcases.yaml?h=stable/euphrates
+ * https://git.opnfv.org/functest/tree/docker/parser/testcases.yaml?h=stable/euphrates
+
Test results
============
-Test results are available in:
+The Functest scenario status on December 15, 2017 can be seen on
+http://testresults.opnfv.org/functest/euphrates/
+
+Test logs are available in:
- - test results document: http://artifacts.opnfv.org/functest
+ - test results logs from CI: http://artifacts.opnfv.org (within different directories 'logs_functest_X')
- jenkins logs on CI: https://build.opnfv.org/ci/view/functest/
+------------------+-----------------------------------------------+
| JIRA | Description |
+==================+===============================================+
-| `FUNCTEST-446`_ | Cleanup ODL-SFC output in Functest execution |
-| | Impact on odl_l2-sfc scenarios |
-+------------------+-----------------------------------------------+
-| `FUNCTEST-454`_ | Cleanup failures when using HA networks in |
-| | Neutron |
+| | |
+| | |
+------------------+-----------------------------------------------+
+All the tickets that are not blocking have been fixed or postponed
+the next release.
+
+
Useful links
============
- wiki project page: https://wiki.opnfv.org/opnfv_functional_testing
- - wiki Functest Colorado page: https://wiki.opnfv.org/display/functest/Functest+Colorado
+ - wiki Functest Euphrates page: https://wiki.opnfv.org/display/functest/5.+Euphrates
- Functest repo: https://git.opnfv.org/cgit/functest
- Functest IRC chan: #opnfv-functest
- - Functest reporting: http://testresults.opnfv.org/reporting
-
- - Functest test configuration: https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml
-
- - Functest Colorado user guide: http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html
-
- - Functest installation/configuration guide: http://artifacts.opnfv.org/functest/colorado/docs/configguide/index.html
-
- - Functest developer guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html
-
-.. _`FUNCTEST-446` : https://jira.opnfv.org/browse/FUNCTEST-446
-
-.. _`FUNCTEST-454` : https://jira.opnfv.org/browse/FUNCTEST-454
-
-.. _`apex-res-6` : http://testresults.opnfv.org/test/api/v1/results?build_tag=bd-push-daily-colorado-6
-
-.. _`apex-res-174` : http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-174
-
-.. _`apex-res-175`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-175
-
-.. _`apex-res-176`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-176
-
-.. _`apex-res-217`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-217
-
-.. _`apex-res-235`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-235
-
-.. _`apex-res-423`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-423
-
-.. _`compass-res-55`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-55
-
-.. _`compass-res-59`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-59
-
-.. _`compass-res-73`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-73
-
-.. _`compass-res-76`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-76
-
-.. _`compass-res-77`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-77
-
-.. _`compass-res-567`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-master-567
-
-.. _`compass-res-285`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-master-285
-
-.. _`fuel-res-8`: https://build.opnfv.org/ci/view/functest/job/functest-fuel-virtual-suite-colorado/lastSuccessfulBuild/console
-
-.. _`fuel-res-115`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-115
-
-.. _`fuel-res-117`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-117
-
-.. _`fuel-res-119`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-119
-
-.. _`fuel-res-123`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-123
-
-.. _`fuel-res-124`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-124
-
-.. _`fuel-res-128`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-128
-
-.. _`fuel-res-129`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-129
-
-.. _`fuel-res-154`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-154
-
-.. _`fuel-res-155`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-155
-
-.. _`fuel-res-160`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-160
-
-.. _`fuel-res-161`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-161
-
-.. _`fuel-res-162`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-162
-
-.. _`fuel-res-164`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-164
-
-.. _`fuel-res-166`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-166
-
-.. _`fuel-res-213`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-213
-
-.. _`fuel-res-219`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-219
-
-.. _`fuel-res-376`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-376
-
-.. _`fuel-res-492`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-492
-
-.. _`fuel-arm-res-128`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/128/console
-
-.. _`fuel-arm-res-122`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/122/console
-
-.. _`fuel-arm-res-129`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/129/console
-
-.. _`fuel-arm-res-135`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/135/console
-
-.. _`joid-res-93`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-93
-
-.. _`joid-res-91`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-91
-
-.. _`joid-res-99`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-99
-
-.. _`joid-res-97`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-97
-
-.. _`joid-res-102`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-102
-
-.. _`joid-res-103`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-103
-
-.. _`joid-res-104`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-104
-
-.. _`joid-res-345`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-345
-
+ - Reporting page: http://testresults.opnfv.org/reporting/euphrates.html
Code Review / functest.git / blobdiff