-==============================================
-OPNFV Brahmaputra3.0 release note for Functest
-==============================================
-
-Abstract
-========
-
-This document describes the release note of Functest project.
-
+=======
License
=======
-OPNFV Brahmaputra release note for Functest Docs
+OPNFV Colorado release note for Functest Docs
(c) by Morgan Richomme (Orange)
-OPNFV Brahmaputra release note for Functest Docs
+OPNFV Colorado release note for Functest Docs
are licensed under a Creative Commons Attribution 4.0 International License.
You should have received a copy of the license along with this.
If not, see <http://creativecommons.org/licenses/by/4.0/>.
+===========================================
+OPNFV Colorado1.0 release note for Functest
+===========================================
+
+Abstract
+========
+
+This document describes the release note of Functest project.
+
+
Version history
===============
| **Date** | **Ver.** | **Author** | **Comment** |
| | | | |
+------------+----------+------------------+------------------------+
-| 2016-02-25 | 1.0.0 | Morgan Richomme | Functest for B release |
-| | | (Orange) | |
-+------------+----------+------------------+------------------------+
-| 2016-04-27 | 3.0.0 | Morgan Richomme | Add scenarios |
-| | | (Orange) | |
+| 2016-08-17 | 1.0.0 | Morgan Richomme | Functest for Colorado |
+| | | (Orange) | release |
+------------+----------+------------------+------------------------+
-OPNFV Brahmaputra Release
+OPNFV Colorado Release
=========================
Functest deals with functional testing of the OPNFV solution.
The internal test cases are:
+ * healthcheck
* vPing ssh
* vPing userdata
- * Tempest
- * Rally
- * vIMS
+ * Tempest Smoke Serial
+ * Rally Sanity
* ODL
+ * Tempest full parallel
+ * Rally full
+ * vIMS
The OPNFV projects integrated into Functest framework for automation are:
- * Promise
+ * bgpvpn
+ * Copper
* Doctor
+ * Domino
+ * Moon
+ * Multisite
* ONOSFW
- * bgpvpn
+ * ONOS-sfc
+ * ODL-sfc
+ * Parser
+ * Promise
+ * Security scan
+
+The validation of a scenario requires a subset of these tests depending
+on the installer and the scenario.
+
+The 3 last internal test cases (tempest full parallel, Rally full and
+vIMS) are not considered for scenario validation.
Release Data
============
| **Project** | functest |
| | |
+--------------------------------------+--------------------------------------+
-| **Repo/tag** | brahmaputra.1.0 |
+| **Repo/tag** | colorado.1.0 |
| | |
+--------------------------------------+--------------------------------------+
-| **Release designation** | Brahmaputra base release |
+| **Release designation** | Colorado base release |
| | |
+--------------------------------------+--------------------------------------+
-| **Release date** | February 26 2016 |
+| **Release date** | September 22 2016 |
| | |
+--------------------------------------+--------------------------------------+
-| **Purpose of the delivery** | Brahmaputra base release |
+| **Purpose of the delivery** | Colorado base release |
| | |
+--------------------------------------+--------------------------------------+
Software
--------
- - The Functest Docker image
+ - The Functest Docker image: https://hub.docker.com/r/opnfv/functest (tag: colorado.1.0)
+
+ - The testapi Docker image: https://hub.docker.com/r/opnfv/testapi (tag:colorado.1.0)
+
Documents
---------
- - Installation/configuration guide
+ - Installation/configuration guide: http://artifacts.opnfv.org/functest/colorado/docs/configguide/index.html
- - User Guide
+ - User Guide: http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html
- - Developer Guide
+ - Developer Guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html
- - Test results per scenario
Version change
==============
Feature evolution
-----------------
- - support new scenarios
+ - refactoring of ODL functional tests (with upstream modifications)
-New features
-------------
+ - refactoring of testapi (update, swagger documentation, dockerization)
- - minor bug fixes (formating)
+ - jenkins logs improvement
- - Modification of the configuration to support vPing_userdata on ONOS scenario
+ - update integration of Doctor, Promise and SDNVPN projects
- - Use serial option in Tempest to improve success rate
+ - split Tempest and rally into 2 different tests: smoke and full
-Scenario Matrix
-===============
+ - vIMS test suite integration
-For Brahmaputra 3.0, Functest was succesfully tested on the following scenarios:
+ - adoption of Kibana for dashboarding
-+----------------+---------+---------+---------+---------+
-| Scenario | Apex | Compass | Fuel | Joid |
-+================+=========+=========+=========+=========+
-| odl_l2 | X | X | X | X |
-+----------------+---------+---------+---------+---------+
-| onos | | X | X | |
-+----------------+---------+---------+---------+---------+
-| nosdn | | X | X | |
-+----------------+---------+---------+---------+---------+
-| ovs (dpdk) | | | X | |
-+----------------+---------+---------+---------+---------+
-| kvm | | | X | |
-+----------------+---------+---------+---------+---------+
-| bgpvpn | X | | X | |
-+----------------+---------+---------+---------+---------+
-| sfc | | | X | |
-+----------------+---------+---------+---------+---------+
-Functest defines the success criteria when having at least 4 consecutive
-successful runs of a given scenario from the Continuous Integration.
+New features
+------------
-The success criteria is defined as follows:
+ - Functest CLI to prepare and run the tests
- * vPing SSH 100% OK
- * vPing userdata 100% OK
- * Tempest success rate > 90%
- * Rally success rate > 90%
- * ODL success rate = 100%
- * ONOSFW success rate = 100%
- * Promise success rate = 100%
- * Bgpvpn success rate = 100%
- * vIMS: deployement of the orchestrator and the vIMS VNF successful
+ - creation of the healthcheck test case
-Other scenarios are currently available but did not meet success criteria for
-the release but might be added in the incremental scenario update of the
-release.
+ - support new scenarios (ocl, odl_l2-sfc, onos-sfc, lxd, moon, fdio, multisite)
-Brahmaputra limitations
------------------------
+ - integration of new OPNFV feature projects (copper, domino, multisite,
+ moon, parser, onos-sfc, odl-sfc, security scan)
-- Fuel and Apex Tempest success rate was below 90% but above 80% on some
-scenarios. Some of the error causes were identified (workers, lack of IP)
+ - introduction of test tiers in functest framework
-- vIMS failed in CI for joid/odl_l2 scenario
+ - automatic reporting
-- vPing userdata and vIMS excluded from onos scenario
+ - introduction of a jenkins summary table
-- vPing_ssh and vIMS excluded from bgpvpn and kvm scenario
+ - support of ARM architecture
-- None of the odl_l3 scenarios has been successful due to vPing ssh issue (ODL
-bug reported https://bugs.opendaylight.org/show_bug.cgi?id=5582)
-- apex/nosdn never run (not a target scenario) but probably succesful
+Scenario Matrix
+===============
-- vPing SSH and vPing userdata no more run on CI since modification of bgpvpn
-configuration regex.
+For Colorado 1.0, Functest was tested on the following scenarios (if not
+precised, the scenario is a HA scenario):
+
++---------------------+---------+---------+---------+---------+
+| Scenario | Apex | Compass | Fuel | Joid |
++=====================+=========+=========+=========+=========+
+| nosdn | X | X | X | X |
++---------------------+---------+---------+---------+---------+
+| nosdn-noha | | | X | X |
++---------------------+---------+---------+---------+---------+
+| kvm | | | X | |
++---------------------+---------+---------+---------+---------+
+| kvm-noha | | | X | |
++---------------------+---------+---------+---------+---------+
+| lxd | | | | X |
++---------------------+---------+---------+---------+---------+
+| lxd-noha | | | | X |
++---------------------+---------+---------+---------+---------+
+| ovs-noha (dpdk) | | | X | |
++---------------------+---------+---------+---------+---------+
+| odl_l2 | X | X | X | X |
++---------------------+---------+---------+---------+---------+
+| odl_l2-noha | | | X | |
++---------------------+---------+---------+---------+---------+
+| odl_l2-bgpvpn | X | | X | |
++---------------------+---------+---------+---------+---------+
+| odl_l2-bgpvpn-noha| | | X | |
++---------------------+---------+---------+---------+---------+
+| odl_l2-fdio-noha | X | | | |
++---------------------+---------+---------+---------+---------+
+| odl_l2-moon | | X | | |
++---------------------+---------+---------+---------+---------+
+| odl_l2-sfc-noha | X | | X | |
++---------------------+---------+---------+---------+---------+
+| odl_l3 | X | X | X | |
++---------------------+---------+---------+---------+---------+
+| odl_l3-noha | | | X | |
++---------------------+---------+---------+---------+---------+
+| onos | | X | X | X |
++---------------------+---------+---------+---------+---------+
+| onos-noha | | | X | |
++---------------------+---------+---------+---------+---------+
+| onos-sfc | | X | X | X |
++---------------------+---------+---------+---------+---------+
+| onos-sfc-noha | | | X | |
++---------------------+---------+---------+---------+---------+
+| multisite | | | X | |
++---------------------+---------+---------+---------+---------+
+
+For Colorado 2.0, the following scenarios have been re-released:
+
++---------------------+---------+---------+---------+---------+
+| Scenario | Apex | Compass | Fuel | Joid |
++=====================+=========+=========+=========+=========+
+| onos | X | X | X | X |
++---------------------+---------+---------+---------+---------+
+| odl_l2-sfc-ha | | | X | |
++---------------------+---------+---------+---------+---------+
+
+
+In Colorado, the functional tests have been sliced in 6 different
+categories:
+
++----------------+-----------------------------------------------+
+| Category | Description |
++================+===============================================+
+| healthcheck | Basic OpenStack commands |
++----------------+-----------------------------------------------+
+| smoke | vPings, Tempest and rally smoke tests |
++----------------+-----------------------------------------------+
+| sdn_suites | Specific SDN feature tests |
++----------------+-----------------------------------------------+
+| features | OPNFV feature project functional test suites |
++----------------+-----------------------------------------------+
+| openstack | Advanced, long duration OpenStack tests |
+| | (Tempest and Rally full suite). Each test may |
+| | last several hours |
++----------------+-----------------------------------------------+
+| vnf | Complex scenarios dealing with orchestration, |
+| | VNF deployment and tests (e.g. vIMS) |
++----------------+-----------------------------------------------+
+
+For the scenario validation, we consider only the healthcheck, smoke,
+sdn_suites and features categories. These tests are run systematically
+in the CI daily loops.
+
+Success criteria have been defined for these test cases, they can be
+PASS/FAIL or a success rate may be declared (100%, > 90%)
+All the criteria, as well as the test dependencies are declared in the
+ci/testcases.yaml file.
+
+The scoring for the Colorado release per installer can be described as
+follows.
+
+The scoring is an indicator showing how many feature project test suites
+have been integrated on the scenario.
+
+The scoring equals the number of tests * succesful iteration of each
+test [0-3]. The scoring method is described in https://wiki.opnfv.org/pages/viewpage.action?pageId=6828617
+
+ e.g.
+ apex/odl_l2-nofeature-ha
+ tests = vping_ssh+vping_userdata+tempest+rally+odl+doctor+copper
+ Scoring = 21/21 = 7 * 3
+
+By default, if not specified, the scenarios are HA.
+HA means OpenStack High Availability (main services). Note that not
+all VIM (e.g. OpenStack) services are deployed in HA mode, as that
+depends upon support of the specific service for HA deployment.
+For example, in the Colorado release, the Congress service
+is deployed in non-HA mode even for HA OPNFV scenarios, as explicit
+support for HA operation has not yet been verified.
+See the release notes of the installers for more details.
+
+
+Apex
+----
+
++------------------+---------+---------+-------------------+
+| Scenario | Scoring | Success | Results |
+| | | rate | |
++==================+=========+=========+===================+
+| nosdn | 17/18 | 95% | `apex-res-174`_ |
++------------------+---------+---------+-------------------+
+| odl_l2 | 21/21 | 100% | `apex-res-175`_ |
++------------------+---------+---------+-------------------+
+| odl_l3 | 15/18 | 83% | `apex-res-176`_ |
++------------------+---------+---------+-------------------+
+| odl_l2-bgpvpn | 14/18 | 78% | `apex-res-235`_ |
++------------------+---------+---------+-------------------+
+| odl_l2-fdio-noha | 12/15 | 80% | `apex-res-6`_ |
++------------------+---------+---------+-------------------+
+| odl_l2-sfc-noha | 18/24 | 75% | `apex-res-217`_ |
++------------------+---------+---------+-------------------+
+| onos-nofeature-ha| 20/21 | 95% | `apex-res-423`_ |
++------------------+---------+---------+-------------------+
+
+
+Compass
+-------
+
++------------------+---------+---------+-------------------+
+| Scenario | Scoring | Success | Results |
+| | | rate | |
++==================+=========+=========+===================+
+| nosdn | 12/12 | 100% | `compass-res-55`_ |
++------------------+---------+---------+-------------------+
+| odl_l2 | 15/15 | 100% | `compass-res-59`_ |
++------------------+---------+---------+-------------------+
+| odl_l3 | 9/12 | 75% | `compass-res-73`_ |
++------------------+---------+---------+-------------------+
+| odl_l2-moon | 15/18 | 83% | `compass-res-567`_|
++------------------+---------+---------+-------------------+
+| onos-ha | 15/15 | 100% | `compass-res-285`_|
++------------------+---------+---------+-------------------+
+| onos-sfc-ha | 17/18 | 95% | `compass-res-76`_ |
++------------------+---------+---------+-------------------+
+
+Note: all the Compass tests for Colorado have been executed on virtual
+environment. Bare metal resources were used for Master branch.
+
+
+Fuel
+----
+
++---------------------+---------+---------+-------------------+
+| Scenario | Scoring | Success | Results |
+| | | rate | |
++=====================+=========+=========+===================+
+| nosdn | 18/18 | 100% | `fuel-res-129`_ |
++---------------------+---------+---------+-------------------+
+| nosdn-noha | 15/15 | 100% | `fuel-res-154`_ |
++---------------------+---------+---------+-------------------+
+| nosdn-kvm | 18/18 | 100% | `fuel-res-128`_ |
++---------------------+---------+---------+-------------------+
+| nosdn-kvm-noha | 15/15 | 100% | `fuel-res-161`_ |
++---------------------+---------+---------+-------------------+
+| nosdn-ovs | 12/18 | 67%* | `fuel-res-213`_ |
++---------------------+---------+---------+-------------------+
+| nosdn-ovs-noha | 15/15 | 100% | `fuel-res-162`_ |
++---------------------+---------+---------+-------------------+
+| odl_l2 | 21/21 | 100% | `fuel-res-123`_ |
++---------------------+---------+---------+-------------------+
+| odl_l2-noha | 17/18 | 94% | `fuel-res-155`_ |
++---------------------+---------+---------+-------------------+
+| odl_l2-bgpvpn | 14/18 | 78% | `fuel-res-119`_ |
++---------------------+---------+---------+-------------------+
+| odl_l2-bgpvpn-noha | 14/15 | 93% | `fuel-res-160`_ |
++---------------------+---------+---------+-------------------+
+| odl_l2-sfc-noha | 6/21 | 29% | `fuel-res-219`_ |
++---------------------+---------+---------+-------------------+
+| odl_l2-sfc-ha | 16/21 | 76% | `fuel-res-376`_ |
++---------------------+---------+---------+-------------------+
+| odl_l3 | 15/18 | 83% | `fuel-res-115`_ |
++---------------------+---------+---------+-------------------+
+| odl_l3-noha | 12/15 | 80% | `fuel-res-164`_ |
++---------------------+---------+---------+-------------------+
+| onos | 20/21 | 95% | `fuel-res-492`_ |
++---------------------+---------+---------+-------------------+
+| onos-noha | 18/18 | 100% | `fuel-res-166`_ |
++---------------------+---------+---------+-------------------+
+| onos-sfc | 24/24 | 100% | `fuel-res-124`_ |
++---------------------+---------+---------+-------------------+
+| onos-sfc-noha | 21/21 | 100% | `fuel-res-129`_ |
++---------------------+---------+---------+-------------------+
+| multisite | N.R | 100% | `fuel-res-8`_ |
++---------------------+---------+---------+-------------------+
+
+*: all results passed, lacking iterations to reach the full score
+
+Results of Functest on AArch64 Colorado 3.0
+
++---------------------+---------+---------+----------------------+
+| Scenario | Scoring | Success | Results |
+| | | rate | |
++=====================+=========+=========+======================+
+| nosdn | 18/18 | 100% | `fuel-arm-res-128`_ |
++---------------------+---------+---------+----------------------+
+| odl_l2 | 21/21 | 100% | `fuel-arm-res-122`_ |
++---------------------+---------+---------+----------------------+
+| odl_l2-noha | 18/18 | 100% | `fuel-arm-res-129`_ |
++---------------------+---------+---------+----------------------+
+| odl_l3 | 17/18 | 95% | `fuel-arm-res-135`_ |
++---------------------+---------+---------+----------------------+
+
+
+Joid
+----
+
++---------------------+---------+---------+-----------------+
+| Scenario | Scoring | Success | Results |
+| | | rate | |
++=====================+=========+=========+=================+
+| nosdn | 18/18 | 100% | `joid-res-102`_ |
++---------------------+---------+---------+-----------------+
+| nosdn-noha | 17/18 | 95% | `joid-res-93`_ |
++---------------------+---------+---------+-----------------+
+| nosdn-lxd | 12/12 | 100% | `joid-res-104`_ |
++---------------------+---------+---------+-----------------+
+| nosdn-lxd-noha | 12/12 | 100% | `joid-res-91`_ |
++---------------------+---------+---------+-----------------+
+| odl_l2 | 21/21 | 100% | `joid-res-103`_ |
++---------------------+---------+---------+-----------------+
+| onos | 21/21 | 100% | `joid-res-345`_ |
++---------------------+---------+---------+-----------------+
+| onos-sfc | 24/24 | 100% | `joid-res-97`_ |
++---------------------+---------+---------+-----------------+
+
+It is highly recommended to install a json viewer in your browser
+(e.g. https://addons.mozilla.org/fr/firefox/addon/jsonview/)
+
+You can get additional details through test logs on http://artifacts.opnfv.org/.
+As no search engine is available on the OPNFV artifact web site you must
+retrieve the pod identifier on which the tests have been executed (see
+field pod in any of the results) then click on the selected POD and look
+for the date of the test you are interested in.
+
+The reporting pages can be found at:
+
+ * apex: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-apex.html
+ * compass: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-compass.html
+ * fuel: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-fuel.html
+ * joid: http://testresults.opnfv.org/reporting/functest/release/colorado/index-status-joid.html
+
+Colorado known restrictions/issues
+==================================
+
++-----------+-----------+----------------------------------------------+
+| Installer | Scenario | Issue |
++===========+===========+==============================================+
+| any | odl_l3-* | Tempest cases related to using floating IP |
+| | | addresses fail because of a known ODL bug. |
+| | | vPing_ssh test case is excluded for the same |
+| | | reason. |
+| | | https://jira.opnfv.org/browse/APEX-112 |
+| | | https://jira.opnfv.org/browse/FUNCTEST-445 |
++-----------+-----------+----------------------------------------------+
+| apex/fuel | *-bgpvpn | vPing_ssh (floating ips not supported) and |
+| | | vIMS excluded. Some Tempest cases related to |
+| | | floating ips also excluded. Some performance |
+| | | issues have been detected in this scenario |
+| | | (i.e. BGPVPN extension enabled) when running |
+| | | commands against the OpenStack APIs, thus |
+| | | Rally sanity test case has been disabled. |
+| | | Performance issues seem to be connected to |
+| | | the ODL version. It is planned to reintroduce|
+| | | Rally sanity in Colorado 2.0 with the |
+| | | adoption of ODL Boron release. |
++-----------+-----------+----------------------------------------------+
+| apex | *-fdio | Due to late integration, fdio decided to |
+| | | focus on mandatory tests and exclude feature |
+| | | tests (copper, doctor, security_scan) from |
+| | | its scenarios |
++-----------+-----------+----------------------------------------------+
+| compass | moon | First ODL test FAILS because ODL/Openstack |
+| | | federation done in moon is partial. Only |
+| | | MD-SAL is federated (not AD-SAL) |
++-----------+-----------+----------------------------------------------+
+| fuel | any | TestServerBasicOps test case skipped |
+| | | https://gerrit.opnfv.org/gerrit/#/c/19635/ |
++-----------+-----------+----------------------------------------------+
+| fuel | kvm | vPing_ssh and vIMS excluded (metadata related|
+| | | scenarios) |
++-----------+-----------+----------------------------------------------+
+| fuel | multisite | random errors running multisite. A known bug |
+| | | in keystone mitaka, due to which memcache |
+| | | raises exception and keystone becomes |
+| | | unresponsive |
+| | | bugs.launchpad.net/keystone/+bug/1600394 |
+| | | workaround consists in restarting memcache on|
+| | | server |
++-----------+-----------+----------------------------------------------+
+| joid | any | Tempest cases related to object storage |
+| | | excluded |
+| | | https://gerrit.opnfv.org/gerrit/#/c/17871/ |
++-----------+-----------+----------------------------------------------+
+| joid | domino | Domino tests are skipped in CI. However the |
+| | | test case can be run by manually setting |
+| | | IS_IPandKEY_CONFIGURED=true after manually |
+| | | setting properly the IP addresses of the 3 |
+| | | Controller nodes in the configuration file |
+| | | /repos/domino/tests/run_multinode.sh |
++-----------+-----------+----------------------------------------------+
-See known issues section for details
Test and installer/scenario dependencies
========================================
-It is not always possible to run all the test cases on all the scenarios. The
-following table details the dependencies of the test cases per scenario.
-
-+----------------+-------------+-------------+-------------+-------------+
-| Test cases | Apex | Compass | Fuel | Joid |
-+================+=============+=============+=============+=============+
-| vPing SSH | all | all | all | all |
-+----------------+-------------+-------------+-------------+-------------+
-| vPing userdata | all | all | all | all |
-+----------------+-------------+-------------+-------------+-------------+
-| Tempest | all | all | all | all |
-+----------------+-------------+-------------+-------------+-------------+
-| Rally | all | all | all | all |
-+----------------+-------------+-------------+-------------+-------------+
-| ODL | all ODL | all ODL | all ODL | all ODL |
-+----------------+-------------+-------------+-------------+-------------+
-| ONOS | ONOS | ONOS | ONOS | ONOS |
-+----------------+-------------+-------------+-------------+-------------+
-| Promise | no | no | all | all |
-+----------------+-------------+-------------+-------------+-------------+
-| vIMS | all except | all except | all except | all except |
-| | ONOS | ONOS | ONOS | ONOS |
-+----------------+-------------+-------------+-------------+-------------+
-| Doctor | all | no | no | no |
-+----------------+-------------+-------------+-------------+-------------+
-| Bgpvpn | all | no | all | no |
-+----------------+-------------+-------------+-------------+-------------+
+It is not always possible to run all the test cases on all the scenarios.
+The following table details the dependencies of the test cases per
+scenario. The scenario dependencies (installer or scenario) are detailed
+in https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml
Test results
============
Test results are available in:
- - test results document: http://artifacts.opnfv.org/functest/docs/results/index.html
+ - test results document: http://artifacts.opnfv.org/functest
- jenkins logs on CI: https://build.opnfv.org/ci/view/functest/
- - Test dashboards: http://testresults.opnfv.org/dashboard
+ - jenkins logs on ARM CI: https://build.opnfv.org/ci/view/armband/
-Known issues
-------------
- - IPv6 issues in tempest suite:
- - tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_port_in_allowed_allocation_pools
+Open JIRA tickets
+=================
- - Tempest code which doesn't reserve big enough allocation
++------------------+-----------------------------------------------+
+| JIRA | Description |
++==================+===============================================+
+| `FUNCTEST-446`_ | Cleanup ODL-SFC output in Functest execution |
+| | Impact on odl_l2-sfc scenarios |
++------------------+-----------------------------------------------+
+| `FUNCTEST-454`_ | Cleanup failures when using HA networks in |
+| | Neutron |
++------------------+-----------------------------------------------+
- - https://bugs.launchpad.net/tempest/+bug/1514457
+Useful links
+============
- - Lack of IP addresses available lead to several errors in different test cases
+ - wiki project page: https://wiki.opnfv.org/opnfv_functional_testing
- - vIMS (http://testresults.opnfv.org/reporting/vims/):
+ - wiki Functest Colorado page: https://wiki.opnfv.org/display/functest/Functest+Colorado
- - the VM needs to have access to OpenStack API.
+ - Functest repo: https://git.opnfv.org/cgit/functest
- - Technical architecture may not allow this access (for security reasons)
+ - Functest CI dashboard: https://build.opnfv.org/ci/view/functest/
- - Orchestrator can be deployed but the vIMS VNF cannot
+ - JIRA dashboard: https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10611
- - That is the reason why it fails on joid/odl_l2 scenario on Orange POD 2
+ - Functest IRC chan: #opnfv-functest
- - case needs to be consolidated on new scenaios (bgpvpn, sfc, ovs)
+ - Functest reporting: http://testresults.opnfv.org/reporting
-Open JIRA tickets
-=================
+ - Functest test configuration: https://git.opnfv.org/cgit/functest/tree/ci/testcases.yaml
-+------------------+-----------------------------------------+
-| JIRA | Description |
-+==================+=========================================+
-| FUNCTEST-231 | vPing SSH no more run systematically |
-| | in CI |
-+------------------+-----------------------------------------+
-| FUNCTEST-230 | Heat issues in Rally scenarios |
-+------------------+-----------------------------------------+
-| FUNCTEST-229 | Extend reporting to brahmaputra |
-+------------------+-----------------------------------------+
-| FUNCTEST-139 | prepare_env failed due to |
-| | https://pypi.python.org/samples is not |
-| | accessible |
-+------------------+-----------------------------------------+
-| FUNCTEST-135 | vPing scenario failed in odl_l3 scenario|
-+------------------+-----------------------------------------+
+ - Functest Colorado user guide: http://artifacts.opnfv.org/functest/colorado/docs/userguide/index.html
-Useful links
-============
+ - Functest installation/configuration guide: http://artifacts.opnfv.org/functest/colorado/docs/configguide/index.html
- - wiki project page: https://wiki.opnfv.org/opnfv_functional_testing
+ - Functest developer guide: http://artifacts.opnfv.org/functest/colorado/docs/devguide/index.html
- - Functest repo: https://git.opnfv.org/cgit/functest
+.. _`FUNCTEST-446` : https://jira.opnfv.org/browse/FUNCTEST-446
- - Functest CI dashboard: https://build.opnfv.org/ci/view/functest/
+.. _`FUNCTEST-454` : https://jira.opnfv.org/browse/FUNCTEST-454
- - JIRA dashboard: https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10611
+.. _`apex-res-6` : http://testresults.opnfv.org/test/api/v1/results?build_tag=bd-push-daily-colorado-6
+
+.. _`apex-res-174` : http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-174
+
+.. _`apex-res-175`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-175
+
+.. _`apex-res-176`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-176
+
+.. _`apex-res-217`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-217
+
+.. _`apex-res-235`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-235
+
+.. _`apex-res-423`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-apex-apex-daily-colorado-daily-colorado-423
+
+.. _`compass-res-55`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-55
+
+.. _`compass-res-59`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-59
+
+.. _`compass-res-73`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-73
+
+.. _`compass-res-76`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-76
+
+.. _`compass-res-77`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-colorado-77
+
+.. _`compass-res-567`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-master-567
+
+.. _`compass-res-285`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-compass-virtual-daily-master-285
+
+.. _`fuel-res-8`: https://build.opnfv.org/ci/view/functest/job/functest-fuel-virtual-suite-colorado/lastSuccessfulBuild/console
+
+.. _`fuel-res-115`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-115
+
+.. _`fuel-res-117`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-117
+
+.. _`fuel-res-119`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-119
+
+.. _`fuel-res-123`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-123
+
+.. _`fuel-res-124`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-124
+
+.. _`fuel-res-128`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-128
+
+.. _`fuel-res-129`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-baremetal-daily-colorado-129
+
+.. _`fuel-res-154`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-154
+
+.. _`fuel-res-155`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-155
+
+.. _`fuel-res-160`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-160
+
+.. _`fuel-res-161`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-161
+
+.. _`fuel-res-162`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-162
+
+.. _`fuel-res-164`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-164
+
+.. _`fuel-res-166`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-166
+
+.. _`fuel-res-213`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-213
+
+.. _`fuel-res-219`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-219
+
+.. _`fuel-res-376`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-376
+
+.. _`fuel-res-492`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-fuel-virtual-daily-colorado-492
+
+.. _`fuel-arm-res-128`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/128/console
+
+.. _`fuel-arm-res-122`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/122/console
+
+.. _`fuel-arm-res-129`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/129/console
+
+.. _`fuel-arm-res-135`: https://build.opnfv.org/ci/view/armband/job/functest-fuel-armband-baremetal-daily-colorado/135/console
+
+.. _`joid-res-93`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-93
+
+.. _`joid-res-91`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-91
+
+.. _`joid-res-99`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-99
+
+.. _`joid-res-97`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-97
+
+.. _`joid-res-102`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-102
- - Wiki page for B Release: https://wiki.opnfv.org/functest_release_2
+.. _`joid-res-103`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-103
- - Functest IRC chan: #opnfv-testperf
+.. _`joid-res-104`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-104
- - Test dashboard: https://www.opnfv.org/opnfvtestgraphs/summary
+.. _`joid-res-345`: http://testresults.opnfv.org/test/api/v1/results?build_tag=jenkins-functest-joid-baremetal-daily-colorado-345
- - Functest dashboard: http://testresults.opnfv.org/dashboard
Code Review / functest.git / blobdiff