--- /dev/null
+apex
+====
+
+os-nosdn-nofeature-ha
+---------------------
+
+==================== =================== ========== ======== =========
+testcase date pod_name result jira
+==================== =================== ========== ======== =========
+connection_check 2018-04-07 04:52:18 lf-pod1 PASS
+api_check 2018-04-07 05:05:30 lf-pod1 PASS
+snaps_health_check 2018-04-07 05:06:12 lf-pod1 PASS
+vping_ssh 2018-04-07 05:07:49 lf-pod1 PASS
+vping_userdata 2018-04-07 05:08:55 lf-pod1 PASS
+tempest_smoke_serial 2018-04-07 05:24:59 lf-pod1 PASS
+rally_sanity 2018-04-07 05:46:43 lf-pod1 FAIL APEX-564
+refstack_defcore 2018-04-07 05:49:54 lf-pod1 PASS
+patrole 2018-04-07 05:53:10 lf-pod1 PASS
+snaps_smoke 2018-04-07 06:50:35 lf-pod1 FAIL SNAPS-283
+neutron_trunk 2018-04-07 06:53:13 lf-pod1 PASS
+cloudify_ims 2018-04-07 07:19:50 lf-pod1 PASS
+vyos_vrouter 2018-04-07 07:40:53 lf-pod1 PASS
+juju_epc 2018-04-07 08:06:00 lf-pod1 FAIL APEX-570
+==================== =================== ========== ======== =========
+
+os-odl-nofeature-ha
+-------------------
+
+==================== =================== ========== ======== =========
+testcase date pod_name result jira
+==================== =================== ========== ======== =========
+connection_check 2018-04-07 13:18:17 lf-pod1 PASS
+api_check 2018-04-07 13:30:11 lf-pod1 PASS
+snaps_health_check 2018-04-07 13:31:46 lf-pod1 PASS
+vping_ssh 2018-04-07 13:32:46 lf-pod1 PASS
+vping_userdata 2018-04-07 13:33:40 lf-pod1 PASS
+tempest_smoke_serial 2018-04-07 13:53:58 lf-pod1 FAIL
+rally_sanity 2018-04-07 14:12:42 lf-pod1 FAIL
+refstack_defcore 2018-04-07 14:21:20 lf-pod1 FAIL
+patrole 2018-04-07 14:24:48 lf-pod1 FAIL
+snaps_smoke 2018-04-07 15:05:15 lf-pod1 FAIL SNAPS-283
+odl 2018-04-07 14:25:10 lf-pod1 PASS
+neutron_trunk 2018-04-07 15:07:53 lf-pod1 FAIL
+==================== =================== ========== ======== =========
--- /dev/null
+compass
+=======
+
+os-nosdn-nofeature-ha
+---------------------
+
+==================== =================== =========== ======== ===========
+testcase date pod_name result jira
+==================== =================== =========== ======== ===========
+connection_check 2018-04-07 11:59:03 huawei-pod1 PASS
+api_check 2018-04-07 12:09:28 huawei-pod1 PASS
+snaps_health_check 2018-04-07 12:11:01 huawei-pod1 PASS
+vping_ssh 2018-04-07 12:12:45 huawei-pod1 PASS
+vping_userdata 2018-04-07 12:14:32 huawei-pod1 PASS
+tempest_smoke_serial 2018-04-07 12:31:35 huawei-pod1 FAIL COMPASS-588
+rally_sanity 2018-04-07 13:00:49 huawei-pod1 PASS
+refstack_defcore 2018-04-07 13:06:36 huawei-pod1 PASS
+patrole 2018-04-07 13:10:05 huawei-pod1 PASS
+snaps_smoke 2018-04-07 14:02:28 huawei-pod1 FAIL SNAPS-283
+neutron_trunk 2018-04-07 14:04:43 huawei-pod1 FAIL
+cloudify_ims 2018-04-07 14:32:36 huawei-pod1 PASS
+vyos_vrouter 2018-04-07 14:56:46 huawei-pod1 PASS
+juju_epc 2018-04-07 15:52:35 huawei-pod1 PASS
+==================== =================== =========== ======== ===========
+
+os-odl_l3-nofeature-ha
+----------------------
+
+==================== =================== =========== ======== =========
+testcase date pod_name result jira
+==================== =================== =========== ======== =========
+connection_check 2018-04-04 08:53:18 huawei-pod1 PASS
+api_check 2018-04-04 09:03:23 huawei-pod1 PASS
+snaps_health_check 2018-04-04 09:03:59 huawei-pod1 PASS
+vping_ssh 2016-07-23 20:21:55 huawei-pod1 PASS
+vping_userdata 2018-04-04 09:04:43 huawei-pod1 PASS
+tempest_smoke_serial 2018-04-04 09:21:20 huawei-pod1 FAIL
+rally_sanity 2018-04-04 09:50:09 huawei-pod1 PASS
+refstack_defcore 2018-04-04 09:52:32 huawei-pod1 PASS
+patrole 2018-04-04 09:56:08 huawei-pod1 PASS
+snaps_smoke 2018-04-04 10:48:07 huawei-pod1 FAIL SNAPS-283
+odl 2018-04-04 09:56:30 huawei-pod1 PASS
+neutron_trunk 2018-04-04 10:51:10 huawei-pod1 FAIL
+==================== =================== =========== ======== =========
+
+k8-nosdn-nofeature-ha
+---------------------
+
+=============== =================== =========== ======== ======
+testcase date pod_name result jira
+=============== =================== =========== ======== ======
+k8s_smoke 2018-04-07 05:46:58 huawei-pod1 PASS
+k8s_conformance 2018-04-07 06:38:26 huawei-pod1 PASS
+=============== =================== =========== ======== ======
.. SPDX-License-Identifier: CC-BY-4.0
-=============================================
-OPNFV Euphrates 5.1 release note for Functest
-=============================================
+======================================
+OPNFV master release note for Functest
+======================================
Abstract
========
This document contains the release notes of the Functest project.
-OPNFV Euphrates Release
-=======================
+OPNFV master Release
+====================
Functest deals with functional testing of the OPNFV solution.
It includes test cases developed within the project, test cases developed in
other OPNFV projects and it also integrates test cases from other upstream
communities.
+OpenStack
+---------
+
The internal test cases are:
* connection_check
* vping_ssh
* vping_userdata
* tempest_smoke_serial
- * refstack_defcore
- * snaps_smoke
* rally_sanity
+ * refstack_defcore
+ * patrole
* odl
+ * odl-netvirt
+ * snaps_smoke
+ * neutron_trunk
* tempest_full_parallel
* rally_full
* cloudify_ims
* vyos_vrouter
+ * juju_epc
The OPNFV projects integrated into Functest framework for automation are:
- * barometer
- * bgpvpn
* doctor
- * domino
- * fds
+ * bgpvpn
* odl-sfc
- * odl-netvirt
+ * barometer
+ * fds
* parser
- * promise
- * orchestra_openims
- * orchestra_clearwaterims
+
+Kubernetes
+----------
+
+The internal test cases are:
+
+ * k8s_smoke
+ * k8s_conformance
+
+The OPNFV projects integrated into Functest framework for automation are:
+
+ * stor4nfv
+ * clover
Release Data
============
+--------------------------------------+--------------------------------------+
| **Project** | functest |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Repo/tag** | opnfv-5.1.0 |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Release designation** | Euphrates 5.1 release |
-| | |
-+--------------------------------------+--------------------------------------+
-| **Release date** | December 15th 2017 |
-| | |
+--------------------------------------+--------------------------------------+
-| **Purpose of the delivery** | Euphrates second release |
-| | |
+| **Repository branch** | master |
+--------------------------------------+--------------------------------------+
Deliverables
Software
--------
- Functest Docker images:
+ Functest Docker images (OpenStack):
* https://hub.docker.com/r/opnfv/functest-healthcheck
* https://hub.docker.com/r/opnfv/functest-smoke
* https://hub.docker.com/r/opnfv/functest-components
* https://hub.docker.com/r/opnfv/functest-vnf
* https://hub.docker.com/r/opnfv/functest-parser
- * https://hub.docker.com/r/opnfv/functest-restapi
- TestAPI Docker image:
+ Functest Docker images (Kubernetes):
- * https://hub.docker.com/r/opnfv/testapi
+ * https://hub.docker.com/r/opnfv/functest-kubernetes-healthcheck
+ * https://hub.docker.com/r/opnfv/functest-kubernetes-smoke
+ * https://hub.docker.com/r/opnfv/functest-kubernetes-features
-Docker tag for Euphrates 5.1 release: opnfv-5.1.0
-Docker tag for Euphrates with latest bugfixes: euphrates
+Docker tag for master: latest
Documents
---------
- - Installation/configuration guide: http://docs.opnfv.org/en/stable-euphrates/submodules/functest/docs/testing/user/configguide/index.html
-
- - User Guide: http://docs.opnfv.org/en/stable-euphrates/submodules/functest/docs/testing/user/userguide/index.html
-
- - Developer Guide: http://docs.opnfv.org/en/stable-euphrates/submodules/functest/docs/testing/developer/devguide/index.html
-
- - API Docs: http://artifacts.opnfv.org/functest/docs/index.html
-
- - Functest Framework presentation: http://testresults.opnfv.org/functest/framework/index.html
+ * Config Guide: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/configguide/index.html
+ * User Guide: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/user/userguide/index.html
+ * Developer Guide: http://docs.opnfv.org/en/latest/submodules/functest/docs/testing/developer/devguide/index.html
+ * API Docs: http://functest.readthedocs.io/en/latest/
Version change
==============
-Functest now delivers light-weigth Docker images based on Alpine 3.7. The test
-cases are grouped into several categories or tiers and must be run from the
-corresponding container. For example, to run the test case healthcheck, the
-image opnfv/functest-healthcheck shall be used. The tiers and the tests within
-them are explained in detail in the User Guide.
-
-The former Ubuntu image is not longer maintained.
-
-The Parser test case has its own dedicated Docker image since it requires
-libraries released for OpenStack Pike and Euphrates is based on Ocata.
-
-The Docker images do not contain OS images (Cirros, Ubuntu, Centos, ..)
-anymore. A script has been created under the ci directory (download_images.sh)
-which contains all the needed images for all the tests. This file can be
-modified by the user since not all the images might be used. It must be
-executed before starting Functest and attach the needed images as a Docker
-volume. See Configuration Guide for more information.
-
-The requirements have been split into 3 files:
- * requirements.txt : lists all abstract dependencies of the OPNFV packages
- * test-requirements.txt : lists all abstract dependencies required by Functest
- Unit Tests
- * upper-constraints.txt : lists all concrete upstream dependencies required by
- Functest Docker container
-
-OPNFV (test-)requirements.txt have been updated according to stable/ocata
-global-requirements.txt. Functest uses (and completes) stable/ocata
-upper-constraints.txt in Dockerfiles and tox configuration. The project relies
-on pbr, which injects requirements into the install_requires, tests_require
-and/or dependency_links arguments to setup. It also supports conditional
-dependencies which can be added to the requirements
-(e.g. dnspython>=1.14.0;python_version=='2.7')
-
-The way to manage logging has been centralized to a configuration file
-(logging.ini) which might be modified by the user. By default, the output of
-executing the test cases is redirected to log files and is not displayed on the
-console, only result messages and summary tables are displayed.
-
-The framework has been refactored and all the test cases inherit from a core
-class TestCase. For Feature projects who develop test cases, 2 sub-classes
-have been created:
-
- - Feature: it implements all the needed functions and the developer must only
- overwrite the method "execute" (e.g. Barometer)
- - BashFeature: it is used if the third party test case is a shell script. This
- way, the execution command must be specified in testcases.yaml as the
- argument (e.g. Domino, Doctor)
-
-An internal REST API has been introduced in Euphrates. The goal is to trigger
-Functest operations through an API in addition of the CLI. This could be
-considered as a first step towards a pseudo micro services approach where the
-different test projects could expose and consume APIs to the other test
-projects.
-
-Euphrates 5.1 improvements
-==========================
-
-* Alpine images are now supported for ARM (arm64).
-* Added Vyos_router test case.
-* Updated of Rally 0.9.1 and fixed some bugs in cinder scenarios.
-* Patch to allow building containers from a gerrit change.
-* Selection of a subset of SNAPS test cases.
-* Reorder VNF test cases and adjust timeouts in VNFs.
-
-Euphrates 5.1 known restrictions/issues
-=======================================
-
-+--------------+-----------+----------------------------------------------+
-| Installer | Scenario | Issue |
-+==============+===========+==============================================+
-| fuel@aarch64 | any | VNF tier not supported yet. |
-+--------------+-----------+----------------------------------------------+
-| | | The test cases belonging to the VNF tier |
-| any | any | have been only tested on os-nosdn-nofeature |
-| | | scenarios and baremetal deployments. |
-+--------------+-----------+----------------------------------------------+
-| Joid | k8 | Functest does not offer test suites for |
-| Compass | | Kubernetes scenarios yet. |
-+--------------+-----------+----------------------------------------------+
-
-Test and installer/scenario dependencies
-========================================
-
-It is not always possible to run all the test cases on all the scenarios.
-The scenario dependencies (installer or scenario) are detailed
-in the different testcases.yaml for each tier:
-
- * https://git.opnfv.org/functest/tree/docker/healthcheck/testcases.yaml?h=stable/euphrates
- * https://git.opnfv.org/functest/tree/docker/smoke/testcases.yaml?h=stable/euphrates
- * https://git.opnfv.org/functest/tree/docker/features/testcases.yaml?h=stable/euphrates
- * https://git.opnfv.org/functest/tree/docker/components/testcases.yaml?h=stable/euphrates
- * https://git.opnfv.org/functest/tree/docker/vnf/testcases.yaml?h=stable/euphrates
- * https://git.opnfv.org/functest/tree/docker/parser/testcases.yaml?h=stable/euphrates
-
-Test results
-============
-
-The Functest scenario status on December 15, 2017 can be seen on
-http://testresults.opnfv.org/functest/euphrates/
-
-Test logs are available in:
+New test cases
+--------------
- - test results logs from CI: http://artifacts.opnfv.org (within different
- directories 'logs_functest_X')
+ * neutron_trunk
+ * patrole
+ * juju_epc
+ * k8s_smoke,
+ * k8s_conformance
+ * stor4nfv
+ * clover
- - jenkins logs on CI: https://build.opnfv.org/ci/view/functest/
+Key changes
+-----------
- - jenkins logs on ARM CI: https://build.opnfv.org/ci/view/armband/
+ * update test cases and containers to `OpenStack Pike`_
+ * publish the framework in a separate project: Xtesting_
+ * ease testing with default values
+ * clean interfaces with OPNFV Installers and Features
+ * rewrite all vnfs to allow multiple tests in parallel
+ * fully support non-default region names and Keystone v3 domains
+ * refactor all tempest-based test cases (e.g. refstack_defcore)
+ * remove obsolete OpenStack and Functest utils
+ * verify all changes via doc8 and yamllint too
+ * generate reports for all tempest-based test cases
-Open JIRA tickets
-=================
+.. _`OpenStack Pike`: https://raw.githubusercontent.com/openstack/requirements/stable/pike/upper-constraints.txt
+.. _Xtesting: http://xtesting.readthedocs.io/en/latest/
-+------------------+-----------------------------------------------+
-| JIRA | Description |
-+==================+===============================================+
-| | |
-| | |
-+------------------+-----------------------------------------------+
+Code quality
+------------
-All the tickets that are not blocking have been fixed or postponed
-the next release.
+ * pylint: ~9.5/10
+ * code coverage: ~70%
Useful links
============
- - wiki project page: https://wiki.opnfv.org/opnfv_functional_testing
-
- - wiki Functest Euphrates page: https://wiki.opnfv.org/display/functest/5.+Euphrates
-
- - Functest repo: https://git.opnfv.org/cgit/functest
-
- - Functest CI dashboard: https://build.opnfv.org/ci/view/functest/
-
- - JIRA dashboard: https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10611
-
- - Functest IRC chan: #opnfv-functest
-
- - Reporting page: http://testresults.opnfv.org/reporting/euphrates.html
+ * wiki project page: https://wiki.opnfv.org/opnfv_functional_testing
+ * Functest git repository: https://git.opnfv.org/cgit/functest
+ * Functest CI dashboard: https://build.opnfv.org/ci/view/functest/
+ * JIRA dashboard: https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10611
+ * Functest IRC channel: #opnfv-functest
+ * Reporting page: http://testresults.opnfv.org/reporting/master/functest/functest.html
Code Review / functest.git / commitdiff