1 ===========================================
2 OPNFV Brahmaputra release note for Functest
3 ===========================================
8 This document describes the release note of Functest project.
13 OPNFV Brahmaputra release note for Functest Docs
14 (c) by Morgan Richomme (Orange)
16 OPNFV Brahmaputra release note for Functest Docs
17 are licensed under a Creative Commons Attribution 4.0 International License.
18 You should have received a copy of the license along with this.
19 If not, see <http://creativecommons.org/licenses/by/4.0/>.
24 +------------+----------+------------------+------------------------+
25 | **Date** | **Ver.** | **Author** | **Comment** |
27 +------------+----------+------------------+------------------------+
28 | 2016-02-25 | 1.0.0 | Morgan Richomme | Functest for B release |
30 +------------+----------+------------------+------------------------+
32 OPNFV Brahmaputra Release
33 =========================
35 Functest deals with functional testing of the OPNFV solution.
36 It includes test cases developed within the project and test cases developed in
37 other OPNFV projects and other upstream communities.
39 The internal test cases are:
48 The OPNFV projects integrated into Functest framework for automation are:
57 +--------------------------------------+--------------------------------------+
58 | **Project** | functest |
60 +--------------------------------------+--------------------------------------+
61 | **Repo/tag** | brahmaputra.1.0 |
63 +--------------------------------------+--------------------------------------+
64 | **Release designation** | Brahmaputra base release |
66 +--------------------------------------+--------------------------------------+
67 | **Release date** | February 26 2016 |
69 +--------------------------------------+--------------------------------------+
70 | **Purpose of the delivery** | Brahmaputra base release |
72 +--------------------------------------+--------------------------------------+
80 - The Functest Docker image
85 - Installation/configuration guide
91 - Test results per scenario
99 - dockerization of Functest
101 - renaming of vPing into vPing userdata
103 - Tempest update, use custom list of test cases
105 - Rally update, use global scenario
107 - Update jenkins logs
109 - support of the different scenarios
114 - introduction of a new vPing test case vPing ssh
116 - introduction of vIMS test case
124 - scenario management system
126 - creation of a Test collection API
128 - creation of the Test dashboard: https://www.opnfv.org/opnfvtestgraphs/summary
130 - creation of Functest dashboard: http://testresults.opnfv.org/dashboard/
135 For Brahmaputra, Functest supports the following scenarios:
137 +----------------+---------+---------+---------+---------+
138 | Scenario | Apex | Compass | Fuel | Joid |
139 +================+=========+=========+=========+=========+
140 | odl_l2 | X | X | X | X |
141 +----------------+---------+---------+---------+---------+
143 +----------------+---------+---------+---------+---------+
144 | nosdn | | X | X | |
145 +----------------+---------+---------+---------+---------+
146 | ovs (dpdk) | | | X | |
147 +----------------+---------+---------+---------+---------+
149 Functest defines the success criteria when having at least 4 consecutive
150 successful runs of a given scenario from the Continuous Integration.
152 The success criteria is defined as follows:
155 * vPing userdata 100% OK
156 * Tempest success rate > 90%
157 * Rally success rate > 90%
158 * ODL success rate = 100%
159 * ONOSFW success rate = 100%
160 * Promise success rate = 100%
161 * vIMS: deployement of the orchestrator and the vIMS VNF successful
163 Other scenarios are currently available but did not meet success criteria for
164 the release but might be added in the incremental scenario update of the
167 Brahmaputra limitations
168 -----------------------
170 - Fuel and Apex Tempest success rate was below 90% but above 80%. Some of the
171 error causes were identified (workers, lack of IP)
173 - vIMS failed in CI for joid/odl_l2 scenario
175 - vPing userdata and vIMS excluded from onos scenario
177 - None of the odl_l3 scenarios has been successful due to vPing ssh issue.
179 - joid/nosdn successful but the complete scenario not run 4 times in a raw
181 - apex/nosdn never run (not a target scenario) but probably succesful
183 See known issues section for details
185 Test and installer/scenario dependencies
186 ========================================
188 It is not always possible to run all the test cases on all the scenarios. The
189 following table details the dependencies of the test cases per scenario.
191 +----------------+-------------+-------------+-------------+-------------+
192 | Test cases | Apex | Compass | Fuel | Joid |
193 +================+=============+=============+=============+=============+
194 | vPing SSH | all | all | all | all |
195 +----------------+-------------+-------------+-------------+-------------+
196 | vPing userdata | all except | all except | all except | all except |
197 | | ONOS | ONOS | ONOS | ONOS |
198 +----------------+-------------+-------------+-------------+-------------+
199 | Tempest | all | all | all | all |
200 +----------------+-------------+-------------+-------------+-------------+
201 | Rally | all | all | all | all |
202 +----------------+-------------+-------------+-------------+-------------+
203 | ODL | all ODL | all ODL | all ODL | all ODL |
204 +----------------+-------------+-------------+-------------+-------------+
205 | ONOS | ONOS | ONOS | ONOS | ONOS |
206 +----------------+-------------+-------------+-------------+-------------+
207 | Promise | no | no | all | all |
208 +----------------+-------------+-------------+-------------+-------------+
209 | vIMS | all except | all except | all except | all except |
210 | | ONOS | ONOS | ONOS | ONOS |
211 +----------------+-------------+-------------+-------------+-------------+
212 | Doctor | all | no | no | no |
213 +----------------+-------------+-------------+-------------+-------------+
218 Test results are available in:
220 - test results document: http://artifacts.opnfv.org/functest/docs/results/index.html
222 - jenkins logs on CI: https://build.opnfv.org/ci/view/functest/
224 - Test dashboards: http://testresults.opnfv.org/dashboard
229 - nova metadata service not supported in ONOS:
231 - it excludes vPing userdata and vIMS test cases
233 - Rally worker issues:
235 - A workaround has been implemented to run Tempest test cases sequentially
237 - It may explain why Tempest scenarios (mainly on apex) do not run the 210 tests
239 - https://bugs.launchpad.net/testrepository/+bug/1538941
241 - IPv6 issues in tempest suite:
243 - tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_port_in_allowed_allocation_pools
245 - Tempest code which doesn't reserve big enough allocation
247 - https://bugs.launchpad.net/tempest/+bug/1514457
249 - Lack of IP addresses available lead to several errors in different test cases
253 - the VM needs to have access to OpenStack API.
255 - Technical architecture may not allow this access (for security reasons)
257 - Orchestrator can be deployed but the vIMS VNF cannot
259 - That is the reason why it fails on joid/odl_l2 scenario on Orange POD 2
264 +------------------+-----------------------------------------+
265 | JIRA | Description |
266 +==================+=========================================+
267 | FUNCTEST-139 | prepare_env failed due to |
268 | | https://pypi.python.org/samples is not |
270 +------------------+-----------------------------------------+
271 | FUNCTEST-137 | Tempest success rate below 90 on apex |
272 +------------------+-----------------------------------------+
273 | FUNCTEST-136 | Tempest success rate below 90 on fuel |
274 +------------------+-----------------------------------------+
275 | FUNCTEST-135 | vPing scenario failed in odl_l3 scenario|
276 +------------------+-----------------------------------------+
277 | FUNCTEST-124 | odl test suite troubleshooting |
278 +------------------+-----------------------------------------+
283 - wiki project page: https://wiki.opnfv.org/opnfv_functional_testing
285 - Functest repo: https://git.opnfv.org/cgit/functest
287 - Functest CI dashboard: https://build.opnfv.org/ci/view/functest/
289 - JIRA dashboard: https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10611
291 - Wiki page for B Release: https://wiki.opnfv.org/functest_release_2
293 - Functest IRC chan: #opnfv-testperf
295 - Test dashboard: https://www.opnfv.org/opnfvtestgraphs/summary
297 - Functest dashboard: http://testresults.opnfv.org/dashboard