1 ==============================================
2 OPNFV Brahmaputra3.0 release note for Functest
3 ==============================================
8 This document describes the release note of Functest project.
13 OPNFV Brahmaputra release note for Functest Docs
14 (c) by Morgan Richomme (Orange)
16 OPNFV Brahmaputra release note for Functest Docs
17 are licensed under a Creative Commons Attribution 4.0 International License.
18 You should have received a copy of the license along with this.
19 If not, see <http://creativecommons.org/licenses/by/4.0/>.
24 +------------+----------+------------------+------------------------+
25 | **Date** | **Ver.** | **Author** | **Comment** |
27 +------------+----------+------------------+------------------------+
28 | 2016-02-25 | 1.0.0 | Morgan Richomme | Functest for B release |
30 +------------+----------+------------------+------------------------+
31 | 2016-04-27 | 3.0.0 | Morgan Richomme | Add scenarios |
33 +------------+----------+------------------+------------------------+
35 OPNFV Brahmaputra Release
36 =========================
38 Functest deals with functional testing of the OPNFV solution.
39 It includes test cases developed within the project and test cases developed in
40 other OPNFV projects and other upstream communities.
42 The internal test cases are:
51 The OPNFV projects integrated into Functest framework for automation are:
61 +--------------------------------------+--------------------------------------+
62 | **Project** | functest |
64 +--------------------------------------+--------------------------------------+
65 | **Repo/tag** | brahmaputra.1.0 |
67 +--------------------------------------+--------------------------------------+
68 | **Release designation** | Brahmaputra base release |
70 +--------------------------------------+--------------------------------------+
71 | **Release date** | February 26 2016 |
73 +--------------------------------------+--------------------------------------+
74 | **Purpose of the delivery** | Brahmaputra base release |
76 +--------------------------------------+--------------------------------------+
84 - The Functest Docker image
89 - Installation/configuration guide
95 - Test results per scenario
103 - support new scenarios
108 - minor bug fixes (formating)
110 - Modification of the configuration to support vPing_userdata on ONOS scenario
112 - Use serial option in Tempest to improve success rate
117 For Brahmaputra 3.0, Functest was succesfully tested on the following scenarios:
119 +----------------+---------+---------+---------+---------+
120 | Scenario | Apex | Compass | Fuel | Joid |
121 +================+=========+=========+=========+=========+
122 | odl_l2 | X | X | X | X |
123 +----------------+---------+---------+---------+---------+
125 +----------------+---------+---------+---------+---------+
126 | nosdn | | X | X | |
127 +----------------+---------+---------+---------+---------+
128 | ovs (dpdk) | | | X | |
129 +----------------+---------+---------+---------+---------+
131 +----------------+---------+---------+---------+---------+
132 | bgpvpn | X | | X | |
133 +----------------+---------+---------+---------+---------+
135 +----------------+---------+---------+---------+---------+
137 Functest defines the success criteria when having at least 4 consecutive
138 successful runs of a given scenario from the Continuous Integration.
140 The success criteria is defined as follows:
143 * vPing userdata 100% OK
144 * Tempest success rate > 90%
145 * Rally success rate > 90%
146 * ODL success rate = 100%
147 * ONOSFW success rate = 100%
148 * Promise success rate = 100%
149 * Bgpvpn success rate = 100%
150 * vIMS: deployement of the orchestrator and the vIMS VNF successful
152 Other scenarios are currently available but did not meet success criteria for
153 the release but might be added in the incremental scenario update of the
156 Brahmaputra limitations
157 -----------------------
159 - Fuel and Apex Tempest success rate was below 90% but above 80% on some
160 scenarios. Some of the error causes were identified (workers, lack of IP)
162 - vIMS failed in CI for joid/odl_l2 scenario
164 - vPing userdata and vIMS excluded from onos scenario
166 - vPing_ssh and vIMS excluded from bgpvpn and kvm scenario
168 - None of the odl_l3 scenarios has been successful due to vPing ssh issue (ODL
169 bug reported https://bugs.opendaylight.org/show_bug.cgi?id=5582)
171 - apex/nosdn never run (not a target scenario) but probably succesful
173 - vPing SSH and vPing userdata no more run on CI since modification of bgpvpn
176 See known issues section for details
178 Test and installer/scenario dependencies
179 ========================================
181 It is not always possible to run all the test cases on all the scenarios. The
182 following table details the dependencies of the test cases per scenario.
184 +----------------+-------------+-------------+-------------+-------------+
185 | Test cases | Apex | Compass | Fuel | Joid |
186 +================+=============+=============+=============+=============+
187 | vPing SSH | all | all | all | all |
188 +----------------+-------------+-------------+-------------+-------------+
189 | vPing userdata | all | all | all | all |
190 +----------------+-------------+-------------+-------------+-------------+
191 | Tempest | all | all | all | all |
192 +----------------+-------------+-------------+-------------+-------------+
193 | Rally | all | all | all | all |
194 +----------------+-------------+-------------+-------------+-------------+
195 | ODL | all ODL | all ODL | all ODL | all ODL |
196 +----------------+-------------+-------------+-------------+-------------+
197 | ONOS | ONOS | ONOS | ONOS | ONOS |
198 +----------------+-------------+-------------+-------------+-------------+
199 | Promise | no | no | all | all |
200 +----------------+-------------+-------------+-------------+-------------+
201 | vIMS | all except | all except | all except | all except |
202 | | ONOS | ONOS | ONOS | ONOS |
203 +----------------+-------------+-------------+-------------+-------------+
204 | Doctor | all | no | no | no |
205 +----------------+-------------+-------------+-------------+-------------+
206 | Bgpvpn | all | no | all | no |
207 +----------------+-------------+-------------+-------------+-------------+
212 Test results are available in:
214 - test results document: http://artifacts.opnfv.org/functest/docs/results/index.html
216 - jenkins logs on CI: https://build.opnfv.org/ci/view/functest/
218 - Test dashboards: http://testresults.opnfv.org/dashboard
223 - IPv6 issues in tempest suite:
225 - tempest.api.network.test_ports.PortsIpV6TestJSON.test_create_port_in_allowed_allocation_pools
227 - Tempest code which doesn't reserve big enough allocation
229 - https://bugs.launchpad.net/tempest/+bug/1514457
231 - Lack of IP addresses available lead to several errors in different test cases
233 - vIMS (http://testresults.opnfv.org/reporting/vims/):
235 - the VM needs to have access to OpenStack API.
237 - Technical architecture may not allow this access (for security reasons)
239 - Orchestrator can be deployed but the vIMS VNF cannot
241 - That is the reason why it fails on joid/odl_l2 scenario on Orange POD 2
243 - case needs to be consolidated on new scenaios (bgpvpn, sfc, ovs)
248 +------------------+-----------------------------------------+
249 | JIRA | Description |
250 +==================+=========================================+
251 | FUNCTEST-231 | vPing SSH no more run systematically |
253 +------------------+-----------------------------------------+
254 | FUNCTEST-230 | Heat issues in Rally scenarios |
255 +------------------+-----------------------------------------+
256 | FUNCTEST-229 | Extend reporting to brahmaputra |
257 +------------------+-----------------------------------------+
258 | FUNCTEST-139 | prepare_env failed due to |
259 | | https://pypi.python.org/samples is not |
261 +------------------+-----------------------------------------+
262 | FUNCTEST-135 | vPing scenario failed in odl_l3 scenario|
263 +------------------+-----------------------------------------+
268 - wiki project page: https://wiki.opnfv.org/opnfv_functional_testing
270 - Functest repo: https://git.opnfv.org/cgit/functest
272 - Functest CI dashboard: https://build.opnfv.org/ci/view/functest/
274 - JIRA dashboard: https://jira.opnfv.org/secure/Dashboard.jspa?selectPageId=10611
276 - Wiki page for B Release: https://wiki.opnfv.org/functest_release_2
278 - Functest IRC chan: #opnfv-testperf
280 - Test dashboard: https://www.opnfv.org/opnfvtestgraphs/summary
282 - Functest dashboard: http://testresults.opnfv.org/dashboard