3 [Cédric Ollivier](mailto:cedric.ollivier@orange.com)
11 ![OPNFV](https://docs.opnfv.org/en/stable-fraser/_images/OPNFV_testing_working_group1.png)
12 <!-- .element: style="border: 0; width: 90%" -->
16 ## Verify OpenStack and Kubernetes
19 ### Functest in a nutshell
21 - verify any kind of OpenStack and Kubernetes deployments
22 - conform with upstream rules (OpenStack gate jobs and Kubernetes conformance
24 - ensure that the platforms meet Network Functions Virtualization requirements
29 - as many upstream functional tests as possible (e.g. Tempest,
30 neutron-tempest-api, Barbican, Patrole...)
31 - upstream API and dataplane benchmarking tools (Rally, Vmtp and Shaker)
32 - additional VNF deployments and testing (vIMS, vRouter and vEPC)
34 **Dovetail only runs few Functest functional tests and then we have to verify
35 any OPNFV-certified scenarios via Functest anyway**
38 ### What's new in Functest?
40 - new testcases were quickly integrated in Functest: Patrole, Barbican, Shaker,
42 - all testcases can run in parallel to decrease the overall duration
43 - the resources cleaning has been improved
44 - **our testcases may be run vs VIM in production**
45 - **it includes most of the OpenStack gate jobs**
48 ### Support of OS and K8s master
50 | Functest | OpenStack | Kubernetes |
51 | :------: | :---------: | :--------: |
52 | master | master | master |
53 | hunter | rocky | v1.11.3 |
54 | gambia | queens | v1.11.3 |
59 ![Raspberry PI](raspberrypi.jpg)
60 <!-- .element: style="border: 0; width: 70%" -->
67 ### Xtesting in a nutshell
69 - allow the developer to work only on the test suites without diving into CI/CD
71 - simplify test integration in a complete LFN-based CI/CD toolchain (e.g.
72 Jenkins, Testing Containers, Test API and dashboard)
73 - allow a proper design and verify multiple components in the same CI/CD
74 toolchain (OpenStack, Kubernetes, ONAP, etc.)
76 **Easy to use and very useful for any CI/CD toochain (unlinked to Infrastrure)**
81 - all tests are run by a specialized Docker container(**<100 MB**) instead of
82 the classical ONAP testing virtual machine (**> 1GB**).
83 - the container mainly inherits from opnfv/xtesting and is completed by:
85 - all ONAP Robot Framework files retrieved from the original repositories
86 - testcases.yaml describing the testcases
88 [Orange-OpenSource/xtesting-onap-robot](https://github.com/Orange-OpenSource/xtesting-onap-robot/)
91 ### What's new in Xtesting?
93 - new ansible roles and playbooks have been developed to allow **deploying your
94 full CI/CD toolchains in few minutes** (Jenkins, Minio, TestAPI, MongoDB and
99 . xtesting/bin/activate
100 pip install ansible docker
101 ansible-galaxy install collivier.xtesting
102 git clone https://gerrit.opnfv.org/gerrit/functest-xtesting functest-xtesting-src
103 ansible-playbook functest-xtesting-src/ansible/site.yml
107 **They are already reused in Functest and by Orange out of the
115 ### OPNFV Test Database in a nutshell
117 - it's a fair comparison of Neutron implementations (Agents vs SDN
119 - it stores all verification results and all performance data from different
120 hardware over the world which could be easily postprocessed
121 - it could be very useful to select the adequate opensource solutions
122 regarding metrics and capabilities
125 ### which Neutron backend?
127 - most Neutron standalone and OVN scenarios pass Functest decently
128 - no ODL scenarios pass the advanced testcases (benchmarking tools and
129 VNFs). It's still unclear if it's due to the Installers, ODL or POD
131 - no Tungsten Fabric is released in Gambia
133 **We expected that ODL results would have improved before the first Gambia
137 ### Contrail testing (out of OPNFV)
139 - **3.X 4.X**: mostly verified except some functional tests about
140 visibility which fail due to the falsy admin role (they can be easily
142 - **5.X**: a limited set of bugs in Contrail mostly forbid running few
143 functional tests and benchmarking tools:
144 - wrong external network listing
145 - Contrail doesn't allow booting a VM without network (and elects the wrong
151 - OVS DPDK is not fully integrated by an OPNFV installer (Fuel is in a good
153 - the only scenario including VPP is not part of Gambia
155 **From the time being, we can't evaluate the benefits of OVS DPDK or VPP thanks
165 - Functest and Xtesting are powerful and easy to (re)use (containers, jenkins
166 jobs, ansible playbooks, Raspberry PI, etc.).
167 - the number of installers and scenarios decreased in Gambia (it's still
168 unclear regarding the overall quality). **What about OPNFV Test Database if
169 it decreases again in 2019?**
174 - test frameworks are now considered as crucial for OPNFV (see [Last OPNFV Marketing update](https://wiki.opnfv.org/download/attachments/2925933/OPNFV%20Marketing%20Update%20091818.pptx?version=1&modificationDate=1537228648000&api=v2)) and Functest and Xtesting could be
175 already widely reused out of OPNFV
176 - the new test-driven approach as proposed by the [OPNFV Strategic Plan](https://wiki.opnfv.org/download/attachments/2925933/OPNFV%20Strategy%20and%20Plan%20v0.5.pptx?version=1&modificationDate=1540961098000&api=v2) could increase **the quality of all
177 scenarios**. But we are also suggesting to let the installers decide their