--- /dev/null
+# Functest on steroids
+
+[Cédric Ollivier](mailto:cedric.ollivier@orange.com)
+
+2019/01/08
+
+
+
+## OPNFV
+
+
+<!-- .element: style="border: 0; width: 90%" -->
+
+
+
+## Verify OpenStack and Kubernetes
+
+
+### Functest in a nutshell
+
+- verify any kind of OpenStack and Kubernetes deployments
+- conform with upstream rules (OpenStack gate jobs and Kubernetes conformance
+ tests)
+- ensure that the platforms meet Network Functions Virtualization requirements
+
+
+### Functest suites
+
+- as many upstream functional tests as possible (e.g. Tempest,
+ neutron-tempest-api, Barbican, Patrole...)
+- upstream API and dataplane benchmarking tools (Rally, Vmtp and Shaker)
+- additional VNF deployments and testing (vIMS, vRouter and vEPC)
+
+**Dovetail only runs few Functest functional tests and then we have to verify
+any OPNFV-certified scenarios via Functest anyway**
+
+
+### What's new in Functest?
+
+- new testcases were quickly integrated in Functest: Patrole, Barbican, Shaker,
+ ...
+- all testcases can run in parallel to decrease the overall duration
+- the resources cleaning has been improved
+- **our testcases may be run vs VIM in production**
+- **it includes most of the OpenStack gate jobs**
+
+
+### Support of OS and K8s master
+
+| Functest | OpenStack | Kubernetes |
+| :------: | :---------: | :--------: |
+| master | master | master |
+| hunter | rocky | v1.11.3 |
+| gambia | queens | v1.11.3 |
+
+
+### < 50 Euros
+
+
+<!-- .element: style="border: 0; width: 70%" -->
+
+
+
+## Reuse of OPNFV
+
+
+### Xtesting in a nutshell
+
+- allow the developer to work only on the test suites without diving into CI/CD
+ integration
+- simplify test integration in a complete LFN-based CI/CD toolchain (e.g.
+ Jenkins, Testing Containers, Test API and dashboard)
+- allow a proper design and verify multiple components in the same CI/CD
+ toolchain (OpenStack, Kubernetes, ONAP, etc.)
+
+**Easy to use and very useful for any CI/CD toochain (unlinked to Infrastrure)**
+
+
+### A user story ONAP
+
+- all tests are run by a specialized Docker container(**<100 MB**) instead of
+the classical ONAP testing virtual machine (**> 1GB**).
+- the container mainly inherits from opnfv/xtesting and is completed by:
+ - Python dependencies
+ - all ONAP Robot Framework files retrieved from the original repositories
+ - testcases.yaml describing the testcases
+
+[Orange-OpenSource/xtesting-onap-robot](https://github.com/Orange-OpenSource/xtesting-onap-robot/)
+
+
+### What's new in Xtesting?
+
+- new ansible roles and playbooks have been developed to allow **deploying your
+ full CI/CD toolchains in few minutes** (Jenkins, Minio, TestAPI, MongoDB and
+ Docker registry)
+
+```bash
+virtualenv xtesting
+. xtesting/bin/activate
+pip install ansible docker
+ansible-galaxy install collivier.xtesting
+git clone https://gerrit.opnfv.org/gerrit/functest-xtesting functest-xtesting-src
+ansible-playbook functest-xtesting-src/ansible/site.yml
+deactivate
+```
+
+**They are already reused in Functest and by Orange out of the
+Infrastrure domain**
+
+
+
+## Collect results
+
+
+### OPNFV Test Database in a nutshell
+
+- it's a fair comparison of Neutron implementations (Agents vs SDN
+ controller)
+- it stores all verification results and all performance data from different
+ hardware over the world which could be easily postprocessed
+- it could be very useful to select the adequate opensource solutions
+ regarding metrics and capabilities
+
+
+### which Neutron backend?
+
+- most Neutron standalone and OVN scenarios pass Functest decently
+- no ODL scenarios pass the advanced testcases (benchmarking tools and
+ VNFs). It's still unclear if it's due to the Installers, ODL or POD
+ misconfigurations.
+- no Tungsten Fabric is released in Gambia
+
+**We expected that ODL results would have improved before the first Gambia
+ corrective**
+
+
+### Contrail testing (out of OPNFV)
+
+- **3.X 4.X**: mostly verified except some functional tests about
+ visibility which fail due to the falsy admin role (they can be easily
+ blacklisted)
+- **5.X**: a limited set of bugs in Contrail mostly forbid running few
+ functional tests and benchmarking tools:
+ - wrong external network listing
+ - Contrail doesn't allow booting a VM without network (and elects the wrong
+ network)
+
+
+### And performance?
+
+- OVS DPDK is not fully integrated by an OPNFV installer (Fuel is in a good
+ shape to support it)
+- the only scenario including VPP is not part of Gambia
+
+**From the time being, we can't evaluate the benefits of OVS DPDK or VPP thanks
+ to OPNFV**
+
+
+
+## Conclusion
+
+
+### Gambia
+
+- Functest and Xtesting are powerful and easy to (re)use (containers, jenkins
+ jobs, ansible playbooks, Raspberry PI, etc.).
+- the number of installers and scenarios decreased in Gambia (it's still
+ unclear regarding the overall quality). **What about OPNFV Test Database if
+ it decreases again in 2019?**
+
+
+### And beyond
+
+- test frameworks are now considered as crucial for OPNFV (see [Last OPNFV Marketing update](https://wiki.opnfv.org/download/attachments/2925933/OPNFV%20Marketing%20Update%20091818.pptx?version=1&modificationDate=1537228648000&api=v2)) and Functest and Xtesting could be
+already widely reused out of OPNFV
+- the new test-driven approach as proposed by the [OPNFV Strategic Plan](https://wiki.opnfv.org/download/attachments/2925933/OPNFV%20Strategy%20and%20Plan%20v0.5.pptx?version=1&modificationDate=1540961098000&api=v2) could increase **the quality of all
+scenarios**. But we are also suggesting to let the installers decide their
+test cases.
--- /dev/null
+<html>
+<head>
+<title>Functest on steroids</title>
+<meta name="author" content="Cédric Ollivier">
+<meta name="viewport"
+ content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no">
+<link rel="stylesheet" href="../reveal.js/css/reveal.css">
+<link rel="stylesheet" href="../reveal.js/css/theme/white.css">
+<link rel="stylesheet" href="../reveal.js/lib/css/zenburn.css">
+<script>
+var link = document.createElement( 'link' );
+link.rel = 'stylesheet';
+link.type = 'text/css';
+link.href = window.location.search.match( /print-pdf/gi ) ? '../reveal.js/css/print/pdf.css' : '../reveal.js/css/print/paper.css';
+document.getElementsByTagName( 'head' )[0].appendChild( link );
+</script>
+</head>
+<body>
+ <div class="reveal">
+ <div class="slides">
+ <section data-markdown="gambia.md" data-separator="^\n\n\n"
+ data-separator-vertical="^\n\n" data-separator-notes="^Note:"></section>
+ </div>
+ </div>
+ <script src="../reveal.js/lib/js/head.min.js"></script>
+ <script src="../reveal.js/js/reveal.js"></script>
+ <script>
+ Reveal.initialize({
+ dependencies : [ {
+ src : '../reveal.js/plugin/markdown/marked.js',
+ condition : function() {
+ return !!document.querySelector('[data-markdown]');
+ }
+ }, {
+ src : '../reveal.js/plugin/markdown/markdown.js',
+ condition : function() {
+ return !!document.querySelector('[data-markdown]');
+ }
+ }, {
+ src: '../reveal.js/plugin/highlight/highlight.js',
+ async: true,
+ callback: function() {
+ hljs.initHighlightingOnLoad();
+ }
+ }, {
+ src: '../reveal.js/plugin/notes/notes.js',
+ async: true
+ } ]
+ });
+ </script>
+</body>
+</html>
Code Review / functest.git / commitdiff
