1 .. This work is licensed under a Creative Commons Attribution 4.0 International License.
2 .. SPDX-License-Identifier: CC-BY-4.0
4 ***********************
5 Testing developer guide
6 ***********************
17 The OPNFV testing ecosystem is wide.
19 The goal of this guide consists in providing some guidelines for new developers
20 involved in test areas.
22 For the description of the ecosystem, see `[1]`_.
28 There are several ways to join test projects as a developer. In fact you may:
30 * Develop new test cases
32 * Develop tooling (reporting, dashboards, graphs, middleware,...)
34 These different tasks may be done within a specific project or as a shared
35 resource accross the different projects.
37 If you develop new test cases, the best practice is to contribute upstream as
38 much as possible. You may contact the testing group to know which project - in
39 OPNFV or upstream - would be the best place to host the test cases. Such
40 contributions are usually directly connected to a specific project, more details
41 can be found in the user guides of the testing projects.
43 Each OPNFV testing project provides test cases and the framework to manage them.
44 As a developer, you can obviously contribute to them. The developer guide of
45 the testing projects shall indicate the procedure to follow.
47 Tooling may be specific to a project or generic to all the projects. For
48 specific tooling, please report to the test project user guide. The tooling used
49 by several test projects will be detailed in this document.
51 Be involved in the testing group
52 ================================
54 The testing group is a self organized working group. The OPNFV projects dealing
55 with testing are invited to participate in order to elaborate and consolidate a
56 consistant test strategy (test case definition, scope of projects, resources for
57 long duration, documentation, ...) and align tooling or best practices.
59 A weekly meeting is organized, the agenda may be amended by any participant.
60 2 slots have been defined (US and APAC). Agendas and minutes are public.See
62 The testing group IRC channel is #opnfv-testperf
67 All the test projects do not have the same maturity and/or number of
68 contributors. The nature of the test projects may be also different. The
69 following best practices may not be acurate for all the projects and are only
70 indicative. Contact the testing group for further details.
76 Most of the projects have a similar structure, which can be defined as follows::
90 | `-- Dockerfile.aarch64.patch
105 Test projects are installing tools and triggering tests. When it is possible it
106 is recommended to implement an API in order to perform the different actions.
108 Each test project should be able to expose and consume APIs from other test
109 projects. This pseudo micro service approach should allow a flexible use of
110 the different projects and reduce the risk of overlapping. In fact if project A
111 provides an API to deploy a traffic generator, it is better to reuse it rather
112 than implementing a new way to deploy it. This approach has not been implemented
113 yet but the prerequisite consiting in exposing and API has already been done by
114 several test projects.
119 Most of the test projects provide a docker as deliverable. Once connected, it is
120 possible to prepare the environement and run tests through a CLI.
125 Dockerization has been introduced in Brahmaputra and adopted by the test
126 projects. Docker containers are pulled on the jumphost of OPNFV POD.
139 ======================================
140 Testing group configuration parameters
141 ======================================
147 The testing group defined several categories also known as tiers. These
148 categories can be used to group test suites.
150 +----------------+-------------------------------------------------------------+
151 | Category | Description |
152 +================+=============================================================+
153 | Healthcheck | Simple and quick healthcheck tests case |
154 +----------------+-------------------------------------------------------------+
155 | Smoke | Set of smoke test cases/suites to validate the release |
156 +----------------+-------------------------------------------------------------+
157 | Features | Test cases that validate a specific feature on top of OPNFV.|
158 | | Those come from Feature projects and need a bit of support |
159 | | for integration |
160 +----------------+-------------------------------------------------------------+
161 | Components | Tests on a specific component (e.g. OpenStack, OVS, DPDK,..)|
162 | | It may extend smoke tests |
163 +----------------+-------------------------------------------------------------+
164 | Performance | Performance qualification |
165 +----------------+-------------------------------------------------------------+
166 | VNF | Test cases related to deploy an open source VNF including |
167 | | an orchestrator |
168 +----------------+-------------------------------------------------------------+
169 | Stress | Stress and robustness tests |
170 +----------------+-------------------------------------------------------------+
171 | In Service | In service testing |
172 +----------------+-------------------------------------------------------------+
177 The domains deal with the technical scope of the tests. It shall correspond to
178 domains defined for the certification program:
192 One of the goals of the testing working group is to identify the poorly covered
193 areas and avoid testing overlap.
194 Ideally based on the declaration of the test cases, through the tags, domains
195 and tier fields, it shall be possible to create heuristic maps.
202 The OPNFV testing group created a test collection database to collect
203 the test results from CI:
206 http://testresults.opnfv.org/test/swagger/spec.html
208 Any test project running on any lab integrated in CI can push the
209 results to this database.
210 This database can be used to see the evolution of the tests and compare
211 the results versus the installers, the scenarios or the labs.
212 It is used to produce a dashboard with the current test status of the project.
217 The Test result management can be summarized as follows::
219 +-------------+ +-------------+ +-------------+
221 | Test | | Test | | Test |
222 | Project #1 | | Project #2 | | Project #N |
224 +-------------+ +-------------+ +-------------+
227 +-----------------------------------------+
229 | Test Rest API front end |
231 +-----------------------------------------+
234 | +-------------------------+
236 | | Test Results DB |
239 | +-------------------------+
242 +----------------------+
246 +----------------------+
250 The TestAPI is used to declare pods, projects, test cases and test
251 results. Pods are the sets of bare metal or virtual servers and networking
252 equipments used to run the tests.
254 The results pushed in the database are related to pods, projects and test cases.
255 If you try to push results of test done on non referenced pod, the API will
256 return an error message.
258 An additional method dashboard has been added to post-process
259 the raw results in release Brahmaputra (deprecated in Colorado).
261 The data model is very basic, 5 objects are created:
269 The code of the API is hosted in the releng repository `[6]`_.
270 The static documentation of the API can be found at `[7]`_.
271 The TestAPI has been dockerized and may be installed locally in your
272 lab. See `[15]`_ for details.
274 The deployment of the TestAPI has been automated.
275 A jenkins job manages:
277 * the unit tests of the TestAPI
278 * the creation of a new docker file
279 * the deployment of the new TestAPI
280 * the archive of the old TestAPI
281 * the backup of the Mongo DB
283 TestAPI Authorization
284 ---------------------
286 PUT/DELETE/POST operations of the TestAPI now require token based authorization. The token needs
287 to be added in the request using a header 'X-Auth-Token' for access to the database.
290 headers['X-Auth-Token']
292 The value of the header i.e the token can be accessed in the jenkins environment variable
293 *TestApiToken*. The token value is added as a masked password.
295 .. code-block:: python
297 headers['X-Auth-Token'] = os.environ.get('TestApiToken')
299 The above example is in Python. Token based authentication has been added so that only ci pods
300 jenkins job can have access to the database.
302 Please note that currently token authorization is implemented but is not yet enabled.
304 ===============================
305 Feedback from the testing group
306 ================================
311 A test case catalog has been realized. Roll over the project then click to get
312 the list of test cases, click on the case to get more details.
315 :url: http://testresults.opnfv.org/reporting2/reporting/index.html#!/select/visual
320 An automatic reporting page has been created in order to provide a
321 consistent view of the scenarios.
323 In this page, each scenario is evaluated according to test criteria.
324 The code for the automatic reporting is available at `[8]`_.
326 The results are collected from the centralized database every day and,
327 per scenario. A score is calculated based on the results from the last
333 Dashboard is used to provide a consistent view of the results collected in CI.
334 The results showed on the dashboard are post processed from the Database,
335 which only contains raw results.
337 It can be used in addition of the reporting page (high level view) to allow
338 the creation of specific graphs according to what the test owner wants to show.
340 In Brahmaputra, a basic home made dashboard was created in Functest.
341 In Colorado, Yardstick adopted Grafana (time based graphs) and ELK (complex
343 Since Danube, the testing community decided to adopt ELK framework and to rely
344 on bitergia. It was not implemented for Danube but it is planned for Euphrates.
346 Bitergia already provides a dashboard for code and infrastructure.
347 A new Test tab will be added. The dataset will be built by consuming
350 See `[3]`_ for details.
357 Where can I find information on the different test projects?
358 ===========================================================
361 How can I contribute to a test project?
362 =======================================
365 Where can I find hardware resources?
366 ====================================
369 How do I integrate my tests in CI?
370 ==================================
373 How to declare my tests in the test Database?
374 =============================================
377 How to push your results into the Test Database?
378 ================================================
380 The test database is used to collect test results. By default it is
381 enabled only for CI tests from Production CI pods.
383 Please note that it is possible to create your own local database.
385 A dedicated database is for instance created for each plugfest.
387 The architecture and associated API is described in previous chapter.
388 If you want to push your results from CI, you just have to call the API
389 at the end of your script.
391 You can also reuse a python function defined in functest_utils.py `[5]`_
394 Where can I find the documentation on the test API?
395 ===================================================
397 http://artifacts.opnfv.org/releng/docs/testapi.html
401 I have tests, to which category should I declare them?
402 ======================================================
406 The main ambiguity could be between features and VNF.
407 In fact sometimes you have to spawn VMs to demonstrate the capabilities of the
408 feature you introduced.
409 We recommend to declare your test in the feature category.
411 VNF category is really dedicated to test including:
413 * creation of resources
414 * deployement of an orchestrator/VNFM
415 * deployment of the VNF
419 The goal is not to study a particular feature on the infrastructure but to have
420 a whole end to end test of a VNF automatically deployed in CI.
421 Moreover VNF are run in weekly jobs (one a week), feature tests are in daily
422 jobs and use to get a scenario score.
424 Where are the logs of CI runs?
425 ==============================
427 Logs and configuration files can be pushed to artifact server from the CI under
428 http://artifacts.opnfv.org/<project name>
435 _`[1]`: http://docs.opnfv.org/en/stable-danube/testing/ecosystem/overview.html
437 _`[2]`: http://www.opnfv.org
439 _`[3]`: https://wiki.opnfv.org/display/testing/Result+alignment+for+ELK+post-processing
441 _`[4]`: https://wiki.opnfv.org/display/INF/CI+Scenario+Naming
443 _`[5]`: https://git.opnfv.org/functest/tree/functest/utils/functest_utils.py#176
445 _`[6]`: https://git.opnfv.org/functest/tree/releng
447 _`[7]`: http://artifacts.opnfv.org/releng/docs/testapi.html
449 _`[8]`: https://wiki.opnfv.org/display/meetings/Test+Working+Group+Weekly+Meeting
452 IRC support chan: #opnfv-testperf