1 ******************************
2 OPNFV FUNCTEST developer guide
3 ******************************
14 This document describes how feature projects aiming to run functional tests can
15 be integrated into FuncTest framework.
18 ================================
19 Functest High level architecture
20 ================================
22 Functest is project delivering a test container dedicated to OPNFV.
23 It includes the tools, the scripts and the test scenarios.
25 Functest can be described as follow::
27 +----------------------+
29 | +--------------+ | +-------------------+
31 | | Tools | +------------------+ OPNFV |
32 | | Scripts | | | System Under Test |
33 | | Scenarios | +------------------+ |
34 | | | | Management | |
35 | +--------------+ | +-------------------+
39 +----------------------+
41 Functest deals with internal and external test cases.
42 The Internal test cases in Brahmaputra are:
50 The external tescases are:
57 see `[2]`_ for details.
59 Functest can also be considered as a framework that may include external OPNFV
61 This framework will ease the integration of the feature test suite to the CI.
67 The installation and the launch of the Functest docker image is described in
70 The Functest docker directories are::
93 +-----------+-------------------+---------------------------------------------+
94 | Directory | Subdirectory | Comments |
95 +-----------+-------------------+---------------------------------------------+
96 | | <project>/conf | All the useful configuration file(s) for |
97 | | | <project> the openstack creds are put there |
99 | | | It is recommended to push it there when |
100 | | | passing the credentials to container through|
101 | | | the -v option |
102 | +-------------------+---------------------------------------------+
103 | opnfv | <project>/data | Usefull data, images for <projects> |
104 | | | By default we put a cirros image: |
105 | | | cirros-0.3.4-x86_64-disk.img |
106 | | | This image can be used by any projects |
107 | +-------------------+---------------------------------------------+
108 | | <project>/results | Local result directory of project <project> |
109 +-----------+-------------------+---------------------------------------------+
111 | +-------------------+ +
113 | +-------------------+ +
115 | +-------------------+ +
116 | | odl_integration | |
117 | +-------------------+ Clone of the useful repositories +
118 | | onos | These repositories may include: |
119 | +-------------------+ - tooling +
120 | | promise | - scenario |
121 | +-------------------+ - scripts +
123 | +-------------------+ +
125 | +-------------------+ +
127 | +-------------------+ +
128 | | <your project> | |
129 +-----------+-------------------+---------------------------------------------+
131 Before running the test suite, you must prepare the environement by running::
133 $ /home/opnfv/repos/functest/docker/prepare_env.sh
135 By running prepare_env.sh, you build the test environement required by the tests
136 including the retrieval and sourcing of OpenStack credentials.
137 This is an example of the env variables we have in the docker container:
139 * HOSTNAME=373f77816eb0
140 * INSTALLER_TYPE=fuel
141 * repos_dir=/home/opnfv/repos
142 * INSTALLER_IP=10.20.0.2
143 * PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
147 * NODE_NAME=opnfv-jump-2
148 * creds=/home/opnfv/functest/conf/openstack.creds
151 The prepare_env.sh will source the credentials, so once run you should have
152 access to the following env variables::
154 root@373f77816eb0:~# env|grep OS_
155 OS_REGION_NAME=RegionOne
156 OS_PROJECT_NAME=admin
158 OS_AUTH_STRATEGY=keystone
159 OS_AUTH_URL=http://172.30.10.70:5000/v2.0
162 OS_ENDPOINT_TYPE=internalURL
166 Then you may run the test suite by running::
168 $ /home/opnfv/repos/functest/docker/run_tests.sh -t <your project>
170 see `[2]`_ for details.
174 =========================
175 How to integrate Functest
176 =========================
178 The files of the Functest repository you must modify to integrate Functest are:
180 * functest/docker/Dockerfile
181 * functest/docker/common.sh
182 * functest/docker/requirements.pip
183 * functest/docker/run_tests.sh
184 * functest/docker/prepare_env.sh
185 * functest/config_funtest.yaml
191 This file lists the repositories to be cloned in the Functest container.
192 The repositories can be internal or external::
194 RUN git clone https://gerrit.opnfv.org/gerrit/<your porject> ${repos_dir}/<your project>
200 This file can be used to declare the branch and commit variables of your
203 <YOUR_PROJECT>_BRANCH=$(cat $config_file | grep -w <your project>_branch | awk 'END {print $NF}')
204 <YOUR_PROJECT>_COMMIT=$(cat $config_file | grep -w <your project>_commit | awk 'END {print $NF}')
206 echo "<YOUR_PROJECT>_BRANCH=${<YOUR_PROJECT>_BRANCH}"
207 echo "<YOUR_PROJECT>_COMMIT=${<YOUR_PROJECT>_COMMIT}"
213 This file can be used to preloaded all the needed Python libraries (and avoid
214 that each project does it)
215 The current libraries used in Functest container are::
217 # cat requirements.pip
220 python-neutronclient==2.6.0
221 python-novaclient==2.28.1
222 python-glanceclient==1.1.0
223 python-cinderclient==1.4.0
224 python-ceilometerclient==1.5.1
225 python-keystoneclient==1.6.0
229 robotframework==2.9.1
230 robotframework-requests==0.3.8
231 robotframework-sshlibrary==2.1.1
242 This script can be adapted if you need to set up a specific environment before
248 This script is used to run the tests. You must thus complete the cases with your
253 info "Running PROMISE test case..."
257 info "Running Doctor test..."
258 python ${FUNCTEST_REPO_DIR}/testcases/features/doctor.py
261 info "Running <your project> test..."
262 # your script that launchs your tests...
265 And do not forget to update also the help line::
267 -t|--test run specific set of tests
268 <test_name> one or more of the following: vping,odl,rally,tempest,vims,onos, promise. Separated by comma.
274 This file is the main configuration file of Functest. You must add the
275 references to your project::
279 dir_repo_<your project>: /home/opnfv/repos/<your project>
281 # branch and commit ID to which the repos will be reset (HEAD)
282 <your project>_branch: master
283 <your project>_commit: latest
290 The OPNFV testing group created a test collection database to collect the test
292 Any test project running on any lab integrated in CI can push the results to
294 This database can be used afterwards to see the evolution of the tests and
295 compare the results versus the installers, the scenario or the labs.
297 You can find more information about the dashboard from Testing Dashboard wiki
304 The Test result management in Brahmaputra can be summarized as follows::
306 +-------------+ +-------------+ +-------------+
308 | Test | | Test | | Test |
309 | Project #1 | | Project #2 | | Project #N |
311 +-------------+ +-------------+ +-------------+
314 +-----------------------------------------+
316 | Test Rest API front end |
317 | http://testresults.opnfv.org/testapi |
319 +-----------------------------------------+
322 | +-------------------------+
324 | | Test Results DB |
327 | +-------------------------+
330 +----------------------+
334 +----------------------+
336 The Test dashboard URL is: TODO LF
337 A alternate Test dashboard has been realized to provide a view per installer and
338 per scenario for Brahmaputra release::
340 http://testresults.opnfv.org/proto/brahma/
342 This Dashboard consumes the results retrieved thanks to the Test API.
347 The Test API is used to declare pods, projects, test cases and test results. An
348 additional method dashboard has been added to post-process the raw results. The
349 data model is very basic, 4 objects are created:
360 "details": <URL description of the POD>,
361 "creation_date": YYYY-MM-DD HH:MM:SS ,
362 "name": <The POD Name>,
363 "mode": <metal or virtual>
370 "name": <Name of the Project>,
371 "creation_date": "YYYY-MM-DD HH:MM:SS",
372 "description": <Short description>
379 "name":<Name of the test case>,
380 "creation_date": "YYYY-MM-DD HH:MM:SS",
381 "description": <short description>,
382 "url":<URL for longer description>
389 "project_name": <Reference to project>,
390 "pod_name": <Reference to POD where the test was executed>,
391 "version": <Scenario on which the test was executed>,
392 "installer": <Installer Apex or Compass or Fuel or Joid>,
393 "description": <Short description>,
394 "creation_date": "YYYY-MM-DD HH:MM:SS",
395 "case_name": <Reference to the test case>
397 <- the results to be put here ->
401 For Brahmaputra, we got:
407 The projects and the test cases have been frozen in December.
408 But all were not ready for Brahmaputra.
412 The API can described as follows:
416 +--------+--------------------------+-----------------------------------------+
417 | Method | Path | Description |
418 +========+==========================+=========================================+
419 | GET | /version | Get API version |
420 +--------+--------------------------+-----------------------------------------+
425 +--------+--------------------------+-----------------------------------------+
426 | Method | Path | Description |
427 +========+==========================+=========================================+
428 | GET | /pods | Get the list of declared Labs (PODs) |
429 +--------+--------------------------+-----------------------------------------+
430 | POST | /pods | Declare a new POD |
431 | | | Content-Type: application/json |
433 | | | "name": "pod_foo", |
434 | | | "creation_date": "YYYY-MM-DD HH:MM:SS"|
436 +--------+--------------------------+-----------------------------------------+
440 +--------+--------------------------+-----------------------------------------+
441 | Method | Path | Description |
442 +========+==========================+=========================================+
443 | GET | /test_projects | Get the list of test projects |
444 +--------+--------------------------+-----------------------------------------+
445 | GET |/test_projects/{project} | Get details on {project} |
447 +--------+--------------------------+-----------------------------------------+
448 | POST | /test_projects | Add a new test project |
449 | | | Content-Type: application/json |
451 | | | "name": "project_foo", |
452 | | | "description": "whatever you want" |
454 +--------+--------------------------+-----------------------------------------+
455 | PUT | /test_projects/{project} | Update a test project |
457 | | | Content-Type: application/json |
459 | | | <the field(s) you want to modify> |
461 +--------+--------------------------+-----------------------------------------+
462 | DELETE | /test_projects/{project} | Delete a test project |
463 +--------+--------------------------+-----------------------------------------+
468 +--------+--------------------------+-----------------------------------------+
469 | Method | Path | Description |
470 +========+==========================+=========================================+
471 | GET | /test_projects/{project}/| Get the list of test cases of {project} |
473 +--------+--------------------------+-----------------------------------------+
474 | POST | /test_projects/{project}/| Add a new test case to {project} |
475 | | cases | Content-Type: application/json |
477 | | | "name": "case_foo", |
478 | | | "description": "whatever you want" |
479 | | | "creation_date": "YYYY-MM-DD HH:MM:SS"|
480 | | | "url": "whatever you want" |
482 +--------+--------------------------+-----------------------------------------+
483 | PUT | /test_projects/{project}?| Modify a test case of {project} |
484 | | case_name={case} | |
485 | | | Content-Type: application/json |
487 | | | <the field(s) you want to modify> |
489 +--------+--------------------------+-----------------------------------------+
490 | DELETE | /test_projects/{project}/| Delete a test case |
491 | | case_name={case} | |
492 +--------+--------------------------+-----------------------------------------+
496 +--------+--------------------------+-----------------------------------------+
497 | Method | Path | Description |
498 +========+==========================+=========================================+
499 | GET |/results/project={project}| Get the test results of {project} |
500 +--------+--------------------------+-----------------------------------------+
501 | GET |/results/case={case} | Get the test results of {case} |
502 +--------+--------------------------+-----------------------------------------+
503 | GET |/results?pod={pod} | get the results on pod {pod} |
504 +--------+--------------------------+-----------------------------------------+
505 | GET |/results?installer={inst} | Get the test results of installer {inst}|
506 +--------+--------------------------+-----------------------------------------+
507 | GET |/results?version={version}| Get the test results of scenario |
508 | | | {version}. Initially the version param |
509 | | | was reflecting git version, in Functest |
510 | | | it was decided to move to scenario |
511 +--------+--------------------------+-----------------------------------------+
512 | GET |/results?project={project}| Get all the results of the test case |
513 | |&case={case} | {case} of the project {project} with |
514 | |&version={scenario} | version {scenario} installed by |
515 | |&installer={installer} | {installer} on POD {pod} stored since |
516 | |&pod={pod} | {days} days |
517 | | | {project_name} and {case_name} are |
518 | |&period={days} | mandatory, the other parameters are |
520 +--------+--------------------------+-----------------------------------------+
521 | POST | /results | Add a new test results |
522 | | | Content-Type: application/json |
524 | | | "project_name": "project_foo", |
525 | | | "case_name": "case_foo", |
526 | | | "pod_name": "pod_foo", |
527 | | | "installer": "installer_foo", |
528 | | | "version": "scenario_foo", |
529 | | | "details": <your results> |
531 +--------+--------------------------+-----------------------------------------+
536 +--------+--------------------------+-----------------------------------------+
537 | Method | Path | Description |
538 +========+==========================+=========================================+
539 | GET |/dashboard? | Get all the dashboard ready results of |
540 | |&project={project} | {case} of the project {project} |
541 | |&case={case} | version {scenario} installed by |
542 | |&version={scenario} | {installer} on POD {pod} stored since |
543 | |&installer={installer} | {days} days |
545 | |&period={days} | {project_name} and {case_name} are |
546 | | | mandatory, the other parameters are |
548 +--------+--------------------------+-----------------------------------------+
551 The results with dashboard method are post-processed from raw results.
552 Please note that dashboard results are not stored. Only raw results are stored.
555 ===============================================
556 How to push your results into the Test Database
557 ===============================================
559 The test database is used to collect test results. By default it is enabled only
560 in Continuous Integration.
561 The architecture and associated API is described in `[2]`_.
562 If you want to push your results from CI, you just have to use the API at the
565 You can also reuse a python function defined in functest_utils.py::
567 def push_results_to_db(db_url, case_name, logger, pod_name,version, payload):
569 POST results to the Result target DB
571 url = db_url + "/results"
572 installer = get_installer_type(logger)
573 params = {"project_name": "functest", "case_name": case_name,
574 "pod_name": pod_name, "installer": installer,
575 "version": version, "details": payload}
577 headers = {'Content-Type': 'application/json'}
579 r = requests.post(url, data=json.dumps(params), headers=headers)
584 print "Error [push_results_to_db('%s', '%s', '%s', '%s', '%s')]:" \
585 % (db_url, case_name, pod_name, version, payload), e
594 .. _`[1]`: http://artifacts.opnfv.org/functest/docs/configguide/index.html Functest configuration guide URL
595 .. _`[2]`: http://artifacts.opnfv.org/functest/docs/userguide/index.html functest user guide URL
596 .. _`[3]`: https://wiki.opnfv.org/opnfv_test_dashboard
599 OPNFV main site: opnfvmain_.
601 OPNFV functional test page: opnfvfunctest_.
603 IRC support chan: #opnfv-testperf
605 .. _opnfvmain: http://www.opnfv.org
606 .. _opnfvfunctest: https://wiki.opnfv.org/opnfv_functional_testing
607 .. _`OpenRC`: http://docs.openstack.org/user-guide/common/cli_set_environment_variables_using_openstack_rc.html
608 .. _`Rally installation procedure`: https://rally.readthedocs.org/en/latest/tutorial/step_0_installation.html
609 .. _`config_test.py` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.py
610 .. _`config_functest.yaml` : https://git.opnfv.org/cgit/functest/tree/testcases/config_functest.yaml